Using Nginx and Apache2 in separate distributed servers (ie, Amazon EC2 instances)

We are running one frontend running NGINX and several app servers running Apache2. There are several issues we have come across but right now I'll be documenting one of them. I'll be completing this article when I get more time.

Mirroring Assets folder

Assets are "generated" on the Apache2 side. Therefore, if you have a frontend nginx server, the /assets folder will not be populated with the desired files created on one of your Apache2 servers. To handle this situation you can make use of one of NGINX's features that functions like a mirror for files.

Supposing you already know how to create NGINX uplinks and you have it all configured, in your server config (ie: in ubuntu will be sites-enabled/default) you have to configure it like this:


server {

    root  /var/www/yoursite.com/public_html/;

    server_name yoursite.com;

    location / {
        # appservers is the name of our uplink, configured in nginx.conf
        proxy_pass http://appservers$request_uri;
    }

    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_read_timeout 300;
    proxy_connect_timeout 300;

    location /css {
        expires         30d;
    }

    location ~ /assets/ {
        try_files $uri $uri/ @assets;
    }

    location @assets {
        internal;
        proxy_pass              http://appservers$request_uri;
        proxy_store             on;
        proxy_store_access      user:rw  group:rw  all:r;
        proxy_temp_path         /var/www/yoursite.com/temp/;
    }

    location /assets {
        expires         30d;
    }

    location /js {
        expires         30d;
    }

    # Serve certain files directly from Nginx
    location ~* ^.+\.(jpg|jpeg|gif|png|css|zip|pdf|txt|js|flv|swf|html|htm|mp3)$
    {
    }

    error_page  404 500 502 503 504 /maint/index.html;

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    location ~ /\.ht {
            deny  all;
    }
}

So, notice the /assets configuration. It tries to retrieve you assets from your NGINX server. If it is not found, it forwards the request to the @asset location. In that location, you fetch the assets from the appserver, and store a copy of the assets in the corresponding relative location.

In our nginx.conf we have:

upstream appservers {
    server server1.compute.amazonaws.com:8080 weight=4; #Apache2 server 01
    server server2.compute.amazonaws.com:8080 weight=4; #Apache2 server 02
    server server3.compute.amazonaws.com:8080 weight=4; #Apache2 server 03
}

In Apache2 /assets folder, you should disable gzip compression, and enable gzip compression in Nginx. If you leave compresssion for the /assets folder in Apache2, your files will be compressed twice! You can do it adding a .htaccess in /assets with this content:

SetEnv no-gzip dont-vary

Finally, don't forget to create the temp folder configured in proxy_temp_path with sufficient permissions for the user running nginx process.