Recently I noticed that an application I'm hosting on AWS Elastic Beanstalk wasn't logging the Client IP Address of my users. After a lot of digging I found the issue was to do with the configuration of the Nginx servers running on the EC2 instances and in my application's Docker container, and in the TCP passthrough configuration of the Elastic Load Balancer.

So here's a quick post about my particular setup and the configuration I used to fix this issue of Client IP Addresses not being forwarded.

Elastic Beanstalk Setup

In a little more detail, my setup consists of an application hosted within a Docker container running on EC2 instances configured by Elastic Beanstalk, which sit behind an Elastic Load Balancer (ELB). Because of issues with Certificate support in ELB, I'm not using AWS's Certificate Manager or using the built-in SSL termination in ELB, but instead using TCP passthrough on the ELB, and using a custom Nginx configuration on each EC2 instance to terminate the SSL connection. The Docker container on each instance also has it's own Nginx configuration, which proxies the application server, in my case UWSGI.

The steps that a request takes to get to the application are:

1. User makes an HTTPS request.
2. ELB uses TCP passthrough to pass the encrypted request to an EC2 instance.
3. Nginx in EC2 decrypts the HTTPS request and passes the HTTP to it's Docker container.
4. The Nginx server on Docker proxies the request to UWSGI.
5. The application hosted by UWSGI handles the request.

In all, the parts that you need to configure to forward the Client IP Address are the TCP passthrough on ELB and each of the two Nginx servers.

Proxy Protocol

The first problem is that if you're using a TCP load balancer to pass through the request, the load balancer will not add an X-Forwarded-For header, and so the downstream Nginx server will only see the IP Address of the load balancer. The problem and part of the solution is described in better detail here. The linked solution is a little different from what I'm going to show below, because I'm proxying the request another hop down the line, but please read the link to better understand what's going on.

The first step is to enable Proxy Protocol support on the Elastic Load Balancer. AWS has a good tutorial on how to do that.

Once the ELB has Proxy Protocol support enabled, you can configure the Nginx server on the EC2 instances. The relevant parts of the configuration look something like this:

log_format elb_log '$proxy_protocol_addr - $remote_user'
                    '[$time_local] "$request" '
                    '$status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

server {
    listen 443 ssl http2 proxy_protocol;

    server_name localhost;

    set_real_ip_from  172.31.0.0/20;
    real_ip_header    proxy_protocol;

    access_log /var/log/nginx/elb-access.log elb_log;

    # SSL Certificate settings go here.

    location / {
        proxy_pass          https://docker;
        proxy_http_version  1.1;

        proxy_set_header Connection         "";
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $proxy_protocol_addr;
        proxy_set_header X-Forwarded-For    $proxy_protocol_addr;
        proxy_set_header X-Forwarded-Proto  $scheme;
    }
}

Here's what's happening in the above config:

1. The listen directive now has proxy_protocol.
2. set_real_ip_from tells Nginx the real CIDR range of addresses that the ELB is using.
3. real_ip_header tells us that we're using the proxy_protocol value to access the client headers.
4. X-Real-IP and X-Forwarded-For use the $proxy_protocal_addr, which is the client IP information taken from the proxy protocol.
5. The elb_log format is now using the $proxy_protocal_addr for the client IP.

Also note that the listen directive can support both http2 and proxy_protocol, so if your version of Nginx supports HTTP/2 then you can enable that just fine. However, the proxy_http_version directive in this case is still 1.1, as it doesn't make sense to use HTTP/2 for proxy backends.

If you deployed the above config, you should now see that Nginx server on the EC2 instance is now logging the correct client IP Address.

Multiple Nginx Proxies

Now that we have the Nginx server on the EC2 instance receiving the correct Client IP from the Load Balancer, we need to foward that IP on to the Nginx server in the Docker container.

The X-Forwarded-For and X-Real-IP directives in the above config will properly forward the Client IP from the EC2 instance's Nginx server. However, Nginx appends each proxy's IP address to the X-Forwarded-For header, as described in more detail here.

The solution to this is in the last Nginx proxy configuration is to include the IP address ranges of all previous known proxies in the set_real_ip_from directive. You can then set the real_ip_header directive to X-Forwarded-For. The final piece is to set real_ip_recursive to on, which will return the leftmost (original) IP instead of the rightmost (most recent) IP from the X-Forwarded-For header.

My Nginx configuration for the Docker container now looks like this:

upstream app {
    server unix:///tmp/uwsgi.sock;
}

server {
    listen 80 default_server;

    set_real_ip_from    172.31.0.0/20;    # IP of the ELB
    set_real_ip_from    172.17.0.0/20;    # IP of the EC2 instance
    real_ip_header      X-Forwarded-For;
    real_ip_recursive   on;

    location / {
        uwsgi_pass app;
        include uwsgi_params;
    }
}

You can see that I've included the IP ranges of both the ELB and the EC2 instance. With a config similar to this you should now see the Nginx instance in the Docker container log the correct Client IP, and pass that IP through to your application.