David Beath (Posts about Nginx)

Nginx and ELB Proxy Protocol Forwarding

David Beath — Mon, 17 Apr 2017 10:35:06 GMT

Recently I noticed that an application I'm hosting on AWS Elastic Beanstalk wasn't logging the Client IP Address of my users. After a lot of digging I found the issue was to do with the configuration of the Nginx servers running on the EC2 instances and in my application's Docker container, and in the TCP passthrough configuration of the Elastic Load Balancer.

So here's a quick post about my particular setup and the configuration I used to fix this issue of Client IP Addresses not being forwarded.

Elastic Beanstalk Setup

In a little more detail, my setup consists of an application hosted within a Docker container running on EC2 instances configured by Elastic Beanstalk, which sit behind an Elastic Load Balancer (ELB). Because of issues with Certificate support in ELB, I'm not using AWS's Certificate Manager or using the built-in SSL termination in ELB, but instead using TCP passthrough on the ELB, and using a custom Nginx configuration on each EC2 instance to terminate the SSL connection. The Docker container on each instance also has it's own Nginx configuration, which proxies the application server, in my case UWSGI.

The steps that a request takes to get to the application are:

User makes an HTTPS request.
ELB uses TCP passthrough to pass the encrypted request to an EC2 instance.
Nginx in EC2 decrypts the HTTPS request and passes the HTTP to it's Docker container.
The Nginx server on Docker proxies the request to UWSGI.
The application hosted by UWSGI handles the request.

In all, the parts that you need to configure to forward the Client IP Address are the TCP passthrough on ELB and each of the two Nginx servers.

Proxy Protocol

The first problem is that if you're using a TCP load balancer to pass through the request, the load balancer will not add an X-Forwarded-For header, and so the downstream Nginx server will only see the IP Address of the load balancer. The problem and part of the solution is described in better detail here. The linked solution is a little different from what I'm going to show below, because I'm proxying the request another hop down the line, but please read the link to better understand what's going on.

The first step is to enable Proxy Protocol support on the Elastic Load Balancer. AWS has a good tutorial on how to do that.

Once the ELB has Proxy Protocol support enabled, you can configure the Nginx server on the EC2 instances. The relevant parts of the configuration look something like this:

log_format elb_log '$proxy_protocol_addr - $remote_user'
                    '[$time_local] "$request" '
                    '$status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

server {
    listen 443 ssl http2 proxy_protocol;

    server_name localhost;

    set_real_ip_from  172.31.0.0/20;
    real_ip_header    proxy_protocol;

    access_log /var/log/nginx/elb-access.log elb_log;

    # SSL Certificate settings go here.

    location / {
        proxy_pass          https://docker;
        proxy_http_version  1.1;

        proxy_set_header Connection         "";
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $proxy_protocol_addr;
        proxy_set_header X-Forwarded-For    $proxy_protocol_addr;
        proxy_set_header X-Forwarded-Proto  $scheme;
    }
}

Here's what's happening in the above config:

The listen directive now has proxy_protocol.
set_real_ip_from tells Nginx the real CIDR range of addresses that the ELB is using.
real_ip_header tells us that we're using the proxy_protocol value to access the client headers.
X-Real-IP and X-Forwarded-For use the $proxy_protocal_addr, which is the client IP information taken from the proxy protocol.
The elb_log format is now using the $proxy_protocal_addr for the client IP.

Also note that the listen directive can support both http2 and proxy_protocol, so if your version of Nginx supports HTTP/2 then you can enable that just fine. However, the proxy_http_version directive in this case is still 1.1, as it doesn't make sense to use HTTP/2 for proxy backends.

If you deployed the above config, you should now see that Nginx server on the EC2 instance is now logging the correct client IP Address.

Multiple Nginx Proxies

Now that we have the Nginx server on the EC2 instance receiving the correct Client IP from the Load Balancer, we need to foward that IP on to the Nginx server in the Docker container.

The X-Forwarded-For and X-Real-IP directives in the above config will properly forward the Client IP from the EC2 instance's Nginx server. However, Nginx appends each proxy's IP address to the X-Forwarded-For header, as described in more detail here.

The solution to this is in the last Nginx proxy configuration is to include the IP address ranges of all previous known proxies in the set_real_ip_from directive. You can then set the real_ip_header directive to X-Forwarded-For. The final piece is to set real_ip_recursive to on, which will return the leftmost (original) IP instead of the rightmost (most recent) IP from the X-Forwarded-For header.

My Nginx configuration for the Docker container now looks like this:

upstream app {
    server unix:///tmp/uwsgi.sock;
}

server {
    listen 80 default_server;

    set_real_ip_from    172.31.0.0/20;    # IP of the ELB
    set_real_ip_from    172.17.0.0/20;    # IP of the EC2 instance
    real_ip_header      X-Forwarded-For;
    real_ip_recursive   on;

    location / {
        uwsgi_pass app;
        include uwsgi_params;
    }
}

You can see that I've included the IP ranges of both the ELB and the EC2 instance. With a config similar to this you should now see the Nginx instance in the Docker container log the correct Client IP, and pass that IP through to your application.

Configuring SSL in Nginx

David Beath — Sun, 17 Nov 2013 03:32:49 GMT

This post aims to be a fairly thorough guide to getting SSL up and running on an Nginx server, largely for my own reference in future, but it may prove useful to anyone else who stumbles across it. I'm going to assume that you have a working installation of Nginx, or know how to get one going. If not, then bookmark this page and find a tutorial on how to get it going first, such as the section on Nginx in my post here, or the Installation and Beginner's Guide pages on the official site.

Get an SSL Certificate

The first step in setting up SSL on your server is to procure an SSL Certificate from an approved Certificate Authority (CA). The easiest way to do this is to purchase one from a certificate reseller such as CheapSSLs.com.

For a single site with no subdomains, such as a blog, that doesn't require all the extra features of more expensive certificates, then a certificate such as Comodo Positive SSL will do fine.

Once you've purchased the certificate, you'll need to create a Certificate Signing Request.

Creating a CSR

A Certificate Signing Request (CSR) is a file that contains the application information for your certificate, such as your location and the name of the domain you want to secure, as well as your Public Key. For creating the CSR we'll be using OpenSSL, so if you don't have that on your server you can install it with the following command:

> sudo apt-get install openssl

Before we start creating files, it's recommended that you have a proper location to store them in. On Linux systems this would be something like /etc/ssl/localcerts/, so lets start there.

> sudo mkdir /etc/ssl/localcerts
> sudo cd /etc/ssl/localcerts

To generate the public and private keys that will be used for the CSR, use this command:

> sudo openssl req -nodes -newkey rsa:2048 -days 365 -keyout myserver.key -out server.csr

This will create two files; myserver.key, the private key file, and server.csr, the Certificate Signing Request. When you enter the above command, you should of course set the name of both files to one appropriate for the server you are securing.

Generally your CA should require you to use RSA 2048, this is specified with the -newkey rsa:2048 portion of the command. If you purchased a certificate that is valid for more than a year, then the -days 365 portion should be set to the number of days your certificate is valid.

Once you've entered the above command, you'll be asked to enter the details into your CSR. These details (with examples) are:

Country Name (2 letter code) [AU]: US
State or Province Name (full name) [Some-State]: California
Locality Name (eg, city) []: Los Angeles
Organization Name (eg, company) [Internet Widgits Pty Ltd]: ACME Ltd
Organizational Unit Name (eg, section) []: Widgets
Common Name (eg, YOUR name) []: widgets.acme.com
Email Address []:

And two 'extra', comletely optional details:

A challenge password []:
An optional company name []:

If you enter a period '.' into one of the above fields then the field will be left blank.

One thing that seems to get glossed over is that the Common Name field is the fully qualified domain name (FQDN) of your site. So if you're wanting to secure example.com, then that's what you put in the common name field. If you're securing a subdomain like subdomain.example.com, then make sure you enter it like that. Unless you've bought a wildcard certificate, then your certificate will only cover a single domain or subdomain.

WARNING: Make very sure you've entered the Common Name correctly before you submit your CSR. (Yes, I have made this mistake. Don't do important things when you're tired.)

Once you've created the CSR, go to the site you purchased your certificate from and activate it. You'll be asked to enter your CSR into an online form, so open your CSR file up in a text editor and paste the entire contents into the enrollment form, including the BEGIN and END lines.

Installing your Certificate

Once your CSR has been approved, you'll get an email containing your new certificate, and the Root certificates required. Copy these to the folder where you intend to store your certificates. For this example we'll use the SSL localcerts directory we created previously /etc/ssl/localcerts/.

You may want to combine the certificates together into one file. If so, then you can use the cat command to concatenate the files in reverse order of authority. For example, if you've got a Comodo Positive SSL Certificate, then you should have three files; the certificate for your domain (eg. example.com.crt), the intermediate certificate (eg. PositiveSSLCA2.crt), and the root certificate (eg. AddTrustExternalCARoot.crt). So to concatenate these files the command would be:

> sudo cat example.com.crt PositiveSSLCA2.crt AddTrustExternalCARoot.crt >> example.com.chained.crt

Configuring Nginx

You can now set about adding your certificate to your server configuration in Nginx. If you installed Nginx from a package, it should have SSL enabled already, otherwise if you're building from source you'll have to specifically enable it during configuration.

> ./configure --with-http_ssl_module --with_http_spdy_module

If you're using SSL then you may as well use SPDY to help speed up the connection.

Now you can configure your Virtual Hosts for SSL. In the server block of your configuration, you need to set it to listen on port 443, specify the certificate and private key, and SSL protocols and cipher settings.

This is an example of a basic Nginx Virtual Host file with SSL enabled:

server {
  listen 443 ssl spdy;
  server_name example.com;

  root /srv/www/example.com;
  index index.html;

  ssl_certificate /etc/ssl/localcerts/example.com.chained.crt;
  ssl_certificate_key /etc/ssl/localcerts/example.com.key;
}

The listen directive specifies that the server should listen on the HTTPS port, 443, instead of the usual port 80, that SSL is on, and that it is using the SPDY protocol. The ssl_certificate directive points to the location of the chained certificate we created. The ssl_certificate_key directive points to the private key that was created when we created the certificate signing request.

The above provides the minimum configuration required to enable SSL on Nginx. All you need to do is restart Nginx, and you should be able to visit your site using HTTPS.

SSL Security in Nginx

Of course, the above basic configuration only uses the most basic default security, so you'll probably want to do some additional configuration.

The first thing you'll want to do is to set the SSL Ciphers your server will use. This post has a good explanation on the background of using ciphers. The exact list of ciphers you should use is dependent on how secure you want to be, and what performance considerations you have. The ciphers I use are:

ssl_ciphers ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM;

This gives me an 'A' rating on the SSL Server Test, while still providing decent performance.

The rest of my SSL configuration is as follows:

ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ecdh_curve secp521r1;

ssl_session_timeout 5m; is the time it takes for the client's ssl session information to be removed from the cache. A longer time means that the ssl session won't have to be renegotiated as often, improving performance if there are a lot of connections. 5 minutes is the default time for many client browsers.

ssl_prefer_server_ciphers on; means that the server will prefer to use ciphers specified in the ssl_ciphers directive over the ciphers preferred by clients.

ssl_session_cache shared:SSL:10m; lets Nginx use its own cache instead of the one provided by OpenSSL, thus allowing Nginx to separate SSL jobs between its own workers.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; is the list of supported protocols, which in this case is only TLSv1 or greater. SSLv3 and below are not recommended due to security vulnerabilities, and most browsers support TLS now.

ssl_ecdh_curve secp521r1; is the Elliptic curve key used for the ECDHE cipher.

If you want a much more in-depth look at securely configuring Nginx, then calomel.org has an excellently thorough post detailing nearly everything you need to know.

Redirecting to HTTPS

If you want to redirect all your traffic to the SSL version of your site, then you'll need to specify another Virtual Host to accept HTTP traffic and redirect it to HTTPS.

server {
  listen 80;
  server_name example.com;
  return 301 https://example.com$request_uri;
}

This server block listens on port 80, and then performs an HTTP 301 redirect to the requested URI. As this is a permanent redirect of a full URL then return rather than rewrite is the correct Nginx directive to use.

Conclusion

You should now have a secure implementation of SSL on your website. You may want to use tools such as pagespeed to test the speed of your site, as it may require a bit of tweaking to get the best balance between performance and security.

Installing Tiny Tiny RSS from scratch

David Beath — Wed, 02 Oct 2013 03:43:39 GMT

Considering that a couple of times now I've been a proponent of using an RSS reader, I figured it was time I wrote a tutorial on how to install Tiny Tiny RSS, using Nginx as the server and PostgreSQL for the database. While as with any piece of software there are a number of tutorials and guides already out there, I've found that none of them provided a complete instruction set that didn't have me running into and searching for solutions to multiple errors. To that end, this tutorial aims to both provide as complete a set of fool-proof instructions as I can make, both for newbies to self-hosting, and simply as a reference for my future self should I need to do this again.

This set of instructions assumes that you're running a Debian based Linux distro. I've tested this setup on a clean Ubuntu Server 12.04 virtual machine, and the settings are roughly the same as I'm using on my Debian server. It's assumed that you either know how to set up a server already, or are capable of reading a tutorial to do so (If not, then why are you reading this? Go use Feedly). There are some very good tutorials at Digital Ocean and Linode, and I'd recommend either of them for hosting purposes.

Installation

On to the install. While not strictly necessary, the first thing I recommend is to make sure that your server is using the correct time, so install ntp to keep the system clock updated.

> sudo apt-get install ntp

For most of the commands in this tutorial you'll need super user privileges, so you can either run them as root (not recommended) or use sudo like I've used in this tutorial. You'll also need to edit a bunch of files in a text editor. I've used nano for this tutorial because it's easier to use and you can just copy the commands straight, but I usually use vim.

PHP

Unfortunately, despite the fact that PHP sucks (seriously, read the article, even if you disagree it's an interesting read), Tiny Tiny RSS is built with PHP. However, just because a language sucks doesn't mean that programs written in it necessarily have to, and tt-rss is the best of the self-hosted RSS readers. Still, that means you're going to have to install PHP, and get it working nicely with Nginx.

Install PHP 5 and the components necessary to get it playing nicely with everything else.

> sudo apt-get install php5 php5-fpm php5-curl php5-pgsql php5-gd php5-mcrypt php5-cli

There's one line in php.ini you'll need to change, which will prevent a possible security issue.

> sudo nano /etc/php5/fpm/php.ini

Change this line:

cgi.fix_pathinfo=1

To this:

cgi.fix_pathinfo=0

The other change is to make PHP-FPM listen on a UNIX socket rather than a TCP socket.

> sudo nano /etc/php5/fpm/pool.d/www.conf

Change the line:

listen = 127.0.0.1:9000

To:

listen = /var/run/php5-fpm.sock

Finally, restart PHP.

> sudo /etc/init.d/php5-fpm restart

Nginx

Now that PHP is configured, you'll want to configure Nginx. There's a very good primer by Martin Fjordvald on the basics of Nginx configuration which you can read to get an understanding of what's going on. If you want to dig deeper for various other settings, the documentation for Nginx is quite thorough.

This tutorial is going to have you build Nginx from source, since the .deb packages are not always up to date. It's really not that much harder than setting it up from a package install anyway.

First, install the packages required to compile the source.

> sudo apt-get install gcc make libpcre3-dev openssl libssl-dev

I also prefer to have Nginx use a dedicated system account with no login or password access for a bit of extra security.

> sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then download and unpack the latest Nginx version.

> wget https://nginx.org/download/nginx-1.5.6.tar.gz

> tar -xzvf nginx-1.5.6.tar.gz

> cd nginx-1.5.6

The configure command sets the parameters for the build. This is where you can change the default install directory and the different modules that Nginx uses. Nginx doesn't allow you to change modules without rebuilding, so you'll need to select or deselect any modules here. The only module I'll add is the http_ssl module, which will allow you to add SSL to your site later if you wish. I also prefer to just use the default install directory, so I won't change that, but I'll set the user and group Nginx uses to the one I created earlier.

> ./configure --user=nginx --group=nginx --with-http_ssl_module

Now we can build and install Nginx.

> make && sudo make install

Nginx Configuration

Change to the directory you installed Nginx to:

> cd /usr/local/nginx

There are various methods people use for controlling the sites on their server. The .deb install uses a sites-enabled and sites-disabled style configuration, but since I don't have many sites on my server, I prefer to keep each site configuration file in the conf/ folder, and include each one manually in nginx.conf.

So, for this tutorial, we'll create a ttrss.conf file.

> sudo nano conf/ttrss.conf

My ttrss.conf looks something like this (except I have SSL enabled):

# Tiny Tiny RSS Configuration

server {
  listen 80;
  server_name domainname www.domainname;

  root /var/www/ttrss;
  index index.php;

  error_log /var/log/nginx/ttrss.error.log;
  access_log /var/log/nginx/ttrss.access.log;

  location / {
    try_files $uri $uri/ /index.php;
  }

  location ~ \.php$ {
    include fastcgi.conf; # don't use fastcgi_params
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
  }
}

A brief primer:

listen: the port that the server listens for incoming connections on.
server_name: the FQDN (fully qualified domain name) that the server uses for this connection. I set this to listen on both example.com and www.example.com.
root: the root directory that the site's files are located at.
index: the index file that the server directs traffic to. In this case you only need index.php, but can also include index.html or .htm, or any other file you want to use as a homepage.
error_log: the location of the error log for this site.
access_log: the location of the access log for this site. The first location block accepts all incoming requests:
try_files: this just makes the URL search engine friendly. Not really necessary in this case, but it doesn't hurt. The second location block catches all incoming php requests:
include fastcgi.conf: this file comes with Nginx and includes all the parameters for using fastcgi. It's almost the same as fastcgi_params, but you'll have issues using fastcgi_params in Ubuntu 12.04, unless you include the line SCRIPT_FILENAME $document_root$fastcgi_script_name.
fastcgi_pass: the socket that fastcgi passes requests through.
fastcgi_index: the index file that fastcgi uses.

Next, you'll need to edit nginx.conf.

> sudo nano conf/nginx.conf

The only change that you really need to make to this file is to add the line include ttrss.conf in the http block, but this is my recommended configuration below.

# Nginx Configuration

user nginx nginx;
worker_processes auto; # optimal value is number of cpu cores

error_log /var/log/nginx/error.log

events {
  worker_connections 2048;
}

http {
  # Virtual Host Configuration
  include ttrss.conf;

  # Basic Settings
  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  client_header_timeout 20s;
  client_body_timeout 20s;
  send_timeout 20s;

  # Disable Nginx version number in error pages and Server header
  server_tokens off;

  # Silently block all undefined vhost access
  server {
    server_name _;
    return 444;
  }

  # Gzip Settings
  gzip on;
  gzip_disable "msie6";

  gzip_comp_level 5;
  gzip_min_length 256;

  gzip_vary on;
  gzip_proxied any;
  gzip_types *;
}

Finally, start Nginx.

> sudo /usr/local/nginx/sbin/nginx

If you need to make any changes to Nginx after it's been started, just open and change the config files. Then run this command to reload the Nginx with the new configuration.

> sudo /usr/local/nginx/sbin/nginx -s reload

Postgres

Tiny Tiny RSS recommends using PostgreSQL over MySQL for the database, as PostgreSQL is slightly faster. For a single user, it probably wouldn't make much difference, depending on how many feeds you have. The instructions for installing PostgreSQL are mostly taken from the Ubuntu community wiki here.

Install PostgreSQL.

> sudo apt-get install postgresql

> sudo -u postgres psql postgres

Change the password for the postgres user. Type in the command below, hit enter, then enter as password. Make sure to save all your passwords in a good password manager somewhere. I recommend KeePass.

> \password postgres

While you're still in the postgres command line, create a database and database user for tt-rss. The password must be entered surrounded by single quotes.

postgres=# CREATE USER ttrss WITH PASSWORD 'password';

postgres=# CREATE DATABASE ttrssdb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrssdb to ttrss;

Exit the postgres command line.

postgres=# \q

From this point, you might be able to get tt-rss to access the database, but I wasn't until I edited a line in the postgresql config to allow access.

> sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Add this line to allow tt-rss to use the database:

local all ttrss md5

Now restart PostgreSQL.

> sudo service postgresql restart

Tiny Tiny RSS

Finally, it's now time to install Tiny Tiny RSS itself. Yeah, I know it takes a while to get here. I assure you, it's worth it in the end, and you shouldn't have to worry about it much at all once you've got it properly set up. The installation instructions for tt-rss can be be found here, but once again they seem somewhat incomplete.

First, make a directory for tt-rss to be served from. This should be the same as the directory you specified as root in your ttrss.conf file for Nginx.

> sudo mkdir -p /var/www/ttrss

You'll need to set the owner of the directory to the user that you're installing and running tt-rss as.

> sudo chown -R user:group /var/www/

Now change to the directory above.

> cd /var/www

Download and extract the latest version of Tiny Tiny RSS.

> wget https://github.com/gothfox/Tiny-Tiny-RSS/archive/1.10.tar.gz

> tar -xzvf 1.10.tar.gz

Rename the extracted directory so that it matches the root directory you specified earlier, then open the directory.

> mv Tiny-Tiny-RSS-1.10/ ttrss

> cd /var/www/ttrss

The following folders need to have their permissions changed to be writable by anyone with an account on the system, or tt-rss won't be able to save files there.

> sudo chmod -R 777 cache/images/ cache/js/ cache/export/ cache/upload/ feed-icons/ lock/

Now you should be able to navigate in your browser to the site that you are hosting tt-rss from. This should be the same as the server_name you specified in ttrss.conf. If you installed tt-rss into a sub-folder of the domain, then the address will be https://domainname/ttrss/. If you got the configuration correct, you should see the Tiny Tiny RSS install page.

The installation page will have a few fields to enter. These will be the credentials for tt-rss to use the database, as well as the full URL that you'll access tt-rss from. Once you've entered the database details, click the 'Test Configuration' button. If there are errors, you'll get a message saying so, and it shouldn't be too hard to figure out what to do from there. If it's good to go, you'll see a text box with a lot of configuration lines. Copy that config data to your clipboard, and then into a config file on your server with:

> sudo nano /var/www/ttrss/config.php

Save that file, and reload tt-rss in your browser. You should now be able to login with the default user admin and password password. Once you're logged in, change the admin settings in the preferences at top right.

The final required step is to set up automatic updating of your feeds. You can skip this, but you'll have to manually click on each feed to check for updates if you don't. The screen command will permanently run feed updates in the background, defaulting to once every 30 minutes.

> screen -d -m php ./update_daemon2.php

Conclusion

You should now have a working installation of Tiny Tiny RSS on your server, accessible from anywhere in the world. If you've used RSS Readers before, you can import your feeds from an OPML file, otherwise, find some good blogs and start adding feeds.