David Beath (Posts about tutorial)

Nginx and ELB Proxy Protocol Forwarding

David Beath — Mon, 17 Apr 2017 10:35:06 GMT

Recently I noticed that an application I'm hosting on AWS Elastic Beanstalk wasn't logging the Client IP Address of my users. After a lot of digging I found the issue was to do with the configuration of the Nginx servers running on the EC2 instances and in my application's Docker container, and in the TCP passthrough configuration of the Elastic Load Balancer.

So here's a quick post about my particular setup and the configuration I used to fix this issue of Client IP Addresses not being forwarded.

Elastic Beanstalk Setup

In a little more detail, my setup consists of an application hosted within a Docker container running on EC2 instances configured by Elastic Beanstalk, which sit behind an Elastic Load Balancer (ELB). Because of issues with Certificate support in ELB, I'm not using AWS's Certificate Manager or using the built-in SSL termination in ELB, but instead using TCP passthrough on the ELB, and using a custom Nginx configuration on each EC2 instance to terminate the SSL connection. The Docker container on each instance also has it's own Nginx configuration, which proxies the application server, in my case UWSGI.

The steps that a request takes to get to the application are:

User makes an HTTPS request.
ELB uses TCP passthrough to pass the encrypted request to an EC2 instance.
Nginx in EC2 decrypts the HTTPS request and passes the HTTP to it's Docker container.
The Nginx server on Docker proxies the request to UWSGI.
The application hosted by UWSGI handles the request.

In all, the parts that you need to configure to forward the Client IP Address are the TCP passthrough on ELB and each of the two Nginx servers.

Proxy Protocol

The first problem is that if you're using a TCP load balancer to pass through the request, the load balancer will not add an X-Forwarded-For header, and so the downstream Nginx server will only see the IP Address of the load balancer. The problem and part of the solution is described in better detail here. The linked solution is a little different from what I'm going to show below, because I'm proxying the request another hop down the line, but please read the link to better understand what's going on.

The first step is to enable Proxy Protocol support on the Elastic Load Balancer. AWS has a good tutorial on how to do that.

Once the ELB has Proxy Protocol support enabled, you can configure the Nginx server on the EC2 instances. The relevant parts of the configuration look something like this:

log_format elb_log '$proxy_protocol_addr - $remote_user'
                    '[$time_local] "$request" '
                    '$status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

server {
    listen 443 ssl http2 proxy_protocol;

    server_name localhost;

    set_real_ip_from  172.31.0.0/20;
    real_ip_header    proxy_protocol;

    access_log /var/log/nginx/elb-access.log elb_log;

    # SSL Certificate settings go here.

    location / {
        proxy_pass          https://docker;
        proxy_http_version  1.1;

        proxy_set_header Connection         "";
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $proxy_protocol_addr;
        proxy_set_header X-Forwarded-For    $proxy_protocol_addr;
        proxy_set_header X-Forwarded-Proto  $scheme;
    }
}

Here's what's happening in the above config:

The listen directive now has proxy_protocol.
set_real_ip_from tells Nginx the real CIDR range of addresses that the ELB is using.
real_ip_header tells us that we're using the proxy_protocol value to access the client headers.
X-Real-IP and X-Forwarded-For use the $proxy_protocal_addr, which is the client IP information taken from the proxy protocol.
The elb_log format is now using the $proxy_protocal_addr for the client IP.

Also note that the listen directive can support both http2 and proxy_protocol, so if your version of Nginx supports HTTP/2 then you can enable that just fine. However, the proxy_http_version directive in this case is still 1.1, as it doesn't make sense to use HTTP/2 for proxy backends.

If you deployed the above config, you should now see that Nginx server on the EC2 instance is now logging the correct client IP Address.

Multiple Nginx Proxies

Now that we have the Nginx server on the EC2 instance receiving the correct Client IP from the Load Balancer, we need to foward that IP on to the Nginx server in the Docker container.

The X-Forwarded-For and X-Real-IP directives in the above config will properly forward the Client IP from the EC2 instance's Nginx server. However, Nginx appends each proxy's IP address to the X-Forwarded-For header, as described in more detail here.

The solution to this is in the last Nginx proxy configuration is to include the IP address ranges of all previous known proxies in the set_real_ip_from directive. You can then set the real_ip_header directive to X-Forwarded-For. The final piece is to set real_ip_recursive to on, which will return the leftmost (original) IP instead of the rightmost (most recent) IP from the X-Forwarded-For header.

My Nginx configuration for the Docker container now looks like this:

upstream app {
    server unix:///tmp/uwsgi.sock;
}

server {
    listen 80 default_server;

    set_real_ip_from    172.31.0.0/20;    # IP of the ELB
    set_real_ip_from    172.17.0.0/20;    # IP of the EC2 instance
    real_ip_header      X-Forwarded-For;
    real_ip_recursive   on;

    location / {
        uwsgi_pass app;
        include uwsgi_params;
    }
}

You can see that I've included the IP ranges of both the ELB and the EC2 instance. With a config similar to this you should now see the Nginx instance in the Docker container log the correct Client IP, and pass that IP through to your application.

Express.js 4.0 BasicAuth

David Beath — Wed, 23 Apr 2014 10:01:25 GMT

Express 4.0 was recently released, with one of the major changes being that connect middleware is no longer available on the express module. This means no basic http authentication unless you do it yourself. Other people may find this easy, but I had to look it up, so to help people like me here are some quick instructions on getting it going again.

First, visionmedia has release a package called "basic-auth", but that simply parses req.headers.authorization into a user object like { name: 'foo', pass: 'bar' }. It's useful, so install it.

> npm install basic-auth

Next we need to require basic-auth and create some middleware to handle the actual authentication.

var basicAuth = require('basic-auth');

var auth = function (req, res, next) {
  function unauthorized(res) {
    res.set('WWW-Authenticate', 'Basic realm=Authorization Required');
  return res.send(401);
  };

  var user = basicAuth(req);

  if (!user || !user.name || !user.pass) {
    return unauthorized(res);
  };

  if (user.name === 'foo' && user.pass === 'bar') {
    return next();
  } else {
    return unauthorized(res);
  };
};

Basic-auth parses the req.authorization.header into a user object. If the user object does not exist, then return unauthorized. If the user object exists, then check it against the authorization, if correct return next, else return unauthorized.

All you need to do now is to include the auth variable in the routes you'd like authenticated, like this:

app.get('/', auth, function (req, res) {
  res.send(200, 'Authenticated');
};

And there you have your basic authentication back. The original code for the Express 3 basic auth can be found here.

Testing HTTP Responses in Node.js

David Beath — Mon, 03 Mar 2014 01:29:39 GMT

It generally goes without saying these days that testing is an important part of the development process. However, testing is not always the easiest thing to master, and it can be difficult to know where to start, especially when it comes to learning a new language. Node.js adds an extra twist on this difficulty, in that your code is running Asynchronously, and you will often be dealing with HTTP calls.

Having just been through the process of learning Node.js and working on a simple application that deals with calling and receiving responses from an external API, the most painful thing I found was that there aren't many tutorials on how to test HTTP requests and responses that I find particularly clear. So, this tutorial aims to provide clear information on how to effectively test your requests and responses, with a little bit about testing emitted events.

Setting up the Test Environment

For these tests we'll be using the Mocha testing framework along with the Chai assertion library. We'll also use the popular Request library to send requests to our server.

Create a new folder for you project and install Mocha, Chai, and Request to it by running:

> npm install mocha chai request

By default, Mocha looks for tests in a folder called /test within the project root, so create the folder with:

> mkdir test

In order to run our tests we'll set up a Makefile in the project root, the code of which looks like this:

REPORTER = spec

test:
    @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --ui tdd

test-w:
    @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --growl \
        --ui tdd \
        --watch

.PHONY: test test-w

Be careful when you're creating the Makefile that you indent the lines with Tab characters and not spaces.

You can now run your tests once with:

> make test

You should see something like 0 passing (2ms), which indicates that the test suite ran but didn't find any tests. If you see an error then the most likely options are probably that you've got spaces in your Makefile, or that Make can't find Mocha, in which case check that you have a /node_modules directory in your project folder and that Mocha is installed to it.

Alternatively, you can also run the tests continuously in the background with:

> make test-w

The make test command should also be added to the scripts section of your /package.json file, and Mocha, Chai, and Request to devdependencies; like so:

{
  "scripts": {
    "test": "make test"
  },
  "devDependencies": {
    "mocha": "*",
    "chai": "*",
    "request": "*"
  }
}

Creating our Server

Next we'll create the server that we'll be testing (Yes, by most standards writing the code before the tests is backasswards, but this is for demo purposes). This server will listen for requests and respond according to the content of the request header. If the request has a Content-Type header of 'text/plain' it will respond with an HTTP 200 "OK" code and emit an Event called success containing the body of the request. For any other Content-Type header we will return an HTTP 400 "Bad Request" error.

Create a server.js file in a new lib folder in your project directory, with the code below:

// ./lib/server.js

var http = require('http');

var server = module.exports = http.createServer(function (req, res) {

  if (req.headers['content-type'] === 'text/plain') {

    var body = '';
    req.on('data', function (chunk) {
      body += chunk.toString();
    });

    req.on('end', function () {
      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('correct header');
      server.emit('success', body);
    });

  } else {
    res.writeHead(400, {'Content-Type': 'text/plain'});
    res.end('wrong header');
  };
});

Setting up the Tests

Before we write our first test, we need to do a little more setup. The first thing we will need to do is to require Chai's expect function in our test suite. So we'll create a file in our /test folder called tests.js:

> touch test/tests.js

And with the first two lines we'll require the expect function from Chai:

var expect = require('Chai').expect;

You may have noticed that our server doesn't start automatically, as it has no listen call. The second line of our server code contains a module export that allows our server to be imported into another file. We can then start the server before our tests run, and close it when they finish. To do this we need to import our server into tests.js, and within the main 'describe' function in which we write our tests we'll include 'before' and 'after' functions.

// ./test/tests.js

var server = require('../lib/server.js');

describe('server response', function () {
  before(function () {
    server.listen(8000);
  });

  after(function () {
    server.close();
  });
});

Writing the Response Tests

We can now get round to actually writing our tests. We'll write three tests, in order to test each response from our server; the incorrect header response, the successful header response, and the data emitted on a successful response. First, we need to require the Request library at the beginning of tests.js:

var request = require('request');

Now we can include the test within our 'describe' function, immediately below the 'after' function:

it('should return 400', function (done) {
  request.get('https://localhost:8000', function (err, res, body){
    expect(res.statusCode).to.equal(400);
    expect(res.body).to.equal('wrong header');
    done();
  });
});

The it statement, just like describe, before, and after, follows the BDD standard for writing tests. We begin by describing what the test should return, and then a function containing the tests. Because our code is Asynchronous, we need to invoke a callback (in this case called done) so that Mocha knows to wait until the test is complete.

The second line begins our request to the server. We're calling request with a GET (though it could just as well be POST) and passing in the address of our server; in this case localhost listening on port 8000. There is also a function with error, response, and body callbacks. Within the function we're listening for a response, and this is where the test really happens.

The Chai assertion library works by chaining together natural language assertions, making the tests fairly easy to read. In this case we're expecting the StatusCode of the response to equal 400 and the body of the response to read wrong header. The final part of the code is the done() callback; without which the test will exit with a timeout error.

Our second test will test the response if the request contains the Content-Type: text/plain header.

it('should return 200', function (done) {
  var options = {
    url: 'https://localhost:8000',
    headers: {
      'Content-Type': 'text/plain'
    }
  };
  request.get(options, function (err, res, body) {
    expect(res.statusCode).to.equal(200);
    expect(res.body).to.equal('correct header');
    done();
  });
});

This test is much like the first, expect we've added an options object containing the destination url and headers of our request. This is passed as the first parameter of the request, so that our request will be sent to https://localhost:8000 with the correct Content-Type header. By creating an options object like this it become very easy to use the Request library to create a request with any parameters we'd like.

These two tests show the basic form for creating thorough HTTP Server tests. By adding more expect statements and adding different parameters to the chain, you can easily test all the different parts of your responses.

Testing Emitters

Now that we can test our responses, I figure that the next step might be to test what our code is emitting internally. While not technically part of the scope of this article, if you're developing and testing HTTP Requests and Responses in Node.js, at some point your going to be using event emitters and listeners, and in fact you already have. In our server the code req.on('data', function () {}); is an event listener, listening for the data event that is emitted when we receive data. Same with the req.on('end') function.

As you can see if you look back at our server code, once we receive the end event, we send out our own success event that contains the body of the received request. To test that we're successfully emitting, we'll add a body to our request, and we'll need to set a timeout in our test to make sure that the event is firing within a reasonable time.

it('should emit request body', function (done) {
  var options = {
    url: 'https://localhost:8000',
    headers: {
      'Content-Type': 'text/plain'
    },
    body: 'successfully emitted request'
  };
  var eventFired = false;

  request.get(options, function (err, res, body) {});

  server.on('success', function (data) {
    eventFired = true;
    expect(data).to.equal('successfully emitted request');
  });

  setTimeout( function () {
    expect(eventFired).to.equal(true);
    done();
  }, 10);
});

We have now attached a body to our request, that should be emitted back to us, and an eventFired variable set to false, that will be set to true when the event fires. The expect statements are no longer in the request function, as we are not testing the HTTP response. The code server.on('success', function (data) {}) listens for the success event, and passes the data from that event into a function. Within the function we set eventFired to true, and test to see that the emitted data is correct. The setTimeout function is required in order to properly test that the event fired. If the event does not fire, then the test will fail gracefully rather than with a timeout error.

Conclusion

We now have some basic tests that should reasonably cover the server code. This basic setup should work with most Node.js projects, and the general idea of this sort of testing is an essential skill in any programming language. If you'd like to have a look at or download and test the full code yourself then it can be found here.

Configuring SSL in Nginx

David Beath — Sun, 17 Nov 2013 03:32:49 GMT

This post aims to be a fairly thorough guide to getting SSL up and running on an Nginx server, largely for my own reference in future, but it may prove useful to anyone else who stumbles across it. I'm going to assume that you have a working installation of Nginx, or know how to get one going. If not, then bookmark this page and find a tutorial on how to get it going first, such as the section on Nginx in my post here, or the Installation and Beginner's Guide pages on the official site.

Get an SSL Certificate

The first step in setting up SSL on your server is to procure an SSL Certificate from an approved Certificate Authority (CA). The easiest way to do this is to purchase one from a certificate reseller such as CheapSSLs.com.

For a single site with no subdomains, such as a blog, that doesn't require all the extra features of more expensive certificates, then a certificate such as Comodo Positive SSL will do fine.

Once you've purchased the certificate, you'll need to create a Certificate Signing Request.

Creating a CSR

A Certificate Signing Request (CSR) is a file that contains the application information for your certificate, such as your location and the name of the domain you want to secure, as well as your Public Key. For creating the CSR we'll be using OpenSSL, so if you don't have that on your server you can install it with the following command:

> sudo apt-get install openssl

Before we start creating files, it's recommended that you have a proper location to store them in. On Linux systems this would be something like /etc/ssl/localcerts/, so lets start there.

> sudo mkdir /etc/ssl/localcerts
> sudo cd /etc/ssl/localcerts

To generate the public and private keys that will be used for the CSR, use this command:

> sudo openssl req -nodes -newkey rsa:2048 -days 365 -keyout myserver.key -out server.csr

This will create two files; myserver.key, the private key file, and server.csr, the Certificate Signing Request. When you enter the above command, you should of course set the name of both files to one appropriate for the server you are securing.

Generally your CA should require you to use RSA 2048, this is specified with the -newkey rsa:2048 portion of the command. If you purchased a certificate that is valid for more than a year, then the -days 365 portion should be set to the number of days your certificate is valid.

Once you've entered the above command, you'll be asked to enter the details into your CSR. These details (with examples) are:

Country Name (2 letter code) [AU]: US
State or Province Name (full name) [Some-State]: California
Locality Name (eg, city) []: Los Angeles
Organization Name (eg, company) [Internet Widgits Pty Ltd]: ACME Ltd
Organizational Unit Name (eg, section) []: Widgets
Common Name (eg, YOUR name) []: widgets.acme.com
Email Address []:

And two 'extra', comletely optional details:

A challenge password []:
An optional company name []:

If you enter a period '.' into one of the above fields then the field will be left blank.

One thing that seems to get glossed over is that the Common Name field is the fully qualified domain name (FQDN) of your site. So if you're wanting to secure example.com, then that's what you put in the common name field. If you're securing a subdomain like subdomain.example.com, then make sure you enter it like that. Unless you've bought a wildcard certificate, then your certificate will only cover a single domain or subdomain.

WARNING: Make very sure you've entered the Common Name correctly before you submit your CSR. (Yes, I have made this mistake. Don't do important things when you're tired.)

Once you've created the CSR, go to the site you purchased your certificate from and activate it. You'll be asked to enter your CSR into an online form, so open your CSR file up in a text editor and paste the entire contents into the enrollment form, including the BEGIN and END lines.

Installing your Certificate

Once your CSR has been approved, you'll get an email containing your new certificate, and the Root certificates required. Copy these to the folder where you intend to store your certificates. For this example we'll use the SSL localcerts directory we created previously /etc/ssl/localcerts/.

You may want to combine the certificates together into one file. If so, then you can use the cat command to concatenate the files in reverse order of authority. For example, if you've got a Comodo Positive SSL Certificate, then you should have three files; the certificate for your domain (eg. example.com.crt), the intermediate certificate (eg. PositiveSSLCA2.crt), and the root certificate (eg. AddTrustExternalCARoot.crt). So to concatenate these files the command would be:

> sudo cat example.com.crt PositiveSSLCA2.crt AddTrustExternalCARoot.crt >> example.com.chained.crt

Configuring Nginx

You can now set about adding your certificate to your server configuration in Nginx. If you installed Nginx from a package, it should have SSL enabled already, otherwise if you're building from source you'll have to specifically enable it during configuration.

> ./configure --with-http_ssl_module --with_http_spdy_module

If you're using SSL then you may as well use SPDY to help speed up the connection.

Now you can configure your Virtual Hosts for SSL. In the server block of your configuration, you need to set it to listen on port 443, specify the certificate and private key, and SSL protocols and cipher settings.

This is an example of a basic Nginx Virtual Host file with SSL enabled:

server {
  listen 443 ssl spdy;
  server_name example.com;

  root /srv/www/example.com;
  index index.html;

  ssl_certificate /etc/ssl/localcerts/example.com.chained.crt;
  ssl_certificate_key /etc/ssl/localcerts/example.com.key;
}

The listen directive specifies that the server should listen on the HTTPS port, 443, instead of the usual port 80, that SSL is on, and that it is using the SPDY protocol. The ssl_certificate directive points to the location of the chained certificate we created. The ssl_certificate_key directive points to the private key that was created when we created the certificate signing request.

The above provides the minimum configuration required to enable SSL on Nginx. All you need to do is restart Nginx, and you should be able to visit your site using HTTPS.

SSL Security in Nginx

Of course, the above basic configuration only uses the most basic default security, so you'll probably want to do some additional configuration.

The first thing you'll want to do is to set the SSL Ciphers your server will use. This post has a good explanation on the background of using ciphers. The exact list of ciphers you should use is dependent on how secure you want to be, and what performance considerations you have. The ciphers I use are:

ssl_ciphers ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM;

This gives me an 'A' rating on the SSL Server Test, while still providing decent performance.

The rest of my SSL configuration is as follows:

ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ecdh_curve secp521r1;

ssl_session_timeout 5m; is the time it takes for the client's ssl session information to be removed from the cache. A longer time means that the ssl session won't have to be renegotiated as often, improving performance if there are a lot of connections. 5 minutes is the default time for many client browsers.

ssl_prefer_server_ciphers on; means that the server will prefer to use ciphers specified in the ssl_ciphers directive over the ciphers preferred by clients.

ssl_session_cache shared:SSL:10m; lets Nginx use its own cache instead of the one provided by OpenSSL, thus allowing Nginx to separate SSL jobs between its own workers.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; is the list of supported protocols, which in this case is only TLSv1 or greater. SSLv3 and below are not recommended due to security vulnerabilities, and most browsers support TLS now.

ssl_ecdh_curve secp521r1; is the Elliptic curve key used for the ECDHE cipher.

If you want a much more in-depth look at securely configuring Nginx, then calomel.org has an excellently thorough post detailing nearly everything you need to know.

Redirecting to HTTPS

If you want to redirect all your traffic to the SSL version of your site, then you'll need to specify another Virtual Host to accept HTTP traffic and redirect it to HTTPS.

server {
  listen 80;
  server_name example.com;
  return 301 https://example.com$request_uri;
}

This server block listens on port 80, and then performs an HTTP 301 redirect to the requested URI. As this is a permanent redirect of a full URL then return rather than rewrite is the correct Nginx directive to use.

Conclusion

You should now have a secure implementation of SSL on your website. You may want to use tools such as pagespeed to test the speed of your site, as it may require a bit of tweaking to get the best balance between performance and security.

Ubuntu Post-Install Guide

David Beath — Fri, 15 Nov 2013 01:13:44 GMT

Having reinstalled Ubuntu a few times on various machines, I figured it's about time that I recorded a few of the post installation steps I take, so that I don't have to keep looking them up. The instructions below are for Ubuntu 13.10, and can be done in any order.

Install Bumblebee

Bumblebee is a package that prevents laptops that run Nvidia Optimus graphics from running the dedicated graphics card permanently. Installing this is the first thing I do, otherwise things get quite toasty. Further instructions here.

> sudo add-apt-repository ppa:bumblebee/stable
> sudo apt-get update
> sudo apt-get install bumblebee bumblebee-nvidia primus linux-headers-generic

> sudo shutdown -r now

Disable Apport

Apport is the error reporting service for Ubuntu. Most of the errors it pops up with are pretty minor, and Canonical knows about them by now anyway. You can either get rid of it entirely or just disable it. Instructions from here.

To disable Apport:

> sudo service apport stop

> sudo nano /etc/default/apport

enabled=0

To remove Apport entirely:

> sudo apt-get purge apport

Install Unity-Tweak-Tool

Unity Tweak Tool is always good for tweaking the Unity interface to your preferences. I think the default launcher is much too large, and I like to set the font-size smaller too.

> sudo apt-get install unity-tweak-tool

Install Sublime Text 3

Sublime Text 3 seems to be everyone's favourite text editor these days, and it's what I use to write these blog posts. Instructions courtesy of webupd8.org.

> sudo add-apt-repository ppa:webupd8team/sublime-text-3
> sudo apt-get update
> sudo apt-get install sublime-text-installer

To prevent annoying update notifications, once Sublime Text is open add the following code in: Preferences > Settings - User:

"update_check": false

Install Numix GTK3 Theme

Numix is my favourite GTK3 Theme, and looks good in Unity. Check it out at their website.

> sudo add-apt-repository ppa:numix/ppa
> sudo apt-get update
> sudo apt-get install numix-gtk-theme

Alternatively, for a few other themes as well if you're running 13.10:

> sudo apt-get install shimmer-themes

Remove Ubuntu One

Ubuntu One doesn't really seem to be under active development anymore, and there are better cloud solutions out there, so you might as well remove it. Instructions from here.

> killall ubuntuone-login ubuntuone-preferences ubuntuone-syncdaemon
> sudo rm -rf ~/.local/share/ubuntuone
> rm -rf ~/.cache/ubuntuone
> rm -rf ~/.config/ubuntuone
> mv ~/Ubuntu\ One/ ~/UbuntuOne_old/

> sudo apt-get purge ubuntuone-client python-ubuntuone-*

Disable Login Drum Sounds

I hate login sounds, and a quick search shows a lot of other people do as well, yet Ubuntu no longer has any options to disable them. So if you can't stop it from playing the sound, just remove the file. These instructions should work regardless of version.

> cd /usr/share/sounds/ubuntu/stereo
> mv system-ready.ogg system-ready.ogg.old

Install VLC Player

VLC is the best video player, enough said.‎

> sudo apt-get install vlc

Disable Amazon Shopping

A lot of people really dislike the direction the Canonical has taken by integrating online shopping searches into Unity Lens search, and I agree. There is a button to turn it off in the settings, but I'd just as soon remove it altogether.

Paste the following into your terminal:

> gsettings set com.canonical.Unity.Lenses disabled-scopes "['more_suggestions-amazon.scope', 'more_suggestions-u1ms.scope', 'more_suggestions-populartracks.scope', 'music-musicstore.scope', 'more_suggestions-ebay.scope', 'more_suggestions-ubuntushop.scope', 'more_suggestions-skimlinks.scope']"

Improve Battery Life

Some instructions for a tool that may help improve laptop battery life. Warning: May cause issues with external devices due to powering off USB ports.

> sudo apt-add-repository ppa:linrunner/tlp
> sudo apt-get update
> sudo apt-get install tlp tlp-rdw
> sudo tlp start

Development Tools

So I don't forget, these are some of the packages that are useful to have to development purposes.

> sudo apt-get install python-software-properties python python-pip g++ make python-dev git

Instructions for installing nodejs.

> sudo add-apt-repository ppa:chris-lea/node.js
> sudo apt-get update
> sudo apt-get install nodejs

KeePass

KeePass is my preferred password manager. Keep the password file in Dropbox and you can access it anywhere. The install instructions for Ubuntu are here.

> sudo apt-add-repository ppa:jtaylor/keepass
> sudo apt-get update
> sudo apt-get install keepass2

Dropbox

You can find the download for Dropbox on their site as usual. In Ubuntu 13.10 you may come across an issue where the Dropbox icon isn't showing in the status bar. If so, just install the following package:

> sudo apt-get install libappindicator1

Reading XBee RSSI with Arduino

David Beath — Fri, 11 Oct 2013 01:47:53 GMT

Last year I spent quite a lot of time trying to read the Recieved Signal Strength Indicator from an XBee for a project I was working on. I had planned to blog about the process, but never got round to writing any more than two posts. However, I had a look at the nascent blog for the first time in a while, and it surprised me that it was getting a respectable number of views, especially for something that wasn't particularly informative and hadn't been updated in so long. Having a look at the analytics, I realised that like I been doing, there are still a lot of people trying to figure out how to read RSSI from an XBee to an Arduino. So here's a tutorial and some pointers for getting XBees set up for signal strength reading.

The first thing you're going to need is of course a couple of XBee units. You'll also need an XBee usb connector like this to configure them from your computer. For this tutorial I'll explain how to read the signal strength with an Arduino, though you can just use a PC.

PLEASE NOTE: Unless you want to set up a Wireless mesh network, you should use the XBee 802.15.4 modules, also known as the Series 1 modules. The Series 1 operates a standard point to point network. The Series 2 modules, the XBee ZB, use the ZigBee mesh protocol. Because of this, they do not include RSSI information in the packet, and anyway, the RSSI is only good for the last hop. Make sure you know your required use-case before ordering these modules. You have been warned.

XBee Configuration

Before doing anything with your XBees, read this list of common XBee mistakes. It will save you a lot of time wondering why things aren't working as they should. I also recommend you read the official Getting Started Guide, and consult the Product Manual.

In order to configure your XBees, by far the easiest way is to use X-CTU. Unfortunately this only runs on Windows, though it may be possible to get it working with WINE in Linux. For very basic instructions on how to use X-CTU, look here. Alternatively, you check out the instructions here on how to configure them from a terminal.

Getting the XBees talking to each other should be a fairly simple matter. You'll want the PAN ID and Channel settings to be the same on each XBee you're using, and of course the MY address should be unique to each XBee. If you're using only two XBees to talk to each other, then set the Destination address high to 0 and the Destination low address to the MY address of the other XBee. Otherwise, if you're wanting to broadcast to all XBees listening on the same Channel and PAN ID, then set the Destinaton high address to 0, and the Destination low address to FFFF to enable broadcast mode.

The final setting is to change the API Enable setting to "2", which will allow the Arduino to control the XBee using API commands.

Arduino Configuration

I'm going to assume that if you're reading this tutorial, then you already have a basic understanding of Arduino code, and how to use them. You'll need to download the xbee-arduino library and put it in your Arduino library folder.

In your Arduino code include and intitialise the xbee-arduino library.

#include <XBee.h>

XBee xbee = XBee()

Then the start the serial connection to the XBee and computer in setup.

void setup()
{
  // XBee serial connection
  xbee.begin(9600);

  // Computer serial connection
  Serial.begin(9600);
}

For the purposes of this tutorial, we're going to have one XBee send a packet, and the other read the packet and output the RSSI value. You can have the sending XBee just send from a terminal, and use only one Arduino for receiving, but I'm going to show the code for having both XBees attached to an Arduino. The code about applies to both sender and receiver.

Sending Packets

On your sending XBee/Arduino, before the setup, initialise the payload. You can of course change the payload anywhere within the loop. I'm going to set the payload to "Hi".

uint8_t payload[] = { 'H', 'i' };

Then specify the destination address that you're going to send to. This is the Serial High and Serial Low address of the recieving XBee. If you're broadcasting, then you can skip this step.

XBeeAddress64 addr64 = XBeeAddress64(0x0013a200, 0x403e0f30);

In this example, because we're not dealing with sending updated data, we can create the transmit request in setup. The Tx16Request takes three parameters; the address it's being sent to, the payload, and the size of the payload. If you're broadcasting, then change the address parameter to 0xFFFF.

Tx16Request tx16 = Tx16Request(addr64, payload, sizeof(payload));

Finally, in your loop, send the packet. I'm going to set it to send every 50 milliseconds.

void loop()
{
  xbee.send(tx16);
  delay(50);
}

Receiving Packets and reading RSSI

On your receiver, before setup, you'll first need to initialise a Response object.

  Rx16Response rx16 = Rx16Response();

In the loop, your XBee will first wait for an incoming packet. The value within the brackets specifies how long it shoud continue waiting before continuing through the loop. For this example, because we're only doing the one thing, listening for and reading the signal strength of a packet, we can afford to wait a while.

  xbee.readPacket(100);

Once a packet is received, in order to prevent errors we first check if the response is available, then check if it matches the response type we want. Only then do we actually read the packet.

// Check if available
if (xbee.getResponse().isAvailable())
{
  // Check if packet is correct type
  if (xbee.getResponse.getApiId() == RX_16_RESPONSE)
  {
    // Read the packet
    xbee.getResponse().getRx16Response(rx16);
  }
}

Now you can read the signal strength from the packet and send it to the computer. If you don't want to print out the previous RSSI value every time through the loop whenever a packet isn't received (which you shouldn't), then this code should go within your if statements immediately after you've read the packet.

Serial.print( rx16.getRssi() );

And if you want to print out the payload:

for (int i = 0; i < rx16.getDataLength(); i++)
{
  Serial.print( rx16.getData(i), HEX );
}

Final Code

Your code for the sender should look something like this:

#include <XBee.h>

XBee xbee = XBee()

uint8_t payload[] = { 'H', 'i' };

XBeeAddress64 addr64 = XBeeAddress64(0x0013a200, 0x403e0f30);

Tx16Request tx16 = Tx16Request(addr64, payload, sizeof(payload));

void setup()
{
  xbee.begin(9600);
}

void loop()
{
  xbee.send( tx16 );
  delay(50);
}

And the code for the receiver something like this:

#include <XBee.h>

XBee xbee = XBee()

Rx16Response rx16 = Rx16Response();

void setup()
{
  xbee.begin(9600);
  Serial.begin(9600);
}

void loop()
{
  xbee.readPacket(100);
  if (xbee.getResponse().isAvailable())
  {
    if (xbee.getResponse().getApiId() == RX_16_RESPONSE)
    {
      xbee.getResponse().getRx16Response(rx16);
      Serial.print( rx16.getRssi() );
    }
  }
}

You should now have a couple of XBees that are talking to each other, and can read the signal strength of the packets. For a full example of working code, where a whole bunch of senders broadcast to multiple receivers, which then send the data to an aggregator, check out my Github repository.

Installing Tiny Tiny RSS from scratch

David Beath — Wed, 02 Oct 2013 03:43:39 GMT

Considering that a couple of times now I've been a proponent of using an RSS reader, I figured it was time I wrote a tutorial on how to install Tiny Tiny RSS, using Nginx as the server and PostgreSQL for the database. While as with any piece of software there are a number of tutorials and guides already out there, I've found that none of them provided a complete instruction set that didn't have me running into and searching for solutions to multiple errors. To that end, this tutorial aims to both provide as complete a set of fool-proof instructions as I can make, both for newbies to self-hosting, and simply as a reference for my future self should I need to do this again.

This set of instructions assumes that you're running a Debian based Linux distro. I've tested this setup on a clean Ubuntu Server 12.04 virtual machine, and the settings are roughly the same as I'm using on my Debian server. It's assumed that you either know how to set up a server already, or are capable of reading a tutorial to do so (If not, then why are you reading this? Go use Feedly). There are some very good tutorials at Digital Ocean and Linode, and I'd recommend either of them for hosting purposes.

Installation

On to the install. While not strictly necessary, the first thing I recommend is to make sure that your server is using the correct time, so install ntp to keep the system clock updated.

> sudo apt-get install ntp

For most of the commands in this tutorial you'll need super user privileges, so you can either run them as root (not recommended) or use sudo like I've used in this tutorial. You'll also need to edit a bunch of files in a text editor. I've used nano for this tutorial because it's easier to use and you can just copy the commands straight, but I usually use vim.

PHP

Unfortunately, despite the fact that PHP sucks (seriously, read the article, even if you disagree it's an interesting read), Tiny Tiny RSS is built with PHP. However, just because a language sucks doesn't mean that programs written in it necessarily have to, and tt-rss is the best of the self-hosted RSS readers. Still, that means you're going to have to install PHP, and get it working nicely with Nginx.

Install PHP 5 and the components necessary to get it playing nicely with everything else.

> sudo apt-get install php5 php5-fpm php5-curl php5-pgsql php5-gd php5-mcrypt php5-cli

There's one line in php.ini you'll need to change, which will prevent a possible security issue.

> sudo nano /etc/php5/fpm/php.ini

Change this line:

cgi.fix_pathinfo=1

To this:

cgi.fix_pathinfo=0

The other change is to make PHP-FPM listen on a UNIX socket rather than a TCP socket.

> sudo nano /etc/php5/fpm/pool.d/www.conf

Change the line:

listen = 127.0.0.1:9000

To:

listen = /var/run/php5-fpm.sock

Finally, restart PHP.

> sudo /etc/init.d/php5-fpm restart

Nginx

Now that PHP is configured, you'll want to configure Nginx. There's a very good primer by Martin Fjordvald on the basics of Nginx configuration which you can read to get an understanding of what's going on. If you want to dig deeper for various other settings, the documentation for Nginx is quite thorough.

This tutorial is going to have you build Nginx from source, since the .deb packages are not always up to date. It's really not that much harder than setting it up from a package install anyway.

First, install the packages required to compile the source.

> sudo apt-get install gcc make libpcre3-dev openssl libssl-dev

I also prefer to have Nginx use a dedicated system account with no login or password access for a bit of extra security.

> sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then download and unpack the latest Nginx version.

> wget https://nginx.org/download/nginx-1.5.6.tar.gz

> tar -xzvf nginx-1.5.6.tar.gz

> cd nginx-1.5.6

The configure command sets the parameters for the build. This is where you can change the default install directory and the different modules that Nginx uses. Nginx doesn't allow you to change modules without rebuilding, so you'll need to select or deselect any modules here. The only module I'll add is the http_ssl module, which will allow you to add SSL to your site later if you wish. I also prefer to just use the default install directory, so I won't change that, but I'll set the user and group Nginx uses to the one I created earlier.

> ./configure --user=nginx --group=nginx --with-http_ssl_module

Now we can build and install Nginx.

> make && sudo make install

Nginx Configuration

Change to the directory you installed Nginx to:

> cd /usr/local/nginx

There are various methods people use for controlling the sites on their server. The .deb install uses a sites-enabled and sites-disabled style configuration, but since I don't have many sites on my server, I prefer to keep each site configuration file in the conf/ folder, and include each one manually in nginx.conf.

So, for this tutorial, we'll create a ttrss.conf file.

> sudo nano conf/ttrss.conf

My ttrss.conf looks something like this (except I have SSL enabled):

# Tiny Tiny RSS Configuration

server {
  listen 80;
  server_name domainname www.domainname;

  root /var/www/ttrss;
  index index.php;

  error_log /var/log/nginx/ttrss.error.log;
  access_log /var/log/nginx/ttrss.access.log;

  location / {
    try_files $uri $uri/ /index.php;
  }

  location ~ \.php$ {
    include fastcgi.conf; # don't use fastcgi_params
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
  }
}

A brief primer:

listen: the port that the server listens for incoming connections on.
server_name: the FQDN (fully qualified domain name) that the server uses for this connection. I set this to listen on both example.com and www.example.com.
root: the root directory that the site's files are located at.
index: the index file that the server directs traffic to. In this case you only need index.php, but can also include index.html or .htm, or any other file you want to use as a homepage.
error_log: the location of the error log for this site.
access_log: the location of the access log for this site. The first location block accepts all incoming requests:
try_files: this just makes the URL search engine friendly. Not really necessary in this case, but it doesn't hurt. The second location block catches all incoming php requests:
include fastcgi.conf: this file comes with Nginx and includes all the parameters for using fastcgi. It's almost the same as fastcgi_params, but you'll have issues using fastcgi_params in Ubuntu 12.04, unless you include the line SCRIPT_FILENAME $document_root$fastcgi_script_name.
fastcgi_pass: the socket that fastcgi passes requests through.
fastcgi_index: the index file that fastcgi uses.

Next, you'll need to edit nginx.conf.

> sudo nano conf/nginx.conf

The only change that you really need to make to this file is to add the line include ttrss.conf in the http block, but this is my recommended configuration below.

# Nginx Configuration

user nginx nginx;
worker_processes auto; # optimal value is number of cpu cores

error_log /var/log/nginx/error.log

events {
  worker_connections 2048;
}

http {
  # Virtual Host Configuration
  include ttrss.conf;

  # Basic Settings
  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  client_header_timeout 20s;
  client_body_timeout 20s;
  send_timeout 20s;

  # Disable Nginx version number in error pages and Server header
  server_tokens off;

  # Silently block all undefined vhost access
  server {
    server_name _;
    return 444;
  }

  # Gzip Settings
  gzip on;
  gzip_disable "msie6";

  gzip_comp_level 5;
  gzip_min_length 256;

  gzip_vary on;
  gzip_proxied any;
  gzip_types *;
}

Finally, start Nginx.

> sudo /usr/local/nginx/sbin/nginx

If you need to make any changes to Nginx after it's been started, just open and change the config files. Then run this command to reload the Nginx with the new configuration.

> sudo /usr/local/nginx/sbin/nginx -s reload

Postgres

Tiny Tiny RSS recommends using PostgreSQL over MySQL for the database, as PostgreSQL is slightly faster. For a single user, it probably wouldn't make much difference, depending on how many feeds you have. The instructions for installing PostgreSQL are mostly taken from the Ubuntu community wiki here.

Install PostgreSQL.

> sudo apt-get install postgresql

> sudo -u postgres psql postgres

Change the password for the postgres user. Type in the command below, hit enter, then enter as password. Make sure to save all your passwords in a good password manager somewhere. I recommend KeePass.

> \password postgres

While you're still in the postgres command line, create a database and database user for tt-rss. The password must be entered surrounded by single quotes.

postgres=# CREATE USER ttrss WITH PASSWORD 'password';

postgres=# CREATE DATABASE ttrssdb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrssdb to ttrss;

Exit the postgres command line.

postgres=# \q

From this point, you might be able to get tt-rss to access the database, but I wasn't until I edited a line in the postgresql config to allow access.

> sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Add this line to allow tt-rss to use the database:

local all ttrss md5

Now restart PostgreSQL.

> sudo service postgresql restart

Tiny Tiny RSS

Finally, it's now time to install Tiny Tiny RSS itself. Yeah, I know it takes a while to get here. I assure you, it's worth it in the end, and you shouldn't have to worry about it much at all once you've got it properly set up. The installation instructions for tt-rss can be be found here, but once again they seem somewhat incomplete.

First, make a directory for tt-rss to be served from. This should be the same as the directory you specified as root in your ttrss.conf file for Nginx.

> sudo mkdir -p /var/www/ttrss

You'll need to set the owner of the directory to the user that you're installing and running tt-rss as.

> sudo chown -R user:group /var/www/

Now change to the directory above.

> cd /var/www

Download and extract the latest version of Tiny Tiny RSS.

> wget https://github.com/gothfox/Tiny-Tiny-RSS/archive/1.10.tar.gz

> tar -xzvf 1.10.tar.gz

Rename the extracted directory so that it matches the root directory you specified earlier, then open the directory.

> mv Tiny-Tiny-RSS-1.10/ ttrss

> cd /var/www/ttrss

The following folders need to have their permissions changed to be writable by anyone with an account on the system, or tt-rss won't be able to save files there.

> sudo chmod -R 777 cache/images/ cache/js/ cache/export/ cache/upload/ feed-icons/ lock/

Now you should be able to navigate in your browser to the site that you are hosting tt-rss from. This should be the same as the server_name you specified in ttrss.conf. If you installed tt-rss into a sub-folder of the domain, then the address will be https://domainname/ttrss/. If you got the configuration correct, you should see the Tiny Tiny RSS install page.

The installation page will have a few fields to enter. These will be the credentials for tt-rss to use the database, as well as the full URL that you'll access tt-rss from. Once you've entered the database details, click the 'Test Configuration' button. If there are errors, you'll get a message saying so, and it shouldn't be too hard to figure out what to do from there. If it's good to go, you'll see a text box with a lot of configuration lines. Copy that config data to your clipboard, and then into a config file on your server with:

> sudo nano /var/www/ttrss/config.php

Save that file, and reload tt-rss in your browser. You should now be able to login with the default user admin and password password. Once you're logged in, change the admin settings in the preferences at top right.

The final required step is to set up automatic updating of your feeds. You can skip this, but you'll have to manually click on each feed to check for updates if you don't. The screen command will permanently run feed updates in the background, defaulting to once every 30 minutes.

> screen -d -m php ./update_daemon2.php

Conclusion

You should now have a working installation of Tiny Tiny RSS on your server, accessible from anywhere in the world. If you've used RSS Readers before, you can import your feeds from an OPML file, otherwise, find some good blogs and start adding feeds.

Using Node Modules with Meteor

David Beath — Fri, 20 Sep 2013 09:52:15 GMT

After playing with Meteor for the past week or so, which overall I really enjoyed, the one thing that really bugged me was how to use node.js modules in a Meteor app. There are plenty of instructions out there that sort of tell you what to do, but I found all of them to be variously confusing and unclear. Of course, this isn't helped by the fact that at the time of writing Meteor is only version 0.6.5.1, so things are still in flux. The aim of this post is therefore to provide a comprehensive and clear guide to how I was able to get Meteor to work with Node modules.

The first thing of course is to initialise a new Meteor app. Feel free to skip this step if you're using an existing one.

> meteor create myapp
> cd myapp

Then create a folder called "packages" within the root of your app.

> mkdir packages && cd packages

For each node module or custom package you want to use, create a new folder. For this example I'm going to use feedparser, as that's what I was using.

> mkdir feedparser && cd feedparser

You'll then want to create two javascript files, one called package.js, and the other the name of the module.

> touch package.js
> touch feedparser.js

In package.js, declare the module you want to use, making sure to specify the version like so:

Package.describe({
    summary: "Reads RSS feeds"
});
Npm.depends({feedparser: "0.16.1"});

Next, add the following lines, or the equivalent for your requirements, to package.js. You can find more information about what you can do here in the Meteor documentation, and by looking at the packages directory in the Meteor source tree. For me, these lines were what I required to access the module within my Meteor app.

Package.on_use(function(api){
    api.add_files('feedparser.js', 'server');
    if(api.export)
        api.export('Feedparser');
});

Now, just create your variable to export by calling Npm.require() on it in the other .js file you created.

Feedparser = Npm.require('feedparser');

Finally, in your application root folder, add the package to your app:

> meteor add feedparser

Now Meteor will update the dependencies for the package. Once this is done, you should be able to use the module within your app by calling the api you exported.

var parser = Feedparser;

Hopefully now you'll have a working Node module within your Meteor app. You can view the full code for this example here. If you know a better or easier way to do this, or a more proper way, then please email me to say so.