David Beath (Posts about IT)

Feedsearch API Updates

David Beath — Thu, 07 Nov 2019 16:50:14 GMT

Over the past year I've been slowly working on improvements to the Feedsearch API that I published in late 2017, and wrote about in my previous post.

The biggest difference is that the old Feedsearch library has now been replaced with the new Feedsearch Crawler library, which is available for use in other projects as a PyPI package. The crawler is now able to make HTTP requests asyncronously, drastically speeding up the number of URLs that can be checked for feeds in the same amount of time, which in turn increases the feed detection rate in cases where the feed URL is not properly advertised.

The second new feature is that all discovered feeds are now saved to a database, along with the website and the URL paths where they were discovered. In this way we can now return a list of all feeds that have been discovered on a site, and remove the need to crawl a site or URL every time a feed search is requested.

In addition to these, I've extended the results provided by the API to add few new fields:

content_length: The length of the feed in bytes at the time it was last seen.
is_podcast: Whether or not the feed contains valid podcast elements and enclosures.
last_seen: The date that the feed was last crawled by Feedsearch.
last_updated: The date that the feed was last updated by the publisher.
velocity: A calculation of the mean number of entries per day in the feed, at the time that the feed was last crawled.

Feedsearch Crawler

The Feedsearch Crawler is built as a separate library from the API, so that it's available for others to use, fork, and extend as they see fit.

I had been thinking for a few years now that what was really needed for feed detection was to use a proper Web crawler. As I kept upgrading the original Feedsearch library to improve detection rates and add features such as Favicon fetching, the time needed perform a search kept increasing, due to the synchronous design of the Requests HTTP library. It was at the time that I started adding asynchronous HTTP requests to Feedsearch, and therefore requiring that I keep track of which URLs have been requested, that I realised I was just building a crappier Web crawler, without a proper architecture. So I built my own, because why not?

The design and features of the base crawler itself are nothing special, but the FeedsearchSpider built on top of it is meant to be as discriminating as possible in what it crawls, while still finding as many feeds as possible. In order to do this the spider uses a number of regular expressions to filter page links that may lead to feeds, and then prioritises them for the crawl queue.

Feedsearch Gateway API

The API is built as the Feedsearch Gateway project, and is responsible for all the custom functionality that is not required in the generic crawler library, such as providing the actual API and dealing with saving results.

When crawling websites, it's not particularly polite to use a lot of resources, and so I needed the ability to cache the results. Because I already had Feedsearch running on AWS Lambda, I decided to use DynamoDB for the database. Whenever a search is completed, the following information is saved to the database:

All feeds found in the crawl and their metadata.
The host URL of the site and the time it was crawled.
The URL of the path that was crawled, the time it was crawled, and the URLs of the feeds that were found from that path.

When a search is started, the API first queries the database for the feeds that were already found and the site information. If the search is for the host URL of the site (e.g. https://example.com), then the site is only crawled again if it hasn't been crawled recently, and all feeds that have ever been found to belong to the site are returned.

If the search contains a path (e.g. https://example.com/path/testing/) then the database is checked to see if that path has already been crawled. If the path has been crawled recently, then the list of URLs found at that path are checked against the sites feeds, and the matching feeds are returned. When the path is searched, only feeds found from the crawl of that path are returned. This is done to increase the chance that only feeds relevant to that particular query are returned, especially if the site contains a lot of feeds.

Finally, the Feedsearch site itself is run behind Cloudflare Workers. Cloudflare doesn't cache HTML by default, but because the API homepage is designed to be as static as possible, it makes sense to cache it with the Workers cache API, so that we don't have to make a request to the Lambda function every time the page is requested. Instead, only requests to the API route are forwarded to the Lambda.

Feedsearch API

David Beath — Fri, 08 Dec 2017 10:03:53 GMT

I've just published a new microsite and API called Feedsearch, which provides an easy way to search websites for RSS, Atom, and JSON feeds.

Feedsearch is primarily meant to be a thin public API wrapper around the Feedsearch Python package, which I wrote to support feed search capabilities in my side project Auctorial. I couldn't find a public library or service that both searched for feeds and returned feed and site metadata, so I wrote this one. In addition to the API, the site allows users to search for feeds using a standard webform.

The code and documentation for the Feedsearch package is hosted in the Feedsearch Github repository. The site and API code is also on Github. The API runs on the Flask framework, and is converted with the ridiculously easy to use Zappa library to run on AWS Lambda.

Acknowledgements are due to Dan Foreman-Mackey for his Feedfinder2 library, from which I stole and extended the original code for Feedsearch, and to Dave Winer for giving us RSS in the first place.

I hope people keep using and writing code for RSS. I think it's important that RSS continues to thrive, both on a personal level (it's how I get most of my reading material), and on a professional level (it's used heavily in Auctorial). RSS is by far the best way we currently have to propagate news and articles around the web; Facebook, Twitter, and Reddit aren't suited for the kind of aggregation that RSS enables.

Nginx and ELB Proxy Protocol Forwarding

David Beath — Mon, 17 Apr 2017 10:35:06 GMT

Recently I noticed that an application I'm hosting on AWS Elastic Beanstalk wasn't logging the Client IP Address of my users. After a lot of digging I found the issue was to do with the configuration of the Nginx servers running on the EC2 instances and in my application's Docker container, and in the TCP passthrough configuration of the Elastic Load Balancer.

So here's a quick post about my particular setup and the configuration I used to fix this issue of Client IP Addresses not being forwarded.

Elastic Beanstalk Setup

In a little more detail, my setup consists of an application hosted within a Docker container running on EC2 instances configured by Elastic Beanstalk, which sit behind an Elastic Load Balancer (ELB). Because of issues with Certificate support in ELB, I'm not using AWS's Certificate Manager or using the built-in SSL termination in ELB, but instead using TCP passthrough on the ELB, and using a custom Nginx configuration on each EC2 instance to terminate the SSL connection. The Docker container on each instance also has it's own Nginx configuration, which proxies the application server, in my case UWSGI.

The steps that a request takes to get to the application are:

User makes an HTTPS request.
ELB uses TCP passthrough to pass the encrypted request to an EC2 instance.
Nginx in EC2 decrypts the HTTPS request and passes the HTTP to it's Docker container.
The Nginx server on Docker proxies the request to UWSGI.
The application hosted by UWSGI handles the request.

In all, the parts that you need to configure to forward the Client IP Address are the TCP passthrough on ELB and each of the two Nginx servers.

Proxy Protocol

The first problem is that if you're using a TCP load balancer to pass through the request, the load balancer will not add an X-Forwarded-For header, and so the downstream Nginx server will only see the IP Address of the load balancer. The problem and part of the solution is described in better detail here. The linked solution is a little different from what I'm going to show below, because I'm proxying the request another hop down the line, but please read the link to better understand what's going on.

The first step is to enable Proxy Protocol support on the Elastic Load Balancer. AWS has a good tutorial on how to do that.

Once the ELB has Proxy Protocol support enabled, you can configure the Nginx server on the EC2 instances. The relevant parts of the configuration look something like this:

log_format elb_log '$proxy_protocol_addr - $remote_user'
                    '[$time_local] "$request" '
                    '$status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

server {
    listen 443 ssl http2 proxy_protocol;

    server_name localhost;

    set_real_ip_from  172.31.0.0/20;
    real_ip_header    proxy_protocol;

    access_log /var/log/nginx/elb-access.log elb_log;

    # SSL Certificate settings go here.

    location / {
        proxy_pass          https://docker;
        proxy_http_version  1.1;

        proxy_set_header Connection         "";
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $proxy_protocol_addr;
        proxy_set_header X-Forwarded-For    $proxy_protocol_addr;
        proxy_set_header X-Forwarded-Proto  $scheme;
    }
}

Here's what's happening in the above config:

The listen directive now has proxy_protocol.
set_real_ip_from tells Nginx the real CIDR range of addresses that the ELB is using.
real_ip_header tells us that we're using the proxy_protocol value to access the client headers.
X-Real-IP and X-Forwarded-For use the $proxy_protocal_addr, which is the client IP information taken from the proxy protocol.
The elb_log format is now using the $proxy_protocal_addr for the client IP.

Also note that the listen directive can support both http2 and proxy_protocol, so if your version of Nginx supports HTTP/2 then you can enable that just fine. However, the proxy_http_version directive in this case is still 1.1, as it doesn't make sense to use HTTP/2 for proxy backends.

If you deployed the above config, you should now see that Nginx server on the EC2 instance is now logging the correct client IP Address.

Multiple Nginx Proxies

Now that we have the Nginx server on the EC2 instance receiving the correct Client IP from the Load Balancer, we need to foward that IP on to the Nginx server in the Docker container.

The X-Forwarded-For and X-Real-IP directives in the above config will properly forward the Client IP from the EC2 instance's Nginx server. However, Nginx appends each proxy's IP address to the X-Forwarded-For header, as described in more detail here.

The solution to this is in the last Nginx proxy configuration is to include the IP address ranges of all previous known proxies in the set_real_ip_from directive. You can then set the real_ip_header directive to X-Forwarded-For. The final piece is to set real_ip_recursive to on, which will return the leftmost (original) IP instead of the rightmost (most recent) IP from the X-Forwarded-For header.

My Nginx configuration for the Docker container now looks like this:

upstream app {
    server unix:///tmp/uwsgi.sock;
}

server {
    listen 80 default_server;

    set_real_ip_from    172.31.0.0/20;    # IP of the ELB
    set_real_ip_from    172.17.0.0/20;    # IP of the EC2 instance
    real_ip_header      X-Forwarded-For;
    real_ip_recursive   on;

    location / {
        uwsgi_pass app;
        include uwsgi_params;
    }
}

You can see that I've included the IP ranges of both the ELB and the EC2 instance. With a config similar to this you should now see the Nginx instance in the Docker container log the correct Client IP, and pass that IP through to your application.

Testing HTTP Responses in Node.js

David Beath — Mon, 03 Mar 2014 01:29:39 GMT

It generally goes without saying these days that testing is an important part of the development process. However, testing is not always the easiest thing to master, and it can be difficult to know where to start, especially when it comes to learning a new language. Node.js adds an extra twist on this difficulty, in that your code is running Asynchronously, and you will often be dealing with HTTP calls.

Having just been through the process of learning Node.js and working on a simple application that deals with calling and receiving responses from an external API, the most painful thing I found was that there aren't many tutorials on how to test HTTP requests and responses that I find particularly clear. So, this tutorial aims to provide clear information on how to effectively test your requests and responses, with a little bit about testing emitted events.

Setting up the Test Environment

For these tests we'll be using the Mocha testing framework along with the Chai assertion library. We'll also use the popular Request library to send requests to our server.

Create a new folder for you project and install Mocha, Chai, and Request to it by running:

> npm install mocha chai request

By default, Mocha looks for tests in a folder called /test within the project root, so create the folder with:

> mkdir test

In order to run our tests we'll set up a Makefile in the project root, the code of which looks like this:

REPORTER = spec

test:
    @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --ui tdd

test-w:
    @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --growl \
        --ui tdd \
        --watch

.PHONY: test test-w

Be careful when you're creating the Makefile that you indent the lines with Tab characters and not spaces.

You can now run your tests once with:

> make test

You should see something like 0 passing (2ms), which indicates that the test suite ran but didn't find any tests. If you see an error then the most likely options are probably that you've got spaces in your Makefile, or that Make can't find Mocha, in which case check that you have a /node_modules directory in your project folder and that Mocha is installed to it.

Alternatively, you can also run the tests continuously in the background with:

> make test-w

The make test command should also be added to the scripts section of your /package.json file, and Mocha, Chai, and Request to devdependencies; like so:

{
  "scripts": {
    "test": "make test"
  },
  "devDependencies": {
    "mocha": "*",
    "chai": "*",
    "request": "*"
  }
}

Creating our Server

Next we'll create the server that we'll be testing (Yes, by most standards writing the code before the tests is backasswards, but this is for demo purposes). This server will listen for requests and respond according to the content of the request header. If the request has a Content-Type header of 'text/plain' it will respond with an HTTP 200 "OK" code and emit an Event called success containing the body of the request. For any other Content-Type header we will return an HTTP 400 "Bad Request" error.

Create a server.js file in a new lib folder in your project directory, with the code below:

// ./lib/server.js

var http = require('http');

var server = module.exports = http.createServer(function (req, res) {

  if (req.headers['content-type'] === 'text/plain') {

    var body = '';
    req.on('data', function (chunk) {
      body += chunk.toString();
    });

    req.on('end', function () {
      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('correct header');
      server.emit('success', body);
    });

  } else {
    res.writeHead(400, {'Content-Type': 'text/plain'});
    res.end('wrong header');
  };
});

Setting up the Tests

Before we write our first test, we need to do a little more setup. The first thing we will need to do is to require Chai's expect function in our test suite. So we'll create a file in our /test folder called tests.js:

> touch test/tests.js

And with the first two lines we'll require the expect function from Chai:

var expect = require('Chai').expect;

You may have noticed that our server doesn't start automatically, as it has no listen call. The second line of our server code contains a module export that allows our server to be imported into another file. We can then start the server before our tests run, and close it when they finish. To do this we need to import our server into tests.js, and within the main 'describe' function in which we write our tests we'll include 'before' and 'after' functions.

// ./test/tests.js

var server = require('../lib/server.js');

describe('server response', function () {
  before(function () {
    server.listen(8000);
  });

  after(function () {
    server.close();
  });
});

Writing the Response Tests

We can now get round to actually writing our tests. We'll write three tests, in order to test each response from our server; the incorrect header response, the successful header response, and the data emitted on a successful response. First, we need to require the Request library at the beginning of tests.js:

var request = require('request');

Now we can include the test within our 'describe' function, immediately below the 'after' function:

it('should return 400', function (done) {
  request.get('https://localhost:8000', function (err, res, body){
    expect(res.statusCode).to.equal(400);
    expect(res.body).to.equal('wrong header');
    done();
  });
});

The it statement, just like describe, before, and after, follows the BDD standard for writing tests. We begin by describing what the test should return, and then a function containing the tests. Because our code is Asynchronous, we need to invoke a callback (in this case called done) so that Mocha knows to wait until the test is complete.

The second line begins our request to the server. We're calling request with a GET (though it could just as well be POST) and passing in the address of our server; in this case localhost listening on port 8000. There is also a function with error, response, and body callbacks. Within the function we're listening for a response, and this is where the test really happens.

The Chai assertion library works by chaining together natural language assertions, making the tests fairly easy to read. In this case we're expecting the StatusCode of the response to equal 400 and the body of the response to read wrong header. The final part of the code is the done() callback; without which the test will exit with a timeout error.

Our second test will test the response if the request contains the Content-Type: text/plain header.

it('should return 200', function (done) {
  var options = {
    url: 'https://localhost:8000',
    headers: {
      'Content-Type': 'text/plain'
    }
  };
  request.get(options, function (err, res, body) {
    expect(res.statusCode).to.equal(200);
    expect(res.body).to.equal('correct header');
    done();
  });
});

This test is much like the first, expect we've added an options object containing the destination url and headers of our request. This is passed as the first parameter of the request, so that our request will be sent to https://localhost:8000 with the correct Content-Type header. By creating an options object like this it become very easy to use the Request library to create a request with any parameters we'd like.

These two tests show the basic form for creating thorough HTTP Server tests. By adding more expect statements and adding different parameters to the chain, you can easily test all the different parts of your responses.

Testing Emitters

Now that we can test our responses, I figure that the next step might be to test what our code is emitting internally. While not technically part of the scope of this article, if you're developing and testing HTTP Requests and Responses in Node.js, at some point your going to be using event emitters and listeners, and in fact you already have. In our server the code req.on('data', function () {}); is an event listener, listening for the data event that is emitted when we receive data. Same with the req.on('end') function.

As you can see if you look back at our server code, once we receive the end event, we send out our own success event that contains the body of the received request. To test that we're successfully emitting, we'll add a body to our request, and we'll need to set a timeout in our test to make sure that the event is firing within a reasonable time.

it('should emit request body', function (done) {
  var options = {
    url: 'https://localhost:8000',
    headers: {
      'Content-Type': 'text/plain'
    },
    body: 'successfully emitted request'
  };
  var eventFired = false;

  request.get(options, function (err, res, body) {});

  server.on('success', function (data) {
    eventFired = true;
    expect(data).to.equal('successfully emitted request');
  });

  setTimeout( function () {
    expect(eventFired).to.equal(true);
    done();
  }, 10);
});

We have now attached a body to our request, that should be emitted back to us, and an eventFired variable set to false, that will be set to true when the event fires. The expect statements are no longer in the request function, as we are not testing the HTTP response. The code server.on('success', function (data) {}) listens for the success event, and passes the data from that event into a function. Within the function we set eventFired to true, and test to see that the emitted data is correct. The setTimeout function is required in order to properly test that the event fired. If the event does not fire, then the test will fail gracefully rather than with a timeout error.

Conclusion

We now have some basic tests that should reasonably cover the server code. This basic setup should work with most Node.js projects, and the general idea of this sort of testing is an essential skill in any programming language. If you'd like to have a look at or download and test the full code yourself then it can be found here.

Bittorrent Sync is not FOSS

David Beath — Tue, 10 Dec 2013 08:26:33 GMT

Bittorrent Sync has been getting a lot of press lately for being a really good file synchronisation solution. I was thinking about installing it on my systems today, to act as a secure, decentralised alternative to Dropbox. Then I had a sudden thought: "Wait, is Bittorrent Sync open source?".

Turns out no, it's not.

Well, crap.

It sounds like such a good system, and just because it's not FOSS doesn't mean it's not a really good piece of software, and as perfectly secure as it's possible to be these days. After all, Dropbox is closed and everyone (including me) trusts them to a certain extent (with non-critical stuff), though it doesn't exactly have the strongest reputation for security. The problem is that anyone who knows anything about computer security and cryptography also knows that a security solution that no-one else can inspect is a potentially insecure solution.

While I would probably never do any more than glance at the source code if it was open, I would feel much safer in the knowledge that people much more clued up than me about security had most likely pored over the code looking for bugs and security holes. I trust KeePass with my passwords because anyone can check the code to make sure it's not transmitting all my passwords to the NSA, GCHQ, GCSB, any other alphabet soup agency, Google, Facebook, or the Russian Mafia (to list just a few possible nefarious organisations). Likewise for the other open source software I use, I trust that there's enough people looking at the code that someone would raise a stink if something was awry.

Looking through the Bittorrent forums, this has all been debated to death. The BT Sync team have stated that they're considering the option of taking it open source. I hope they do. In the meantime, I figure I might as well go ahead and use it; the only files of mine that really require strong encryption is the aforementioned KeePass database, and that's already encrypted.

However, if I do ever find myself in a position where my life and liberty depend on secure communications, I won't be using anything that isn't open source.

Configuring SSL in Nginx

David Beath — Sun, 17 Nov 2013 03:32:49 GMT

This post aims to be a fairly thorough guide to getting SSL up and running on an Nginx server, largely for my own reference in future, but it may prove useful to anyone else who stumbles across it. I'm going to assume that you have a working installation of Nginx, or know how to get one going. If not, then bookmark this page and find a tutorial on how to get it going first, such as the section on Nginx in my post here, or the Installation and Beginner's Guide pages on the official site.

Get an SSL Certificate

The first step in setting up SSL on your server is to procure an SSL Certificate from an approved Certificate Authority (CA). The easiest way to do this is to purchase one from a certificate reseller such as CheapSSLs.com.

For a single site with no subdomains, such as a blog, that doesn't require all the extra features of more expensive certificates, then a certificate such as Comodo Positive SSL will do fine.

Once you've purchased the certificate, you'll need to create a Certificate Signing Request.

Creating a CSR

A Certificate Signing Request (CSR) is a file that contains the application information for your certificate, such as your location and the name of the domain you want to secure, as well as your Public Key. For creating the CSR we'll be using OpenSSL, so if you don't have that on your server you can install it with the following command:

> sudo apt-get install openssl

Before we start creating files, it's recommended that you have a proper location to store them in. On Linux systems this would be something like /etc/ssl/localcerts/, so lets start there.

> sudo mkdir /etc/ssl/localcerts
> sudo cd /etc/ssl/localcerts

To generate the public and private keys that will be used for the CSR, use this command:

> sudo openssl req -nodes -newkey rsa:2048 -days 365 -keyout myserver.key -out server.csr

This will create two files; myserver.key, the private key file, and server.csr, the Certificate Signing Request. When you enter the above command, you should of course set the name of both files to one appropriate for the server you are securing.

Generally your CA should require you to use RSA 2048, this is specified with the -newkey rsa:2048 portion of the command. If you purchased a certificate that is valid for more than a year, then the -days 365 portion should be set to the number of days your certificate is valid.

Once you've entered the above command, you'll be asked to enter the details into your CSR. These details (with examples) are:

Country Name (2 letter code) [AU]: US
State or Province Name (full name) [Some-State]: California
Locality Name (eg, city) []: Los Angeles
Organization Name (eg, company) [Internet Widgits Pty Ltd]: ACME Ltd
Organizational Unit Name (eg, section) []: Widgets
Common Name (eg, YOUR name) []: widgets.acme.com
Email Address []:

And two 'extra', comletely optional details:

A challenge password []:
An optional company name []:

If you enter a period '.' into one of the above fields then the field will be left blank.

One thing that seems to get glossed over is that the Common Name field is the fully qualified domain name (FQDN) of your site. So if you're wanting to secure example.com, then that's what you put in the common name field. If you're securing a subdomain like subdomain.example.com, then make sure you enter it like that. Unless you've bought a wildcard certificate, then your certificate will only cover a single domain or subdomain.

WARNING: Make very sure you've entered the Common Name correctly before you submit your CSR. (Yes, I have made this mistake. Don't do important things when you're tired.)

Once you've created the CSR, go to the site you purchased your certificate from and activate it. You'll be asked to enter your CSR into an online form, so open your CSR file up in a text editor and paste the entire contents into the enrollment form, including the BEGIN and END lines.

Installing your Certificate

Once your CSR has been approved, you'll get an email containing your new certificate, and the Root certificates required. Copy these to the folder where you intend to store your certificates. For this example we'll use the SSL localcerts directory we created previously /etc/ssl/localcerts/.

You may want to combine the certificates together into one file. If so, then you can use the cat command to concatenate the files in reverse order of authority. For example, if you've got a Comodo Positive SSL Certificate, then you should have three files; the certificate for your domain (eg. example.com.crt), the intermediate certificate (eg. PositiveSSLCA2.crt), and the root certificate (eg. AddTrustExternalCARoot.crt). So to concatenate these files the command would be:

> sudo cat example.com.crt PositiveSSLCA2.crt AddTrustExternalCARoot.crt >> example.com.chained.crt

Configuring Nginx

You can now set about adding your certificate to your server configuration in Nginx. If you installed Nginx from a package, it should have SSL enabled already, otherwise if you're building from source you'll have to specifically enable it during configuration.

> ./configure --with-http_ssl_module --with_http_spdy_module

If you're using SSL then you may as well use SPDY to help speed up the connection.

Now you can configure your Virtual Hosts for SSL. In the server block of your configuration, you need to set it to listen on port 443, specify the certificate and private key, and SSL protocols and cipher settings.

This is an example of a basic Nginx Virtual Host file with SSL enabled:

server {
  listen 443 ssl spdy;
  server_name example.com;

  root /srv/www/example.com;
  index index.html;

  ssl_certificate /etc/ssl/localcerts/example.com.chained.crt;
  ssl_certificate_key /etc/ssl/localcerts/example.com.key;
}

The listen directive specifies that the server should listen on the HTTPS port, 443, instead of the usual port 80, that SSL is on, and that it is using the SPDY protocol. The ssl_certificate directive points to the location of the chained certificate we created. The ssl_certificate_key directive points to the private key that was created when we created the certificate signing request.

The above provides the minimum configuration required to enable SSL on Nginx. All you need to do is restart Nginx, and you should be able to visit your site using HTTPS.

SSL Security in Nginx

Of course, the above basic configuration only uses the most basic default security, so you'll probably want to do some additional configuration.

The first thing you'll want to do is to set the SSL Ciphers your server will use. This post has a good explanation on the background of using ciphers. The exact list of ciphers you should use is dependent on how secure you want to be, and what performance considerations you have. The ciphers I use are:

ssl_ciphers ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM;

This gives me an 'A' rating on the SSL Server Test, while still providing decent performance.

The rest of my SSL configuration is as follows:

ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ecdh_curve secp521r1;

ssl_session_timeout 5m; is the time it takes for the client's ssl session information to be removed from the cache. A longer time means that the ssl session won't have to be renegotiated as often, improving performance if there are a lot of connections. 5 minutes is the default time for many client browsers.

ssl_prefer_server_ciphers on; means that the server will prefer to use ciphers specified in the ssl_ciphers directive over the ciphers preferred by clients.

ssl_session_cache shared:SSL:10m; lets Nginx use its own cache instead of the one provided by OpenSSL, thus allowing Nginx to separate SSL jobs between its own workers.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; is the list of supported protocols, which in this case is only TLSv1 or greater. SSLv3 and below are not recommended due to security vulnerabilities, and most browsers support TLS now.

ssl_ecdh_curve secp521r1; is the Elliptic curve key used for the ECDHE cipher.

If you want a much more in-depth look at securely configuring Nginx, then calomel.org has an excellently thorough post detailing nearly everything you need to know.

Redirecting to HTTPS

If you want to redirect all your traffic to the SSL version of your site, then you'll need to specify another Virtual Host to accept HTTP traffic and redirect it to HTTPS.

server {
  listen 80;
  server_name example.com;
  return 301 https://example.com$request_uri;
}

This server block listens on port 80, and then performs an HTTP 301 redirect to the requested URI. As this is a permanent redirect of a full URL then return rather than rewrite is the correct Nginx directive to use.

Conclusion

You should now have a secure implementation of SSL on your website. You may want to use tools such as pagespeed to test the speed of your site, as it may require a bit of tweaking to get the best balance between performance and security.

Ubuntu Post-Install Guide

David Beath — Fri, 15 Nov 2013 01:13:44 GMT

Having reinstalled Ubuntu a few times on various machines, I figured it's about time that I recorded a few of the post installation steps I take, so that I don't have to keep looking them up. The instructions below are for Ubuntu 13.10, and can be done in any order.

Install Bumblebee

Bumblebee is a package that prevents laptops that run Nvidia Optimus graphics from running the dedicated graphics card permanently. Installing this is the first thing I do, otherwise things get quite toasty. Further instructions here.

> sudo add-apt-repository ppa:bumblebee/stable
> sudo apt-get update
> sudo apt-get install bumblebee bumblebee-nvidia primus linux-headers-generic

> sudo shutdown -r now

Disable Apport

Apport is the error reporting service for Ubuntu. Most of the errors it pops up with are pretty minor, and Canonical knows about them by now anyway. You can either get rid of it entirely or just disable it. Instructions from here.

To disable Apport:

> sudo service apport stop

> sudo nano /etc/default/apport

enabled=0

To remove Apport entirely:

> sudo apt-get purge apport

Install Unity-Tweak-Tool

Unity Tweak Tool is always good for tweaking the Unity interface to your preferences. I think the default launcher is much too large, and I like to set the font-size smaller too.

> sudo apt-get install unity-tweak-tool

Install Sublime Text 3

Sublime Text 3 seems to be everyone's favourite text editor these days, and it's what I use to write these blog posts. Instructions courtesy of webupd8.org.

> sudo add-apt-repository ppa:webupd8team/sublime-text-3
> sudo apt-get update
> sudo apt-get install sublime-text-installer

To prevent annoying update notifications, once Sublime Text is open add the following code in: Preferences > Settings - User:

"update_check": false

Install Numix GTK3 Theme

Numix is my favourite GTK3 Theme, and looks good in Unity. Check it out at their website.

> sudo add-apt-repository ppa:numix/ppa
> sudo apt-get update
> sudo apt-get install numix-gtk-theme

Alternatively, for a few other themes as well if you're running 13.10:

> sudo apt-get install shimmer-themes

Remove Ubuntu One

Ubuntu One doesn't really seem to be under active development anymore, and there are better cloud solutions out there, so you might as well remove it. Instructions from here.

> killall ubuntuone-login ubuntuone-preferences ubuntuone-syncdaemon
> sudo rm -rf ~/.local/share/ubuntuone
> rm -rf ~/.cache/ubuntuone
> rm -rf ~/.config/ubuntuone
> mv ~/Ubuntu\ One/ ~/UbuntuOne_old/

> sudo apt-get purge ubuntuone-client python-ubuntuone-*

Disable Login Drum Sounds

I hate login sounds, and a quick search shows a lot of other people do as well, yet Ubuntu no longer has any options to disable them. So if you can't stop it from playing the sound, just remove the file. These instructions should work regardless of version.

> cd /usr/share/sounds/ubuntu/stereo
> mv system-ready.ogg system-ready.ogg.old

Install VLC Player

VLC is the best video player, enough said.‎

> sudo apt-get install vlc

Disable Amazon Shopping

A lot of people really dislike the direction the Canonical has taken by integrating online shopping searches into Unity Lens search, and I agree. There is a button to turn it off in the settings, but I'd just as soon remove it altogether.

Paste the following into your terminal:

> gsettings set com.canonical.Unity.Lenses disabled-scopes "['more_suggestions-amazon.scope', 'more_suggestions-u1ms.scope', 'more_suggestions-populartracks.scope', 'music-musicstore.scope', 'more_suggestions-ebay.scope', 'more_suggestions-ubuntushop.scope', 'more_suggestions-skimlinks.scope']"

Improve Battery Life

Some instructions for a tool that may help improve laptop battery life. Warning: May cause issues with external devices due to powering off USB ports.

> sudo apt-add-repository ppa:linrunner/tlp
> sudo apt-get update
> sudo apt-get install tlp tlp-rdw
> sudo tlp start

Development Tools

So I don't forget, these are some of the packages that are useful to have to development purposes.

> sudo apt-get install python-software-properties python python-pip g++ make python-dev git

Instructions for installing nodejs.

> sudo add-apt-repository ppa:chris-lea/node.js
> sudo apt-get update
> sudo apt-get install nodejs

KeePass

KeePass is my preferred password manager. Keep the password file in Dropbox and you can access it anywhere. The install instructions for Ubuntu are here.

> sudo apt-add-repository ppa:jtaylor/keepass
> sudo apt-get update
> sudo apt-get install keepass2

Dropbox

You can find the download for Dropbox on their site as usual. In Ubuntu 13.10 you may come across an issue where the Dropbox icon isn't showing in the status bar. If so, just install the following package:

> sudo apt-get install libappindicator1

Reading XBee RSSI with Arduino

David Beath — Fri, 11 Oct 2013 01:47:53 GMT

Last year I spent quite a lot of time trying to read the Recieved Signal Strength Indicator from an XBee for a project I was working on. I had planned to blog about the process, but never got round to writing any more than two posts. However, I had a look at the nascent blog for the first time in a while, and it surprised me that it was getting a respectable number of views, especially for something that wasn't particularly informative and hadn't been updated in so long. Having a look at the analytics, I realised that like I been doing, there are still a lot of people trying to figure out how to read RSSI from an XBee to an Arduino. So here's a tutorial and some pointers for getting XBees set up for signal strength reading.

The first thing you're going to need is of course a couple of XBee units. You'll also need an XBee usb connector like this to configure them from your computer. For this tutorial I'll explain how to read the signal strength with an Arduino, though you can just use a PC.

PLEASE NOTE: Unless you want to set up a Wireless mesh network, you should use the XBee 802.15.4 modules, also known as the Series 1 modules. The Series 1 operates a standard point to point network. The Series 2 modules, the XBee ZB, use the ZigBee mesh protocol. Because of this, they do not include RSSI information in the packet, and anyway, the RSSI is only good for the last hop. Make sure you know your required use-case before ordering these modules. You have been warned.

XBee Configuration

Before doing anything with your XBees, read this list of common XBee mistakes. It will save you a lot of time wondering why things aren't working as they should. I also recommend you read the official Getting Started Guide, and consult the Product Manual.

In order to configure your XBees, by far the easiest way is to use X-CTU. Unfortunately this only runs on Windows, though it may be possible to get it working with WINE in Linux. For very basic instructions on how to use X-CTU, look here. Alternatively, you check out the instructions here on how to configure them from a terminal.

Getting the XBees talking to each other should be a fairly simple matter. You'll want the PAN ID and Channel settings to be the same on each XBee you're using, and of course the MY address should be unique to each XBee. If you're using only two XBees to talk to each other, then set the Destination address high to 0 and the Destination low address to the MY address of the other XBee. Otherwise, if you're wanting to broadcast to all XBees listening on the same Channel and PAN ID, then set the Destinaton high address to 0, and the Destination low address to FFFF to enable broadcast mode.

The final setting is to change the API Enable setting to "2", which will allow the Arduino to control the XBee using API commands.

Arduino Configuration

I'm going to assume that if you're reading this tutorial, then you already have a basic understanding of Arduino code, and how to use them. You'll need to download the xbee-arduino library and put it in your Arduino library folder.

In your Arduino code include and intitialise the xbee-arduino library.

#include <XBee.h>

XBee xbee = XBee()

Then the start the serial connection to the XBee and computer in setup.

void setup()
{
  // XBee serial connection
  xbee.begin(9600);

  // Computer serial connection
  Serial.begin(9600);
}

For the purposes of this tutorial, we're going to have one XBee send a packet, and the other read the packet and output the RSSI value. You can have the sending XBee just send from a terminal, and use only one Arduino for receiving, but I'm going to show the code for having both XBees attached to an Arduino. The code about applies to both sender and receiver.

Sending Packets

On your sending XBee/Arduino, before the setup, initialise the payload. You can of course change the payload anywhere within the loop. I'm going to set the payload to "Hi".

uint8_t payload[] = { 'H', 'i' };

Then specify the destination address that you're going to send to. This is the Serial High and Serial Low address of the recieving XBee. If you're broadcasting, then you can skip this step.

XBeeAddress64 addr64 = XBeeAddress64(0x0013a200, 0x403e0f30);

In this example, because we're not dealing with sending updated data, we can create the transmit request in setup. The Tx16Request takes three parameters; the address it's being sent to, the payload, and the size of the payload. If you're broadcasting, then change the address parameter to 0xFFFF.

Tx16Request tx16 = Tx16Request(addr64, payload, sizeof(payload));

Finally, in your loop, send the packet. I'm going to set it to send every 50 milliseconds.

void loop()
{
  xbee.send(tx16);
  delay(50);
}

Receiving Packets and reading RSSI

On your receiver, before setup, you'll first need to initialise a Response object.

  Rx16Response rx16 = Rx16Response();

In the loop, your XBee will first wait for an incoming packet. The value within the brackets specifies how long it shoud continue waiting before continuing through the loop. For this example, because we're only doing the one thing, listening for and reading the signal strength of a packet, we can afford to wait a while.

  xbee.readPacket(100);

Once a packet is received, in order to prevent errors we first check if the response is available, then check if it matches the response type we want. Only then do we actually read the packet.

// Check if available
if (xbee.getResponse().isAvailable())
{
  // Check if packet is correct type
  if (xbee.getResponse.getApiId() == RX_16_RESPONSE)
  {
    // Read the packet
    xbee.getResponse().getRx16Response(rx16);
  }
}

Now you can read the signal strength from the packet and send it to the computer. If you don't want to print out the previous RSSI value every time through the loop whenever a packet isn't received (which you shouldn't), then this code should go within your if statements immediately after you've read the packet.

Serial.print( rx16.getRssi() );

And if you want to print out the payload:

for (int i = 0; i < rx16.getDataLength(); i++)
{
  Serial.print( rx16.getData(i), HEX );
}

Final Code

Your code for the sender should look something like this:

#include <XBee.h>

XBee xbee = XBee()

uint8_t payload[] = { 'H', 'i' };

XBeeAddress64 addr64 = XBeeAddress64(0x0013a200, 0x403e0f30);

Tx16Request tx16 = Tx16Request(addr64, payload, sizeof(payload));

void setup()
{
  xbee.begin(9600);
}

void loop()
{
  xbee.send( tx16 );
  delay(50);
}

And the code for the receiver something like this:

#include <XBee.h>

XBee xbee = XBee()

Rx16Response rx16 = Rx16Response();

void setup()
{
  xbee.begin(9600);
  Serial.begin(9600);
}

void loop()
{
  xbee.readPacket(100);
  if (xbee.getResponse().isAvailable())
  {
    if (xbee.getResponse().getApiId() == RX_16_RESPONSE)
    {
      xbee.getResponse().getRx16Response(rx16);
      Serial.print( rx16.getRssi() );
    }
  }
}

You should now have a couple of XBees that are talking to each other, and can read the signal strength of the packets. For a full example of working code, where a whole bunch of senders broadcast to multiple receivers, which then send the data to an aggregator, check out my Github repository.

Installing Tiny Tiny RSS from scratch

David Beath — Wed, 02 Oct 2013 03:43:39 GMT

Considering that a couple of times now I've been a proponent of using an RSS reader, I figured it was time I wrote a tutorial on how to install Tiny Tiny RSS, using Nginx as the server and PostgreSQL for the database. While as with any piece of software there are a number of tutorials and guides already out there, I've found that none of them provided a complete instruction set that didn't have me running into and searching for solutions to multiple errors. To that end, this tutorial aims to both provide as complete a set of fool-proof instructions as I can make, both for newbies to self-hosting, and simply as a reference for my future self should I need to do this again.

This set of instructions assumes that you're running a Debian based Linux distro. I've tested this setup on a clean Ubuntu Server 12.04 virtual machine, and the settings are roughly the same as I'm using on my Debian server. It's assumed that you either know how to set up a server already, or are capable of reading a tutorial to do so (If not, then why are you reading this? Go use Feedly). There are some very good tutorials at Digital Ocean and Linode, and I'd recommend either of them for hosting purposes.

Installation

On to the install. While not strictly necessary, the first thing I recommend is to make sure that your server is using the correct time, so install ntp to keep the system clock updated.

> sudo apt-get install ntp

For most of the commands in this tutorial you'll need super user privileges, so you can either run them as root (not recommended) or use sudo like I've used in this tutorial. You'll also need to edit a bunch of files in a text editor. I've used nano for this tutorial because it's easier to use and you can just copy the commands straight, but I usually use vim.

PHP

Unfortunately, despite the fact that PHP sucks (seriously, read the article, even if you disagree it's an interesting read), Tiny Tiny RSS is built with PHP. However, just because a language sucks doesn't mean that programs written in it necessarily have to, and tt-rss is the best of the self-hosted RSS readers. Still, that means you're going to have to install PHP, and get it working nicely with Nginx.

Install PHP 5 and the components necessary to get it playing nicely with everything else.

> sudo apt-get install php5 php5-fpm php5-curl php5-pgsql php5-gd php5-mcrypt php5-cli

There's one line in php.ini you'll need to change, which will prevent a possible security issue.

> sudo nano /etc/php5/fpm/php.ini

Change this line:

cgi.fix_pathinfo=1

To this:

cgi.fix_pathinfo=0

The other change is to make PHP-FPM listen on a UNIX socket rather than a TCP socket.

> sudo nano /etc/php5/fpm/pool.d/www.conf

Change the line:

listen = 127.0.0.1:9000

To:

listen = /var/run/php5-fpm.sock

Finally, restart PHP.

> sudo /etc/init.d/php5-fpm restart

Nginx

Now that PHP is configured, you'll want to configure Nginx. There's a very good primer by Martin Fjordvald on the basics of Nginx configuration which you can read to get an understanding of what's going on. If you want to dig deeper for various other settings, the documentation for Nginx is quite thorough.

This tutorial is going to have you build Nginx from source, since the .deb packages are not always up to date. It's really not that much harder than setting it up from a package install anyway.

First, install the packages required to compile the source.

> sudo apt-get install gcc make libpcre3-dev openssl libssl-dev

I also prefer to have Nginx use a dedicated system account with no login or password access for a bit of extra security.

> sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then download and unpack the latest Nginx version.

> wget https://nginx.org/download/nginx-1.5.6.tar.gz

> tar -xzvf nginx-1.5.6.tar.gz

> cd nginx-1.5.6

The configure command sets the parameters for the build. This is where you can change the default install directory and the different modules that Nginx uses. Nginx doesn't allow you to change modules without rebuilding, so you'll need to select or deselect any modules here. The only module I'll add is the http_ssl module, which will allow you to add SSL to your site later if you wish. I also prefer to just use the default install directory, so I won't change that, but I'll set the user and group Nginx uses to the one I created earlier.

> ./configure --user=nginx --group=nginx --with-http_ssl_module

Now we can build and install Nginx.

> make && sudo make install

Nginx Configuration

Change to the directory you installed Nginx to:

> cd /usr/local/nginx

There are various methods people use for controlling the sites on their server. The .deb install uses a sites-enabled and sites-disabled style configuration, but since I don't have many sites on my server, I prefer to keep each site configuration file in the conf/ folder, and include each one manually in nginx.conf.

So, for this tutorial, we'll create a ttrss.conf file.

> sudo nano conf/ttrss.conf

My ttrss.conf looks something like this (except I have SSL enabled):

# Tiny Tiny RSS Configuration

server {
  listen 80;
  server_name domainname www.domainname;

  root /var/www/ttrss;
  index index.php;

  error_log /var/log/nginx/ttrss.error.log;
  access_log /var/log/nginx/ttrss.access.log;

  location / {
    try_files $uri $uri/ /index.php;
  }

  location ~ \.php$ {
    include fastcgi.conf; # don't use fastcgi_params
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
  }
}

A brief primer:

listen: the port that the server listens for incoming connections on.
server_name: the FQDN (fully qualified domain name) that the server uses for this connection. I set this to listen on both example.com and www.example.com.
root: the root directory that the site's files are located at.
index: the index file that the server directs traffic to. In this case you only need index.php, but can also include index.html or .htm, or any other file you want to use as a homepage.
error_log: the location of the error log for this site.
access_log: the location of the access log for this site. The first location block accepts all incoming requests:
try_files: this just makes the URL search engine friendly. Not really necessary in this case, but it doesn't hurt. The second location block catches all incoming php requests:
include fastcgi.conf: this file comes with Nginx and includes all the parameters for using fastcgi. It's almost the same as fastcgi_params, but you'll have issues using fastcgi_params in Ubuntu 12.04, unless you include the line SCRIPT_FILENAME $document_root$fastcgi_script_name.
fastcgi_pass: the socket that fastcgi passes requests through.
fastcgi_index: the index file that fastcgi uses.

Next, you'll need to edit nginx.conf.

> sudo nano conf/nginx.conf

The only change that you really need to make to this file is to add the line include ttrss.conf in the http block, but this is my recommended configuration below.

# Nginx Configuration

user nginx nginx;
worker_processes auto; # optimal value is number of cpu cores

error_log /var/log/nginx/error.log

events {
  worker_connections 2048;
}

http {
  # Virtual Host Configuration
  include ttrss.conf;

  # Basic Settings
  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  client_header_timeout 20s;
  client_body_timeout 20s;
  send_timeout 20s;

  # Disable Nginx version number in error pages and Server header
  server_tokens off;

  # Silently block all undefined vhost access
  server {
    server_name _;
    return 444;
  }

  # Gzip Settings
  gzip on;
  gzip_disable "msie6";

  gzip_comp_level 5;
  gzip_min_length 256;

  gzip_vary on;
  gzip_proxied any;
  gzip_types *;
}

Finally, start Nginx.

> sudo /usr/local/nginx/sbin/nginx

If you need to make any changes to Nginx after it's been started, just open and change the config files. Then run this command to reload the Nginx with the new configuration.

> sudo /usr/local/nginx/sbin/nginx -s reload

Postgres

Tiny Tiny RSS recommends using PostgreSQL over MySQL for the database, as PostgreSQL is slightly faster. For a single user, it probably wouldn't make much difference, depending on how many feeds you have. The instructions for installing PostgreSQL are mostly taken from the Ubuntu community wiki here.

Install PostgreSQL.

> sudo apt-get install postgresql

> sudo -u postgres psql postgres

Change the password for the postgres user. Type in the command below, hit enter, then enter as password. Make sure to save all your passwords in a good password manager somewhere. I recommend KeePass.

> \password postgres

While you're still in the postgres command line, create a database and database user for tt-rss. The password must be entered surrounded by single quotes.

postgres=# CREATE USER ttrss WITH PASSWORD 'password';

postgres=# CREATE DATABASE ttrssdb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrssdb to ttrss;

Exit the postgres command line.

postgres=# \q

From this point, you might be able to get tt-rss to access the database, but I wasn't until I edited a line in the postgresql config to allow access.

> sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Add this line to allow tt-rss to use the database:

local all ttrss md5

Now restart PostgreSQL.

> sudo service postgresql restart

Tiny Tiny RSS

Finally, it's now time to install Tiny Tiny RSS itself. Yeah, I know it takes a while to get here. I assure you, it's worth it in the end, and you shouldn't have to worry about it much at all once you've got it properly set up. The installation instructions for tt-rss can be be found here, but once again they seem somewhat incomplete.

First, make a directory for tt-rss to be served from. This should be the same as the directory you specified as root in your ttrss.conf file for Nginx.

> sudo mkdir -p /var/www/ttrss

You'll need to set the owner of the directory to the user that you're installing and running tt-rss as.

> sudo chown -R user:group /var/www/

Now change to the directory above.

> cd /var/www

Download and extract the latest version of Tiny Tiny RSS.

> wget https://github.com/gothfox/Tiny-Tiny-RSS/archive/1.10.tar.gz

> tar -xzvf 1.10.tar.gz

Rename the extracted directory so that it matches the root directory you specified earlier, then open the directory.

> mv Tiny-Tiny-RSS-1.10/ ttrss

> cd /var/www/ttrss

The following folders need to have their permissions changed to be writable by anyone with an account on the system, or tt-rss won't be able to save files there.

> sudo chmod -R 777 cache/images/ cache/js/ cache/export/ cache/upload/ feed-icons/ lock/

Now you should be able to navigate in your browser to the site that you are hosting tt-rss from. This should be the same as the server_name you specified in ttrss.conf. If you installed tt-rss into a sub-folder of the domain, then the address will be https://domainname/ttrss/. If you got the configuration correct, you should see the Tiny Tiny RSS install page.

The installation page will have a few fields to enter. These will be the credentials for tt-rss to use the database, as well as the full URL that you'll access tt-rss from. Once you've entered the database details, click the 'Test Configuration' button. If there are errors, you'll get a message saying so, and it shouldn't be too hard to figure out what to do from there. If it's good to go, you'll see a text box with a lot of configuration lines. Copy that config data to your clipboard, and then into a config file on your server with:

> sudo nano /var/www/ttrss/config.php

Save that file, and reload tt-rss in your browser. You should now be able to login with the default user admin and password password. Once you're logged in, change the admin settings in the preferences at top right.

The final required step is to set up automatic updating of your feeds. You can skip this, but you'll have to manually click on each feed to check for updates if you don't. The screen command will permanently run feed updates in the background, defaulting to once every 30 minutes.

> screen -d -m php ./update_daemon2.php

Conclusion

You should now have a working installation of Tiny Tiny RSS on your server, accessible from anywhere in the world. If you've used RSS Readers before, you can import your feeds from an OPML file, otherwise, find some good blogs and start adding feeds.

Web Design: A good rant

David Beath — Sun, 15 Sep 2013 10:00:48 GMT

A really interesting talk by Matthew Butterick about the state of Web Design.

It’s now or never for the web. The web is a medium for creators, including designers. But after 20 years, the web still has no culture of design excellence. Why is that? Because design excellence is inhibited by two structural flaws in the web. First flaw: the web is good at making information free, but terrible at making it expensive. So the web has had to rely largely on an advertising economy, which is weakening under the strain. Second flaw: the process of adopting and enforcing web standards, as led by the W3C, is hopelessly broken. Evidence of both these flaws can be seen in a) the low design quality across the web, and b) the speed with which publishers, developers, and readers are migrating away from the web, and toward app platforms and media platforms. This evidence strongly suggests that the web is on its way to becoming a second-class platform. To address these flaws, I propose that the W3C be disbanded, and that the leadership of the web be reorganized around open-source software principles. I also encourage designers to advocate for a better web, lest they find themselves confined to a shrinking territory of possibilities.

I really sympathise with his views on this. The design of most major websites is crap. Unfortunately, he's correct in his assertion that this is largely to do with the fact that advertising drives most of the web. It's going to take a lot of work to find ways around this.

As I'm currently working on plans for my new idea for a useful site, so this kind of thing is something that I'm doing a lot of thinking about lately. It's so easy to just copy the base format that most sites use without thinking about whether that design is the best for your current situation.

A Failure of Ideology

David Beath — Tue, 03 Sep 2013 03:48:17 GMT

There's an article in the New Zealand Herald today, highlighting the ridiculousness of the current government's intervention in the broadband market. Feel free to skip the rest of this post if you wish, but I recommend you at least read the article. It does a good job of going over the issues from a technical, legal, and governmental angle, but there is one area it doesn't cover, which journalists mostly seem to shy away from, and that is the ideological angle.

For those who don't know, a quick backstory first. Currently in New Zealand our broadband internet services are sorely lacking compared to other OECD countries, against which we always compare ourselves. The current government, led by the National party, has thus initiated a plan to roll out 100 Mbps Fibre Broadband to the majority of New Zealand by 2020. The vast majority of the tender for the work went to Chorus, which was spun off from Telecom NZ as New Zealand's main infrastructure provider. Subsequently, as part of a long running effort to reduce the costs of broadband, the Commerce Commission decided that the prices Chorus charges for use of the copper network should more accurately reflect the cost of running the network. Now the government, represented by the Minister for Communications, Amy Adams, has decided that the wholesale price for copper should reflect the price for fibre, citing concerns that if the cost of copper is lowered, then consumers won't upgrade to fibre.

The article I listed at the beginning of this post does a good job of demolishing the incredibly spurious reasoning going on behind the government's decision. However, what I find interesting is what this action says about the government's underlying ideology. The National party is generally described as a Center-Right party in political terms, with the main opposition party, Labour, being Center-Left. As you'd expect from a right-leaning party, National frequently espouses pro business and pro free market values. This then should cause National some serious ideological problems when it comes to actions such as the above.

The Pro Business angle

Lets start with the pro business angle. Commonly accepted doctrine states that monopolies are generally detrimental to smaller businesses, having the money and power to outcompete in price and features, or simply buy them out. In extreme cases, the larger companies can even lobby to get the law changed in their favour, or at least get a few nice kickbacks. This latter case seems to be the situation we are currently faced with.

In bowing to Chorus's wishes in raising the price of copper, the government is effectively providing a subsidy to Chorus at the expense of the rest of us. There are a number of smaller ISP's who would like to compete on price, but Chorus having control of the vast majority of copper and fibre backbone in New Zealand means that they can't. Keeping the price of copper high also keeps costs higher for the many small businesses here that require an internet connection, and increases the cost for those business to expand. It should also be fairly uncontroversially accepted that higher prices also retard the development of internet based businesses in New Zealand.

The government can therefore be said to be subsidising Chorus and their shareholders at the expense of the thousands more small business owners. That would seem to me to go against their stated values of supporting small business.

The Pro Free Market angle

Espousing free market values generally means that one believes that government regulation should be limited, or even removed entirely, in order to allow the market to more freely adjust to an optimum state without regulatory distortion. This is the staple of modern neo-liberal philosophy, yet this National government's actions seem to run opposite to this principle. If you believe that the free market should in most cases provide the best outcome, then why would you then make a policy that distorts prices in favour of a single monopolistic company?

Now, many people would argue that the Commerce Commission's decision to lower the price of copper is also regulatory distortion, and they would be correct. The problem we have in New Zealand is that we're so small that there simply isn't room for the market to work effectively at the scale required for major infrastructure projects. Thus you see situations here where you're practically required to be a monopoly in order to afford to roll out infrastructure. In this case the Commerce Commission's job is to ensure that in the absense of price competition, the prices these monopolistic companies are allowed to charge closer reflect the market costs of building and running the infrastructure, and not the monopoly rents they could otherwise charge. Whether you agree with regulation or not, the actions of the Commerce Commission should allow the situation to more accurately reflect a free market.

By explicity saying that the price of copper should be higher, for no other reason than that it will discourage consumers from moving to fibre, the government is effectively stating that they do not in fact believe in the free market. They don't believe that the advantages of fibre are enough to outweigh the costs in relation to copper. If they believe that the market thinks fibre isn't good enough to justify it's increased cost, then why is the government paying to roll it out in the first place? In fact, as explained in the above article, fibre uptake is proceeding as expected, and Chorus projects that it's worth the money, else they wouldn't have tendered for what is basically a long term loan.

There are many more potential examples, and it would be possible to go much deeper into the various philosophies surrounding this issue, but I think I've given a decently broad overview of why this policy represents a failure on the part of the National party to stick to it's stated ideologies. Those on the left should have no trouble explaining why they think it's bad that the government is regulating in favour of a large corporation, while I've just outlined why I think those on the right should also be against this policy.

A New Blog

David Beath — Fri, 23 Aug 2013 09:23:54 GMT

So, I guess it's time I started to write a blog. I've always read that it's recommended in IT to write a blog, or maybe in any industry these days, in order to have some sort of recognisable professional presence. Or, at the very least, to keep your rants off Facebook.

At any rate, there are plenty of things I could talk about, and I will. I tend mostly to have strong opinions on Politics, Information Technology, Economics, the Environment, and any intersection between the bunch, though not necessarily in that order. I think I'll start off properly with a politics post next.

Anyway, I figure I might as well write about the tools I used to set this blog up. Starting from the bottom of the stack, my server is a VPS hosted by Linode. I like that I have full control over my machine, without having to worry about the hardware. Even the lowest tier plan gets me more than enough machine for my needs, and the documentation and tools they have are really good. Definitely recommended.

The server OS is Debian, because for me it was either that or Ubuntu Server for the easy and familiar option. Fanboys can feel free to flame away at whatever choice is made, but I like something stable.

For the HTTP server itself, I'm using nginx. Apache is just to mainstream, and from everything I've read it doesn't scale as well. Not that scaling should be a problem for me for a long time, if ever, but you never know. I've found nginx to be just as easy, if not easier, to understand and configure compared to Apache anyway.

Finally, I use Nikola to generate this blog. Nikola is a static blog generator, so all the files on this site are just plain HTML, no CMS needed. The site design is basically just an unmodified Foundation theme, which for now looks good enough. Besides, if you're not using an RSS reader to read this blog (once you've first visited and decided to keep reading of course), then you damned well should be. Why on earth would you manually visit blogs to read?

On the non-essential side of things, analytics on the blog are handled by Piwik, so that I don't have to send all my data to Google Analytics. The thing I use this server for the most though, is Tiny Tiny RSS. I find that using an RSS reader is essential for keeping up with the world. Like most everyone else, I used to use Google Reader, but of course anyone who hasn't been living under a rock this year knows the story of how they shut it down, and the gnashing and wailing of teeth that ensued. I contributed my part of that, on my tiny corner of Google Plus and Facebook.

Just like many others, I decided that if I couldn't trust even Google with my stuff, then it might be better to go self-hosted. Tiny Tiny RSS was the self hosted solution with the most recommendations, and I've been pretty happy with it so far. It's nice to know that you've got a service that no-one is going to arbitrarily remove, just so long as you have a good backup policy in place.

Someday soon, I might get round to routing my mail through my server as well, though I might like to have it on a different one. A single point of failure for everything I rely on makes me nervous.

In conclusion, Welcome to my Blog, and don't be afraid to look into self hosting if you're at all technically inclined. It's the 21st century, and all the information you need is out there.