David Beath (Posts about RSS)

Feedsearch API Updates

David Beath — Thu, 07 Nov 2019 16:50:14 GMT

Over the past year I've been slowly working on improvements to the Feedsearch API that I published in late 2017, and wrote about in my previous post.

The biggest difference is that the old Feedsearch library has now been replaced with the new Feedsearch Crawler library, which is available for use in other projects as a PyPI package. The crawler is now able to make HTTP requests asyncronously, drastically speeding up the number of URLs that can be checked for feeds in the same amount of time, which in turn increases the feed detection rate in cases where the feed URL is not properly advertised.

The second new feature is that all discovered feeds are now saved to a database, along with the website and the URL paths where they were discovered. In this way we can now return a list of all feeds that have been discovered on a site, and remove the need to crawl a site or URL every time a feed search is requested.

In addition to these, I've extended the results provided by the API to add few new fields:

content_length: The length of the feed in bytes at the time it was last seen.
is_podcast: Whether or not the feed contains valid podcast elements and enclosures.
last_seen: The date that the feed was last crawled by Feedsearch.
last_updated: The date that the feed was last updated by the publisher.
velocity: A calculation of the mean number of entries per day in the feed, at the time that the feed was last crawled.

Feedsearch Crawler

The Feedsearch Crawler is built as a separate library from the API, so that it's available for others to use, fork, and extend as they see fit.

I had been thinking for a few years now that what was really needed for feed detection was to use a proper Web crawler. As I kept upgrading the original Feedsearch library to improve detection rates and add features such as Favicon fetching, the time needed perform a search kept increasing, due to the synchronous design of the Requests HTTP library. It was at the time that I started adding asynchronous HTTP requests to Feedsearch, and therefore requiring that I keep track of which URLs have been requested, that I realised I was just building a crappier Web crawler, without a proper architecture. So I built my own, because why not?

The design and features of the base crawler itself are nothing special, but the FeedsearchSpider built on top of it is meant to be as discriminating as possible in what it crawls, while still finding as many feeds as possible. In order to do this the spider uses a number of regular expressions to filter page links that may lead to feeds, and then prioritises them for the crawl queue.

Feedsearch Gateway API

The API is built as the Feedsearch Gateway project, and is responsible for all the custom functionality that is not required in the generic crawler library, such as providing the actual API and dealing with saving results.

When crawling websites, it's not particularly polite to use a lot of resources, and so I needed the ability to cache the results. Because I already had Feedsearch running on AWS Lambda, I decided to use DynamoDB for the database. Whenever a search is completed, the following information is saved to the database:

All feeds found in the crawl and their metadata.
The host URL of the site and the time it was crawled.
The URL of the path that was crawled, the time it was crawled, and the URLs of the feeds that were found from that path.

When a search is started, the API first queries the database for the feeds that were already found and the site information. If the search is for the host URL of the site (e.g. https://example.com), then the site is only crawled again if it hasn't been crawled recently, and all feeds that have ever been found to belong to the site are returned.

If the search contains a path (e.g. https://example.com/path/testing/) then the database is checked to see if that path has already been crawled. If the path has been crawled recently, then the list of URLs found at that path are checked against the sites feeds, and the matching feeds are returned. When the path is searched, only feeds found from the crawl of that path are returned. This is done to increase the chance that only feeds relevant to that particular query are returned, especially if the site contains a lot of feeds.

Finally, the Feedsearch site itself is run behind Cloudflare Workers. Cloudflare doesn't cache HTML by default, but because the API homepage is designed to be as static as possible, it makes sense to cache it with the Workers cache API, so that we don't have to make a request to the Lambda function every time the page is requested. Instead, only requests to the API route are forwarded to the Lambda.

Feedsearch API

David Beath — Fri, 08 Dec 2017 10:03:53 GMT

I've just published a new microsite and API called Feedsearch, which provides an easy way to search websites for RSS, Atom, and JSON feeds.

Feedsearch is primarily meant to be a thin public API wrapper around the Feedsearch Python package, which I wrote to support feed search capabilities in my side project Auctorial. I couldn't find a public library or service that both searched for feeds and returned feed and site metadata, so I wrote this one. In addition to the API, the site allows users to search for feeds using a standard webform.

The code and documentation for the Feedsearch package is hosted in the Feedsearch Github repository. The site and API code is also on Github. The API runs on the Flask framework, and is converted with the ridiculously easy to use Zappa library to run on AWS Lambda.

Acknowledgements are due to Dan Foreman-Mackey for his Feedfinder2 library, from which I stole and extended the original code for Feedsearch, and to Dave Winer for giving us RSS in the first place.

I hope people keep using and writing code for RSS. I think it's important that RSS continues to thrive, both on a personal level (it's how I get most of my reading material), and on a professional level (it's used heavily in Auctorial). RSS is by far the best way we currently have to propagate news and articles around the web; Facebook, Twitter, and Reddit aren't suited for the kind of aggregation that RSS enables.

Installing Tiny Tiny RSS from scratch

David Beath — Wed, 02 Oct 2013 03:43:39 GMT

Considering that a couple of times now I've been a proponent of using an RSS reader, I figured it was time I wrote a tutorial on how to install Tiny Tiny RSS, using Nginx as the server and PostgreSQL for the database. While as with any piece of software there are a number of tutorials and guides already out there, I've found that none of them provided a complete instruction set that didn't have me running into and searching for solutions to multiple errors. To that end, this tutorial aims to both provide as complete a set of fool-proof instructions as I can make, both for newbies to self-hosting, and simply as a reference for my future self should I need to do this again.

This set of instructions assumes that you're running a Debian based Linux distro. I've tested this setup on a clean Ubuntu Server 12.04 virtual machine, and the settings are roughly the same as I'm using on my Debian server. It's assumed that you either know how to set up a server already, or are capable of reading a tutorial to do so (If not, then why are you reading this? Go use Feedly). There are some very good tutorials at Digital Ocean and Linode, and I'd recommend either of them for hosting purposes.

Installation

On to the install. While not strictly necessary, the first thing I recommend is to make sure that your server is using the correct time, so install ntp to keep the system clock updated.

> sudo apt-get install ntp

For most of the commands in this tutorial you'll need super user privileges, so you can either run them as root (not recommended) or use sudo like I've used in this tutorial. You'll also need to edit a bunch of files in a text editor. I've used nano for this tutorial because it's easier to use and you can just copy the commands straight, but I usually use vim.

PHP

Unfortunately, despite the fact that PHP sucks (seriously, read the article, even if you disagree it's an interesting read), Tiny Tiny RSS is built with PHP. However, just because a language sucks doesn't mean that programs written in it necessarily have to, and tt-rss is the best of the self-hosted RSS readers. Still, that means you're going to have to install PHP, and get it working nicely with Nginx.

Install PHP 5 and the components necessary to get it playing nicely with everything else.

> sudo apt-get install php5 php5-fpm php5-curl php5-pgsql php5-gd php5-mcrypt php5-cli

There's one line in php.ini you'll need to change, which will prevent a possible security issue.

> sudo nano /etc/php5/fpm/php.ini

Change this line:

cgi.fix_pathinfo=1

To this:

cgi.fix_pathinfo=0

The other change is to make PHP-FPM listen on a UNIX socket rather than a TCP socket.

> sudo nano /etc/php5/fpm/pool.d/www.conf

Change the line:

listen = 127.0.0.1:9000

To:

listen = /var/run/php5-fpm.sock

Finally, restart PHP.

> sudo /etc/init.d/php5-fpm restart

Nginx

Now that PHP is configured, you'll want to configure Nginx. There's a very good primer by Martin Fjordvald on the basics of Nginx configuration which you can read to get an understanding of what's going on. If you want to dig deeper for various other settings, the documentation for Nginx is quite thorough.

This tutorial is going to have you build Nginx from source, since the .deb packages are not always up to date. It's really not that much harder than setting it up from a package install anyway.

First, install the packages required to compile the source.

> sudo apt-get install gcc make libpcre3-dev openssl libssl-dev

I also prefer to have Nginx use a dedicated system account with no login or password access for a bit of extra security.

> sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then download and unpack the latest Nginx version.

> wget https://nginx.org/download/nginx-1.5.6.tar.gz

> tar -xzvf nginx-1.5.6.tar.gz

> cd nginx-1.5.6

The configure command sets the parameters for the build. This is where you can change the default install directory and the different modules that Nginx uses. Nginx doesn't allow you to change modules without rebuilding, so you'll need to select or deselect any modules here. The only module I'll add is the http_ssl module, which will allow you to add SSL to your site later if you wish. I also prefer to just use the default install directory, so I won't change that, but I'll set the user and group Nginx uses to the one I created earlier.

> ./configure --user=nginx --group=nginx --with-http_ssl_module

Now we can build and install Nginx.

> make && sudo make install

Nginx Configuration

Change to the directory you installed Nginx to:

> cd /usr/local/nginx

There are various methods people use for controlling the sites on their server. The .deb install uses a sites-enabled and sites-disabled style configuration, but since I don't have many sites on my server, I prefer to keep each site configuration file in the conf/ folder, and include each one manually in nginx.conf.

So, for this tutorial, we'll create a ttrss.conf file.

> sudo nano conf/ttrss.conf

My ttrss.conf looks something like this (except I have SSL enabled):

# Tiny Tiny RSS Configuration

server {
  listen 80;
  server_name domainname www.domainname;

  root /var/www/ttrss;
  index index.php;

  error_log /var/log/nginx/ttrss.error.log;
  access_log /var/log/nginx/ttrss.access.log;

  location / {
    try_files $uri $uri/ /index.php;
  }

  location ~ \.php$ {
    include fastcgi.conf; # don't use fastcgi_params
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
  }
}

A brief primer:

listen: the port that the server listens for incoming connections on.
server_name: the FQDN (fully qualified domain name) that the server uses for this connection. I set this to listen on both example.com and www.example.com.
root: the root directory that the site's files are located at.
index: the index file that the server directs traffic to. In this case you only need index.php, but can also include index.html or .htm, or any other file you want to use as a homepage.
error_log: the location of the error log for this site.
access_log: the location of the access log for this site. The first location block accepts all incoming requests:
try_files: this just makes the URL search engine friendly. Not really necessary in this case, but it doesn't hurt. The second location block catches all incoming php requests:
include fastcgi.conf: this file comes with Nginx and includes all the parameters for using fastcgi. It's almost the same as fastcgi_params, but you'll have issues using fastcgi_params in Ubuntu 12.04, unless you include the line SCRIPT_FILENAME $document_root$fastcgi_script_name.
fastcgi_pass: the socket that fastcgi passes requests through.
fastcgi_index: the index file that fastcgi uses.

Next, you'll need to edit nginx.conf.

> sudo nano conf/nginx.conf

The only change that you really need to make to this file is to add the line include ttrss.conf in the http block, but this is my recommended configuration below.

# Nginx Configuration

user nginx nginx;
worker_processes auto; # optimal value is number of cpu cores

error_log /var/log/nginx/error.log

events {
  worker_connections 2048;
}

http {
  # Virtual Host Configuration
  include ttrss.conf;

  # Basic Settings
  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  client_header_timeout 20s;
  client_body_timeout 20s;
  send_timeout 20s;

  # Disable Nginx version number in error pages and Server header
  server_tokens off;

  # Silently block all undefined vhost access
  server {
    server_name _;
    return 444;
  }

  # Gzip Settings
  gzip on;
  gzip_disable "msie6";

  gzip_comp_level 5;
  gzip_min_length 256;

  gzip_vary on;
  gzip_proxied any;
  gzip_types *;
}

Finally, start Nginx.

> sudo /usr/local/nginx/sbin/nginx

If you need to make any changes to Nginx after it's been started, just open and change the config files. Then run this command to reload the Nginx with the new configuration.

> sudo /usr/local/nginx/sbin/nginx -s reload

Postgres

Tiny Tiny RSS recommends using PostgreSQL over MySQL for the database, as PostgreSQL is slightly faster. For a single user, it probably wouldn't make much difference, depending on how many feeds you have. The instructions for installing PostgreSQL are mostly taken from the Ubuntu community wiki here.

Install PostgreSQL.

> sudo apt-get install postgresql

> sudo -u postgres psql postgres

Change the password for the postgres user. Type in the command below, hit enter, then enter as password. Make sure to save all your passwords in a good password manager somewhere. I recommend KeePass.

> \password postgres

While you're still in the postgres command line, create a database and database user for tt-rss. The password must be entered surrounded by single quotes.

postgres=# CREATE USER ttrss WITH PASSWORD 'password';

postgres=# CREATE DATABASE ttrssdb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrssdb to ttrss;

Exit the postgres command line.

postgres=# \q

From this point, you might be able to get tt-rss to access the database, but I wasn't until I edited a line in the postgresql config to allow access.

> sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Add this line to allow tt-rss to use the database:

local all ttrss md5

Now restart PostgreSQL.

> sudo service postgresql restart

Tiny Tiny RSS

Finally, it's now time to install Tiny Tiny RSS itself. Yeah, I know it takes a while to get here. I assure you, it's worth it in the end, and you shouldn't have to worry about it much at all once you've got it properly set up. The installation instructions for tt-rss can be be found here, but once again they seem somewhat incomplete.

First, make a directory for tt-rss to be served from. This should be the same as the directory you specified as root in your ttrss.conf file for Nginx.

> sudo mkdir -p /var/www/ttrss

You'll need to set the owner of the directory to the user that you're installing and running tt-rss as.

> sudo chown -R user:group /var/www/

Now change to the directory above.

> cd /var/www

Download and extract the latest version of Tiny Tiny RSS.

> wget https://github.com/gothfox/Tiny-Tiny-RSS/archive/1.10.tar.gz

> tar -xzvf 1.10.tar.gz

Rename the extracted directory so that it matches the root directory you specified earlier, then open the directory.

> mv Tiny-Tiny-RSS-1.10/ ttrss

> cd /var/www/ttrss

The following folders need to have their permissions changed to be writable by anyone with an account on the system, or tt-rss won't be able to save files there.

> sudo chmod -R 777 cache/images/ cache/js/ cache/export/ cache/upload/ feed-icons/ lock/

Now you should be able to navigate in your browser to the site that you are hosting tt-rss from. This should be the same as the server_name you specified in ttrss.conf. If you installed tt-rss into a sub-folder of the domain, then the address will be https://domainname/ttrss/. If you got the configuration correct, you should see the Tiny Tiny RSS install page.

The installation page will have a few fields to enter. These will be the credentials for tt-rss to use the database, as well as the full URL that you'll access tt-rss from. Once you've entered the database details, click the 'Test Configuration' button. If there are errors, you'll get a message saying so, and it shouldn't be too hard to figure out what to do from there. If it's good to go, you'll see a text box with a lot of configuration lines. Copy that config data to your clipboard, and then into a config file on your server with:

> sudo nano /var/www/ttrss/config.php

Save that file, and reload tt-rss in your browser. You should now be able to login with the default user admin and password password. Once you're logged in, change the admin settings in the preferences at top right.

The final required step is to set up automatic updating of your feeds. You can skip this, but you'll have to manually click on each feed to check for updates if you don't. The screen command will permanently run feed updates in the background, defaulting to once every 30 minutes.

> screen -d -m php ./update_daemon2.php

Conclusion

You should now have a working installation of Tiny Tiny RSS on your server, accessible from anywhere in the world. If you've used RSS Readers before, you can import your feeds from an OPML file, otherwise, find some good blogs and start adding feeds.

Typography: New Style and Fonts

David Beath — Wed, 25 Sep 2013 00:51:40 GMT

I've spent a few hours over the last few days reading Butterick's Practical Typography, which I felt might be a good idea considering that I enjoyed his rant about Web Design that I posted a while back. I feel that in some ways I damn myself as the worst (or best, depending on how twisted your perceptions are) type of geek for being interested in and recommending a book on the intricacies of typography. Though really, much as there is on the site, it's still only a primer, and I don't think I'm going to go much farther down the rabbit hole than this.

The further I read through the book, the more little details in my site started to annoy me a bit. As a result, I've spent even more hours slowly and meticulously changing the styling and fonts of this blog. The changes are relatively minor, but while you won't be able to do a direct comparison, as obviously I've already updated the site, I assure you they make a big difference in the quality of this site's appearance.

The biggest difference comes simply from changing the font. While I'm still using the Foundation framework, I've changed the font from it's trusty default, Helvetica Neue, to Charter. Ordinarily I've always tended to use Sans-Serif fonts whenever I've actually thought about it and gone beyond the default Times New Roman or Arial in my writing, but this time I figured a good Serif font looks more professional and easier to read. A serif font also allows for better use of emphasis. Doing side by side comparisons between the old and new versions of my site, I find Helvetica Neue to be slightly blurry to my eyes, while Charter looks relatively crisp and sharp.

For code blocks I've switched from the default css monospace to Inconsolata.

def helloworld():
    print "Hello World"

It's a nicely legible monospace font, and quite importantly when it comes to code, makes it very easy to distinguish between characters such as O and 0. While any good font will do this, it's especially important in a monospace font when you're reading code, and trying to figure out what character you're dealing with at a quick glance. Witness the relatively subtle difference between O and 0 in Charter. While the capital O is slightly wider than the number 0, it can be confusing in the absence of other references. The strikethrough in Inconsolata's zero makes the difference clear. Many fonts have the same issue between capital I and lowercase l as well. Most monospace fonts take pains to make these differences plain, so it largely comes down to a matter of personal preference.

One small change that makes a big difference is increasing the whitespace by slightly separating the lines with the line-height value. This separation just makes each line a little more distinct and less cramped feeling, making for easier reading. I also increased the margins a touch around the code in the code blocks, making the code stand out a little more, and combined with the line-height change makes the code in my previous post stand out nicely.

Finally with the typography changes, centering the titles of each post provides a clearer visual indication of the start of a new post. It also makes the difference between the title of a post and the subheadings stand out more, especially if I should want to start a new post with a subheading, or have one after a single paragraph.

The last little change I've made is to include a couple of cool little icons from Font Awesome. I always wanted an RSS icon on my RSS feed button, and now I have one. There are also icons for the links on my About page, which I really should update with a bit more info sometime.

One final point: If you're still reading this far, you'll hopefully be wondering what the hell I'm talking about fonts and spacing for, because you should be reading this in a proper RSS reader, and all my typography has been stripped out. I'll forgive you if this is your first time visiting, or you've come from a link posted somewhere. Otherwise, go be a good geek and learn how to host and run Tiny Tiny RSS, or if you're not into self hosting, then I recommend Feedly. And if you're into Web Design, do any typing or publishing, or need to write a résumé, then read Practical Typography.

A New Blog

David Beath — Fri, 23 Aug 2013 09:23:54 GMT

So, I guess it's time I started to write a blog. I've always read that it's recommended in IT to write a blog, or maybe in any industry these days, in order to have some sort of recognisable professional presence. Or, at the very least, to keep your rants off Facebook.

At any rate, there are plenty of things I could talk about, and I will. I tend mostly to have strong opinions on Politics, Information Technology, Economics, the Environment, and any intersection between the bunch, though not necessarily in that order. I think I'll start off properly with a politics post next.

Anyway, I figure I might as well write about the tools I used to set this blog up. Starting from the bottom of the stack, my server is a VPS hosted by Linode. I like that I have full control over my machine, without having to worry about the hardware. Even the lowest tier plan gets me more than enough machine for my needs, and the documentation and tools they have are really good. Definitely recommended.

The server OS is Debian, because for me it was either that or Ubuntu Server for the easy and familiar option. Fanboys can feel free to flame away at whatever choice is made, but I like something stable.

For the HTTP server itself, I'm using nginx. Apache is just to mainstream, and from everything I've read it doesn't scale as well. Not that scaling should be a problem for me for a long time, if ever, but you never know. I've found nginx to be just as easy, if not easier, to understand and configure compared to Apache anyway.

Finally, I use Nikola to generate this blog. Nikola is a static blog generator, so all the files on this site are just plain HTML, no CMS needed. The site design is basically just an unmodified Foundation theme, which for now looks good enough. Besides, if you're not using an RSS reader to read this blog (once you've first visited and decided to keep reading of course), then you damned well should be. Why on earth would you manually visit blogs to read?

On the non-essential side of things, analytics on the blog are handled by Piwik, so that I don't have to send all my data to Google Analytics. The thing I use this server for the most though, is Tiny Tiny RSS. I find that using an RSS reader is essential for keeping up with the world. Like most everyone else, I used to use Google Reader, but of course anyone who hasn't been living under a rock this year knows the story of how they shut it down, and the gnashing and wailing of teeth that ensued. I contributed my part of that, on my tiny corner of Google Plus and Facebook.

Just like many others, I decided that if I couldn't trust even Google with my stuff, then it might be better to go self-hosted. Tiny Tiny RSS was the self hosted solution with the most recommendations, and I've been pretty happy with it so far. It's nice to know that you've got a service that no-one is going to arbitrarily remove, just so long as you have a good backup policy in place.

Someday soon, I might get round to routing my mail through my server as well, though I might like to have it on a different one. A single point of failure for everything I rely on makes me nervous.

In conclusion, Welcome to my Blog, and don't be afraid to look into self hosting if you're at all technically inclined. It's the 21st century, and all the information you need is out there.