Saturday, 16 May 2009

Yahoo Pipes - a web service for the customisation of RSS and Atom feeds

In the last couple of years I've started subscribing to more and more RSS and Atom feeds preferring interesting articles and information to come to me rather than having to go out trawling a list of well known websites on a fairly ad hoc basis for new articles of interest.

Google Reader is my feed reader of choice mainly because it's online and I can access it from anywhere without having to maintain multiple installs of feed reader software and settings.

I'm fairly ruthless with my feed subscriptions preferring feeds that are reasonably low volume (1 of 2 posts a week) with high quality content. There is usually a single human being on the other end with a limited amount of bandwidth.

Quite a few feeds are aggregates of multiple contributions (e.g. news websites) or are automated to provide information when changes to some underlying system occur (e.g. check ins for a code hosting site). They contain useful information but you sometimes quickly get bogged down by the sheer volume and frequency of updates coming through. I don't stay subscribed to these feeds for long when all I'm doing is clicking on "Mark All As Read" as hundreds of unread entries pile up.

There are certain feeds in this latter category that I would still like to follow, if only I could reduce the noise by either adding an include or exclude filter and possibly generating sub feeds that I could publish and make available to myself and others.

This was a pipe dream until this morning when I came across Yahoo Pipes :-)

You rarely come across technology that does exactly what you want out of the box without requiring a decent amount of customisation first, either through configuration or some coding. Yahoo Pipes allowed me solve a problem that has been nagging me for ages and it was dead simple to do.

As its name suggests Yahoo Pipes makes use concepts familiar to those who know and love UNIX command line with I/O processing pipelines, redirection and a set of options similar to tools like grep, sed and awk. These are all provided through a great visual designer and editor that make settings up and publishing your own custom feeds really easy and most importantly very quick to crank out. You can literally build and publish a custom feed in 5 minutes or less without much prior experience of Yahoo Pipes. That is real power (hats off to the designers - this kind of thing is not easy to do).

Here is a sample feed that is already saving me loads of time filtering out stuff I'm not interested in :-

Source feed :-

PyPI Package Update RSS feed

Resulting feed :-

PyPI Package Update RSS feed (without Zope or Plone packages)

You can organise your new feeds by giving them sensible and meaningful URLs too (another great feature).

So, if you subscribe to a lot of feeds and find that you want to tame them this is definitely the way forward™.

Check out these impressive tech demo videos for some real "wow" moments :-

No comments: