Twit World Report: From PHP to Erlang

When you use a web app to interact with Twitter you are almost always going via a sort of 'proxy' system, as browsers have lots of security to prevent Cross-Site Scripting Attacks. On Twitterfall, this proxying was performed by a piece of PHP code running via a FastCGI server, as well as other cross-site operations such as shortening URLs. We have encountered some big disadvantages with this method:

The Twitter API is slow. At peak times it can take seconds, not milliseconds, to respond. As a result, we have to run lots of PHP processes on our server that have to sit maintaining connections, using up lots of memory (This isn't what PHP was designed for!). When Twitterfall first started, this used to be a huge problem, now it's more of a niggling pain, but still there.

The PHP code is basic, it doesn't forward errors to the client (it just silently fails), and is very difficult to debug. It's been around since the start of Twitterfall, so understandably it's got a bit long in the tooth.

There's no easy way of monitoring it, and adding monitoring to an already bulky process would just slow it down more.

So we've worked on a replacement, and we're pleased to say it's been running in production successfully now for quite some time. We've built it on several great components available:

Erlang, which is great at running hundreds, thousands, even millions of concurrent processes, great for the slow Twitter API.

Mochiweb, a lightweight HTTP server framework for Erlang that powers Facebook Chat, and Hydrometeor.

erlang-oauth, an Erlang library for dealing with oAuth requests. We have modified it to support:

ibrowse, an HTTP client written in Erlang that replaces Erlang's HTTP client, which we've experienced big problems with when dealing with multiple concurrent requests to the same server.

erlang-collectd, an Erlang library for logging data to collectd, a monitoring daemon.

This means:

We can handle loads of concurrent connections to Twitter without breaking a sweat.

We can easily monitor the status of the connection between Twitterfall and Twitter, which makes it easier to inform our users when Twitter have a problem (and they think it's us).

We can monitor how people use Twitterfall to help improve the user experience.

Twitterfall feels more responsive.

Thanks to this new component, we've made the following observations:

The Twitter API errors a lot. Sometimes it will return 404 errors on known API endpoints for no apparent reason, and quite often it will return capacity errors.

The Twitter API is really slow when America are awake. Here's a graph of response time when people have made authenticated requests to the Twitter API via Twitterfall:

People have lists in their Twitterfall that don't exist any more, so we've added a visual cue to show when you have a list in your Twitterfall that doesn't exist.

People don't notice their tweets are too long, so we've made it so that you are warned if you try to send a tweet that is more than 140 characters.