³ÉÈËÂÛ̳

is a prototype to highlight the most shared ³ÉÈËÂÛ̳ webpages on Twitter, a digest to link people to the hottest ³ÉÈËÂÛ̳ pages. The project is part of a larger area of exploration to see how the ³ÉÈËÂÛ̳ can use real-time trending data to enrich user experiences. One of our recent projects shows how the artists played on ³ÉÈËÂÛ̳ radio are trending on other music services, such as and .

We developed Zeitgeist as a simple information source for users and to provide insight into users' interests and behaviours for our production teams. There are some interesting commercial alternatives available such as , , and , which are worth checking out but we had some specific requirements for our prototype.

The system combines a custom built ingest chain using to search for tweets containing a ³ÉÈËÂÛ̳ URL. As it's running in real-time these links come and go depending on what Twitter users are talking about. You can see the 'liveness' in the view or take a broader view of the .

Zeitgeist uses the web page's URL and metadata to determine where it comes from and assign it a category, e.g , , or . These give links a context for the user and a means of navigating deeper.

The links are ranked by a tweet count (including retweets) for the chosen time period. Each entry details the page title, category, media type, short description and when it was first tweeted. The date of publication is indicated where available as it's not just new links that seem to get picked up on Twitter.

We have a different view for ³ÉÈËÂÛ̳ employees (shown below), which allows us to see; the tweet history of each page, a full list of tweets, most retweeted messages, hashtags and keywords. We are unable to show this to everyone as the messages would need to be moderated.

We use the Twitter streaming API to access the Gardenhose sample stream, which provides a subset of the full Twitter message stream, at a rate of about 100 messages per second and to track "³ÉÈËÂÛ̳" as a keyword. These messages are then fed into a pipeline of processes written in connected by queues provided by , a fast and reliable messaging server.

These are the stages that each incoming tweets goes through:

1. Twitter combines retweets with it's original tweet, these are split to deliver both messages to the pipeline
2. A tweet from the API contains a lot of extraneous data which needs to be removed, such as the user's page background colour
3. Links in the message are extracted and resolved following through redirections and expanding shortened links, provide a for this
4. Only tweets containing links to ³ÉÈËÂÛ̳ pages are kept. Automatically generated ³ÉÈËÂÛ̳ tweets from accounts such as are filtered out and links to the are also removed as they skew the results
5. These are saved to the database
6. The link category is determined by its domain and in-page metadata

We split these steps into separate processes for two reasons: it's easier to develop and test a process if it does only one thing; and more importantly, it allows us to balance different parts of the system depending on load. For example, there is only one process required to strip data out of tweets, but ten to resolve the URL. By load balancing this way, we can maintain a steady throughput of messages that does not get overloaded at any point.

To make Zeitgeist, we have had to handle large data sets at high speed. As a rough guide, the Zeitgeist ingest chain handles about 300,000 tweets an hour, of that 900 contain links, 500 of which link to the ³ÉÈËÂÛ̳. Finally, short lists work well as there's a steep drop-off of tweets lower down the chart and as you might expect the majority of links point to ³ÉÈËÂÛ̳ News articles.

Zeitgeist is now up and running for a limited period and we trust that you'll find it an interesting resource. We think a system like this could feed into ³ÉÈËÂÛ̳ Search as a ranking algorithm, as an additional real-time feed for News recommendations, or as a 'news on the move' mobile service. In any case it shows how audiences can help shape and prioritise content.

Visit the ³ÉÈËÂÛ̳ prototype