How we built A Feed Apart (using Django, Twitter, and Flickr)

UPDATE (5/2010): Developer Steve Losh talks about rebuilding A Feed Apart here. Check it out!

If you're just tuning in, A Feed Apart is a web app that captured tweets & photos that were created during An Event Apart Boston 2009. Nick Sergeant and I wrote it the night before An Event Apart, and launched it with reasonable success.

This post is to feed the curiosity of the programmer folk who asked "how we did that."

Server & Software

A Feed Apart was built on top of Django, and uses the Twitter and Flickr APIs. It's hosted on a 512Mb Slice (from Slicehost), which it shares with Lion Burger's home page, Snipt and a couple small apps.

These applications are served by Apache2 running mod_python, and proxied by Nginx. All of this is installed on a 512Mb Slicehost slice running Ubuntu 8.10 (Intrepid Ibex).

Application Design

We used django-syncr to capture and store tweets and photos during the conference. Django-syncr is a django module that supplied us with the API frameworks and object models that we used to archive tweets & photos. We didn't require a lot of the template code or additional APIs that django-syncr came with. We just added syncr.flickr & syncr.twitter to our INSTALLED_APPS, ran python manage.py syncdb, and then accessed their FlickrSyncr and TwitterSyncr classes directly.

At this point, we were ready to start fetching tweets & photos. After a little bit of planning (we had to decide which tags to search for, how we should sort/categorize the content, etc.), we were ready to roll. I started by pinging twitter's search API, and parsing the tweets from the response [see example].

With that code in place, I could access individual tweets by iterating over TwitterSearchFeed.entries. The next step was to store these tweets in the DB with the TwitterSync class provided by django-syncr.

After a few iterations of this, we ran into the infamous Twitter API rate limit (100 requests per hour; read more about it). When TwitterSyncr is fed a tweet_id, it uses Twitter's REST API to collect all of the metadata for that tweet. Clearing out the DB and refreshing our tweet searches over and over again put a screeching halt to Twitter development a few times. In hindsight, we should have applied for the Twitter whitelist when we started the project.

We added a couple more searches for other AEA terms, and had to handle the occasional http error response from Twitter. Besides that (and the rate limit issue), our Tweet collection mechanism was in good shape.

The Flickr photo fetching code functions similarly:

• Perform a search through the Flickr API
• Pass a list of photos to a FlickrSyncr instance
• FlickrSyncr collects metadata & stores it in the DB
• Ice cream sundaes

The next step was to retrieve, sort, and categorize the tweets & photos for corresponding templates. There are two ways that information is surfaced on A Feed Apart: the live feed (home page), and the session pages.

The content on the live feed was sorted so that the very newest information was at the top, and newly added updates (via AJAX) took on a colored tint to signal the user that this information was fresh.

The session content was intended to act as an archive, and a such wanted users to experience the tweets chronologically as they scrolled down. We toyed with the idea of featuring the photos for each session separately. We ended up grouping photos together by the photographer, and placed them in the session timeline by the time that the last photo was taken. IMHO, the execution of photos on the site has yet to meet its potential.

Getting tweets & photos to share a chronologically ordered list was something of an obstacle. Rather than abstract this at the DB level, or add to the existing photo/tweet models, we decided to just sort them together, and pass them to the template as a list of dicts [see example].

If you have any suggestions for refactoring this into something more efficient, I'm all ears (and the AEA community would thank you, too).

What's next?

We're pretty sure you'll see an appearance of A Feed Apart at future AEA conferences :). Until then, feedback is always welcome, and ideas for new features are welcome-er. Thanks, everyone!