FriendFeed to XMPP Bridge – Why, how and the plan.

Yesterday I took about and hour and created a FriendFeed to XMPP bridge. The advantage I had was that I was already running a XMPP server and had a working XMPP bot – these existed for my own personal use as well as for cosinity.

Why?

Primarily this is about information discovery. The ability to participate is challenging (see below) as FriendFeed is an aggregator.

As I said in an earlier post – I believe FriendFeed’s Real-Time functionality is missing two important capabilities.

1. The ability to filter the feed by keyword(s).
2. The ability to get real-time for the public timeline.

While there is nothing I can do about access to the public timeline – I can, via the API, create the ability to filter a feed by keyword(s).

Additionally, XMPP is a clean way to distribute the information – so why not have it published via XMPP?

How?

Since FriendFeed had not released any sample code or a Python/PHP library for real-time yet I had to modify the existing libraries to support real time. This turned out to be trivial – I simply made a new {fetch_RT} function which looks much like the existing {fetch} function. With that complete I could simply use this modified PHP library to grab a user’s real-time stream.

As I mentioned earlier I already had an XMPP server running and had created a bot for other reasons. I simply re-used that bot and the over the network XML API defined by the bot to send the FriendFeed stream out as XMPP messages.

There are several important things to note here about the architecture:

1. De-Coupling the FriendFeed real-time processes from the XMPP bot processes is a very efficient model.
2. Keeping the XMPP stream bot a “dumb pipe” (i.e., outbound message streaming only) is also very efficient.
   1. Clicking on the link at the front of the XMPP message opens the FriendFeed “conversation” in a browser.
3. Since FriendFeed is an aggregator, participation via XMPP will be very problematic.
   1. For Example: If the message is a tweet do you want to reply with a tweet – or comment on the tweet in FriendFeed’s conversation?
   2. The bot would have to track a large number of variables for each message in order to properly distribute your comment/message/etc.
   3. This overhead would directly affect the ability of the service to scale.
4. The scope of the feed (e.g., the apple room or My Home feed or my “the cool people list”) and the keywords for which you’d like an XMPP notification would be configured via a web interface.
   1. This – again – goes directly to keeping the service as scalable as possible.
   2. The down side is you can’t change scope or keywords via XMPP messages.
   3. There is the possibility of creating a separate “ffstreamcontrol” bot which would take these commands. It remains to seen if this is really required or a nice to have.
5. Since the stream bot acts as a dumb pipe – it can rapidly scale by running multiple instances with the same UID and a different resource string. You won’t know (or care) which bot you are connected to – they all look the same. If one dies – the FriendFeed real-time processes will be programed to try the next one… and the next one… etc.
   1. Again – this is a very efficient model.
   2. Will provide high levels of redundancy
6. The bots will all connect to a dedicated XMPP server – the XMPP messages will then be distributed via XMPP federation.
   1. XMPP federation protocols are very efficient.

The Plan:

My current plan is to build something similar to what you see below. The user will create XMPP streams based on scope (i.e., FriendFeed scope – For Example: the Debates 2008 room) and keyword(s). The user will be allowed multiple XMPP streams. All XMPP stream creation, deletion and modification will be done via a web interface.

The XMPP bot(s) will serve as a dumb pipe for content delivery.

Participation is problematic – and as such will be set aside for now. User’s can still participate by clicking on the link in the XMPP message – this link will open the FriendFeed conversation (here is one for example). This will allow the user to participate, subscribe to the user’s involved in the conversation, etc.

Please feel free to comment – let me know if you think this is headed in the right direction.
NOTE: This was written very quickly based on a prototype made yesterday. Please excuse any less than completely thought through concepts… it is a work in progress.