Twitter, @replies and Multicasting

Twitter made a change to how @ replies are handled yesterday and the response has been, well, loud. Essentially you used to be able to see @ replies to everyone you followed, even if you didn’t follow the sender. Under Twitter’s new rules you only get to see @ replies to people you follow IF you follow the sender.

To many this may seem like a trivial change, but to those who use Twitter to discover interesting people… it is a very big deal. There has been rampant speculation (ignited, near as I can tell, by this post from Jesse Stay of SocialToo) that this change “kills” Follow Friday.

I’m including my comment on Jesse’s post here:

Ok… just to clear this one up. I checked (as you know I use justSignal to track Follow Friday for http://followfridays.com).

I have every Follow Friday tweet from 4/13 – 5/13 (last 30 days). There are a total of 771,244 Tweets. Of those 220,166 BEGIN with @username. That is 28.5% of the Tweets.

So definitely NOT most… but a very significant percentage.

Also, we’ve added User Filters to justSignal on FollowFridays. What does that mean? You can filter the total Follow Friday Tweet Stream by what it is about people you want to discover. Want to find PR folks to follow… add a User Filter for PR…

Follow Friday is most certainly NOT dead.

The short version of my take on this is that there are much, much better ways to find interesting people to follow than to rely on the people you follow to tell you about them. Don’t get me wrong, I believe in the power of recommendations, but I also believe in the power of context.

You see, my fictional Twitter Friend Joe is really, really into late 13th century birth control. I – for whatever it is worth – find his updates about walrus skin condoms entertaining. Beyond that Joe and I don’t have much in common. There is nothing atypical about this Twitter relationship… in fact, if I had to bet with my own money I would guess that 90% of Twitter relationships are pretty similar to this. So, Joe’s Follow Friday recommendations are only valuable to me in a very, very narrow window of my interests… so if I don’t see them all am I really missing anything?

What I really want is to see the recommendations from the 10% of the people I follow with whom I have a much broader affinity AND people recommending others because they are really smart about (CONTEXT). Whatever my context of interest happens to be right now.

That is why we’ve added User Filters to the justSignal Tracker on FollowFridays to allow you to filter the live Twitter Stream for your context, your 10%, or whatever else makes Follow Friday work for you. That is what we call – Getting Signal.

Multicasting & @ Replies:

UPDATE: 05/13/2009 @ 4:10PM Pacific Time

I blew it… the below examples misrepresent what the @ reply option did. I’m not going to redact it or change it… because:

  1. I blew it… and that happens and is ok with me.
  2. The example of Tweet distribution as it relates to a multicast system vs. a non-multicast system is still 100% relevant and correct.

END UPDATE

Biz made an update to the Twitter blog letting us know that the change to the @ reply system wasn’t really about user confusion – but a technical issue. This is a reason that actually makes sense. It makes sense for one simple reason – the “web” and Twitter were not built to efficiently implement multicasting. The problem with these @ replies is they create a burst of Tweet traffic… why?

If you create a normal Follow Friday tweet like:

#followfriday @joe @mary @steve @mike @larry @meg @biz @wally

That tweet will be sent to joe, mary, steve, mike, larry, meg, biz and wally. Let’s say each of them are followed by 400 people. And let’s say 60% of those people have “show me all replies to those I follow” checked.

This tweet will be sent 1,928 times:

  • 8 – The original 8
  • 240 – Joe’s followers who’ve checked show me all replies
  • 240 – Mary’s followers who’ve checked show me all replies
  • 240 – Steve’s followers who’ve checked show me all replies
  • 240 – Mike’s followers who’ve checked show me all replies
  • 240 – Larry’s followers who’ve checked show me all replies
  • 240 – Meg’s followers who’ve checked show me all replies
  • 240 – Biz’s followers who’ve checked show me all replies
  • 240 – Wally’s followers who’ve checked show me all replies

Now lets say we create the following Follow Friday tweet:

#followfriday @scoblizer @mashable @aplusk @oprah @cnnbrk

Again, let’s round things off and say each of these people are followed by 700,000 each and 60% have “show me all replies to those I follow” checked.

This tweet will be sent 2,100,005 times:

  • 5 – The original 5
  • 420,o0- – Scoble’s followers who’ve checked show me all replies
  • 420,000 – Mashable’s followers who’ve checked show me all replies
  • 420,000 – AplusK’s followers who’ve checked show me all replies
  • 420,000 – Oprah’s followers who’ve checked show me all replies
  • 420,000 – CNNBRK’s followers who’ve checked show me all replies

YIKES!!! Now imagine that tweet getting re-tweeted a couple hundred times. This is both why Twitter (I’m speculating here but with a high degree of confidence) “edited” trending topics to exclude Follow Friday AND why the @ reply change was made.

Why Multicast Matters:

Today Twitter (because they don’t implement efficient multicasting and instead rely on publish/subscribe architectures) has to effectively send each of those 2,100,005 tweets serially, one at a time. This consumes massive amounts of computing resources.

Multicast would allow them to send it once to many destinations – thereby removing that bottleneck from their scalability and tweet processing systems.

Those of you who are familiar with large scale real-time communications systems (i.e., VoIP) will immediately recognize this problem – it was central to creating a large scale VoIP service platform in the late 1990’s. The VoIP community resolved it (not without much effort) and now a very small system can efficiently multicast hundreds of millions of status updates/changes per hour.

I totaly buy Twitter pulling @ replies due to technical issues – as exposed by Follow Friday combined with celebrity adoption and massive follower counts. But if you are going to be the real-time web backbone disabling useful things because they create too much load instead of implementing a more efficient architecture isn’t the right answer.