cosinity, Tech

Twitter, @replies and Multicasting

Twitter made a change to how @ replies are handled yesterday and the response has been, well, loud. Essentially you used to be able to see @ replies to everyone you followed, even if you didn’t follow the sender. Under Twitter’s new rules you only get to see @ replies to people you follow IF you follow the sender.

To many this may seem like a trivial change, but to those who use Twitter to discover interesting people… it is a very big deal. There has been rampant speculation (ignited, near as I can tell, by this post from Jesse Stay of SocialToo) that this change “kills” Follow Friday.

I’m including my comment on Jesse’s post here:

Ok… just to clear this one up. I checked (as you know I use justSignal to track Follow Friday for http://followfridays.com).

I have every Follow Friday tweet from 4/13 – 5/13 (last 30 days). There are a total of 771,244 Tweets. Of those 220,166 BEGIN with @username. That is 28.5% of the Tweets.

So definitely NOT most… but a very significant percentage.

Also, we’ve added User Filters to justSignal on FollowFridays. What does that mean? You can filter the total Follow Friday Tweet Stream by what it is about people you want to discover. Want to find PR folks to follow… add a User Filter for PR…

Follow Friday is most certainly NOT dead.

The short version of my take on this is that there are much, much better ways to find interesting people to follow than to rely on the people you follow to tell you about them. Don’t get me wrong, I believe in the power of recommendations, but I also believe in the power of context.

You see, my fictional Twitter Friend Joe is really, really into late 13th century birth control. I – for whatever it is worth – find his updates about walrus skin condoms entertaining. Beyond that Joe and I don’t have much in common. There is nothing atypical about this Twitter relationship… in fact, if I had to bet with my own money I would guess that 90% of Twitter relationships are pretty similar to this. So, Joe’s Follow Friday recommendations are only valuable to me in a very, very narrow window of my interests… so if I don’t see them all am I really missing anything?

What I really want is to see the recommendations from the 10% of the people I follow with whom I have a much broader affinity AND people recommending others because they are really smart about (CONTEXT). Whatever my context of interest happens to be right now.

That is why we’ve added User Filters to the justSignal Tracker on FollowFridays to allow you to filter the live Twitter Stream for your context, your 10%, or whatever else makes Follow Friday work for you. That is what we call – Getting Signal.

Multicasting & @ Replies:

UPDATE: 05/13/2009 @ 4:10PM Pacific Time

I blew it… the below examples misrepresent what the @ reply option did. I’m not going to redact it or change it… because:

  1. I blew it… and that happens and is ok with me.
  2. The example of Tweet distribution as it relates to a multicast system vs. a non-multicast system is still 100% relevant and correct.

END UPDATE

Biz made an update to the Twitter blog letting us know that the change to the @ reply system wasn’t really about user confusion – but a technical issue. This is a reason that actually makes sense. It makes sense for one simple reason – the “web” and Twitter were not built to efficiently implement multicasting. The problem with these @ replies is they create a burst of Tweet traffic… why?

If you create a normal Follow Friday tweet like:

#followfriday @joe @mary @steve @mike @larry @meg @biz @wally

That tweet will be sent to joe, mary, steve, mike, larry, meg, biz and wally. Let’s say each of them are followed by 400 people. And let’s say 60% of those people have “show me all replies to those I follow” checked.

This tweet will be sent 1,928 times:

  • 8 – The original 8
  • 240 – Joe’s followers who’ve checked show me all replies
  • 240 – Mary’s followers who’ve checked show me all replies
  • 240 – Steve’s followers who’ve checked show me all replies
  • 240 – Mike’s followers who’ve checked show me all replies
  • 240 – Larry’s followers who’ve checked show me all replies
  • 240 – Meg’s followers who’ve checked show me all replies
  • 240 – Biz’s followers who’ve checked show me all replies
  • 240 – Wally’s followers who’ve checked show me all replies

Now lets say we create the following Follow Friday tweet:

#followfriday @scoblizer @mashable @aplusk @oprah @cnnbrk

Again, let’s round things off and say each of these people are followed by 700,000 each and 60% have “show me all replies to those I follow” checked.

This tweet will be sent 2,100,005 times:

  • 5 – The original 5
  • 420,o0- – Scoble’s followers who’ve checked show me all replies
  • 420,000 – Mashable’s followers who’ve checked show me all replies
  • 420,000 – AplusK’s followers who’ve checked show me all replies
  • 420,000 – Oprah’s followers who’ve checked show me all replies
  • 420,000 – CNNBRK’s followers who’ve checked show me all replies

YIKES!!! Now imagine that tweet getting re-tweeted a couple hundred times. This is both why Twitter (I’m speculating here but with a high degree of confidence) “edited” trending topics to exclude Follow Friday AND why the @ reply change was made.

Why Multicast Matters:

Today Twitter (because they don’t implement efficient multicasting and instead rely on publish/subscribe architectures) has to effectively send each of those 2,100,005 tweets serially, one at a time. This consumes massive amounts of computing resources.

Multicast would allow them to send it once to many destinations – thereby removing that bottleneck from their scalability and tweet processing systems.

Those of you who are familiar with large scale real-time communications systems (i.e., VoIP) will immediately recognize this problem – it was central to creating a large scale VoIP service platform in the late 1990’s. The VoIP community resolved it (not without much effort) and now a very small system can efficiently multicast hundreds of millions of status updates/changes per hour.

I totaly buy Twitter pulling @ replies due to technical issues – as exposed by Follow Friday combined with celebrity adoption and massive follower counts. But if you are going to be the real-time web backbone disabling useful things because they create too much load instead of implementing a more efficient architecture isn’t the right answer.

Published by briantroy

entrepreneur, technologist, foodie & dad - technology leadership consulting @ partition tolerance

16 thoughts on “Twitter, @replies and Multicasting”

  1. This isn't what the problem was at all. If some person I don't follow tweets about someone I follow, I'm not going to see that. If I follow @scobleizer and you (who I don't follow) tweet about @scobleizer, I would never see your tweet, no matter what my setting was. Twitter never did that, and probably never will.The “show me all replies” checkbox was for viewing @replies from people I do follow directed at people I don't follow. So it's trimming @replies from people you already follow.http://twittercism.com/all-replies/

    Like

    Reply

    1. Not entirely true that is the case (per the post you cited and others):'Williams explained how this was now an important part of the network and went on to mention that 98 per cent of Twitter users left their replies on the ‘default’ setting – that is, they only saw @ replies to the people they are following.”That being said… I may have this wrong in terms of who sees what based on an @ reply… but the issue of multicast in highly scalable real-time systems remains the same.If I do have this wrong I'll update the post (once I can be sure). Part of my confusion is I never had All @ Replies turned on.

      Like

      Reply

      2. Entirely possible. It does not seem CLEAR in any of the posts I've ready. The previous see all @ replies setting is both documented poorly and explained multiple ways by everyone. From the post Luigi cites:What this means is that whenever you open a tweet with @, Twitter reads the submission as a reply. This:@Sheamus is a great guy!Is a very different message to this:I tell you who's a great guy – @Sheamus!Twitter absorbs these messages in different ways. The first, the reply, will be seen by the following people: 1. @Sheamus 2. Anybody who is following @Sheamus and the sender of the message 3. Anybody who, previously, had Twitter set to see all repliesThat actually supports my supposition. But hey.. If I'm wrong I'm wrong… thanks for pointing it out.

        Like

      3. You never had “all @ replies” on? If you noticed a difference with Twitter's settings update, then you must have. If you had “@ replies to the people I'm following” selected, your stream should be the same now as it was before.I have never seen nor heard about people seeing @ replies in their streams FROM people they don't follow. Your home screen (& statuses/friends_timeline API calls) should only show tweets from people you follow. Depending on the setting you had selected, you would either see 100% of tweets from those you follow (“all @ replies”), most tweets excluding those from your friends TO people outside of your network (“@ replies to the people I'm following”), or only non-conversational tweets (“no @ replies”).Also, it would seem strange for Twitter to implement the multicast network protocol to replicate objects in a relational database model… Are you implying each user should have an independent db or table for the tweets of those they follow?

        Like

      4. No, I never had it on. And I didn't notice a difference, I was commenting on:1) The idea that the change would kill Follow Fridays2) The fact that (which isn't changed by my misunderstanding the issue) multicast is vital to any large scale real-time system.The same situation (regarding multicast) existed with VoIP. Multicast isn't about replicating the object in the DB – as a matter of fact the “object” should only exist once in the DB. It is about propagating the “object” to 1 or many endpoints/clients efficiently. Think of it this way, the tweet only exists once (like a packet of real-time video) – but it has to get to a multitude of “viewers”. We know the efficient way to do that is multicast. Iterating through each “viewer” and sending the same packet over and over again just won't work at scale. The content of the “packet” isn't relevant – could be status, tweet, video, audio, etc. What is relevant is that you have to send it to many endpoints in the most efficient way possible.I'll update the post to indicate I misunderstood the @ reply feature… My Bad.

        Like

      5. I think the part I'm stuck on is that yes, Twitter is a near-real-time system, but with the exception of the SMS gateway, Twitter is not a push mechanism.My background with multicast is from a live video perspective. The theory is great: I have one server and thousands of clients waiting to be pushed the same data at the same instant.With Twitter, my Twitter update doesn't get pushed to X people simultaneously. It gets saved into a database and waits until clients come by at their convenience and request/pull the data. Luckily, Twitter updates are immutable and can be indexed, cached, and optimized six ways from Sunday.

        Like

      6. I understand. But if Twitter is going to be a real-time service it will have to do EXACTLY what you need to do for video (or what I've done for VoIP conferenece calls). That is one of my fundamental annoyances with Twitter and the “Real-Time” web community – they want real-time but only if it works within the constraints of the publish/subscribe architectures they already use. We need to re-think that.XMPP is already building in Multicast (http://xmpp.org/extensions/xep-0033.html) SIP/SIMPLE has had it for years.I'm not suggesting that part of Twitter won't continue to be publish/subscribe. I'm suggesting that if they really want to be real-time publish/subscribe won't get that done.

        Like

  2. This isn't what the problem was at all. If some person I don't follow tweets about someone I follow, I'm not going to see that. If I follow @scobleizer and you (who I don't follow) tweet about @scobleizer, I would never see your tweet, no matter what my setting was. Twitter never did that, and probably never will.The “show me all replies” checkbox was for viewing @replies from people I do follow directed at people I don't follow. So it's trimming @replies from people you already follow.http://twittercism.com/all-replies/

    Like

    Reply

  3. Not entirely sure that is the case (per the post you cited and others):'Williams explained how this was now an important part of the network and went on to mention that 98 per cent of Twitter users left their replies on the ‘default’ setting – that is, they only saw @ replies to the people they are following.”That being said… I may have this wrong in terms of who sees what based on an @ reply… but the issue of multicast in highly scalable real-time systems remains the same.If I do have this wrong I'll update the post (once I can be sure). Part of my confusion is I never had All @ Replies turned on.

    Like

    Reply

  5. You never had “all @ replies” on? If you noticed a difference with Twitter's settings update, then you must have. If you had “@ replies to the people I'm following” selected, your stream should be the same now as it was before.I have never seen nor heard about people seeing @ replies in their streams FROM people they don't follow. Your home screen (& statuses/friends_timeline API calls) should only show tweets from people you follow. Depending on the setting you had selected, you would either see 100% of tweets from those you follow (“all @ replies”), most tweets excluding those from your friends TO people outside of your network (“@ replies to the people I'm following”), or only non-conversational tweets (“no @ replies”).Also, it would seem strange for Twitter to implement the multicast network protocol to replicate objects in a relational database model… Are you implying each user should have an independent db or table for the tweets of those they follow?

    Like

    Reply

  6. Entirely possible. It does not seem CLEAR in any of the posts I've ready. The previous see all @ replies setting is both documented poorly and explained multiple ways by everyone. From the post Luigi cites:What this means is that whenever you open a tweet with @, Twitter reads the submission as a reply. This:@Sheamus is a great guy!Is a very different message to this:I tell you who's a great guy – @Sheamus!Twitter absorbs these messages in different ways. The first, the reply, will be seen by the following people: 1. @Sheamus 2. Anybody who is following @Sheamus and the sender of the message 3. Anybody who, previously, had Twitter set to see all repliesThat actually supports my supposition. But hey.. If I'm wrong I'm wrong… thanks for pointing it out.

    Like

    Reply

  7. No, I never had it on. And I didn't notice a difference, I was commenting on:1) The idea that the change would kill Follow Fridays2) The fact that (which isn't changed by my misunderstanding the issue) multicast is vital to any large scale real-time system.The same situation (regarding multicast) existed with VoIP. Multicast isn't about replicating the object in the DB – as a matter of fact the “object” should only exist once in the DB. It is about propagating the “object” to 1 or many endpoints/clients efficiently. Think of it this way, the tweet only exists once (like a packet of real-time video) – but it has to get to a multitude of “viewers”. We know the efficient way to do that is multicast. Iterating through each “viewer” and sending the same packet over and over again just won't work at scale. The content of the “packet” isn't relevant – could be status, tweet, video, audio, etc. What is relevant is that you have to send it to many endpoints in the most efficient way possible.I'll update the post to indicate I misunderstood the @ reply feature… My Bad.

    Like

    Reply

  8. I think the part I'm stuck on is that yes, Twitter is a near-real-time system, but with the exception of the SMS gateway, Twitter is not a push mechanism.My background with multicast is from a live video perspective. The theory is great: I have one server and thousands of clients waiting to be pushed the same data at the same instant.With Twitter, my Twitter update doesn't get pushed to X people simultaneously. It gets saved into a database and waits until clients come by at their convenience and request/pull the data. Luckily, Twitter updates are immutable and can be indexed, cached, and optimized six ways from Sunday.

    Like

    Reply

  9. I understand. But if Twitter is going to be a real-time service it will have to do EXACTLY what you need to do for video (or what I've done for VoIP conferenece calls). That is one of my fundamental annoyances with Twitter and the “Real-Time” web community – they want real-time but only if it works within the constraints of the publish/subscribe architectures they already use. We need to re-think that.XMPP is already building in Multicast (http://xmpp.org/extensions/xep-0033.html) SIP/SIMPLE has had it for years.I'm not suggesting that part of Twitter won't continue to be publish/subscribe. I'm suggesting that if they really want to be real-time publish/subscribe won't get that done.

    Like

    Reply

Leave a comment