Micro-Messaging – Data Interchange Standards and Track

I’ve been hard at work over the past week. Having your own company which you are attempting to bootstrap in this economy and sponsoring an academic project with ASU Polytechnic and – in my spare time – working on the challenges of real-time information discovery and participation is exhausting. Never-mind the two children under 6.

I’ve listened to what everyone has had to say regarding the “fire-hose” – or as I tend to refer to it – the question of trackable scope. Karoli took the time to write a very persuasive and passionate post – which you can read here. While we still may not agree weather or not the “fire-hose” is required to make track – I think we understand each other’s point of view. We agree on what is important – if not in which order and why. That is enough for me.

Apparently I was mentioned on the Gillmor Gang on 11/11/2008 – I’ve included the podcast below:

PodCast courtesy of The Gillmor Gang

icon for podpress Standard Podcast [60:12m] Download (746)

The discussion turns to track for the last half or so of the hour. After sitting with my latte this morning and listening (to some parts more than once) I believe I have a clearer understanding of Steve Gillmor’s perspective on the issue.

I completely agree with Steve that establishing a base mechanism for data interchange between real-time/near real-time social media services is going to be critical to the ultimate value delivered. As I’ve discussed on identi.ca we need a real-time data “bus” which moves data in real time from publishers to subscribers. Much the same way an electrical bus moves electricity from generators to consumers. At some point that bus – when widely adopted – will become a standard.

I’ll be posting more about the bus early next week.

I believe – and I am quite certain history bears this out – that standards develop because they benefit the services that implement them. In most cases this is because the interchange of data in some structured way is required to unlock the full value of a particular service or solution. We’ve seen this evolution in the past – email is an excellent example. Prior to SMTP every major producer of email systems had a “standard” for routing email between users. SMTP became dominant because it became more valuable to have email that could be exchanged with anyone than to have email without that capability. As a matter of fact it became a deal breaker if you couldn’t send email to anyone.

A counter example can be found in the world of Instant Messaging. After nearly 10 years there is no dominant standard. Each network implements it’s own standard and perhaps bridges messages to other standards. AIM uses OSCAR, GTalk uses XMPP, MSN uses SIP/SIMPLE. You want them all – you need a clever developer who creates a client that can talk to all 3.

There are many reasons that these standards either emerge or fail to emerge. But I’m fairly certain that it has rarely been the case that the standard was implemented because a small, vocal community of users insisted on it. I am very certain that the majority of standards become dominant is because there is a business imperative which makes using a standard more valuable than not.

Call me cynical – but that is how the world works. The question isn’t should there be a widely implemented standard for real-time micro-messaging, the question is what is the win-win? What is the business imperative that will drive widespread adoption? Specifically – how does it benefit Twitter to publish everything to the real-time messaging bus?

My contention is – as I’ve said before:

When compelling and broadly adopted services exist, which demand real-time un-scoped access to multiple underlying services, the individual services will have no choice but to “open their kimono” or face massive user defection.

The key part of that statement is “broadly adopted services exist“. My opinion is that we have to focus on the value proposition. What are the problems being solved and why are the valuable to users?

There are many – and some can be solved today (and as Karoli knows – some that can’t) – without the fire-hose. If I did not believe that to be true I wouldn’t be attempting to solve them. Will they be imperfect? Yes – but the goal isn’t perfection on day one – it is making a situation incrementally better by solving the important problems facing the user.

FriendFeed offers an interesting case – since they base their business model on being an aggregator. And, at least in theory, aggregation is one way to establish a real-time messaging bus and standard. It, however, requires not a network of peers but a single massive aggregator serving as the gateway/hub for access to information.

What I know – with complete certainty – is that the marketplace has ways of working these types of issues out. There will be a winner (or winners). They may or may not be the best technical solution. The real-time micro-messaging bus will be created to support the solutions that gain traction in the market. The solutions will not constrain themselves to 140 characters or any other standard which impedes the ability to solve important problems.

In short – until we hash out the types of services and how they deliver value AND the business imperative which drives a broadly implemented standard… there will be no standard (beyond paper standards).

So I’m going back to work creating value and solving important problems using the power of real-time (or near real-time) information discovery and participation… you in?

The myth of “everything” – Responses to my view of Track

My post yesterday – specifically regarding the Great Track Debate – received several responses on Twitter, identi.ca and friendfeed. Many of these – really all of them were positive – but the debate is far from settled.

I learned something very important from one discussion in particular – that there is a myth that permeates the conversation. The myth of “everything”.

This myth is the adherence to two ideas:

  1. That somehow the “fire-hose” (as discussed here) represents the complete information about any given topic – that somehow the “fire-hose” is everything.
    1. At the root of it, this is the idea that everything is attainable.
  2. That – in order to monetize track – having everything is essential. For example, if you are trying to manage your brand you need every reference to it in real-time.
    1. At the root of it, this is the idea that everything is required and valuable.

The reason I refer to these two ideas collectively as the “myth of everything” is because when they are clearly stated and examined they are immediately recognizable as inconsistent with reality.

So let’s take the two ideas one at a time and examine how tenuous their attachment to reality is.

First, that the “fire-hose” represents everything about any given topic. The fire-hose is the sum total of what is said on a given service. In order for that to be everything that service would need to be participated in by everyone. That is a hard enough hurdle to overcome, but there is more – not only would it need to be everyone, it would need to be the only method by which they communicate their thoughts, ideas and feelings.

Even if you were able to combine the fire-hose of every service available – every social networking site, every blog, every micro-blog, every IM service, every news site – you would still be far short of everything.

So the idea that track only becomes valuable when it can capture “everything” is a myth. Everything is unattainable.

Second, that having everything is essential. Let’s suppose we were to allow that somehow everything was achievable. Even if it were would it be required? Would it provide value commensurate with the effort required to collect it?

Let’s evaluate this in terms of brand management. The assumption which underlies this is that it is required to respond in real time to every post which is misleading, false, or damaging. This assumption is flawed – the reality is that it is required to respond to a statistically relevant sample of those posts. You aren’t trying to refute every post – you are trying to move (or keep from moving) the average (or perhaps mean) opinion.

If any company were forced to staff enough positions to actively monitor and respond to every post made about them – they would immediately cease to be profitable. It isn’t scalable, and more importantly it isn’t required.

This holds true equally for politics.

So the idea that track delivers value because having access to everything is required and the primary driver of value is a myth. Everything is neither required nor valuable in real terms.

What the track community should be focused on – again IMHO – is not the fire-hose and the attendant myth of everything, but creating systems which can attain enough trackable scope to provide a statistically relevant sample of the posts in the social media universe.

I understand that, emotionally, it feels good to tap into some perceived “everything” and refute any and all posts that you think are misleading, false, biased, or offensive. But this isn’t about what feels good – at the end of the day it will be about what is effective. And to be effective everything is neither required nor valuable.

These two conclusions – that everything is unattainable and that – even if attained – is neither required nor valuable should allow us to dispense with the “myth of everything” and return to the point of track:

  1. Real-Time Information Discovery
  2. Real-Time Participation