I’ve been thinking about writing a series of posts about Big Data for months now… which is entirely too much thinking an not enough doing, so here we go.

And, by Big Data we mean…

Wikipedia offers a very computer science oriented explanation of Big Data – but while the size of the dataset is a factor in Big Data there are several others worth considering:

• The arrival pattern (how data is created in the large dataset)
• Dimensions (how the data is grouped, sub-grouped and organized for analysis)
• Transformation (how the data is re-dimensioned or transformed to meet new analysis needs)
• Consumption (how will the data be consumed by client applications and ultimately humans)

Classically Big Data was regarded is extremely large datasets, usually acquired over a long period of time, involving historical data leveraged for predictive analysis of future events. In the simplest terms, we use hundreds of years of historical weather data to predict future weather. Big Data isn’t new, my first exposure to these concepts was in the early 1990′s dealing with Contact Center telecom and CTI (computer telephony integration) analytics and predictive analysis of future staffing needs in large footprint (3 to 5 thousand agent) contact centers.

Another example of the classic Big Data problem is the traditional large operational dataset. In the 90′s a 2 Terabyte dataset – like the one I worked with at Bank of America – was so massive that it created the need for a special computing solution. Today large operational datasets are measured in Petrabytes and are becoming fairly common. The primary challenges with these types of datasets is maintaining acceptable operational performance for the standard CRUD operations.

There are Big Data storage models emerging and maturing today with the two default options being hadoop – which relies on a distributed file system and distributed key value store systems such as Cassandra and CouchDB. These systems (referred to as NoSQL solutions) differ from standard RDBMS systems in two important ways:

• The underlying organization of data is amorphous and does not implement relationships.
• There is no support for Transactional Integrity (also know as ACID).

While all of these are interesting engineering problems, they still lack a crucial component. As a matter of fact, most conversations about Big Data fail to adequately address what is, perhaps, the most important problem with Big Data systems today.

The Intersection of Real Time and Big Data

Today’s big datasets are manageable in RDMBS systems. That being said, a significant amount of complexity is inserted in the managament process, most notably:

• Database Sharding and the complexity of managing multiple databases.
• Downtime associated with Schema changes or complexity of CLOB fields used in a amorphous manner.

Given that, large datasets that change slowly over time, or more accurately, those that have a relatively low volume of creates (including those that occur as a result of large transformations) as compared to read, update and delete can be managed using RDBM systems.

Where Real Time – specifically as related to Social Media, user generated content and other high create applications (such as the Large Hadron Collider) – intersect with Big Data is – to me – the most interesting Big Data topic.

This model presents three distinct challenges:

• High data arrival – potentially hundreds of thousands of new objects “found” per second.
• High transformation rate.
• Real Time Client updates.

The intersection of all three of these challenges was exactly the what we dealt with at justSignal. We needed to consistently collect hundreds of thousands of Social Media objects per second, generate hundreds of meta data elements (transformations) for each object, and make all of that data available in real time to our client applications. This view of Big Data is slightly different. In this view the size of the dataset – while still significant – isn’t as important as the challenges presented by a very high volume of CRUD operations over very short time slices.

The most important thing I’ve learned is that there is no silver bullet. While the traditional relational database isn’t effective in real time big data scenarios neither is a standalone Hadoop or Distributed Key Value Store. Essentially, you must evaluate each use case against the suite of technologies available and select the one best suited to that particular use case. Selecting a NoSQL solution for order processing – which has heavy ACID/Transactional requirements isn’t a good idea. However, staying with your RDBMS for high insert/transform data processing isn’t going to work either.

The approach we took at justSignal – which I will go into in more detail in a future post – was to create a unified data persistence layer designed to leverage the right long/short term data store (and sometimes more than one) based on the requirements of the application. This data persistance layer is made up of:

• MySQL
• memcached
• Cassandra
• Beanstalkd

Each plays a critical role in our ability to collect, transform, process and serve hundreds of thousands of social media mentions per minute.

More to follow…

I’m honestly heartened by the sudden rash of efforts to create a methodology to determine ROI (return on investment) for Social Media efforts. It signals something very important for Social Media – the return of rationality to the debate.

When you consider that a few short months ago the prevailing meme was that creating a basis for your Social Media efforts in terms of ROI was “doing it wrong” – it is impressive how far we’ve come. The realization that moral arguments and scare tactics will only get you so far – and in many cases backfire – has led to an overwhelming need to create an ROI model.

Unfortunately many of these efforts are not really after ROI – they are seeking to justify an already formed point of view.

The reality is we simply don’t know if Social Media has a analytical, fact based ROI. That may sound odd coming from a guy who has bet his personal savings starting a Social Media Engagement and Analytics company – so let me explain both why the ROI hasn’t been proven and why I’m betting it will be.

Social Media is a Niche Opportunity – Today

If you want to know why there is no fact based proven ROI for Social Media investments today, all you need to understand is that Social Media has been adopted in niches. It may be in the Marketing department, or used by your Digital Agency, or perhaps in your Customer Service department. Each of these adoptions was driven out of fear (we have to monitor this and deal with the negative) or the moral (we love our customers – so we are going to do this). The investment was negligible – and in most cases I’d bet it was funded right out of the operating budget of the organization where it was used.

These organizations are beginning to declare victory and are being challenged to prove it. This presents unique challenges, because Social Media runs on anecdotes, not analysis. Dell sells 3 million in product from Dell Outlet after offering those products on Twitter. That is a great anecdote – but it isn’t analysis. When you ask the critical questions:

• What would you have sold without Twitter?
• Was that a 3MM increase in sales – or just 3MM net sales from those links?
• How much did it cost to generate the 3MM in sales and how does that compare to email?
• Is this repeatable – can it be replicated in other parts of the business – and how do you know?

you quickly find that the anecdote doesn’t equate to ROI. It might… but it isn’t there yet.

These types of anecdotes are justifications. They are about proving the correctness of an already made assumption.

I’ve seen this movie before – it exactly parallels the pattern for CRM in the late 1990′s.

NOTE: For simplicity I’ve omitted the case where a technology/methodology has a niche ROI without broader adoption.

We are squarely in the middle of the justification phase for Social Media. This roughly corresponds to the height of the expectations (the big peak on the Gartner Magic Quadrant) and always directly precedes the Trough of Disillusionment. This is a recognizable and predictable pattern for adoption of new technologies and methodologies – and here is why.

The initial opportunity is too good to stay on the sidelines for some early adopter group. They – almost always within existing operating budgets and using the promise as a bulwark defense – adopt the technology/methodology. Once they believe they have seen tangible results they attempt to socialize the “win” outside the organization by creating justifications for what they’ve already done. These justifications bring broader scrutiny.

That scrutiny happens in two phases:

1. Was it worth it?
2. Can it be done systemically – can I forecast a x% increase in metric z if I do this again.

The second is ROI. A systemic way of proving that adoption generates a return. If, and only if, that can be proven will the technology escape the niche application and be applied on a broad scale.

Why does it work this way – because enterprises are first and foremost risk management systems. They systemically avoid large risks.

Why Will Social Media Attain Broad Adoption

The primary reasons I believe Social Media will in fact generate a valid ROI and attain broad adoption:

1. It is measurable.
2. The unrecognized value far exceeds the recognized value.

Measurability

As you might imagine, it is very difficult to justify and create a systemic ROI for something that is exceptionally difficult to measure. Social Media is – in contrast – eminently measurable. Rational decisions must be made about what to measure – and we need more focus on connecting those measures to the core business metrics – but there is no fundamental barrier to creating valuable measures.

The Value Proposition

Today, we’ve put all our Social Media eggs in the PR/Marketing basket. Even the small amount of credibility given to customer service via Social Media has been driven by the (C-Level Down) idea that customer service should “avert disasters” by monitoring Social Media and addressing customer issues. Make no mistake, this is customer service acting in a PR role – the goal isn’t to provide service so much as to avoid negative perceptions.

However, if you take one large step back and think about the opportunity Social Media presents – you can quickly see that the value proposition is in having a huge, open back channel to your market. We’ve had channels to our customers, and sometimes even our prospects – but this is bigger. It is the entire market for your product or service. You get to listen in on what they have to say about what they want and need. You can engage them to better understand their motivations. You can apply what you learn to create incremental improvements in every phase of your business.

Yes, you can send out special offers. Yes, you can address customer concerns. But the real return will come from having a robust back channel with your entire market; and the resulting market intelligence can – if you apply it – help you make every part of your business more appealing to your target market.

So let’s get serious about ROI. Let’s talk about how companies operate and win by continually tuning their processes to better address the needs of their target market. Let’s talk about how Social Media provides them a back channel to that market, a back channel that is an invaluable source of intelligence about the market.

Let’s talk about how a business that applies the intelligence gained via Social Media to all of their decision making processes is faster and more agile in addressing the needs of their market – and thereby wins market share.

David Alston of Radian6 stopped by yesterday and commented on this post.

This video is a detailed explanation of my views on Social Media, Brand Monitoring, and operationalizing and scaling those things in companies.

Video Blog 12-19-2008 | The question of Scale & Social Media for Brand Managment from Brian Roy on Vimeo.

Another cold medication induced video blog today.

Video Blog – 12-17-2008 | Track News and JustSignal Re-Launch from Brian Roy on Vimeo.
Link to Steve Gillmor’s post on Twitter Track
By the way… however Vimeo selects the static frame that shows before you press play… It isn’t working for me.

Trying something new… my very first video blog post – fueled by cold medicine. Don’t say I didn’t warn you.

Video Blog Post – 12-16-2008 | Paid Blog Posts and JustSignal IM from Brian Roy on Vimeo.

As promised here is the link to the FriendFeed Conversation about Paid Blog Posts.

Here is the link to the JustSignal forums on cosinity.com

My post yesterday – specifically regarding the Great Track Debate – received several responses on Twitter, identi.ca and friendfeed. Many of these – really all of them were positive – but the debate is far from settled.

I learned something very important from one discussion in particular – that there is a myth that permeates the conversation. The myth of “everything”.

This myth is the adherence to two ideas:

1. That somehow the “fire-hose” (as discussed here) represents the complete information about any given topic – that somehow the “fire-hose” is everything.
   1. At the root of it, this is the idea that everything is attainable.
2. That – in order to monetize track – having everything is essential. For example, if you are trying to manage your brand you need every reference to it in real-time.
   1. At the root of it, this is the idea that everything is required and valuable.

The reason I refer to these two ideas collectively as the “myth of everything” is because when they are clearly stated and examined they are immediately recognizable as inconsistent with reality.

So let’s take the two ideas one at a time and examine how tenuous their attachment to reality is.

First, that the “fire-hose” represents everything about any given topic. The fire-hose is the sum total of what is said on a given service. In order for that to be everything that service would need to be participated in by everyone. That is a hard enough hurdle to overcome, but there is more – not only would it need to be everyone, it would need to be the only method by which they communicate their thoughts, ideas and feelings.

Even if you were able to combine the fire-hose of every service available – every social networking site, every blog, every micro-blog, every IM service, every news site – you would still be far short of everything.

So the idea that track only becomes valuable when it can capture “everything” is a myth. Everything is unattainable.

Second, that having everything is essential. Let’s suppose we were to allow that somehow everything was achievable. Even if it were would it be required? Would it provide value commensurate with the effort required to collect it?

Let’s evaluate this in terms of brand management. The assumption which underlies this is that it is required to respond in real time to every post which is misleading, false, or damaging. This assumption is flawed – the reality is that it is required to respond to a statistically relevant sample of those posts. You aren’t trying to refute every post – you are trying to move (or keep from moving) the average (or perhaps mean) opinion.

If any company were forced to staff enough positions to actively monitor and respond to every post made about them – they would immediately cease to be profitable. It isn’t scalable, and more importantly it isn’t required.

This holds true equally for politics.

So the idea that track delivers value because having access to everything is required and the primary driver of value is a myth. Everything is neither required nor valuable in real terms.

What the track community should be focused on – again IMHO – is not the fire-hose and the attendant myth of everything, but creating systems which can attain enough trackable scope to provide a statistically relevant sample of the posts in the social media universe.

I understand that, emotionally, it feels good to tap into some perceived “everything” and refute any and all posts that you think are misleading, false, biased, or offensive. But this isn’t about what feels good – at the end of the day it will be about what is effective. And to be effective everything is neither required nor valuable.

These two conclusions – that everything is unattainable and that – even if attained – is neither required nor valuable should allow us to dispense with the “myth of everything” and return to the point of track:

1. Real-Time Information Discovery
2. Real-Time Participation

As those of you who follow me on twitter or friendfeed know FFStream – which I began discussing in this post – has evolved into FF-Filtered. The changes are not dramatic, but they are significant.

FF-Filtered is now focused on providing “your friendfeed – filtered” – and as that implies what it does is filter your friendfeed (home feed to be specific) by a list of keywords. If these keywords match the post title, comment, or user you receive the update in real time – in the browser or via IM, including GTalk, Jabber, AIM and Yahoo.

It isn’t track – as I’ve been repeatedly and vehemently told by the “community” over the last 5 days – more on that later.

Additionally – for the mobility set – we’ve added like, comment, post and filter updates via a mobile web page. If you click on the link in the IM from a mobile platform you get the following mobile web page:

I’ve got a few more tricks up my sleeve – so look for more changes this week including a name change.

Now for the second half of the post… I’m going to talk about Track again… so sharpen your knives (or tongues) and get ready to revel in your abject disdain for my refusal to “go along” or shut up.

Let me say one thing first – if you want to attack my positions and opinions go for it. If you are here to attack my motives or me personally – GO AWAY NOW.

The last 5 days has been very illuminating for me – both in terms of the fervor of the “track community” and in terms of their point of view – which at times verges on dogma.

Let me attempt to “play back” what I’ve heard and then explain – as clearly as possible given my limited skills – my point of view.

Track – by the definition of the “track community” as led today by Steve Gillmor of Gillmor Gang/News Gang Live, is defined by the “fire-hose”. This fire-hose is the complete unabridged stream of posts occurring on any social media site. In the case of Twitter, this is the entire public timeline published in real time.

This is an important distinction – because it states that “track” can not be achieved without the “fire-hose”. More on this latter.

The second component of “track” is the ability to keyword filter the real time stream and deliver the filtered content in real (or near real) time.

The third component of “track” is the ability to insert posts into the public timeline from the same user interface you are viewing the stream in.

Those four things collectively comprise the “holy grail” of track.

I’m sure you will all let me know exactly how wrong I am… but I think those four capture the broad brushstrokes. Ok?

Before I attempt to explain my point of view – let me clarify one point. Regardless of my agreement or disagreement with the track community on any given point, there is one thing we vehemently agree on:

There is massive value in the ability to discover and participate in the social media stream in real (or near) real time. Our objective where that is concerned is the same.

When I consider track – I consider it in terms of the problems it attempts to solve. To me, track is an attempt to solve 2 very important problems:

1. Real-Time Information Discovery
2. Real-Time Participation

Any solution which solves those two problems would – by my definition – fall within the scope of a “track” service.

Now let me explain why (take a deep breath… you can throw something at me later). Where I differ with the “track community” on this issue is on the scope of the track-able data not what happens after the “track service” receives it. As importantly I fundamentally agree that the wider the scope of the data being tracked the more effective the track solution will be.

But, consider this – not every user wants to track the entire social media universe. To the contrary – most IMHO simply want to track their friends, family, co-workers, brand, market makers, influencers, power users, etc.

For those users a limited scope is a good thing. Beyond consideration of the scope of the data being tracked this service solves the exact same problems.

1. Real-Time Information Discovery
2. Real-Time Participation

So apply the duck test. It walks like a duck… it quacks like a duck… why isn’t it a duck?

It is my opinion – and you can feel free to take issue with it – that the track communities’ obsession with the “fire-hose” has actually retarded the growth of alternative track services. The obsession with scope has prevented the creation of useful (if limited by their limited trackable data) solutions under the banner of track – and that is a shame.

Every developer that seeks to solve the two problems should be embraced, encouraged and supported.

The real battle here is one of leverage. And the way to get the social media services to both open up their data and participate in the creation of a standard for doing so is to create a win-win. I believe track services that are useful and solve real problems (e.g., real-time brand monitoring) can and will provide the leverage that causes the change the community has been seeking.

If Twitter wants to pretend they ARE the social media universe – let them. It is abundantly clear from the success of friendfeed that no single service is or will be the social media universe – any service that ignores this will fail.

When compelling and broadly adopted services exist, which demand real-time un-scoped access to multiple underlying services, the individual services will have no choice but to “open their kimono” or face massive user defection.

So stop complaining about the lack of a “fire-hose” and figure out what those services are, who needs them the most, and how to drive that value to as many users as fast as possible. If you do that – you’ll get what you want… not today, not even tomorrow… but relatively soon.

I had intended to discuss standards, what I believe the high level components of an open track environment might look like, and why friendfeed is in the best position to lead standard development… but this has already gotten too long. I’ll set those subjects aside for another day.

If you’ve disagreed with everything else I’ve said – please remember – I share your goal. I’m not saying the outcome you seek isn’t valuable – I am just proposing a different course of action. I hope I’ve done so respectfully and without denigrating anyone or their point of view.

Last Friday I created a FriendFeed Real-Time bridge to XMPP. Over the weekend I’ve had several people receiving their home feed in real time over XMPP (GTalk to be specific). I also wrote a post about what I had done and what I intend to do with this going forward.

One thing I heard from the initial testers was the desire to “make it work like Track”. Consequently – I’ve spent quite a few mental cycles this weekend thinking about Track (as deployed by Twitter) and how that might work for FriendFeed Streams.

There are three basic components of Track

1. The ability to track keywords (topics) and receive any tweets containing those keywords.
2. The ability to post a new tweet.
3. All interaction happens via IM (XMPP to be specific).

As I thought about these three basic components – and how they might be implemented for FriendFeed Streams I quickly realized that Twitter and FriendFeed are very different – and those differences have a huge impact on “Track” for FriendFeed Streams.

Let’s first consider the nature of Twitter.

Twitter is a stream of stateless tweets. Each tweet is an independent entity – bearing no relation to any other tweet in the timeline. Tweets only have relationships to the user that sent the tweet and, optionally, the user the tweet was directed at – via a @ reply. However, even those @ replies are simply stateless tweets – existing as an independent entity in the timeline.

It is that fundamental nature of Twitter that makes Track interesting – and to a certain extent possible. Since all tweets exist as stateless peer entities in a universal timeline – Twitter essentially creates a stateless stream of tweets. Track works by scanning and filtering this stream and injecting tweets into this stream.

Now let’s consider the nature of FriendFeed.

FriendFeed is a social media aggregator. It contains posts created organically (via the create post button in the FriendFeed interface) or via any number of aggregation channels (blogs, RSS Feeds, Twitter, Disqus, Flickr, etc). Each of these posts exist as state-full peer entities on the timeline.

FriendFeed posts are state-full because they exist as containers. They contain a separate groups of entities – Likes and Comments. Likes and Comments change change the state of the Post – and by doing so change the posts place in the timeline. For Example – If I comment on a 4 day old post it will move to the top of the timeline.

The Post as a container creates a conversation. This concept – the conversation – is totally foreign to Twitter.

FriendFeed, then, creates a state-full stream of conversations. Conversations appear in the stream one or more times – each time they are created or added to via a Like or Comment.

Now that we understand how Twitter and FriendFeed are similar and differ we can re-visit Track.

1. The ability to track keywords (topics) and receive any tweets containing those keywords.
2. The ability to post a new tweet.
3. All interaction happens via IM (XMPP to be specific).

What is clear is that component #1 above is still viable and makes sense. It was the fundamental reason I chose to build FFStream. It still solves the problem of information discovery.

What poses a significant challenge is #2. I refer to this as the problem of participation. This is due to the state-fullness of the FriendFeed Post (or conversation). Where FriendFeed is concerned participation is a relative exercise – you can post a new conversation OR you can participate in an existing conversation. It is the relative nature of participation (relative to an existing conversation) that makes participation via IM so problematic.

This schism is completely analogous to the schism between the command line and the graphical user interface. As a matter of fact, an IM/XMPP interface for Social Media streams is simply a Social Media Command Line. You enter commands – often obscure, opaque and archaic commands – and thereby control the behavior of the application. And this model works… until the interaction model becomes so complex that a command line interface becomes impractical.

When the goal is to filter a stateless stream of data (ala grep) and insert data into that stateless stream – a command line works. However, when the stream becomes state-full – and contains entity relationships (i.e., conversations) filtering the stream is sensible – but the complexity of the required command line interface to inject data into the right entity in the stream makes the proposition a fools errand.

In short – FFStream is an ideal information discovery mechanism – far more powerful than Track because the stream is an aggregation of content and because the stream is a stream of conversations. It is not, however, a usable interface (i.e., command line) for participation in Social Media.

So the goal of FFStream will be to enable information discovery and encourage participation by providing a reference to the conversation. That reference will open a browser window with an appropriate interface (i.e., graphical user interface) for the participation in the conversation.

I know some will find this (to put it mildly) lacking. I know some will declare that I don’t get it. But there is a reason people vastly prefer the ease of use provided by a graphical user interface over the command line. If you are one of those who believe I’ve missed the point – please consider the command line vs. graphical user interface schism and how it relates to this conversation before declaring my ignorance.

TechCrunch coverage of Phoenix Startup Weekend… Yeah – I was there working on Reserve Chute!!!

Twitrratr gets a nice bump… nice job guys.

Go Phoenix!

It’s been some time since we last covered Startup Weekend, a series of events that bring a roomful of developers and entrepreneurs together to develop new startups in only 54 hours. When the program originally launched last year, each weekend was geared towards building a single application, of which every participating member was a cofounder. Since then the format has changed – multiple companies are created at each event, and they don’t have to incorporate at the end of the weekend. Here’s a handful of the companies founded at last weekend’s event, which was held in Phoenix.

ReserveChute

With so much of our essential data making its way to the cloud, it’s becoming increasingly difficult to quickly make a local backup. Reserve Chute, an open source web app that will be available for use in the next few weeks, will automatically make backups of popular cloud based services including Gmail, Twitter, and Basecamp.

It’s unlikely that any of these services will be going out of business in the foreseeable future, and they almost certainly have redundant backup systems in place. But the prospect of having my entire photo collection or Email history wiped out is unnerving – having an easy way to back up these services would definitely give users some peace of mind, even if they never had to use it.

My Shelter Helper

My Shelter Helper is web page builder aimed to help animal shelters establish a presence on the web. Shelters can logon and after entering some basic information like their address and telephone number will be presented with a functional and good looking website. It’s a great idea, and I love the tagline: “Helping save animal’s lives.”

For now the service is generating some pretty barebones sites, but will introduce support for donations so that shelters can easily collect from benevolent animal lovers worldwide. I hope the team keeps working on this – anything that helps animals is a good thing, and plenty of people (like my mother, for example) would love to donate to their local animal shelter along with the national organizations.

Twittrratr

Awful domain name aside, the Twittrratr team has actually built a pretty cool Twitter site. After entering any keyword, Twittrratr will find related tweets and attempt to figure out if the subject is being spoke about in a positive or negative light. It’s a good idea but unfortunately it doesn’t work very well – oftentimes words that Twittrratr associates with a negative tweet aren’t being used to describe the keyword that was searched for. The team acknowledges that the system isn’t perfect and is open to suggestions (it’s still pretty impressive for 54 hours from conception to launch).

[From 54 Hours To Build A Company: A Look At Startup Weekend Phoenix]

I hope you had fun. I was at Phoenix Startup Weekend working with a team of smart, energized people to create a new company in a single weekend.

The result – Reserve Chute – is sheer genius.

This one time, at Phoenix Startup Weekend (http://phoenix.startupweekend.com) actually held in Chandler, AZ, a band of brothers, and one sister, came together to create a new company called Reserve Chute. With it’s main offering a personalized data backup service that permits users to grab their data from online sources like GMail and others and save them to a storage device of their choice. This is one cool and innovative service that should be on your “Must Have” list. Your data on your terms for when Web 2.0 becomes Web 2 point Oh no! What more could you ask for. For more information, visit www.reservechute.com and sign-up for this unique and powerful service offering.

If you use online applications to run your business you need to check Reserve Chute out. It might be the best thing you ever do for your business.

The following photos are courtesy of Adam Nollmeyer – Thanks Adam. You can find Adam’s work here.

UPDATE – The photos below are actually courtesy of ccl1111 – you can find his photos here. Adam’s photos are still great… be sure to check them out as well.