We should have seen it coming. When we stopped even thinking about how we store data for our applications, when we just assumed some DBA would give us a database – and some SysAdmin would give us a file system. Sure, we can talk about W-SAN (what WLAN was to the LAN, but for storage) solutions like Amazon S3 and Rackspace Cloud – but they didn’t fundamentally change anything.

Big Data forces us to re-think storage completely. Not just structured/unstructured, relational/non-relational, ACID compliance or not. It forces us – at the application level – to rethink the current model exemplified by

I’m storing this because I may need it again in the future.

Where storage means physical, state aware object persistence and future means anywhere between now and the end of time.

Data Persistance – A Systemic Approach to Big Data for Applications

What Big Data applications require is a systemic approach to data. Instead of applications approaching data as only a set of if/then operations designed to determine what (if any) CRUD operations to perform it demands that applications (or supporting Data Persistence Layers) understand the nature of the data persistence required.

This is a level of complexity developers have been trained to ignore. The CRUD model itself explicitly excludes any dimensionally – or meta information about the persistence. It is all or nothing.

Data Persistance is primarily the idea that data isn’t just stored – it is stored for a specific purpose which is relevant within a specific time slice. These time slices are entirely analogous to those discussed in Preemption. Essentially, any sufficiently large real time Big Data system is simply a loosely aggregated computer system in which any data object may generate multiple tasks each of which have a specific priority.

For example, in a geo location game (Foursquare) the appearance of a new checkin requires multiple tasks which are prioritized based on their purpose, for example:

1. Store the checkin to distribute to “friends” (real-time)
2. Store the checking association with the venue (real-time)
3. Analyze nearby “friends” (real-time)
4. Determine any game mechanics, badges, awards, etc
5. Store the checkin on the user’s activity
6. Store the checkin object

NOTE: Many developers will look at this list above and ask: “Why not a database?” While a traditional database may suffice for a relatively low volume system (5k users, 20k checkins per day) it would not be sufficient at Big Data scale (as discussed here).

This Data Persistence solution is comprised of four vertical persistence types:

Transitory

Transitory persistance is for data persisted only long enough to perform some specific unit of work. Once the unit of work is completed the data is no longer required and can be expunged. For example: Notifying my friends (that want to be notified) that I’m at home.

Generally speaking (and this can vary widely by use case) Transitory persistence must be atomic, extremely fast and fault tolerant.

Volatile

Volatile persistance is for data that is useful but can be lost and rebuilt at any time. Object caching (how memcached is predominantly used) is one type Volatile persistence, but does not describe the entire domain. Other examples of volatile data include process orchestration data, data used to calculate decay for API Rate Limits, data arrival patterns (x/second over the last 30 seconds), etc.

The most important factor for Volatile data persistence is that the data can be rebuilt from normal operations or from long term data storage if it is not found in the Volatile dataset.

Generally speaking, data is stored in Volatile persistence because is offers superior performance, but limited dataset size.

ACID

Relational databases and atomicity, consistency, isolation and durability (ACID) are not obsolete. It is important for specific types of operations – done for specific purposes to maintain transactional compliance and ensure the entire transaction either succeeds in an atomic way, or fails. Examples of this include eCommerce transactions, Account Signup, Billing Information updates, etc.

Generally speaking, this data complies with the old rules of data. It is created/updated slowly over any given time slice, it is read periodically, there is little need to publish the information across a large group of subscribers, etc.

Amorphous

Amorphous persistence is the new kid on the block. NoSQL solutions fit nicely here. This non-volatile storage is amorphous in that the content (think property, not property value) of any object can change at any time. There is no schema, table structure or enforced relationship model. I think of this data persistence model as raw object storage, derived object storage and the transformed data that forms the basis of what Jeff Jones refers to as Counting Systems. Additionally, these systems store data in application consumable objects – with those objects being created on the way in.

Systems in this layer are generally highly scalable, fault tolerant, distributed systems with enhanced write efficiency. They offer the ability to perform the high volume writes required in real time Big Data systems without significant loss of read performance.

What Does All This Mean?

Most notably it means, that after years of obfuscating the underlying data storage from developers, we now need to re-engage application developers in the data storage conversation. No longer can a DBA define the most elegant data model based on the “I’m storing this because I may need it again in the future.” model and expect it to function in the context of a real time Big Data application.

We will hear a chorus of voices who will attack these dis-aggregated data persistence models based on complexity or the CAP Theorem or on the standard “the old way is the best way” defense of ACID and the RDBMS for everything. But all of this strikes me as a perfect illustration of what Henry Ford said:

If I had asked customers what they wanted, they would have told me they wanted a faster horse

I saw this on TechCrunch earlier today and thought it was an awesome add to my big data series. Jeff Jonas is clearly a big thinker and I agree with almost everything he says. The only thing I take issue with is the recurring theme in this interview that Big Data is primarily about commerce – and specifically ad targeting.

In my next post I’ll be talking about Big Data for One – which I think Jeff hints at but never fully develops.

Part 1: What is Data?

Part 2: Why data makes us more ignorant.

Data is actually evidence that you already knew, but failed to act on it. Amnesia.

Part 3: Why Big Data is the next big thing.

From pixels to pictures… This agrees with my idea of Big Data – it isn’t about the size of the dataset, but about using pieces of data in context by understanding context.

Why he goes to ad based is beyond me however…

Part 4: How data makes us average.

Very similar to the points I made here: Living In Public – Facebook, Privacy and Frictionless Distribution

Why the future is irresistible.

I don’t get it… I really don’t. When (and why) did silicon valley abandon the idea that technology – particularly the web based variety – had intrinsic value. The kind of transformative value that people would pay for…

Find me two business plans for an online consumer product that don’t have “and monazite via ad sales” somewhere in the business plan – go ahead, I dare you.

On the backs of Google and their transformation of advertising based on keywords and relevance we’ve become lazy. Why deliver real solutions to the problems of consumers when you can just aggregate eyeballs and information? Make something “cool”, something that will “go viral” and “generate buzzzz”. Get people to LOOK, not buy.

Think about it for a second, how different is Facebook from Yahoo circa 1998? Yahoo had tons of users. Yahoo knew a lot about their users based on what they made part of their “portal”. Yahoo sold ads.

I’m not talking about how advertising is evil, I’m not demonizing marketers. I’m asking why we’d have conversations like these: SEO vs. Usability

I’m also not suggesting that if you simply build a better mousetrap the world will beat a path to your door… I’m just suggesting that we should get back to trying to build better mousetraps – and by extension trying to change the world.

I’ve been thinking about writing a series of posts about Big Data for months now… which is entirely too much thinking an not enough doing, so here we go.

And, by Big Data we mean…

Wikipedia offers a very computer science oriented explanation of Big Data – but while the size of the dataset is a factor in Big Data there are several others worth considering:

• The arrival pattern (how data is created in the large dataset)
• Dimensions (how the data is grouped, sub-grouped and organized for analysis)
• Transformation (how the data is re-dimensioned or transformed to meet new analysis needs)
• Consumption (how will the data be consumed by client applications and ultimately humans)

Classically Big Data was regarded is extremely large datasets, usually acquired over a long period of time, involving historical data leveraged for predictive analysis of future events. In the simplest terms, we use hundreds of years of historical weather data to predict future weather. Big Data isn’t new, my first exposure to these concepts was in the early 1990′s dealing with Contact Center telecom and CTI (computer telephony integration) analytics and predictive analysis of future staffing needs in large footprint (3 to 5 thousand agent) contact centers.

Another example of the classic Big Data problem is the traditional large operational dataset. In the 90′s a 2 Terabyte dataset – like the one I worked with at Bank of America – was so massive that it created the need for a special computing solution. Today large operational datasets are measured in Petrabytes and are becoming fairly common. The primary challenges with these types of datasets is maintaining acceptable operational performance for the standard CRUD operations.

There are Big Data storage models emerging and maturing today with the two default options being hadoop – which relies on a distributed file system and distributed key value store systems such as Cassandra and CouchDB. These systems (referred to as NoSQL solutions) differ from standard RDBMS systems in two important ways:

• The underlying organization of data is amorphous and does not implement relationships.
• There is no support for Transactional Integrity (also know as ACID).

While all of these are interesting engineering problems, they still lack a crucial component. As a matter of fact, most conversations about Big Data fail to adequately address what is, perhaps, the most important problem with Big Data systems today.

The Intersection of Real Time and Big Data

Today’s big datasets are manageable in RDMBS systems. That being said, a significant amount of complexity is inserted in the managament process, most notably:

• Database Sharding and the complexity of managing multiple databases.
• Downtime associated with Schema changes or complexity of CLOB fields used in a amorphous manner.

Given that, large datasets that change slowly over time, or more accurately, those that have a relatively low volume of creates (including those that occur as a result of large transformations) as compared to read, update and delete can be managed using RDBM systems.

Where Real Time – specifically as related to Social Media, user generated content and other high create applications (such as the Large Hadron Collider) – intersect with Big Data is – to me – the most interesting Big Data topic.

This model presents three distinct challenges:

• High data arrival – potentially hundreds of thousands of new objects “found” per second.
• High transformation rate.
• Real Time Client updates.

The intersection of all three of these challenges was exactly the what we dealt with at justSignal. We needed to consistently collect hundreds of thousands of Social Media objects per second, generate hundreds of meta data elements (transformations) for each object, and make all of that data available in real time to our client applications. This view of Big Data is slightly different. In this view the size of the dataset – while still significant – isn’t as important as the challenges presented by a very high volume of CRUD operations over very short time slices.

The most important thing I’ve learned is that there is no silver bullet. While the traditional relational database isn’t effective in real time big data scenarios neither is a standalone Hadoop or Distributed Key Value Store. Essentially, you must evaluate each use case against the suite of technologies available and select the one best suited to that particular use case. Selecting a NoSQL solution for order processing – which has heavy ACID/Transactional requirements isn’t a good idea. However, staying with your RDBMS for high insert/transform data processing isn’t going to work either.

The approach we took at justSignal – which I will go into in more detail in a future post – was to create a unified data persistence layer designed to leverage the right long/short term data store (and sometimes more than one) based on the requirements of the application. This data persistance layer is made up of:

• MySQL
• memcached
• Cassandra
• Beanstalkd

Each plays a critical role in our ability to collect, transform, process and serve hundreds of thousands of social media mentions per minute.

More to follow…

Reposted from the justSignal Blog (http://justsignal.com)

Sometimes the best way to illustrate how justSignal can accelerate your strategy is by providing concrete examples of how others have accelerated theirs.

TweetsForBoobs is raising money for the Susan G. Komen foundation by encouraging folks to tweet the #tweetsforboobs hash tag on Twitter. It is the brain child of Chase Granberry and Josh Strebel – justSignal (and I for that matter) claim no credit for the idea or it’s successful execution.

In the interest of full disclosure, we did donate justSignal to TweetsForBoobs.

TweetsForBoobs needed four things in order to complete their vision.

1. A way to capture Tweets about the site and with the hashtag
2. The ability to put those Tweets on the site in real time.
3. A way to count how many times a Twitter User used the hashtag.
4. A way to measure the effectiveness of their efforts.

justSignal, because we focus on the complete Social Media Lifecycle, was uniquely suited to get them there – fast.

TweetsForBoobs was able to create a Signal that pulled in the content they were interested in. They were also able to – using our Exclusion Engine – remove spam and re-tweet bots from their Signal.

In order to create an engaging user experience on the site TweetsForBoobs dropped in and customized our real-time Twitter widget.   

Using our API service, TweetsForBoobs was able to pull in all mentions of the hashtag and the site in near real-time. This enabled them to count how many times each user tweeted the hash tag and update the site with current pledge totals.

Finally, TweetsForBoobs wanted to have some information (analysis) that gave them some indication of how the campaign was going. Since every Signal comes with our SignalReports (SignalMeter, SignalDensity and User Activity) they had basic who, when, and how much information.

That is execution of the complete Social Media Lifecycle enabled by justSignal.

1. Signal – Getting the #tweetsforboobs content.
2. Engagement – Using our Widgets to put the content on the site, and using the API to create site specific information.
3. Analytics – Using our SignalReports to gauge the effectiveness of the effort.

What makes TweetsForBoobs even more interesting is that they clearly show the benefit of our approach to data and analytics. When Chase wanted to understand the “Reach” of the hashtag he wasn’t confined by the data we provide “out of the box”.

The data is yours, when you have a question we don’t answer – because it is the really, really important kind, those specific to your business, company, product or service – you have access to the entire data set and the unfettered ability to mine out what is important to you.

You can read Chase’s excellent summary of the data and his thoughts on justSignal at the Authority Labs Blog.

What about action?

That is where you come in… head over to TweetsForBoobs or just grab your favorite Twitter client and send a tweet with #tweetsforboobs in it. Every time you do you pledge $1.00 to the Susan G. Komen foundation.

People are obsessed with making their email “broadcast”. Push email is a big deal… but sadly it means you have to lock yourself into a single email service from a single provider. You want near real-time push email… Blackberry or Exchange… or you can pretty much forget about it.

So, why can’t we have an open alternative?

There are a multitude of ways to get real time notifications of new emails, and there is no reason for that mechanism to be an integral part of the email solution itself. Given that I set out to make a push email solution for my Google Apps accounts that would effectively notify on my iPhone or any other device.

As it turns out the solution is actually very simple. I created a simple server process that checks my inbox every few seconds and sends me an Instant Message (IM) when I have a new email. I also incorporated sender and subject filters – so you can get notified only for the emails you want.

I could have made it send me a Twitter DM, or an SMS, or plugged it in to PubSubHubBub, or RSS Cloud…

The point is, the actual mechanism for the notification isn’t really all that important…

More after the jump…

How does it work?

EmailToIM is a PHP daemon that will run on any system that has PHP installed. It periodically checks for new messages on your email server (IMAP only) and sends you an IM if it finds one.

For notifications on my iPhone I use BeeJive IM with Apple’s push notifications turned on.

Where can I get it?

I’ve set up a Google Code project for it here.

It is open source (GPL v3).

If you have any questions let me know.

One of the problems with Twitter is that it encourages trite overly-simplistic statements of fact. Hey, here is one from Tony (@zappos) the CEO of Zappos:

Posted via email from briantroy’s posterous

There is a reason we value art and those who create it. I could write 3 or 4 hundred words about how technology and Social Media are making it impossible for despots to control the message… or I could just point you to the justSignal FreeIran site and get out of the way and let Peter Himmelman (who I’m proud to know) use his art to say just about everything that needs to be said.

I’ve gone through another one of those spells where I just didn’t write anything here – and to say I’ve been busy isn’t enough – hell I’m always busy.

I also can’t claim lack of interesting content – I could write six really compelling pieces on ASU Startup Weekend alone.

So we’ll just agree that my energy for writing posts took a bit of a vacation… but now I’m back.

Before I get back to my regular banality I need to get a bunch of housekeeping taken care of, so this post will catch you up on the justSignal updates, upgrades and new features.

Details after the jump…

Supported Services:

We’ve added Flickr support to our stable of Social Media services we collect from (Twitter, Google Blog Search and Backtype). I’m particularly excited about this addition because it allows you to add a visual element to your Signal. As always there is an accompanying widget which can be fully customized to meet the needs of your site.

Widget:

To get an idea of how all the Widgets can be deployed to create a compelling and interactive fan experience check out The UnOfficial Dbacks Tracker.

We plan to keep adding services as we go… so if you have any particular service you think would be particularly valuable drop me a note in the comments below.

Filtering:

Let’s be honest, the “filtering” provided by the services is ok… but not nearly as robust as it needs to be to really get Signal. More often than not you get a “rough tuning” which still contains considerable noise. I recognize that, and because of that we’ve added our Exclusion Engine.

The Exclusion Engine allows you to exclude items returned from the underlying services. In practical terms, you can create an overly broad filter for your Tracker, and then use the Exclusion Engine to “throw out” items that you don’t want.

For Example: You want to find information about Cobras (you know, snakes), but when you create a filter using the word “cobra” you get back everything – COBRA (health care), cobra (GI Joe), Shelby Cobra (car), Cobra Starship. To get down to just the Signal, you use the Exclusion Engine to define two sets of rules, Must Includes and Must NOT Includes.

Must Includes are words that tell you that you want this content – in our example, “snake”.

Must NOT Includes are words that tell you that you do not want this content – in our example, “ford, starship, GI Joe, health care, …”.

Each of these rules can contain a large number of keywords/phrases – allowing you to really fine tune your Signal.

Content Control:

Social Media gives people a megaphone – and they get to use that megaphone however they want. That does not mean you want to put everything they say on your site – after all, what you allow (and don’t) says a lot about you.

We’ve also all learned the lesson from Skittles and other brands who’ve hacked together some solution which puts social media on their site, only to see the usual school-yard pranks involving every offensive word possible.

We can do better – and we are. We’ve implemented a Dirty Word Filter which you can enable or disable at your discretion. This filter does not suppress the content, it simply replaces a list of offensive terms with (censor).

The content is saved in it’s raw (pun intended) form, we just allow you to not display offensive language. Here is a screen shot:

As always you can click through to see the original content on the service un-censored.

Twitter Widget

I kind of gave it away in the screenshot above, but we’ve also updated the Twitter Widget display photos from TwitPic and YFrog in the widget. User’s no longer have to click through to see the picture – it is displayed directly in your widget.

The Portal

We’ve also implemented (at long last) the Tracker Admin Portal. The portal contains all the tools you need to modify your filter, add or change Exclusion Engine terms, and enable or disable the Dirty Word Filter.

Additionally, it provides you control of each of the underlying services – which you can activate or deactivate.

This is probably my favorite upgrade we’ve done. It – along with our customization mechanisms – gives you complete control over your Tracker from day one. You can see your embed source, provide custom CSS for the iPhone page, and change your hash tag.

Signalytics

This will get a whole post to itself very soon. Signalytics.org will be the site for our Open Source project providing Analytics Tools based on the justSignal Tracker Data. As you know, I believe the data should be free (free as in liberty not beer). You pay us for the Tracker service – so you should be able to get the data we collect and do with it what you like.

Already have a Business Intelligence Vendor/Team? Great – take the data and find what interests you.

We realize we could charge you for custom reports, or Professional Services to extract what you want to know from the data we collect – but that isn’t the business I want to be in. There are hundreds of thousands of Analytics contractors, consultants, and vendors out there who are experts in Analytics. Use them.

I know some “Social Media Experts” will tell you that your analitics vendor has to “understand social media” (whatever that means) as if Social Media were somehow so special that you have to be one of US to derive any value from it. I’m calling bullshit. But that is the topic for the Signalytics post you’ll see shortly.

Until then…

More than any other word “innovator” describes me best. I’m not saying I’m Steve Jobs – I’m just saying I’m constantly fascinated by new, interesting combinations of ideas that add value. This morning – as I watched the Follow Friday Tweets FLY by on the justSignal tracker I began to wonder…

Wouldn’t it be great if I could tap into everything people said was “great” on Twitter. justSignal to the rescue! Here is the justSignal Tracker seeking out everything people Tweet about because it is great.

(I’m not embedding the Tracker here for any number of reasons… but it will live at the link above).

There are three things that make this exceptionally interesting (to me – feel free to suggest your own in the comments).

1. These are 100% authentic – no gaming of the system. Just real people telling us what they think is “great”.
2. Watching the Real-Time Tracker is a great (sorry for the pun) way to discover things that you might think are great too.
3. Being able to collect all these tweets for analysis later is an amazingly rich source of information (mostly because of #1 above)

Just as an example, here is an 100% authentic endorsement of a car repair shop. I have no idea who the Drake Garage is… but @ljpalmer is clearly a fan. I don’t know about you, but I totally trust this endorsement. Why? Because it is a spontaneous, authentic utterance…

I’ll have some more announcements about this experiment over the next few days and weeks. This is a really interesting idea – and further illustrates my point that Twitter Search does not make Twitter a Search Engine. If, however, you can collect the relevant data (for your purpose) you can gain amazing insights.