Big Data – Spike Volume

One aspect of Big Data is the arrival pattern of the data being stored/analyzed.

Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.

Here is what that looks like graphically:

Yankees Loss ALCS Tweets

Texas Rangers ALCS Win Tweets

Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.

Creativity Matters – The Decline of US Creativity

From an interview with Kyung Hee Kim

Decrease in Elaboration after 1984: Elaboration scores (ability to develop and elaborate upon ideas and detailed and reflective thinking and motivation to be creative) decreased more than other subscales of the TTCT. Elaboration scores decreased by 19.41% from 1984 to 1990, by 24.62% from 1984 to 1998, and by 36.80% from 1984 to 2008.

The interview is dense and contains a lot of information and statistics – but what jumped out at me was the section above. Elaboration is a critical skill in making things that solve people’s problems. Without elaboration entrepreneurship suffers.

Following the Trend @dbacksbooth on Twitter

Time for another update on the trend line for @dbacksbooth on Twitter. Not much to say beyond the staggering growth in the number of replies to the booth per day. 

I factored out the early spikes in July and mapped a trend line from August 1st through yesterday. I think the results are pretty impressive especially factoring the DBacks record.

More importantly, It looks like the growth is beginning to “hockey stick”… I’m pretty sure the trend will be well above the 80 or so the trend line predicts. Let’s see what happens over the last few weeks of the season.