Big Data – Spike Volume

One aspect of Big Data is the arrival pattern of the data being stored/analyzed.

Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.

Here is what that looks like graphically:

Yankees Loss ALCS Tweets

Texas Rangers ALCS Win Tweets

Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.

Creativity Matters – The Decline of US Creativity

From an interview with Kyung Hee Kim

Decrease in Elaboration after 1984: Elaboration scores (ability to develop and elaborate upon ideas and detailed and reflective thinking and motivation to be creative) decreased more than other subscales of the TTCT. Elaboration scores decreased by 19.41% from 1984 to 1990, by 24.62% from 1984 to 1998, and by 36.80% from 1984 to 2008.

The interview is dense and contains a lot of information and statistics – but what jumped out at me was the section above. Elaboration is a critical skill in making things that solve people’s problems. Without elaboration entrepreneurship suffers.