One aspect of Big Data is the arrival pattern of the data being stored/analyzed.
Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.
Here is what that looks like graphically:
Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.
From an interview with Kyung Hee Kim
Decrease in Elaboration after 1984: Elaboration scores (ability to develop and elaborate upon ideas and detailed and reflective thinking and motivation to be creative) decreased more than other subscales of the TTCT. Elaboration scores decreased by 19.41% from 1984 to 1990, by 24.62% from 1984 to 1998, and by 36.80% from 1984 to 2008.
The interview is dense and contains a lot of information and statistics – but what jumped out at me was the section above. Elaboration is a critical skill in making things that solve people’s problems. Without elaboration entrepreneurship suffers.
Time for another update on the trend line for @dbacksbooth on Twitter. Not much to say beyond the staggering growth in the number of replies to the booth per day.
Not sure what was going on, and there was no mention of it on the Twitter status blog, but it now appears that Twitter Search (and the search API) is back to full speed. There is no question that all day yesterday and and today up until 12:00 Pacific Twitter search was not indexing all Tweets.
I get hourly emails that show me (graphically) how much justSignal “collects” total from each of the services (Twitter, Blog Search, Backtype, etc). The thing about these graphs is they are amazingly consistent over time – that is why we use them… you can quickly see a change in the normal trend.