Category: Uncategorized
Killer. The key is the Béchamel
One aspect of Big Data is the arrival pattern of the data being stored/analyzed.
Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.
Here is what that looks like graphically:
Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.
From an interview with Kyung Hee Kim
Decrease in Elaboration after 1984: Elaboration scores (ability to develop and elaborate upon ideas and detailed and reflective thinking and motivation to be creative) decreased more than other subscales of the TTCT. Elaboration scores decreased by 19.41% from 1984 to 1990, by 24.62% from 1984 to 1998, and by 36.80% from 1984 to 2008.
The interview is dense and contains a lot of information and statistics – but what jumped out at me was the section above. Elaboration is a critical skill in making things that solve people’s problems. Without elaboration entrepreneurship suffers.
Time for another update on the trend line for @dbacksbooth on Twitter. Not much to say beyond the staggering growth in the number of replies to the booth per day.
I factored out the early spikes in July and mapped a trend line from August 1st through yesterday. I think the results are pretty impressive especially factoring the DBacks record.
More importantly, It looks like the growth is beginning to “hockey stick”… I’m pretty sure the trend will be well above the 80 or so the trend line predicts. Let’s see what happens over the last few weeks of the season.
Not sure what was going on, and there was no mention of it on the Twitter status blog, but it now appears that Twitter Search (and the search API) is back to full speed. There is no question that all day yesterday and and today up until 12:00 Pacific Twitter search was not indexing all Tweets.
Unless of course all the Twitter users just took an 30 hour nap…
We will keep an eye on it and keep you updated here.
I get hourly emails that show me (graphically) how much justSignal “collects” total from each of the services (Twitter, Blog Search, Backtype, etc). The thing about these graphs is they are amazingly consistent over time – that is why we use them… you can quickly see a change in the normal trend.
Yesterday I noticed that Twitter’s graph looked funny and that the volume was way down from a typical day (on a typical day we collect ~150 thousand tweets). This morning that trend has continued:
A normal day is a nice smooth curve up from about 2 or 3 am through mid day… like this:
Also worth noting, our total volume of Tweets collected yesterday was the lowest it has been in 60 days. Normally I would chalk this up to normal variation, but the percentage change from Monday to Tuesday was 50% – and that is unprecedented day over day change. You should also note that http://search.twitter.com provides an exact tweet for tweet match (in our limited sampling) to what we are collecting via the API – so this appears to be a global Twitter Search issue.
This seems to be affecting Trackers with high volume traffic on Twitter – so many of you will not notice any change. If however, your tracker regularly pulls in 10,000 tweets per day it appears that your volume will be cut (roughly) in half.
Obviously all of this is anecdotal and I’ve received no confirmation from Twitter that there is any issue. We will stay on top of it and keep you updated.
NOTE: No, we are not being rate limited – we are, in fact, using less than 10% of our whitelisted request limit per hour. All requests to Twitter Search API are returning normal response codes and, in most cases, tweets – just not nearly as many as just 2 days ago.
Brian Roy 