Crowd-Sourcing a Research Project

OK, Rowdies, I need some creative juice for a research project I’m working on.

Last week, I went to an interesting lecture by Michael Chorost, an author and speaker on science and technology.  Mr. Chorost has written two books, one detailing his experience with a cochlear implant and the other, just out, on the interaction between people and technology and how that exchange is becoming more intimate. The lecture was sponsored by the local PRSA chapter as the first of its John Beardsley lecture series.

One of the interesting ideas Mr. Chorost threw out was that Twitter is the beginning of a global nervous system, at least in the sense of conveying emotions.  When the Arab Spring took hold, for example, the emotions of those tweeting their experiences from Tahrir Square and elsewhere reverberated around the world.

Of particular interest to me, given my endless fascination with “bad things” was the notion that the first news of a sudden event – an earthquake, an explosion, a plane crash, etc. – will travel almost instantly around the world via Twitter not in the form of specific news – “a plane has just crashed” – but in the form of a more emotional “WTF was that?” sort of Tweet.  I’ve seen this anecdotally – the first news of the raid on Osama Bin Laden’s Abbottadad compound was a series of Tweets from a neighbor wondering what was causing all the noise in the middle of the night – but Mr. Chorost’s lecture got me wondering if it would be possible to somehow track such sentiments in realtime and to alert us when there’s a spike in such comments.  This might only give you a couple extra minutes heads up, but in some situations, a couple minutes is a huge advantage.

Thus was born over the last couple of days my first efforts at the “WTF Index.” All rights reserved.

The WTFI is trying to track first notice of incidents by scanning the Twitterverse in more or less realtime for the occurrence of certain terms that would be most likely to be Tweeted in the moments immediately following an adverse event. I’m looking less for specifics words than I am for expressions of surprise, fear, shock, etc.  My assumption is that in such situations, most people won’t immediately know the specifics but they will report “huge explosion” or “bright light in the sky” or simply “Whoa” or “WTF?”

There are a couple of challenges with this.

First is finding the right tool.  The volume of global Tweets per second is staggering – at peak times it can pass 8,000 (as it did for the globally significant event of…wait for it…Beyonce’s pregnancy) – which creates  issues of both bandwidth and processing.  The best place to do something like this would be from inside Twitter itself, but since they’re not likely to offer me a job anytime soon (“Hey, Biz…I’m @jmaustin just in case you’re looking) I have to make do with the tools at hand.  For me, that means Tweetdeck and a collection of search terms.

And that’s where I need your help.

So far I’m tracking:

  • Whoa
  • “what was that”
  • “what the hell”
  • uh-oh
  • “huge explosion”
  • “huge noise”
  • “light in the sky”

And, of course…

  • WTF
  • “what the fuck”

Needless to say, these terms produce a huge number of false positives in the sense that most posts that end up in the net are variations on “WTF…no peach yogurt AGAIN???”  There are also way too many hits per minute – between 25 and 125 in my observation so far – to scan each one. Accordingly, I’m just looking at the total number of Tweets that fit the search terms and using that number to look for moments when the Tweeting activity deviates sharply upward from the normal background noise levels.  I haven’t seen it yet (no global disasters since Thursday shockingly) but my expectation is that I’d see a spike when something happened.

So…I’d love the Crowd’s ideas on what to add to the list of search terms.  If something unexpected happened right now in your vicinity and your first instinct was to reach for your Twitter client, what would you Tweet?

Thanks!

– Austin

6 thoughts on “Crowd-Sourcing a Research Project

Comments are closed.