Just Landed: Processing, Twitter, MetaCarta & Hidden Data

Just Landed - Screenshot

I have a friend who has a Ph.D in bioinformatics. Over a beer last week, we ended up discussing the H1N1 flu virus, epidemic modeling, and countless other fascinating and somewhat scary things. She told me that epidemiologists have been experimenting with alternate methods of creating transmission models – specifically, she talked about a group that was using data from the Where’s George? project to build a computer model for tracking and predicting the spread of contagions (which I read about again in this NYTimes article two days later).

Just Landed - Screenshot

This got me thinking about the data that is hidden in various social network information streams – Facebook & Twitter updates in particular. People share a lot of information in their tweets – some of it shared intentionally, and some of it which could be uncovered with some rudimentary searching. I wondered if it would be possible to extract travel information from people’s public Twitter streams by searching for the term ‘Just landed in…’.

Just Landed - Screenshot

The idea is simple: Find tweets that contain this phrase, parse out the location they’d just landed in, along with the home location they list on their Twitter profile, and use this to map out travel in the Twittersphere (yes, I just used the phrase ‘Twittersphere’). Twitter’s search API gives us an easy way to get a list of tweets containing the phrase – I am working in Processing so I used Twitter4J to acquire the data from Twitter. The next question was a bit trickier – how would I extract location data from a list of tweets like this?:

Queen_Btch: just landed in London heading to the pub for a drink then im of to bed…so tired who knew hooking up on an airplane would be so tiring =S
jjvirgin: Just landed in Maui and I feel better already … Four days here then off to vegas
checrothers: Just landed in Dakar, Senegal… Another 9 hours n I’ll be in South Africa two entire days after I left … Doodles

It turned out to be a lot easier than I thought. MetaCarta offers 2 different APIs that can extract longitude & latitude information from a query. It can take the tweets above and extract locations:

London, London, United Kingdom – “Latitude” : 51.52, “Longitude” : -0.1
Maui, Hawaii, United States – “Latitude” : 20.5819, “Longitude” : -156.375
Dakar, Dakar, Senegal – “Latitude” : 14.72, “Longitude” : -17.48

This seemed perfect, so I signed up for an API key and set to work hooking the APIs up to Processing. This was a little bit tricky, since the APIs require authentication. After a bit of back and forth, I managed to track down the right libraries to implement Basic Authorization in Processing. I ended up writing a set of Classes to talk to MetaCarta – I’ll share these in a follow-up post later this week.

Now I had a way to take a series of tweets, and extract location data from them. I did the same thing with the location information from the Twitter user’s profile page – I could have gotten this via the Twitter API but it would cost one query per user, and Twitter limits requests to 100/hour, so I went the quick and dirty way and scraped this information from HTML. This gave me a pair of location points that could be placed on a map. This was reasonably easy with some assistance from the very informative map projection pages on Wolfram MathWorld.

I’ll admit it took some time to get the whole thing working the way that I wanted it to, but Processing is a perfect environment for this kind of project – bringing in data, implementing 3D, exporting to video – it’s all relatively easy. Here’s a render from the system, showing about 36 hours of Twitter-harvested travel:

Just Landed – 36 Hours from blprnt on Vimeo.

And another, earlier render showing just 4 hours but running a bit slower (I like this pace a lot better – but not the files size of the 36 hour video rendered at this speed!!)

Just Landed – Test Render (4 hrs) from blprnt on Vimeo.

Now, I realize this is a far stretch from a working model to predict epidemics. But, it sure does look cool. I also I think it will be a good base for some more interesting work. Of course, as always, I’d love to hear your feedback and suggestions.

136 thoughts on “Just Landed: Processing, Twitter, MetaCarta & Hidden Data

  1. Excellent post. Thank you. As one who comes from a Library Science background, it's encouraging to see the use of such "tools" as RDF outside of the traditional library environment. In fact, the use of controlled vocabularies such as the Web Ontology Language (OWL) is another example of how traditional information organization tools and techniques are being applied to new technological and information discovery challenges. However, all keep to the same concern of helping people find the right information at the right time, a concern that has concerned librarians for centuries. I will wait for next same post

  2. I am curious as to whether there would be a way to query tweets that are asking for a particular product or service and what that query might look like..?? It would be cool to be able to tweet them back with a link to what they are looking for!

  3. Man!You are a genius. I come from Kenya and internet penetration is only about 25%. But we are working on raising that figure. People are slowly warming up to twitter back here and local mobile developers are thinking of apps that can utilise the power of twitter for local apps and content generation.

    What you have done here is amazing.I would also like to know how you implemented this from a code perspective.

    Great work and thankyou for beaming out the light of knowledge.

  4. What you have done here is amazing. I come from Kenya and Internet penetration back here is around 25% but we are working on increasing that figure. People are slowly warming up to twitter and its potential more so local mobile application developers in generating mobile apps and content generation.This info could really come in handy.

    I would also like to know how you implemented this from a code perspective.

    Thank you for beaming the light of knowledge to us.

  5. Just Landed is the latest project by Jer Thorp, an artist from Vancouver, Canada. Built in Processing, Just Landed visualizes particular tweets containing the phrases ‘just landed in…’ or ‘just arrived in…’. in the form of a 3D geographical map of the world. As phrases are mentioned, the streams of communication represented.Data and text mining boffins like to crunch “big data.” The idea is that the more data one has, the less slop in the wonky “scores” that fancy math slaps on certain “objects.” Individuals think that his / her actions are unique. Not exactly. The more data one has about people, the easier it is to create some conceptual pig pens and push individuals in them. If you don’t know the name and address of the people, no matter. Once a pig pen has enough piggies in it (50 is a minimum I like to use as a lower boundary), I can push anonymous “users” into those pig pens. Once in a pig pen, the piggies do some predictable things. Since I am from farm country, piggies will move toward chow. You get the idea.

  6. To get more data, one could analyse the usage of google, with complete statistics such as ip-address and possibly set cookies, you could easily find out, where people are going if they were working off of their laptops..

    sim only blackberry

  7. Is this project intended to show how much people reveal on Twitter, track flights, or track disease?
    For the first, this is awesome.
    For the second, I would suggest scraping HTML from airlines' flight info.
    For the third, I suggest tracking each user, and trying to figure out where they are and have been and if they are sick.

  8. ive already seen it and have to say i think its really great. Like most of your work you always got the right ideas how to use different apis in the best possible way.

  9. It's very cool, i don't think usa is the center of the world, but in the neo european-american culture it is, i belive that the creators of this maravellous video are extremly influenced by that consumist-capitalist culture.
    Great Job!

  10. This is beautiful! A stunning idea and an equally stunning visualization.

    Landing in a new place is always more exciting than coming back to your own country, so I would expect people to give the former more mentions. In any case, one can see that longer distances (across the globe) are mentioned more than the shorter ones (within the country)

    What if the map was portrayed as a rotating globe and not a plane? Would it be as interesting?

    It would be great if you can monitor the movements a few days before important global events and create specific videos for the same. Like the Olympics, World Cup, the Copenhagen Summit, Commonwealth games, G8 summit, etc.

    Keep up the wonderful work. Looking forward to more.

    Regards,

    Arushi

  11. Jer – this is great stuff, I am a geographer at The Nature Conservancy and I'm looking for a compelling way to visualize the connectivity and migratory routes of birds. There's a fairly large citizen science database of observations (with location) available from ebird.org. Nonetheless if this sounds interesting to you I'd love to chat about it. Nice work!

  12. Really, really great stuff. I shared this with my entire staff. Thank you for taking the time to post. I for one, would love more insight into how specifically you generated this from a code perspective if you're willing to share. :)
    buckle

  13. Hi Jer, what a wonderful visualization!!!
    I have a technical question. How did you get the 3D like visualization? (the flat map with the arcs) I'm trying to use unfolding lib + peasy cam lib but i have some error placing the coordinates. Any recommendation? Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>