I have a friend who has a Ph.D in bioinformatics. Over a beer last week, we ended up discussing the H1N1 flu virus, epidemic modeling, and countless other fascinating and somewhat scary things. She told me that epidemiologists have been experimenting with alternate methods of creating transmission models – specifically, she talked about a group that was using data from the Where’s George? project to build a computer model for tracking and predicting the spread of contagions (which I read about again in this NYTimes article two days later).
This got me thinking about the data that is hidden in various social network information streams – Facebook & Twitter updates in particular. People share a lot of information in their tweets – some of it shared intentionally, and some of it which could be uncovered with some rudimentary searching. I wondered if it would be possible to extract travel information from people’s public Twitter streams by searching for the term ‘Just landed in…’.
The idea is simple: Find tweets that contain this phrase, parse out the location they’d just landed in, along with the home location they list on their Twitter profile, and use this to map out travel in the Twittersphere (yes, I just used the phrase ‘Twittersphere’). Twitter’s search API gives us an easy way to get a list of tweets containing the phrase – I am working in Processing so I used Twitter4J to acquire the data from Twitter. The next question was a bit trickier – how would I extract location data from a list of tweets like this?:
Queen_Btch: just landed in London heading to the pub for a drink then im of to bed…so tired who knew hooking up on an airplane would be so tiring =S
jjvirgin: Just landed in Maui and I feel better already … Four days here then off to vegas
checrothers: Just landed in Dakar, Senegal… Another 9 hours n I’ll be in South Africa two entire days after I left … Doodles
It turned out to be a lot easier than I thought. MetaCarta offers 2 different APIs that can extract longitude & latitude information from a query. It can take the tweets above and extract locations:
London, London, United Kingdom – “Latitude” : 51.52, “Longitude” : -0.1
Maui, Hawaii, United States – “Latitude” : 20.5819, “Longitude” : -156.375
Dakar, Dakar, Senegal – “Latitude” : 14.72, “Longitude” : -17.48
This seemed perfect, so I signed up for an API key and set to work hooking the APIs up to Processing. This was a little bit tricky, since the APIs require authentication. After a bit of back and forth, I managed to track down the right libraries to implement Basic Authorization in Processing. I ended up writing a set of Classes to talk to MetaCarta – I’ll share these in a follow-up post later this week.
Now I had a way to take a series of tweets, and extract location data from them. I did the same thing with the location information from the Twitter user’s profile page – I could have gotten this via the Twitter API but it would cost one query per user, and Twitter limits requests to 100/hour, so I went the quick and dirty way and scraped this information from HTML. This gave me a pair of location points that could be placed on a map. This was reasonably easy with some assistance from the very informative map projection pages on Wolfram MathWorld.
I’ll admit it took some time to get the whole thing working the way that I wanted it to, but Processing is a perfect environment for this kind of project – bringing in data, implementing 3D, exporting to video – it’s all relatively easy. Here’s a render from the system, showing about 36 hours of Twitter-harvested travel:
Just Landed – 36 Hours from blprnt on Vimeo.
And another, earlier render showing just 4 hours but running a bit slower (I like this pace a lot better – but not the files size of the 36 hour video rendered at this speed!!)
Just Landed – Test Render (4 hrs) from blprnt on Vimeo.
Now, I realize this is a far stretch from a working model to predict epidemics. But, it sure does look cool. I also I think it will be a good base for some more interesting work. Of course, as always, I’d love to hear your feedback and suggestions.
53 Comments
Jer, this is gorgeous!
I’d like to see it built into an App that allows live UI tag filtering to be entered, then dynamically updated. Could be great as an installation or performance tool.
Thanks, Jesse.
This actually renders totally fine in real time – but the data harvesting takes a long time, since it has to make a query to MetaCarta twice for every tweet. Ideally I’d like to move this process server-side, and have a PHP script that takes care of harvesting the data, etc. That way you’d be able to use the interface to explore historical data, etc.
-Jer
This is great. I love the way the arcs have that leading pulses. I could see this being useful for visualising not just transmission of flu but also other data that moves over time for example weather.
Interesting.
It looks like most of the arcs in the US go west to east. I’m wondering why this is. It’s possible you collected data during times when flights are mostly west to east, such as coast to coast redeyes? Although it’s 36 hours of data. Or maybe there’s a bug in your renderer? Also, it’s possibly a vast majority of twitter users are west coasters.
What about layovers, return trips, long trips, people in the military?
Nice visualisation!
You can get Lat/Long information from USGS:
http://geonames.usgs.gov
Barry.
I notice that all the arcs start in the west and move to the east. This would seem to imply an assumption in the programming that the target left its home location to arrive at ‘just landed in’. Also the direction of movement may be biased by Twitters user base being mostly North American. Too much inferred by this exercise to really tell much about the actual movement of the twitter targets.
As someone just noted on Slashdot, you’d better re-implement it as a globe with better distance calculation. Great thought it is, as it stands, the Pacific Ocean is either a no-fly zone, or the world really *is* flat!
Is great though. If only people would twitter more accurate information so (for instance) if someone’s flying from one location to another and neither is their home, it’ll still “work”.
Awesome work, nice to see Processing getting some press. Be sure to let the guys at Processing know about this, they’ll find it just as cool as I do.
Nice one!
Very clever! I’m a Vancouverite myself. Question – why is it that we see so many outgoing flights but I don’t think I recall seeing even a single flight heading IN to the U.S?
It’s possible that people are less likely to post something immediately after returning from a trip abroad as they simply want to get home and unpack and relax. To get those results it might be necessary to look for “just got back from…”
However, that would lead to *tons* of results not being international flights and instead be things like, “the movies” or “grandma’s house”. What it boils down to is that mining social networking sites for interpreted data in very dependent on the language that is used to describe different situations and circumstances and also in looking for the right phrases.
Finally, after over a decade of drinking beer together, we come up with something good. This is much, much better than candlelight croquet and the resulting lawn fire.
Thanks for all of the comments. I’ll respond in more detail when I get a chance – but suffice to say I appreciate all of your suggestions & questions.
Right now what you are seeing is ‘away’ pairs – where they are arriving somewhere that isn’t their home location. I’d like to add the ‘home’ pairs – people who are landing at home, but I’m not sure how I want to represent these graphically, yet.
A lot of people were quick to find the 2D weakness – cross-pacific flights are not shown properly. I like the suggestion that this might be due to a no-fly zone or a sea monster – but really it’s because I opted for the 2D map to avoid the typical spinning globe visualization cliché. I think the 2D map works well for a lot of reasons – I might eventually try a globe version but I don’t think so.
Two things to keep in mind here:
1. This is a sketch of a project that I don’t consider final in any real way
2. I am not intending to produce a real simulation of air travel – there are lots of better ways to do that. As I said int he post, I’m interested in exposing hidden information in Twitter feeds. I’m also interested in the possibilities (when implemented by much smarter people with a lot more time) of this kind of concept to real modeling.
In any case, I’m glad that people seem to be enjoying these early steps – and I welcome any and all comment and feedback.
-Jer
Tres cool. What a way to visualize data.
How about sound effects? I hear some sort of up-sliding tone or whoosh for a takeoff, and a downward one for landing (nothing for mid-flight, though); have the pitch / frequency be higher for shorter trips.
This is a really, really cool visualization. My mind is swirling with the possibilities.
Really, really great stuff. I shared this with my entire staff. Thank you for taking the time to post. I for one, would love more insight into how specifically you generated this from a code perspective if you’re willing to share.
All the best,
Jonathan
brilliant idea here
At first, i thought it was too much “traffic” (in both senses) coming from the US, but then i realized, it had to do with the language being parsed. The US would still lead in Twitter-using, but i think if you tried parsing different languages, the whole thing would spread a little more..
To get more data, one could analyse the usage of google, with complete statistics such as ip-address and possibly set cookies, you could easily find out, where people are going if they were working off of their laptops..
oh, and i second Jonathan Lambert. More detail about the code in question would be nice.
Really cool piece of work!
I just wonder what would happen if you were able to take data from travel sites like expedia.com and others as the input for your model.
Best Regards,
Victor
Another suggestion: Just /arrived/ in…
It’s pretty cool but it is too US centric as you would expect from twitter. I don’t think it would make a very accurate model for the spread of a virus.
Otstanding, Jer!!!
Not only cool idea, but also very nice visualisation.
Another issue to consider is the west-coaster who makes a stop on the way to Europe — your model as built would show an LA-NY flight, plus a LA-London flight. I suppose it would take a bit of work to correlate “just landed in” with prior entries from a user, rather than just his home location.
Hi Jer,
That is absolutely amazing. I’ve a proposal for you regarding working on a slightly different data source for this visualisation that would provide for a much more compelling real time visualisation. Please do feel free to drop me an email (its on the form, I assume you can see it) if you’d like to discuss.
cheers
David
You should consider storing user names and locations for two weeks or so in the final product. That way, in the case of jjvrigin or someone else who is traveling multiple places. You could have it plot from their last “just landed in location” rather than from their listed home location. Not entirely a necessity, but if you want to create a tool to predict epidemics accurately, you are definitely going to want to include that information. Plus, this would help out with the return home plot. The search for lets say, “just got home” or “its good to be home”, or something to that effect, could then have the name extracted and compared the user list. You then have the last location and could easily get the home location.
Either way, very cool project and by all definitions quite brilliant. Who could have thought, Twitter made useful? I never thought I’d see the day.
A lot of my friends and business associates just say “LAX to DFW” as their Facebook updates (indicating that they are flying to Dallas from LA). I wonder if you could dig out some good data points using that kind of tagging.
You assume that people have flown directly from their home location to their “just landed in” location. I think that’s a very big assumption that’s only true a fraction of the time.
What about people taking multiple flights throughout a vacation? I’ve travelled that way before. What about people who drove somewhere else first, then flew? Business travel?
This is really cool. Keep up the good work. It looks amazing and I really like the concept!
Jer,
This is phenomenal! From the side at the end it looks almost exactly like a strange attractor – I guess that fractal function is a matter of the distribution of airline travel consumption. Have you considered Buckminster Fuller’s Dymaxion projection? You’d see all travel moving up and down across a plane of nearly continuous landmass.
http://en.wikipedia.org/wiki/Dymaxion_map
Even given various smart questions about what’s being tracked and what it means, I think the visualizations are beautiful. They definitely got the creative, “what-if”? side of my brain working.
Exiting
Beautiful, curious , funny project.
I should be happy if you plan to produce a shareware or why not a toool able to vizualize the Internet traffic in 3D; I presume that many people should be interested in this product.
Good luck with your project.
Reminds me a bit of DEFCON actually.
Fantastic.
Have you considered doing something similar with “I am sick” and “I feel sick” – each entry could have a radius, and overlapping results could create increasing color gradients – similar to a storm on a weather radar.
Thanks again for all of the comments. A lot of other people have done great work visualizing network traffic, airline traffic, etc. In this project I was most interested in using data that people didn’t necessarily intend to share.
Once things calm down a bit (thanks /. !) I’ll sit down and build a version with some of the excellent suggestions that have come out of this thread and the Vimeo thread.
And yes, I do intend to release source for this project – who needs $$$ anyways??
PopeOnABomb – what you are talking about has already been done very very well by Jonathan Harris with http://wefeelfine.org
Ryan – Dymaxion would be a great idea – certainly cool to try. I’ll have to see if I can dig up the formulas.
Awesome stuff there, truly awesome.
As an epidemiologist at the CDC (who has colleagues currently dispatched to work on the Swine Flu, this is really valuable information. What an incredible dramatization too, I’ll pass it on.
Nice job. Hope in the future geolocation services such Latitude bring public anonymized data. All this information will increase the self-knowing of the humanity.
What a great idea!
I just posted on your vimeo page http://vimeo.com/4587178 about doing this kind of visualization for “http” searches in the twitter API to get recently mentioned Web pages and using the MetaCarta GeoTagger to plot the locations mentioned in those Web pages.
MetaCarta also has a search API that you can use to find recent news articles that mention places near any place — perhaps you could use that to add a “word cloud” to each landing site. “Just landed in… Wellington” would then generate “05/02/2009 06:58:00 A Wellington woman who tested positive for Influenza A (H1N1) after arriving in Auckland from Los Angeles on NZ1 on …”
Let me know if you want any help filtering with the APIs.
We love what you’re doing.
jrf
beautiful.
Processing source code?
It’s truly too pretty not to be released!
Amazing work Jer! Which were the two Metacarta APIs you used? And how were you able to get the black and white map? Keep up the great work!
Woww, you’re on Twitter’s Trending Topics, kudos
Indeed Gorgeous Twitter!! =)) Keep it up. Let us know when you release this awesome app
Hello Jer, this is beautiful!
If you want we could do a sound track for the movies, for free. They are lacking it.
You have my e-mail, just get in touc if you are interested.
Cheers,
Fernando
All I can say is awesome! I am thrilled with the direction I see discussed here. Maybe there is hope when people start thinking. Many great uses for this, keep up the excellent creative genius. This is brilliant!
As you can see from my last post I mucked up on my url…lol sorry about that. Thanks
Lovely idea with elegant solution. =)
Keep up with the good work!!
Beautiful
Why assume that they are travelling from their Twitter home location? I realize that you lack a choice, but that's gonna skew the results for multi-city travellers, and the “just landed in” crowd are likely pre-selected for that bunch of people.
Hi Bruce,
First – this very quick project was meant as a proof of concept and was never supposed to be an accurate model of any kind. There has been some interest from the epidemiology community, though, so perhaps it might become something more accurate.
Second, you're right. I think an ideal solution would track back into a user's tweets to try to guess what their itinerary may have been. I don't think that this would be extraordinarily difficult, but would certainly involve a fair number of hits to the Twitter API.
In any case, thanks for your comments.
-Jer
Amazing…I would like to know how integrate the API with Processing.
Very cool way to track this information. While the data could potentially be skewed a bit due to various possibilities, it's got good potential.
52 Trackbacks
[...] Just Landed extracts travel information from Tweets and maps the journeys on a map. The map itself remeains two dimensional but the “flights” are visualized in as three dimensional curves. A chronological order of the Tweets makes it possible to review a certain time period. [...]
[...] Link. Via http://waxy.org/links/ Tags: facebook, processing, programming, social networking, travel, twitter, visualization Posted by Shannon on May 11th, 2009 Filed in Animation, Nerdy, Science, Technology [...]
Twitter Comment
Just Landed: Processing, Twitter, MetaCarta & Hidden Data [link to post] from: @blprnt #cool
– Posted using Chat Catcher
Twitter Comment
I raise epidemiology point over beer last week, my smarty friend rolls with it, and its Slashdotted! @blprnt Just Landed: [link to post]
– Posted using Chat Catcher
[...] Jer Thorp has harnessed the internet’s oversharing tendencies for good, compiling 36 hours worth of [...]
[...] Just Landed: Processing, Twitter, MetaCarta & Hidden Data (blprnt) [...]
[...] Just Landed: Processing, Twitter, MetaCarta & Hidden Data | blprnt.blg (tags: twitter socialmedia blog visualization travel research maps processing) [...]
[...] Hay más detalles de todo el proceso en Just Landed: Processing, Twitter, MetaCarta & Hidden Data. [...]
[...] Jer Thorp has harnessed the internet’s oversharing tendencies for good, assembling 36 hours worth of [...]
[...] Just Landed is a great twitter visualization extracting travel information from Tweets and maps the journeys on a map. The application looks for tweets containing the phrases ‘just landed in…’ or ‘just arrived in…’. The home location for the traveling users are scraped from their Twitter pages. The system then plots these voyages over time. more info here via infostethics [...]
[...] Awesome. Find out more about his methodology and check out some animation here. [...]
[...] describes his work in detail on his blog. But all you really need do is click the videos below to watch tweeters in [...]
[...] describes his work in detail on his blog. But all you really need do is click the videos below to watch tweeters in [...]
[...] naheliegende Übung, wie sie z.B. rivva.de vollführt. Der kanadische Medienkünstler Jer Thorp hat nun Reisen von Twitter-Usern visualisiert – inspiriert von den Modellen, mit denen die Ausbreitung von Seuchen vorhergesagt [...]
[...] Just Landed: Processing, Twitter, MetaCarta & Hidden Data | blprnt.blg (tags: twitter geolocation) [...]
[...] Just Landed: Processing, Twitter, MetaCarta & Hidden Data | blprnt.blg Mapping where people are leaving and arriving based on nothing more than what they said on Twitter. Pretty, and perhaps the beginnings of something quite useful. (tags: data informatics twitter visualisation processing mapping socialmedia ) [...]
[...] interactive “map” has some real promise. Starting to play with Adobe Flex Admiring the “Just Landed” work from Jer Thorp, especially the process he took. Listening to Bat for Lashes Visiting the NGV for the John Brack [...]
[...] the “Just Landed” work from Jer Thorp, especially the process he [...]
[...] Just Landed: Processing, Twitter, MetaCarta & Hidden Data | blprnt.blg "Just Landed: Processing, Twitter, MetaCarta & Hidden Data" – blprnt.blg http://tr.im/lCpJ [from http://twitter.com/kenmat/statuses/1831572517 (tags: tweecious Twitter UnitedStates Hawaii TheNewYorkTimesCompany Applicationprogramminginterface Facebook&Twitter OntheWeb PHP) [...]
[...] 艺术家Jer Thorp 制作了一个漂亮的视频，向人们展示了Twitter 上36小时之内信息在世界范围内的传递。 [...]
[...] there’s almost no location based application I can’t build. Indeed sites such as Just Landed which searches Twitter for the text ‘just landed in’ and geocodes the places in order [...]
[...] comes almost a month after Vancouver-based digital artist blprnt created a video that visualises the flight journeys of thousands of Twitter users by scouring the [...]
[...] those of you interested, an artist in Vancouver has created animations showing the flight paths of Twitter users, using the search term “Just Landed [...]
[...] Jer Thorp describes on his blog the experiment of mapping Twitter arrivals status updates containing the phrase “just landed”. Jer writes: “The idea is simple: Find tweets that contain this phrase, parse out the location they’d just landed in, along with the home location they list on their Twitter profile, and use this to map out travel in the Twittersphere…” [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks out for the location they’d landed in, and the location they were sent from, and show all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks out for the location they’d landed in, and the location they were sent from, and show all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks out for the location they’d landed in, and the location they were sent from, and show all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] 6 gorgeous Twitter visualizations — There are a lot of really cool ways to visualize Twitter and all its tweets, but my favorite might just be Just Landed: Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks for the location they’ve landed in, and the location they were sent from, and shows all this on a 3D map of the world. For more information check out the author’s blog,blprnt.org. [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks for the location they’ve landed in, and the location they were sent from, and shows all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks for the location they’ve landed in, and the location they were sent from, and shows all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] landed in…” heeft hij inzichtelijk kunnen maken hoe mensen de wereld overvliegen. Lees hier meer over zijn [...]
[...] with my friend Jennifer Gardy, whose insights into epidemiology and data modeling led me to build Just Landed. Jennifer is currently working at the BC Centre for Disease Control where, among other things, [...]
[...] siguiente de la lista es Just Landed, que no deja de ser una prueba de concepto, pero hay que destacar la espectacularidad con que se [...]
[...] siguiente de la lista es Just Landed, que no deja de ser una prueba de concepto, pero hay que destacar la espectacularidad con que se [...]
[...] like a tourist in their new city. The other map that I thought was pretty cool to look at was the Just Landed Map. I thought the idea about taking social network information streams pretty crazy, as I was one of [...]
[...] is a very similar system to the one I used for Just Landed – the only real difference here is that the locations and travel paths are mapped onto a [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks for the location they’ve landed in, and the location they were sent from, and shows all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] Of course, all of this information can be useful for the good guys, too. With millions of people active on Twitter, the store of data – and what it can reveal – gets more and more interesting every day. We are already seeing scientists using web data to measure public happiness, but I think we have just scraped the surface of what could be uncovered. (To see a model of how Twitter updates could be used to track travel and disease spread, see my post on Just Landed). [...]
[...] after discovering Processing, I completely forgot about it. Yesterday, I saw this guy’s blog, and also a preview of what Google Chrome can do with [...]
[...] Just Landed is a beautiful geo-visualization of tweets containing the words “Just landed in…”. It finds the tweets containing the phrase, checks for the location they’ve landed in, and the location they were sent from, and shows all this on a 3D map of the world. For more information check out the author’s blog, blprnt.org. [...]
[...] anuncian los buenos días en diferentes idiomas, a lo largo del planeta. El padre de la idea es Jer Thorp, que ya había hecho anteriormente otro experimento similar con las palabras “Just [...]
[...] that displays over 11,000 tweets collected within a 24-hour period (similar to his 'just landed' one a while back). Different colours represent different times the phrase 'Good [...]
[...] It began as a quick idea that emerged out of the discussions following my post about Just Landed, in which several commenters asked to see a global version of that project. This would have been [...]
[...] sociais. Muitos dados úteis podem estar escondidos em meio a informação não estruturada. O Just Landed é um experimento de visualização de dados que utiliza o Twitter para visualizar o trafego de [...]
[...] artist and educator Jer Thorp used a smart mix of tools. Read more here. [ via infothestics [...]
[...] artist and educator Jer Thorp used a smart mix of tools. Read more here. [ via infothestics ] swfobject.embedSWF("http://www.vimeo.com/moogaloop.swf", "vvq-248-vimeo-1", [...]
[...] Just Landed представляет собой симпатичную гео-визуализацию твитов, содержащих фразу “Just landed in…”. Данный сервис ищет все записи с данными словами, определяет местоположение пишущего пользователя и отображает его на 3D карте мира. Чтобы узнать больше о сервисе Just Landed, вы можете прочитать блог разработчика – blprnt.org. [...]
[...] or twitter– using processing. A couple of his recent projects include NY Times 365/360 and Just Landed. There’s lots more speakers,too. You can see the full list here: [...]
[...] Pittsburgh’s conference for interactive designers, developers and artists returns for a second year after last year’s sold out event. When: October 15 & 16th 2009 Where: Sheraton Station Square, 300W Station Square Dr., Pittsburgh, PA 15219 Flashpitt is a conference for interactive designers, developers and artists. It is the only event of it’s kind in Pittsburgh and it’s running for a second year to follow up last year’s sold out event. Flashpitt brings international industry leaders right here to Pittsburgh to benefit the region’s interactive community. These renowned designers, developers, and authors will share inside tips, new technologies and industry insights with attendees. Highlights from this year’s speakers list include Seb Lee-Delisle and Jer Thorp. Seb is from Brighton, UK and will be speaking on his work, including: the Pyro for the People installation and a cutting edge 3D web site for the BBC. Jer, coming from Vancouver, BC, will speak about his art created with open data sources — like the New York Times database and twitter– using processing. His recent projects include NY Times 365/360 and Just Landed. [...]
[...] http://blog.blprnt.com/blog/blprnt/just-landed-processing-twitter-metacarta-hidden-data The idea is simple: Find tweets that contain this phrase, parse out the location they’d just landed in, along with the home location they list on their Twitter profile, and use this to map out travel in the Twittersphere Categories: Uncategorized Reacties (0) Trackbacks (0) Plaats een reactie Trackback [...]
[...] Just Landed by Jer Thorp [...]
[...] Fast schon gruselig – Reisenotizen von Twitter-Nutzern wurden dank API von Jer Thorp visualisiert. Mehr Info: Klick. [...]