7 Days of Source Day #1: GoodMorning!

GoodMorning! Pinheads

Project: GoodMorning!
Date: August, 2009
Language: Processing
Key Concepts: Spherical coordinates, latitude & longitude conversion, Twitter API, MetaCarta API

Overview:

GoodMorning! is a global Twitter visualization tool. It allows tweets to be placed geographically and temporally, showing how a word or phrase is used around the world over a certain time period. You can watch a video here.

The project consists of two Processing sketches: a small sketch called TwitGather to gather tweets from the Twitter API and store them with location data in a JSON file, and a larger sketch which renders that data, and allows for export as .MOV and hi-res bitmap files. I’ve included a sample file which holds about 24 hours of ‘Good morning’ tweets.

This is likely the most complicated project that I will be releasing this week. I have tried to document the sketches as thoroughly as possible, but it may be confusing for beginners. However, there are lots of fairly simple things that can be gleaned by looking at various parts of the code.

To compile this project, you’ll need to make sure you have all of the libraries and other dependencies outlined below.

Getting Started:

Move the sketches into your Processing sketch folder. Open Processing and open the GoodMorning sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (the GoodMorning.pde file).

Thanks:

GoodMorning uses a variety of libraries, to the authors of which I am extremely grateful:

  1. Karsten Schmidt’s toxiclibs coreutils for storing coordinates and doing vector math
  2. Andreas Köberle and Christian Riekoff’s surfaceLib for rendering the planet and cloud layer
  3. Yusuke Yamamoto’s excellent Twitter4J library for talking to the Twitter API (this library is included in the download)
  4. Marius Watz’ TileSaver class, which lets you output huge bitmaps from any Processing sketch.

Download: GoodMorning.zip (10.0MB)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.

16 thoughts on “7 Days of Source Day #1: GoodMorning!

  1. Thanks for sharing Jer. Another way of doing the twitter calls would be to have a thread that polls the Twitter every x secs. You could then have either the same thread or probably another one that then makes the API calls and allows you to spread the calls to the none search API over the day.

    A good example of how to model an async polling thread in processing is http://www.shiffman.net/teaching/a2z/threads/

    This approach is great for when you need to have realtime app with background polling such as getting new tweets during the run of an application.

    1. Hi Ben,

      Yes – threading would make a certain amount of sense. But I don't think it would get around the issue of having to run a gathering program for about 24 hours. As far as I can see, Twitter will only give you the last 1500 results from any query. I haven't found a way to go back further than that.

      Thanks for the Shiffman link – as always, incredibly useful.

  2. Cool! A few months ago I wrote some code I called "Goodnight" to geographically and temporally locate tweets with the phrase "goodnight" (or a variation) in them. I used the data to figure out and graph when people went to bed in their local TZ. I never took the idea past my first working prototype, so it's cool to see you have done something similar and gone further!

  3. Yep. You'd be right wrt it running all day…..useful if you need to be doing processing/accepting user input while it runs. You could almost look at combining the two for a realtime version of good morning or other topic tracking.

  4. Jer- Thanks so much for sharing! I feel like I've learned most of what I know about Processing from your examples. I expect this week to be able to get to a whole new level!

  5. Hi Jer, thanks for releasing the code. I did run into one problem, though. The date parsing code in TwitterEngine.pde assumes that the user is in the US locale, and will give an exception if one runs the program as is. The following code forces the SimpleFormat instance to act as though it the locale really was US:

    SimpleDateFormat df = new SimpleDateFormat(
    "EEE MMM dd kk:mm:ss zzz yyyy",java.util.Locale.US);

    // Instead of:
    // SimpleDateFormat df = new SimpleDateFormat();
    // df.applyPattern("EEE MMM dd kk:mm:ss zzz yyyy");

  6. Hi, Thanks for sharing this!
    I try it out a little bit, but i can't get a lot of results cause of Twitters Rate Limit.
    "Rate limit exceeded. Clients may not make more than 150 requests per hour."

    That did you do? Did you use multiple accounts?

    Thanks
    Jörn

    1. Hi Jörn – rate limiting is always going to be a problem. At any given time, you can retrieve the 1500 latest results – but you can only make that request once an hour. That's why, in this case, I had to build a harvesting application that runs for 24 hours and gathers data. Once we have that data, we can use some pretty hacky HTML scraping to get the locations from users' twitter home pages and then use MetaCarta to get the lat/lon data.

  7. Hi blprnt ! First thanks for sharing. Really helpful in learning processing and reaching the next level. I am having some questions/problems though.
    1. does it work with a demo account on metacarta? or do you need to pay for a full account?
    2. in the twitGather sketch twitgather tab you state:

    ***IMPORTANT***
    You need to enter in your username and password on both the TwitterEngine and TwitterToken tabs!!!

    I cant find where to enter the username and password in the twitterToken tab?

    Problems:
    i have entered the twitter username and password in the twitterEngine tab and the metacarta credentials in the metacartaClient tab but when i run it i get some error messages.

    first it states:

    2010-09-27 19:51:17.189 java[711:c0b] *** Assertion failure in -[CocoaAppWindow _changeJustMain], /SourceCache/AppKit/AppKit-949.54/AppKit.subproj/NSWindow.m:8598
    2010-09-27 19:51:17.192 java[711:c0b] *** CPerformer: ignoring exception 'Invalid parameter not satisfying: [self canBecomeMainWindow]' raised during perform of selector 'requestFocus:' on target 'FocusManager' with args '<AWTComponentSetFocusData: 0x1c5dd0>

    And then:

    GETTING 0
    Couldn't connect: twitter4j.TwitterException: 401:Authentication credentials were missing or incorrect.
    <?xml version="1.0" encoding="UTF-8"?>
    <errors>
    <error code="53">Basic authentication is not supported</error>
    </errors>

    GETTING 1
    Couldn't connect: twitter4j.TwitterException: 401:Authentication credentials were missing or incorrect.
    <?xml version="1.0" encoding="UTF-8"?>
    <errors>
    <error code="53">Basic authentication is not supported</error>
    </errors>

    And so on until i stop the sketch.

    Do you understand where the problem is??
    I have to admit this is a bit over my level of skills so it is difficult for me to iron it out.

    best regards niklas

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>