7 Days of Source Day #6: NYTimes GraphMaker

NYTimes Drug Diptych

Project: NYTimes GraphMaker
Date: Fall, 2009
Language: Processing
Key Concepts: Data vizualization, graphing, NYTimes Article Search API

Overview:

The New York Times Article Search API gives us access to a mountain of data: more than 2.6 million indexed articles. There must be countless discoveries waiting to be made in this vast pile of information – we just need more people with shovels! With that in mind, I wanted to release a really simple example of using Processing to access word trend information from the Article Search API. Since I made this project in February, the clever folks at the NYT research lab have released an online tool to explore word trends, but I think it’s useful to have the Processing code released for those of us who want to poke around the data in a slightly deeper way. Indeed, I hope this sketch can act as a starting point for people to take some more involved forays into the dataset – it is ripe to be customized and changed and improved.

This is the simplest project I’m sharing in this now multi-week source release. It should be a nice starting point for those of you who have some programming experience but haven’t done too much in the way of data visualization. As always, if you have questions, feel free to send me an e-mail or post in the comments section below.

You can see a whole pile of radial and standard bar graphs that I made with this sketch earlier in the year in this Flickr set.

Getting Started:

You’ll need the toxiclibs core, which you can download here. Put the unzipped library into the ‘libraries’ folder in your sketchbook (if there isn’t one already, create one).

Put the folder ‘NYT_GraphMaker’ into your Processing sketch folder. Open Processing and open the sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (theNYT_GraphMaker.pde file).

Thanks:

It’s starting to get a bit repetitive, but once again this file depends on Karsten Schmidt’s toxiclibs. These libraries are so good they should ship with Processing.

Download: GraphMaker.zip(88k)


CC-GNU GPL

This software is licensed under theĀ CC-GNU GPL version 2.0 or later.

16 thoughts on “7 Days of Source Day #6: NYTimes GraphMaker”

  1. This looks awesome!
    Kind of a newbie question: I'm getting an error that the package "toxi" does not exist. I tried downloading toxi math and geom classes from google code and placing folders in my Processing > libraries folder — doesn't seem to help. Do I have the wrong toxi libraries? Am I putting it in the wrong place? (I'm using Processing 1.0.9).

        1. David, APIs are changing sometime and while I'm trying to keep these down to a minimum, they're unavoidable… Furthermore, these cases are not helped by people not familiar with opensource dev practice (i.e. always read the supplied changelog file, it's there for a reason: http://is.gd/d5r7c) but also Processing's lack of intelligent library handling or finding of missing classes (as a proper IDE would do)…

          But in this case the fix should be trivial: The package has simply been renamed into "toxi.geom.mesh" – so change that and you should be fine…

  2. Thanks for these Jer! Is there a way to connect the NYTimes GraphMaker & NYTimes: 365/360 apps to a set of offline data like you did with your BC Budget Visualizations? i.e. Excel .csv?

    1. Definitely possible. I've been meaning to implement some kind of a caching system for a while now.

      I think the most straightforward way of doing it would be to save out the individual JSON requests as you get them into a folder – then before each request you could see if the file exists before you do a new search.

      If I get some time over the weekend I will try this out.

      -Jer

      1. Nice. I am cleaning up a pretty cool dataset and I'd like to use your apps to visualize them. Feel free to drop me an email & let me know how its coming along.

  3. Thank you so much for sharing…as a Processing newbie (going through Shiffamn's book: as we speak) it's a great piece of code to play with. Unfortunately, being a newbie seems to have its cons as i'm unable to change the background color :(it's stays white no matter what color I input). I've tried the following:
    color backColor = #111111;
    tried to change background(backColor); to background(0); but same thing.

    What am I missing?

    1. background(red,green,blue);
      change red green and blue to a value between 0 and 255 thats your colour setting
      background(255) or background(0) gives you white or black

      make sure you read lots of shiffmans book before delving into blprnts code
      :-)

  4. It is very great what you do sharing your knowledge first of all, there is no problem with the Toxiclibs core but i get a NullPointerException at here >> "String JSONStr = join(loadStrings(url), "");", i want to know the math behind those arc works,

    thanks,
    metin

  5. When they originally introduced this API did they cap the number of hits you could make per second? I think I might be getting the same error as those above because I'm maxing out the 10 per second rule (the error happens at a different iteration each time). Could this be the case? I'm currently looking in to a timer method of populating the arrays as opposed to a for loop to see if this fixes the issue.
    Thanks for the great, informative posts about this API. Much appreciated!
    —Jenks

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>