Project: NYTimes GraphMaker
Date: Fall, 2009
Language: Processing
Key Concepts: Data vizualization, graphing, NYTimes Article Search API
Overview:
The New York Times Article Search API gives us access to a mountain of data: more than 2.6 million indexed articles. There must be countless discoveries waiting to be made in this vast pile of information – we just need more people with shovels! With that in mind, I wanted to release a really simple example of using Processing to access word trend information from the Article Search API. Since I made this project in February, the clever folks at the NYT research lab have released an online tool to explore word trends, but I think it’s useful to have the Processing code released for those of us who want to poke around the data in a slightly deeper way. Indeed, I hope this sketch can act as a starting point for people to take some more involved forays into the dataset – it is ripe to be customized and changed and improved.
This is the simplest project I’m sharing in this now multi-week source release. It should be a nice starting point for those of you who have some programming experience but haven’t done too much in the way of data visualization. As always, if you have questions, feel free to send me an e-mail or post in the comments section below.
You can see a whole pile of radial and standard bar graphs that I made with this sketch earlier in the year in this Flickr set.
Getting Started:
You’ll need the toxiclibs core, which you can download here. Put the unzipped library into the ‘libraries’ folder in your sketchbook (if there isn’t one already, create one).
Put the folder ‘NYT_GraphMaker’ into your Processing sketch folder. Open Processing and open the sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (theNYT_GraphMaker.pde file).
Thanks:
It’s starting to get a bit repetitive, but once again this file depends on Karsten Schmidt’s toxiclibs. These libraries are so good they should ship with Processing.
Download: GraphMaker.zip(88k)
This software is licensed under the CC-GNU GPL version 2.0 or later.
15 Comments
This looks awesome!
Kind of a newbie question: I'm getting an error that the package "toxi" does not exist. I tried downloading toxi math and geom classes from google code and placing folders in my Processing > libraries folder — doesn't seem to help. Do I have the wrong toxi libraries? Am I putting it in the wrong place? (I'm using Processing 1.0.9).
Hi,
I am using the toxiclibs core, which you can download from here:
http://code.google.com/p/toxiclibs/
/>
Cheers,
-Jer
Thanks for these Jer! Is there a way to connect the NYTimes GraphMaker & NYTimes: 365/360 apps to a set of offline data like you did with your BC Budget Visualizations? i.e. Excel .csv?
Definitely possible. I've been meaning to implement some kind of a caching system for a while now.
I think the most straightforward way of doing it would be to save out the individual JSON requests as you get them into a folder – then before each request you could see if the file exists before you do a new search.
If I get some time over the weekend I will try this out.
-Jer
I downloaded from the same link you indicated here, but have the same problem with toxi math and geom classes =/
Nice. I am cleaning up a pretty cool dataset and I'd like to use your apps to visualize them. Feel free to drop me an email & let me know how its coming along.
Thank you so much for sharing…as a Processing newbie (going through Shiffamn's book: as we speak) it's a great piece of code to play with. Unfortunately, being a newbie seems to have its cons as i'm unable to change the background color
it's stays white no matter what color I input). I've tried the following:
color backColor = #111111;
tried to change background(backColor); to background(0); but same thing.
What am I missing?
background(red,green,blue);
change red green and blue to a value between 0 and 255 thats your colour setting
background(255) or background(0) gives you white or black
make sure you read lots of shiffmans book before delving into blprnts code
aprove
It is very great what you do sharing your knowledge first of all, there is no problem with the Toxiclibs core but i get a NullPointerException at here >> "String JSONStr = join(loadStrings(url), "");", i want to know the math behind those arc works,
thanks,
metin
Hi Jer,
I'm trying to get this to work using the toxiclibs core here:
http://hg.postspectacular.com/toxiclibs/downloads
Specifically, I'm loading the unzipped "toxiclibscore-0018.zip" into the Libraries folder and I still get this error:
"The package "toxi.geom.util" does not exist."
Any thoughts? Thanks in advance for any help.
David
David, APIs are changing sometime and while I'm trying to keep these down to a minimum, they're unavoidable… Furthermore, these cases are not helped by people not familiar with opensource dev practice (i.e. always read the supplied changelog file, it's there for a reason: http://is.gd/d5r7c) but also Processing's lack of intelligent library handling or finding of missing classes (as a proper IDE would do)…
But in this case the fix should be trivial: The package has simply been renamed into "toxi.geom.mesh" – so change that and you should be fine…
I have a same problem randomly, can you share how to fix this, if you did?
yes me too, I would love to make this work
When they originally introduced this API did they cap the number of hits you could make per second? I think I might be getting the same error as those above because I'm maxing out the 10 per second rule (the error happens at a different iteration each time). Could this be the case? I'm currently looking in to a timer method of populating the arrays as opposed to a for loop to see if this fixes the issue.
Thanks for the great, informative posts about this API. Much appreciated!
—Jenks