Project: NYTimes 365/360
Date: February, 2009
Language: Processing
Key Concepts: Data Visualization, NYTimes Article Search API, HashMaps & ArrayLists
Overview:
Many have you have already seen the series of visualizations that I created early in the year using the newly-released New York Times APIs. The most complex of these were in the 365/360 series in which I tried to distill an entire year of news stories into a single graphic. The resulting visualizations (2009 is picture above) capture the complex relationships – and somewhat tangled mess – that is a year in the news.
This release is a single sketch. I’ll be releasing the Article Search API Processing code as a library later in the week, but I wanted to show this project as it sits, with all of the code intact. The output from this sketch is a set of .PDFs which are suitable for print. Someday I’d like to show the entire series of these as a set of 6′ x 6′ prints – of course, someday I’d also like a solid-gold skateboard and a castle made of cheese.
That said, really nice, archival quality prints from this project (and the one I’ll be releasing tomorrow) are for sale in my Etsy shop. I realize that you’ll all be able to make your own prints now (and you are certainly welcome to do so) – but if you really enjoy the work and want to have a signed print to hang on your wall, you know who to talk to.
Getting Started:
Put the folder ‘NYT_365_360′ into your Processing sketch folder. Open Processing and open the sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (the NYT_365_360.pde file).
Thanks:
Most of the credit for this sketch goes to the clever kids at the NYT who made the amazing Article Search API. This is the gold standard of APIs, and really is a dream to use. As you’ll see if you dig into the code, each of these complicated graphics is made with just 21 calls to the API. I can’t imagine the amount of blood, sweat, and tears that would go into making a graphic like this the old-fashioned way.
Speaking of gold standards, Robert Hodgin got me pointed to ArrayLists in the first place, and has been helpful many times over the last few years as I’ve tried to solve a series of ridiculously simple problems in Processing. Thanks, Robert!
Download: NYT365.zip (140k)
This software is licensed under the CC-GNU GPL version 2.0 or later.
22 Comments
I think you forgot to add the link to the source code! Thanks for posting this! I'm trying to learn to program and seeing code really helps.
Fixed!
Thank you for great blog, it´s a great help!
The link works fine, but the folder behind it is empty.
Really? Works fine for me – try it again. If it doesn't work, send me an e-mail and I'll fire you the .ZIP file.
-Jer
Are you a professional journalist? You write very well.
Got it working! It was some strange Windows based problem with the zip-file.
Sorry that I bothered you.
I think I am having the same problem as Thomas with the zip file. It may or may not be due to the colon placed in the file name, which I think is generally not allowed in Windows. The unzipping utilities will not process anything with an invalid file name – Thomas how did you get around this?
Hi,
I've rebuilt the .ZIP file – hopefully it will work better, now!
-Jer
jake: I unzipped the file with 7-zip.
Hi, sorry to bother you, i downloaded your files, but when i open the patch, an error accord
in the;
String JSONStr = join(loadStrings(url), "");
so i entered:
String JSONStr = join(loadStrings(url), "http://query.nytimes.com/search/sitesearch?query=jobs&less=multimedia&more=date_all"
;
but it still says:
NullPointerException
can you please help?
hi, same problem with null pointer exception ? Did you find any solution
too much symmetry. not really
Ok, so I get the sketch running but hit the wall…
font = createFont("Meta-Normal", 100);
It says it cannot use that font with PDF Export. It is in the data folder. I try to change simply to Helvetica, and still it doesn't work.
Hmm.
If you open a new sketch and just try this:
PFont myFont = createFont(\”Helvetica\”, 100);
Does that work?
-Jer
same wall here with the font meta-normal cannot be used with pdf export?
hi , I get the same font error here as well, the following lines
Use PGraphicsPDF.listFonts() to get a list of available fonts.
processing.app.debug.RunnerException: RuntimeException: The font “Meta-Normal” cannot be used with PDF Export.
hi I get a similar font problem…..here is the error
many thanks
Use PGraphicsPDF.listFonts() to get a list of available fonts.
processing.app.debug.RunnerException: RuntimeException: The font “Meta-Normal” cannot be used with PDF Export.
I just tried using: font = createFont("Arial", 100)
Just wanted to post a comment so that others might know for future use. While poking around with this project I tried running the code and received notification from NYT that I've exceeded their queries per minute – will be throttling the requests in the code; let me know if you would like the update to the code with the throttling and I'll pass it along. Also I'm considering setting up a quick proxy for dev so I'm not always hitting their api. I can post a quick php file for that as well if anyone else is interested.
Thanks,
Heidi
Heidi, That would be great!
Help?
java.io.IOException: Server returned HTTP response code: 403 for URL: http://api.nytimes.com/svc/search/v1/article?quer…
at sun.net .www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at processing.core.PApplet.createInputRaw(PApplet.java:4532)
at processing.core.PApplet.createInput(PApplet.java:4500)
at processing.core.PApplet.loadStrings(PApplet.java:4785)
at NYT_365_360$TimesArticleSearch.doSearch(NYT_365_360.java:482)
at NYT_365_360.connectYear(NYT_365_360.java:142)
at NYT_365_360.setup(NYT_365_360.java:116)
at processing.core.PApplet.handleDraw(PApplet.java:1583)
at processing.core.PApplet.run(PApplet.java:1503)
at java.lang.Thread.run(Thread.java:680)
The file "http://api.nytimes.com/svc/search/v1/article?query=%20publication_year:1984%20desk_facet:Foreign+Desk&fields=+&facets=per_facet,org_facet&api-key=e46*****************************" is missing or inaccessible, make sure the URL is valid or that the file has been added to your sketch and is readable.
processing.app.debug.RunnerException: NullPointerException
at processing.app.Sketch.placeException(Sketch.java:1543)
at processing.app.debug.Runner.findException(Runner.java:583)
at processing.app.debug.Runner.reportException(Runner.java:558)
at processing.app.debug.Runner.exception(Runner.java:498)
at processing.app.debug.EventThread.exceptionEvent(EventThread.java:367)
at processing.app.debug.EventThread.handleEvent(EventThread.java:255)
at processing.app.debug.EventThread.run(EventThread.java:89)
Exception in thread "Animation Thread" java.lang.NullPointerException
at processing.core.PApplet.join(PApplet.java:5841)
at NYT_365_360$TimesArticleSearch.doSearch(NYT_365_360.java:482)
at NYT_365_360.connectYear(NYT_365_360.java:142)
at NYT_365_360.setup(NYT_365_360.java:116)
at processing.core.PApplet.handleDraw(PApplet.java:1583)
at processing.core.PApplet.run(PApplet.java:1503)
at java.lang.Thread.run(Thread.java:680)
I found a solution. If you go to processing tools > create font you see a list of fonts. Pick one of the adobe____ fonts and that should work?
4 Trackbacks
[...] to distill an entire year of news stories into a single graphic. The resulting visualizations (2009 is picture above) capture the complex relationships – and somewhat tangled mess – that is a year in the [...]
[...] Jer Thorp: NY Times visualization [...]
[...] her path to use Processing to visualise the agencies and functions in a spectacular array based on NYTimes 365/360 by Jer Thorp – again, click to download the full [...]
[...] One of a series of visualizations created using the New York Times APIs that plot the complex relationships between news stories over the course of a year. This graphic is for 2009. [...]