Tag Archives: source code

7 Days of Source Day #6: NYTimes GraphMaker

NYTimes Drug Diptych

Project: NYTimes GraphMaker
Date: Fall, 2009
Language: Processing
Key Concepts: Data vizualization, graphing, NYTimes Article Search API

Overview:

The New York Times Article Search API gives us access to a mountain of data: more than 2.6 million indexed articles. There must be countless discoveries waiting to be made in this vast pile of information – we just need more people with shovels! With that in mind, I wanted to release a really simple example of using Processing to access word trend information from the Article Search API. Since I made this project in February, the clever folks at the NYT research lab have released an online tool to explore word trends, but I think it’s useful to have the Processing code released for those of us who want to poke around the data in a slightly deeper way. Indeed, I hope this sketch can act as a starting point for people to take some more involved forays into the dataset – it is ripe to be customized and changed and improved.

This is the simplest project I’m sharing in this now multi-week source release. It should be a nice starting point for those of you who have some programming experience but haven’t done too much in the way of data visualization. As always, if you have questions, feel free to send me an e-mail or post in the comments section below.

You can see a whole pile of radial and standard bar graphs that I made with this sketch earlier in the year in this Flickr set.

Getting Started:

You’ll need the toxiclibs core, which you can download here. Put the unzipped library into the ‘libraries’ folder in your sketchbook (if there isn’t one already, create one).

Put the folder ‘NYT_GraphMaker’ into your Processing sketch folder. Open Processing and open the sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (theNYT_GraphMaker.pde file).

Thanks:

It’s starting to get a bit repetitive, but once again this file depends on Karsten Schmidt’s toxiclibs. These libraries are so good they should ship with Processing.

Download: GraphMaker.zip(88k)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.

7 Days of Source Day #5: smart.rockets

Project: smart.rockets
Date: Summer, 2006
Language: ActionScript 2
Key Concepts: Evolutionary computing, genetic algorithms, rocket science

Overview:

For me, 2009 has undoubtedly been the Year of Data Visualization. Three years ago, I was engrossed in the somewhat different world of evolutionary computing. Having come from a background in genetics, I was very interested in exploring how evolutionary techniques could be applied in a programmatic fashion, particularly to the kinds of creative systems that I had been building over the previous years.

In preparation for a talk about my explorations at FlashBelt that year, I built a system in which a set of rockets evolved their way towards a target, navigating through obstacles by evolving their own individual firing patterns over time. It was a good demo to show how the general concept of evolutionary computing worked, and let me talk about mutation rates, hybridization algorithms, and all kinds of other interesting things.

As it turns out, the idea of combining EC and space travel is a very real idea. NASA uses evolutionary computing techniques to solve all kinds of problems, ranging from radio telescope scheduling to satellite antenna design to (you guessed it) construction or rocket firing patterns. In a future of un-manned space probes, the long-term goal will be to have machines that can evolve to adapt to unexpected conditions. Such a machine might use genetic algorithms to evaluate a huge number of possible strategies, and to choose the one that is most likely to be a success.

I’ll admit, that previous paragraph places this little Flash toy in undeservedly grand company. But smart.rockets does provide a simple example to get you thinking about how evolutionary computing works. This project was built in ActionScript 2 (opening the file left me with a strange mix of nostalgia and terror). It would be fairly trivial to convert this project to AS3 or to Processing – if anyone does end up doing this, let me know and I will post a link.

Getting Started:

You can see a working example of this file here: http://www.blprnt.com/smartrockets. You’ll need Flash or some kind of .SWF compiler to get the files working.

Thanks:

In 2006 I was working as a freelance Flash developer and designer. As you might suspect, there wasn’t a whole lot of demand for commercial projects which used evolutionary computation, or Lindenmayer systems, or Particle Swarm Optimization, or any number of other strange things that I was exploring in my spare time. I owe a great deal of thanks to the organizers of conferences like FlashBelt, who gave me a venue to talk about things like evolving rockets and in turn gave me an excuse to keep working in these interesting (and completely non-lucrative) areas.

Download: SmartRockets.zip (48k)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.

7 Days of Source Day #4 – BC Budget Visualization Tool

BC Budget Visualization Tool

Project: BC Budget Visualization Tool
Date: September, 2009
Language: Processing
Key Concepts: Data visualization, data organization, sticking it to the man

Overview:

More and more data is being released to the public every day. Big initiatives like the US data.gov and the UK’s upcoming data.hmg.gov.uk are resulting in a mountain of interesting data sets. These transparency initiatives are a step in the right direction, but we are quickly going to find ourself with a surfeit of data, and a very limited number of people with the skill set to do something with it.

One solution to this is to standardize the data so that generic tools can be built to dig into the data sets. This is a great idea – but it will take a lot of work, along with something that governments are not typically too good at: consensus.

Until that happens, tools like Processing offer another solution – make small, custom tools for individual data sets which can be built quickly and can be used specifically to work with the characteristics of a specific data set. Because Processing is fairly simple, journalists, researchers and activists can all be empowered to investigate data themselves, without having to rely on expensive or difficult to acquire resources.

This sketch is an example of how this might work. I wanted to investigate the recently announced staggering Arts & Culture cuts in my local government‘s budget, and built a simple tool to do that. All told, it took about 5 hours to gather the data, produce this tool and get the results out on the web – certainly a turnaround time that would be useful for media and for activists looking to be quick with their responses.

Getting Started:

Move the sketches into your Processing sketch folder. Open Processing and open the BCBudget sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (the BCBudget.pde file).

Thanks:

Again, this project uses Karsten Schmidt’s amazing and incredibly useful toxiclibs.

Download: BCBudget.zip (12k)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.

7 Days of Source Day #2: NYTimes 365/360

NYTimes: 365/360 - 2009 (in color)

Project: NYTimes 365/360
Date: February, 2009
Language: Processing
Key Concepts: Data Visualization, NYTimes Article Search API, HashMaps & ArrayLists

Overview:

Many have you have already seen the series of visualizations that I created early in the year using the newly-released New York Times APIs. The most complex of these were in the 365/360 series in which I tried to distill an entire year of news stories into a single graphic. The resulting visualizations (2009 is picture above) capture the complex relationships – and somewhat tangled mess – that is a year in the news.

This release is a single sketch. I’ll be releasing the Article Search API Processing code as a library later in the week, but I wanted to show this project as it sits, with all of the code intact. The output from this sketch is a set of .PDFs which are suitable for print. Someday I’d like to show the entire series of these as a set of 6′ x 6′ prints – of course, someday I’d also like a solid-gold skateboard and a castle made of cheese.

That said, really nice, archival quality prints from this project (and the one I’ll be releasing tomorrow) are for sale in my Etsy shop. I realize that you’ll all be able to make your own prints now (and you are certainly welcome to do so) – but if you really enjoy the work and want to have a signed print to hang on your wall, you know who to talk to.

Getting Started:

Put the folder ‘NYT_365_360′ into your Processing sketch folder. Open Processing and open the sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (the NYT_365_360.pde file).

Thanks:

Most of the credit for this sketch goes to the clever kids at the NYT who made the amazing Article Search API. This is the gold standard of APIs, and really is a dream to use. As you’ll see if you dig into the code, each of these complicated graphics is made with just 21 calls to the API. I can’t imagine the amount of blood, sweat, and tears that would go into making a graphic like this the old-fashioned way.

Speaking of gold standards, Robert Hodgin got me pointed to ArrayLists in the first place, and has been helpful many times over the last few years as I’ve tried to solve a series of ridiculously simple problems in Processing. Thanks, Robert!

Download: NYT365.zip (140k)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.

7 Days of Source Day #1: GoodMorning!

GoodMorning! Pinheads

Project: GoodMorning!
Date: August, 2009
Language: Processing
Key Concepts: Spherical coordinates, latitude & longitude conversion, Twitter API, MetaCarta API

Overview:

GoodMorning! is a global Twitter visualization tool. It allows tweets to be placed geographically and temporally, showing how a word or phrase is used around the world over a certain time period. You can watch a video here.

The project consists of two Processing sketches: a small sketch called TwitGather to gather tweets from the Twitter API and store them with location data in a JSON file, and a larger sketch which renders that data, and allows for export as .MOV and hi-res bitmap files. I’ve included a sample file which holds about 24 hours of ‘Good morning’ tweets.

This is likely the most complicated project that I will be releasing this week. I have tried to document the sketches as thoroughly as possible, but it may be confusing for beginners. However, there are lots of fairly simple things that can be gleaned by looking at various parts of the code.

To compile this project, you’ll need to make sure you have all of the libraries and other dependencies outlined below.

Getting Started:

Move the sketches into your Processing sketch folder. Open Processing and open the GoodMorning sketch from the File > Sketchbook menu. You’ll find detailed instructions in the header of the main tab (the GoodMorning.pde file).

Thanks:

GoodMorning uses a variety of libraries, to the authors of which I am extremely grateful:

  1. Karsten Schmidt’s toxiclibs coreutils for storing coordinates and doing vector math
  2. Andreas Köberle and Christian Riekoff’s surfaceLib for rendering the planet and cloud layer
  3. Yusuke Yamamoto’s excellent Twitter4J library for talking to the Twitter API (this library is included in the download)
  4. Marius Watz’ TileSaver class, which lets you output huge bitmaps from any Processing sketch.

Download: GoodMorning.zip (10.0MB)


CC-GNU GPL

This software is licensed under the CC-GNU GPL version 2.0 or later.