Oh, and Batman. And Wayne Gretzky.

You can watch the video below (it’s about 50 minutes). As always, questions, feedback and bizarre non-sequiturs are welcome.

These days my spare time is typically measured in seconds – so I took advantage of a rare free morning today to put together a very simple list of the some of the work that I have built since the beginning of 2009. You can find a link to this list at the top of the site – or you can just click here. As always, comments and suggestions are encouraged and appreciated.

Accessing information from the Twitter API with Processing is (reasonably) easy. A few people have sent me e-mails asking how it all works, so I thought I’d write a very quick tutorial to get everyone up on their feet.

We don’t need to know too much about how the Twitter API functions, because someone has put together a very useful Java library to do all of the dirty work for us. It’s called twitter4j, and you can download it here. We’ll be using this in the first step of the building section of this tutorial.

1. Authentication

Here’s the tricky part. In the olden days, Twitter used to be happy with us sending it a username and password pair for authentication. Now, all of the Twitter API functionality uses an oAuth system, which allows people to authenticate applications through a login at twitter.com (so that developers never see the user’s login information).

That’s pretty good from a security standpoint, but it makes our job a little bit harder. Now, instead of just needing a short username and password, we need FOUR things to authenticate our sketch: A consumer key & secret, and an access token & secret. You can get all 4 of these things from the twitter developer page.

1. Visit https://dev.twitter.com/ and login with your twitter username and password
2. Click on the ‘Create an app’ button at the bottom right
3. Fill out the form that follows – you can use temporary values (like “Jer’s awesome test application”) for the first three fields.
4. Once you’ve agreed to the developer terms, you’ll arrive at a page which shows the first two of the four things that we need: the Consumer key and the Consumer secret. Copy and paste these somewhere so you have them ready to access.

5. To get the other two values that we need, click on the button that says ‘Recreate my access token’. Copy and paste those two values (Access token and Access token secret) so that we have all four in the same place.

2. Building

1. Open up a new Processing sketch.
2. Import the twitter4j library. In your downloaded twitter4j files, you’ll find the core .jar in a directory called ‘lib’. You’re looking for a file called something like ‘twitter4j-core-2.2.5.jar’. We can add this to our sketch by simply dragging the twitter4j-core-2.2.5.jar file onto your sketch window. If you want to check, you should now see this file in your sketch folder, inside of a new ‘code’ directory.
3. The first thing we’ll do in our new file is to store the credentials that we gathered in the first part of this tutorial into a ConfigurationBuilder object (this is a built-in part of the twitter4j library):

** Don’t use the credentials below – these are fake ones that I made up – use the ones that you copy & pasted in steps 4 & 5 above

ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("lPFSpjBppo5u4KI5xEXaQ");
cb.setOAuthConsumerSecret("SYt3e4xxSHUL1gPfM9bxQIq6Jf34Hln9T1q9KGCPs");
cb.setOAuthAccessToken("17049577-Yyo3AEVsqZZopPTr055TFdySop228pKKAZGbJDtnV");
cb.setOAuthAccessTokenSecret("6ZjJBebElMBiOOeyVeh8GFLsROtXXtKktXALxAT0I");

4. Now we’ll make the main Twitter object that we can use to do pretty much anything you can do on the twitter website – get status updates, run search queries, find follower information, etc. This Twitter object gets built by something called the TwitterFactory, which needs our configuration information that we set above:

ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("lPFSpjBppo5u4KI5xEXaQ");
cb.setOAuthConsumerSecret("SYt3e4xxSHUL1gPfM9bxQIq6Jf34Hln9T1q9KGCPs");
cb.setOAuthAccessToken("17049577-Yyo3AEVsqZZopPTr055TFdySop228pKKAZGbJDtnV");
cb.setOAuthAccessTokenSecret("6ZjJBebElMBiOOeyVeh8GFLsROtXXtKktXALxAT0I");

Twitter twitter = new TwitterFactory(cb.build()).getInstance();

5. Now that we have a Twitter object, we want to build a query to search via the Twitter API for a specific term or phrase. This is code that will not always work – sometimes the Twitter API might be down, or our search might not return any results, or we might not be connected to the internet. The Twitter object in twitter4j handles those types of conditions by throwing back an exception to us; we need to have a try/catch structure ready to deal with that if it happens:

ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("lPFSpjBppo5u4KI5xEXaQ");
cb.setOAuthConsumerSecret("SYt3e4xxSHUL1gPfM9bxQIq6Jf34Hln9T1q9KGCPs");
cb.setOAuthAccessToken("17049577-Yyo3AEVsqZZopPTr055TFdySop228pKKAZGbJDtnV");
cb.setOAuthAccessTokenSecret("6ZjJBebElMBiOOeyVeh8GFLsROtXXtKktXALxAT0I");

Twitter twitter = new TwitterFactory(cb.build()).getInstance();
Query query = new Query("#OWS");

try {
  QueryResult result = twitter.search(query);
}
catch (TwitterException te) {
  println("Couldn't connect: " + te);
};

5. This code is working – but we haven’t done anything with the results. Here, we’ll set the results per page parameter for the query to 100 to get the last 100 results with the term ‘#OWS’ and spit those results into the output panel:

ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("lPFSpjBppo5u4KI5xEXaQ");
cb.setOAuthConsumerSecret("SYt3e4xxSHUL1gPfM9bxQIq6Jf34Hln9T1q9KGCPs");
cb.setOAuthAccessToken("17049577-Yyo3AEVsqZZopPTr055TFdySop228pKKAZGbJDtnV");
cb.setOAuthAccessTokenSecret("6ZjJBebElMBiOOeyVeh8GFLsROtXXtKktXALxAT0I");

Twitter twitter = new TwitterFactory(cb.build()).getInstance();
Query query = new Query("#OWS");
query.setRpp(100);

try {
  QueryResult result = twitter.search(query);
  ArrayList tweets = (ArrayList) result.getTweets();

  for (int i = 0; i < tweets.size(); i++) {
    Tweet t = (Tweet) tweets.get(i);
    String user = t.getFromUser();
    String msg = t.getText();
    Date d = t.getCreatedAt();
    println("Tweet by " + user + " at " + d + ": " + msg);
  };
}
catch (TwitterException te) {
  println("Couldn't connect: " + te);
};

It seems stupid to come all of this way without doing something visual, so let’s throw all of the words that we see in the tweets onto the screen, quick and dirty, at random positions. The following code takes our existing instructions and puts them into the standard Processing setup/draw enclosures so that we can do some animation over time:

//Build an ArrayList to hold all of the words that we get from the imported tweets
ArrayList<String> words = new ArrayList();

void setup() {
  //Set the size of the stage, and the background to black.
  size(550,550);
  background(0);
  smooth();

  //Credentials
  ConfigurationBuilder cb = new ConfigurationBuilder();
  cb.setOAuthConsumerKey("lPFSpjBppo5u4KI5xEXaQ");
  cb.setOAuthConsumerSecret("SYt3e4xxSHUL1gPfM9bxQIq6Jf34Hln9T1q9KGCPs");
  cb.setOAuthAccessToken("17049577-Yyo3AEVsqZZopPTr055TFdySop228pKKAZGbJDtnV");
  cb.setOAuthAccessTokenSecret("6ZjJBebElMBiOOeyVeh8GFLsROtXXtKktXALxAT0I");

  //Make the twitter object and prepare the query
  Twitter twitter = new TwitterFactory(cb.build()).getInstance();
  Query query = new Query("#OWS");
  query.setRpp(100);

  //Try making the query request.
  try {
    QueryResult result = twitter.search(query);
    ArrayList tweets = (ArrayList) result.getTweets();

    for (int i = 0; i < tweets.size(); i++) {
      Tweet t = (Tweet) tweets.get(i);
      String user = t.getFromUser();
      String msg = t.getText();
      Date d = t.getCreatedAt();
      println("Tweet by " + user + " at " + d + ": " + msg);

      //Break the tweet into words
      String[] input = msg.split(" ");
      for (int j = 0;  j < input.length; j++) {
       //Put each word into the words ArrayList
       words.add(input[j]);
      }
    };
  }
  catch (TwitterException te) {
    println("Couldn't connect: " + te);
  };
}

void draw() {
  //Draw a faint black rectangle over what is currently on the stage so it fades over time.
  fill(0,1);
  rect(0,0,width,height);

  //Draw a word from the list of words that we've built
  int i = (frameCount % words.size());
  String word = words.get(i);

  //Put it somewhere random on the stage, with a random size and colour
  fill(255,random(50,150));
  textSize(random(10,30));
  text(word, random(width), random(height));
}

** Extra-coolness: see this example working in the browser with Processing.js!!

That’s it! This example is very simple, but it’s the bare bones of what you need build a project which connects Twitter through Processing.

Shortly I’ll add another post which shows how to use Processing with the Twitter Streaming API, so that we can deal with large volumes of tweets over time.

Good luck!

Near the end of this summer, I was asked by the publishers of Popular Science magazine to produce a visualization piece that explored the archive of their publication. PopSci has a history that spans almost 140 years, so I knew there would be plenty of material to draw from. Working with Mark Hansen, I ended up making a graphic that showed how different technical and cultural terms have come in and out of use in the magazine since its inception.

The graphic is anchored by a kind of molecular chain – decade clusters in turn contain year clusters. Every atom in these year clusters is a single issue of the magazine, and is shaded with colours extracted from the issue covers via a colour clustering routine. The size of the issue-atoms is determined by the number of words in each issue.

Surrounding this chain are about 70 word frequency histograms showing the issue-by-issue usage of different terms (like ‘software’ or ‘bakelite’). I used a simple space-filling algorithm to place these neatly around the molecule chain, and to stack them so that one histogram begins shortly after another ends. This ended up resulting in some interesting word chains that show how technology has progressed – some that make sense (microcomputer to e-mail) and some what are more whimsical (supernatural to periscope to datsun to fax).

Picking out interesting words from all of the available choices (pretty much the entire dictionary) was a tricky part of the process. I built a custom tool in Processing that pre-visualized the frequency plots of each word so that I could go through many, many possibilities and identify the ones that would be interesting to include in the final graphic. This is a really common approach for me to take – building small tools during the process of a project that help me solve specific problems. For this visualization, I actually ended up writing 4 tools in Processing – only one of which contributed visually to the final result.

My working process is riddled with dead-ends, messy errors and bad decisions – the ‘final’ product usually sits on top of a mountain of iterations that rarely see the light of day. To give a bit of insight into the steps between concept and product, I’ve put together a Flickr set showing 134 process images that came out of the development of this visualization. Here are a few images from that set:

Rough beginning

Early molecular chain

Denser chain with test ‘word span’

A diversion

Near-final

Lost in the image records are the steps that involved the data – and there were a lot of them. The archive was text that came from an OCR (optical character recognition) process, and was incredibly messy. To make matters worse, the file names for each issue were machine-generated and didn’t tie to the actual date order of the documents. A great deal of our time was spent cleaning up this data, and compiling customized datasets (many of which never ended up getting used).

While the final graphic is pictured above, it looks much better in print – so make sure you get a copy! Better yet, you can order a poster-sized version of the graphic by clicking here.

[The above graphic is an interactive 3D bar graph. If you can't see it, it's probably because your browser sucks can't render WebGL content. Maybe try Chrome or Firefox?]

Ben Fry and Casey Reas, the initiators of the Processing project, announced at the Eyeo Festival that a new version of Processing, Processing 2.0, will be released in the fall (VIDEO). This 2.0 release brings a lot of really exciting changes, including vastly improved 3D performance, and a brand-new high-performance video library. Perhaps most interesting is the addition of publishing modes – Processing sketches can now be published to other formats other than the standard Java applet. The mode that is going to cause the most buzz around the interweb is Javascript mode: yes, Processing sketches can now be published, with one click, to render in pretty much any web browser. This is very exciting news for those of us who use Processing for data visualization – it means that our sketches can reach a much broader audience, and that the visual power of Processing can be combined with the interactive power of Javascript.

While preview releases of Processing 2.0 will be available for download very soon, you don’t have to wait to experiment with Processing and Javascript. The upcoming JS mode comes thanks to the team behind Processing.js, and we can use it to build sketches for the browser right now.

To get started, you can download this sample file. When you unzip the file, you’ll see a directory with three files – the index.html page, processing.js, and our sketch file, called tutorial.pde. If you open the sketch file in a text editor, you’ll see something like this:

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);
}

void draw() {
  //instructions in here happen over and over again
}

If you’re a Processing user, you’ll be familiar with the setup and draw enclosures. If you’re not, the notes inside of each one explain what’s going on: when we run the sketch, the size is set to 500px by 500px, and the background is set to black. Nothing is happening (yet) in the draw enclosure.

You can see this incredibly boring sketch run if you open the index.html file in your browser (this will work in Firefox and Safari, but not in Chrome due to security settings).

Processing.js is a fully-featured port of Processing – all of the things that you’d expect to be able to do in Processing without importing extra libraries can be done in Processing.js. Which, quite frankly, is amazing! This means that many of your existing sketches will run, with no changes, in the browser.

Since this is a data visualization tutorial, let’s look at a simple representation of numbers. To keep it simple, we’ll start with a set of numbers: 13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9 35.1, 34.3. (These numbers are obesity percentages in the U.S. population, between the 1960s and the present – but we’re really going to use them without context).

Typically, if we were to put these numbers into a Processing sketch, we’d construct an array of floating point numbers (numbers with a decimal place), which would look like this:

float[] myNumbers = {13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3};

However, we’re not in Java-land anymore, so we can make use of Javascript’s easier syntax (one of the truly awesome things about Processing.js is that we can include JS in our sketches, inline):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

Our .pde file, then, looks like this:

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);
}

void draw() {
  //instructions in here happen over and over again
}

Now that we have our helpful list of numbers (called, cleverly, ‘numbers’), we can access any of the numbers using its index. Indexes in Processing (and Javascript) are zero-based, so to get the first number out of the list, we’d use numbers[0]. To get the fourth one, we’d use numbers[3]. Let’s use those two numbers to start doing some very (very) simple visualization. Indeed, the first thing we’ll do is to draw a couple of lines:

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  //Draw two lines
  stroke(255);
  line(0, 50, numbers[0], 50);
  line(0, 100, numbers[4], 100);
}

void draw() {
  //instructions in here happen over and over again
}

OK. I’ll admit. That’s a pretty crappy sketch. But, we can build on it. Let’s start by making those lines into rectangles:

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  //Draw two lines
  stroke(255);
  rect(0, 50, numbers[0], 20);
  rect(0, 100, numbers[4], 20);
}

void draw() {
  //instructions in here happen over and over again
}

At this point, I’m going to introduce my very best Processing friend, the map() function. What the map() function allows us to do is to adjust a number that exists in a certain range so that it fits into a new range. A good real-world example of this would be converting a number (0.5) that exists in a range between 0 and 1 into a percentage:

50 = map(0.5, 0, 1, 0, 100);

We can take any number inside any range, and map it to a new range. Here are a few more easy examples:

500 = map(5, 0, 10, 0, 1000);
0.25 = map(4, 0, 16, 0, 1);
PI = map(180, 0, 360, 0, 2 * PI);

In our sketch, the list of numbers that we’re using ranges from 13.4 to 34.3. This means that our rectangles are pretty short, sitting in the 500 pixel-wide sketch. So, let’s map those numbers to fit onto the width of our screen (minus 50 pixels as for a buffer):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  //Draw two rectangles
  rect(0, 50, map(numbers[0], 0, 34.3, 0, width - 50), 20);
  rect(0, 100, map(numbers[4], 0, 34.3, 0, width - 50),  20);
}

void draw() {
  //instructions in here happen over and over again
}

Two bars! Wee-hoo! Now, let’s use a simple loop to get all of our numbers on the screen (this time we’ll use the max() function to find the biggest number in our list):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  for (var i = 0; i < numbers.length; i++) {
    //calculate the width of the bar by mapping the number to the width of the screen minus 50 pixels
    var w = map(numbers[i], 0, max(numbers), 0, width - 50);
    rect(0, i * 25, w, 20);
  }
}

void draw() {
  //instructions in here happen over and over again
}

We’re still firmly in Microsoft Excel territory, but we can use some of Processing’s excellent visual programming features to make this graph more interesting. For starters, let’s change the colours of the bars. Using the map() function again, we’ll colour the bars from red (for the smallest number) to yellow (for the biggest number):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  for (var i = 0; i < numbers.length; i++) {
    //calculate the amount of green in the colour by mapping the number to 255 (255 red &amp; 255 green = yellow)
    var c = map(numbers[i], min(numbers), max(numbers), 0, 255);
    fill(255, c, 0);
    //calculate the width of the bar by mapping the number to the width of the screen minus 50 pixels
    var w = map(numbers[i], 0, max(numbers), 0, width - 50);
  	rect(0, i * 25, w, 20);
  }
}

void draw() {
  //instructions in here happen over and over again
}

This process of mapping a number to be represented by a certain dimension (in this case, the width of the bars along with the colour) is the basic premise behind data visualization in Processing. Using this simple framework, we can map to a whole variety of dimensions, including size, position, rotation, and transparency.

As an example, here is the same system we saw above, rendered as a radial graph (note here that I’ve added a line before the loop that moves the drawing point to the centre of the screen; I’ve also changed the width of the bars to fit in the screen):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500);
  background(0);

  //move to the center of the sketch before we draw our graph
  translate(width/2, height/2);
  for (var i = 0; i < numbers.length; i++) {
    //calculate the amount of green in the colour by mapping the number to 255 (255 red &amp; 255 green = yellow)
    var c = map(numbers[i], min(numbers), max(numbers), 0, 255);
    fill(255, c, 0);
    //calculate the width of the bar by mapping the number to the half the width of the screen minus 50 pixels
    var w = map(numbers[i], 0, max(numbers), 0, width/2 - 50);
  	rect(0, 0, w, 20);
  	//after we draw each bar, turn the sketch a bit
  	rotate(TWO_PI/numbers.length);
  }
}

void draw() {
  //instructions in here happen over and over again
}

So far we’ve been keeping things very simple. But the possibilities using Processing and Javascript are really exciting. It’s very easy to add interactivity, and it’s even possible to take advantage of Processing’s 3D functionality in the browser.

Here’s our numbers again, rendered in 3D (note the change to the size() function), with the rotation controlled by the mouse (via our new friend the map() function):

var numbers = [13.4, 14.5, 15.0, 23.2, 30.9, 31.3, 32.9, 35.1, 34.3];

void setup() {
  //instructions in here happen once when the sketch is run
  size(500,500,P3D);
  background(0);

}

void draw() {
  //instructions in here happen over and over again
  background(0);
  //turn on the lights so that we see shading on the 3D objects
  lights();
  //move to the center of the sketch before we draw our graph
  translate(width/2, height/2);
  //Tilt about 70 degrees on the X axis - like tilting a frame on the wall so that it's on a table
  rotateX(1.2);
  //Now, spin around the Z axis as the mouse moves. Like spinning that frame on the table around its center
  rotateZ(map(mouseX, 0, width, 0, TWO_PI));
  for (var i = 0; i < numbers.length; i++) {
    //calculate the amount of green in the colour by mapping the number to 255 (255 red &amp; 255 green = yellow)
    var c = map(numbers[i], min(numbers), max(numbers), 0, 255);
    fill(255, c, 0);
    //calculate the height of the bar by mapping the number to the half the width of the screen minus 50 pixels
    var w = map(numbers[i], 0, max(numbers), 0, width/2 - 50);
    //move out 200 pixels from the center
  	pushMatrix();
  		translate(200, 0);
  		box(20,20,w);
  	popMatrix();
  	//after we draw each bar, turn the sketch a bit
  	rotate(TWO_PI/numbers.length);
  }
}

Thirty lines of code, and we have an interactive, 3D data visualization (you can see this sketch running inline in the header). Now, I understand that this isn’t the best use of 3D, and that we’re only visualizing a basic dataset, but the point here is to start to imagine the possibilities of using Processing with Javascript in the browser.

For starters, Javascript lets us very easily import data from any JSON source with no parsing whatsoever – this means we can create interactive graphics that load dynamic data on the fly (more on this in the next tutorial). On top of that, because we can write Javascript inline with our Processing code, and we can integrate external Javascript functions and libraries (lie JQuery) to our sketches, we can incorporate all kinds of advanced interactivity.

I suspect that the release of Processing 2.0 will lead to a burst of Javascript-based visualization projects being released on the web. In the meantime, we can use the basic template from this tutorial to work with Processing.js – have fun!

In late October, 2009, I received an e-mail from Jake Barton from Local Projects, titled plainly ‘Potential Freelance Job’. I read the e-mail, and responded with a reply in two parts: First, I would love to work on the project. Second, I wasn’t sure that it could be done.

The project was to design an algorithm for placement of names on the 9/11 memorial in New York City. In architect Michael Arad‘s vision for the memorial, the names were to be laid according to where people were and who they were with when they died – not alphabetical, nor placed in a grid. Inscribed in bronze parapets, almost three thousand names would stream seamlessly around the memorial pools. Underneath this river of names, though, an arrangement would provide a meaningful framework; one which allows the names of family and friends to exist together. Victims would be linked through what Arad terms ‘meaningful adjacencies’ – connections that would reflect friendships, family bonds, and acts of heroism. through these connections, the memorial becomes a permanent embodiment of not only the many individual victims, but also of the relationships that were part of their lives before those tragic events.

Over several years, staff at the 9/11 Memorial Foundation undertook the painstaking process of collecting adjacency requests from the next of kin of the victims, creating a massive database of requested linkages. Often, several requests were made for each victim. There were more than one thousand adjacency requests in total, a complicated system of connections that all had to be addressed in the final arrangement. In mathematical terms, finding a layout that satisfied as many of these adjacency requests as possible is an optimization problem – a problem of finding the best solution among a myriad of possible ones. To solve this problem and to produce a layout that would give the Memorial Designers a structure to base their final arrangement of the names upon, we built a software tool in two parts: First, an arrangement algorithm that optimized this adjacency problem to find the best possible solution. And second, an interactive tool that allowed for human adjustment of the computer-generated layout.

The Algorithm

The solution for producing a solved layout for the names arrangement sat at the bottom of a precariously balanced stack of complex requirements.

First, there was the basic spatial problem – the names for each pool had to fit, evenly, into a set of 76 panels (18 panels per side plus one corner). 12 of these panels were irregularly shaped (the corners and the panels adjacent to the corners, as seen in the image at the top of this post). Because the names were to appear in one continuous flowing group around each pool, some names had to overlap between panels, crossing a thin, invisible expansion joint between the metal plates. This expansion joint was small enough it would fit in the space between many of the first and last names (or middle initials), but with certain combinations of letterforms (for example, a last name starting with a J, or a first name ending with a y), names were unable to cross this gap. As a result, the algorithm had to consider the typography of each name as it was placed into the layout.

Of course, the most challenging problem was to place the names within the panels while satisfying as many of the requested adjacencies as possible. There were more than 1200 adjacency requests that needed to be addressed. One of the first things that I did was to get an idea of how complex this network of relations was by building some radial maps:

Clearly, there was a dense set of relations that needed to be considered. On top of that, the algorithm needed to be aware of the physical form of each of the names, since longer names could offer more space for adjacency than smaller ones. Because each person could have more than one adjacency request, there were groups of victims who were linked together into large clusters of relationship – the largest one of these contained more than 70 names. Here is a map of the adjacency clusters within one of the major groupings:

On top of these crucial links between victim names, there were a set of larger groupings in which the names were required to be placed: affiliations (usually companies), and sub-affiliations (usually departments within companies). To complicate matters, many of the adjacency requests linked people in different affiliations and subaffiliations, so the ordering of these groupings had to be calculated in order to satisfy both the adjacency requests and the higher-level relationships.

At some point I plan on detailing the inner workings of the algorithm (built in Processing). For the time being, I’ll say that the total process is in fact a combination of several smaller routines: first, a clustering routine to make discrete sets of names in which the adjacency requests are satisfied. Second, a space filling process which places the clusters into the panels and fills available space with names from the appropriate groupings. Finally, there is a placement routine which manages the cross-panel names, and adjusts spacing within and between panels.

The end result from the algorithm is a layout which completes as many of the adjacency requests as completely as possible. With this system, we were able to produce layouts which satisfied more than 98% of the requested adjacencies.

The Tool

Early on in the project, it became clear that the final layout for the names arrangement would not come directly from the algorithm. While the computerized system was able to solve the logistical problems underlying the arrangement, it was not as good at addressing the myriad of aesthetic concerns. The final layout had to be reviewed by hand – the architects needed to be able to meticulously adjust spacing and placement so that the final layout would be precisely as they wanted it. With this in mind, we built a custom software tool which allowed the memorial team to make custom changes to the layout, while still keeping track of all of the adjacencies.

The tool, again built in Processing, allowed users to view the layouts in different modes, easily move names within and between panels, get overall statistics about adjacency fulfillment, and export SVG versions of the entire system for micro-level adjustments in Adobe Illustrator. In the image above, we see an early layout for the South Pool. The colouring here represents two levels of grouping within the names. Main colour blocks represent affiliations, while shading within those colour blocks represents sub-affiliations. Below, we see a close-up view of a single panel, with several names placed in the ‘drawer’ below. Names can be placed in that space when they need to be moved from one panel to another. Note that adjacency indications remain active while the names are in the drawer.

Other features were built in to make the process of finalizing the layout as easy as possible: users could search for individual names, as well as affiliations and sub-affiliations; a change-tracking system allowed users to see how a layout had changed over multiple saved versions, and a variety of interface options allowed for precise placement of names within panels. Here is a video of the tool in use, again using a preliminary layout for the South Pool:

Computer Assisted Design

It would be misleading to say that the layout for the final memorial was produced by an algorithm. Rather, the underlying framework of the arrangement was solved by the algorithm, and humans used that framework to design the final result. This is, I think, a perfect example of something that I’ve believed in for a long time: we should let computers do what computers do best, and let humans do what humans do best. In this case, the computer was able to evaluate millions of possible solutions for the layout, manage a complex relational system, and track a large set of important variables and measurements. Humans, on the other hand, could focus on aesthetic and compositional choices. It would have been very hard (or impossible) for humans to do what the computer did. At the same time, it would have been very difficult to program the computer to handle the tasks that were completed with such dedication and precision by the architects and the memorial team.

The Weight of Data

This post has been a technical documentation of a very challenging project. Of course, on a emotional level, the project was also incredibly demanding. I was very aware throughout the process that the names that I was working with were the names of fathers and sons, mothers, daughters, best friends, and lovers.

Lawrence Davidson. Theresa Munson. Muhammadou Jawara. Nobuhiro Hayatsu. Hernando Salas. Mary Trentini.

In the days and months that I worked on the arrangement algorithm and the placement tool, I found myself frequently speaking these names out loud. Though I didn’t personally know anyone who was killed that day, I would come to feel a connection to each of them.

This project was a very real reminder that information carries weight. While names of the dead may be the heaviest data of all, almost every number or word we work with bears some link to a significant piece of the real world. It’s easy to download a data set – census information, earthquake records, homelessness figures - and forget that the numbers represent real lives. As designers, artists, and researchers, we always need to consider the true source of data, and the moral responsibility which they carry.

Ultimately, my role in the construction of the memorial was a small, and largely invisible one. Rightly, visitors to the site will never know that an algorithm assisted in its design. Still, it is rewarding to think that my work with Local Projects has helped to add meaning to such an important monument.

The 9/11 Memorial will be dedicated on September 11, 2011 and opens to the public with the reservation of a visitor pass on September 12th.

An hour ago, Apple announced that it has released a patch for iOS and iTunes, which reduces the size of the location cache stored on your machine, and prevents an automatic back-up through iTunes.

Good news, right?

I don’t think so. Apple is still collecting this data, still getting this data from you, and still using it. The only difference is that you can’t use your own data.

Location data is extremely useful. That’s why Apple, Google, and Microsoft are collecting it. Over the last year, Apple has, intentionally or not, created what is likely the largest locational database ever. This is a hugely, massively, ridiculously useful database. And with this new update, Apple are the only ones who will be able to get their hands on it. I believe that our data should be… well, our data. We should be able to store it securely, explore it, and use it for any purposes that we might choose. This data would be extraordinarily useful for researchers – people studying how diseases spread, trying to solve traffic-flow problems, and researching human mobility.

With all of this in mind, some colleagues and I have been working on a project for the last week called openpaths.cc. It lets you upload your location data from your iDevice, securely store it, explore it via a map interface, and we’ll eventually offer you a system to directly donate your data to well-deserving research projects.

We’re pushing this project out quickly in hopes that we can gather as many location files as we can before people upgrade iOS and iTunes.

Visit openpaths.cc now to upload, explore, and securely store your iDevice location data.

We are existing a world where data is being collected about us on a massive scale. This data is currently being stored, analyzed and monetized by corporations – there is little or no agency for the people to whom the data actually belongs. I believe that grass-roots initiatives like openpaths.cc can provide a framework for how data sovereignty can be established and managed.

In the short term, I am hoping we can collect and store enough locational data to be of use to researchers. So please, before you upgrade iOS and iTunes, visit openpaths.cc and make your own data your own data. And please (please) – pass this on.

About seven years ago, I had a bit of a career crisis. I was freelancing – working for clients I didn’t care much about on projects that I didn’t care much about, and feeling that there was a huge distance between the work that I was creating and my physical self. I was sick of computers, and was considering a range of (in hindsight) ridiculous vocational changes.

My rescue didn’t come from a new programming language, or a faster computer, or even better clients. It came, instead, from a return to the physical. I learned how to screenprint, and made rock posters for local bands, out of my living room. Every weekend, a friend and I would rack paper, pull squeegees, make an enormous mess – and escape from all of our pixel-based problems. We kept it up for a few years; after I moved into a larger, cleaner, less ink-friendly place I put my screens into storage. Even though I stopped printing, that time I spent screenprinting turned the rest of my career in a more creative direction.

Imagine how happy I was, then, to be asked by curator Christina Vassallo to be part of the inaugural edition of her Random Number Multiple series – a project that would produce screenprints from the work of computational artists and designers. Even better, this first edition would pair me with Marius Watz, an artist who has been a huge inspiration to me over the years, and whose work is exceptional in every way.

Marius and Christina and I spent three days at Bushwick Print Lab printing each of the 200 prints by hand. It was a fantastic experience, and the results, I think, speak for themselves. Marius’ prints are explosions of colour, vivid, dramatic pseudo-random that really capture the eye:

I made two prints. Both are abstractions of my word frequency visualizations that I created using Processing and the NYTimes Article Search API. The first, titled ‘RGB – NYT Word Frequency’, shows usage of the words ‘red’, ‘green’, ‘blue’ in the Times between 1981 and 2011 (you can see a series of details from the print here):

This print turned out even better than I could have expected. The fine detail is amazing, the colours are rich and vivid, and the half-toning on the individual bars creates a jewel-like halo in the center that is fascinating to look at up close.

My second print visualizes the terms ‘hope’ and ‘crisis’ over the same time period (again, more detailed views can be found here). This print was made with a semi-reflective ink, so it has a unique shimmer to it when viewed in the light:

Overall, I was surprised and delighted by how well this computer-generated work translated to the traditional medium of screenprint. I will definitely be looking to make more prints in the future.

In the meantime, a limited number of both of these prints are available for sale at on the Random Number Multiples site. Prints are $100, made with entirely acid-free media, and ship with a signed certificate of authenticity.

I have been spending far too much of my time over the last week or so playing Angry Birds. It’s a simple, clever, and addictive game which seems to have captivated large part of the human population (just yesterday, I heard a woman on the subway exclaim: “Take that, you %$!# pigs!”). Along with immense popularity, the game has also brought in millions upon millions of dollars for Rovio, the Finnish game company who created the concept and built the game. Though exact figures are unclear, the game has had upwards of 50M downloads, and brings in $1M per month in ad revenue alone. It’s likely that Angry Birds has made upwards of 100 million dollars.

For those that haven’t played, the gameplay is straight-forward: you launch birds out of a slingshot, in an attempt to collapse structures that are protecting egg-thieving pigs (it’s more fun that it sounds). A large part of the effectiveness of the game comes from the fact that these structures – built from blocks, triangles, and cylinders of various materials, react with realistic physics; the blocks knock each other over, the cylinders roll down hills, and the triangles act as convenient ramps.

I can’t say this for sure, but it’s very very likely that Angry Birds is built on top of Erin Catto’s excellent Box2D physics engine. Box2D is a set of libraries for C++ which makes it easy to build rigid body physics simulations – which is, essentially, what Angry Birds is. [EDIT: It has been confirmed that Angry Birds does in fact use Box2D]

Box2D is released with a very liberal open source license – it can basically be used by anyone, for anything. The only requirement is that, if and when source is released, the code has to be attributed to Erin, and can’t be claimed as original. There is absolutely no legal requirement for anyone using Box2D to pay for it in any way.

But is there an ethical requirement? The founders of Rovio are very, very, very rich men – thanks to Angry Birds. If, indeed, Angry Birds relies as heavily on Box2D as I suspect, they are also very, very, very rich men thanks to Erin and Box2D.

I’ve often thought of open source as a gratitude economy – people maintain and distribute projects largely because they are fueled by the thanks that they receive. By releasing Box2D with such a generous license, Erin Catto was clearly aware that he wouldn’t be profiting from its use. Likely, he’s been happy seeing his project used in so many interesting ways. I can’t imagine that he ever expected someone to make such a ridiculous fortune from his code, on the scale of Rovio and Angry Birds.

Of course, Rovio may have built their own physics engine for Angry Birds. And, even if they didn’t, they are perfectly within their rights to keep every penny, and not say a word about Erin or about Box2D.

With Christmas just around the corner, though, I can’t help but imagine an ideal world – in which Erin Catto receives some kind of an unexpected bonus. An open source holiday wish?

Munich designer Christian Flaccus used my tree.growth source code to create this amazing print, which includes, along with a hugely detailed tree, all of the Processing code used to render the tree. It’s giant – about 84 x 120cm, and is printed at 400dpi.

Digital artists talk a lot about preservation techniques for flaky digital storage media – and indeed a solution that comes up fairly often is to transfer the code onto a much more permanent, more archive-friendly medium – paper. So I suppose this poster serves as an archive as well – I’ll hope that Christian takes good care of it.