Visualizing Pressible: Blog Clusters & Growth of a System from blprnt on Vimeo.

For July and half of August, I’m the artist-in-residence at EdLab, a research group which is part of Teachers College at Columbia University.

I’ll be working with data from Pressible, a network of sites published by Teachers College students, faculty and staff. I’m interested in looking at the growth of this system, and in examining intertextuality between content in a network with a broad range of research interests.

I’ll posting about my process in as much detail as I can on my own Pressible site: blprnt.pressible.org. On that site, you’ll already find some early aesthetic and structural explorations, and you’ll be able to follow along with the project as it moves towards completion.

This residency will keep me in NYC until at least the end of August; if you are in the area and would like to connect, please get in touch.

In the August issue, out on newsstands right now, I had a chance to work with some spectacular data and extremely talented people. The piece looks at a very, very big data set – cellular phone records from a pool of 10 million users in an anonymous European country. This data came (under a very strict layer of confidentiality) from Barabási Lab in Boston, where they have been using this information to find out some fascinating things about human mobility patterns.

In this post, I’ll walk through the process of creating this piece. Along the way, I’ll show some draft images and unused experiments that eventually evolved into the final project.

Working With Big Data

I can’t get into a lot of detail about the specifics of the data set, but needless to say, phone records for 10 million individuals take up a lot of space. All told, the data for this project consisted of more than 5.5GB of flattened text files. I should say, at this point, that I don’t work on a supercomputer – I churn out all of my work from an often overheated 2.33GHZ MacBook Pro. Since the deadline was reasonably tight on this project, I decided to rule out a distributed computing approach to get at all of this data, and instead chose to work with a subset of the full list of records. Working in Processing, I built a simple script that could filter out a smaller dataset from the complete files. I built several of these at varying file sizes, giving me a nice set of data to work with both in prototyping and in production stages. This is a strategy that I often employ, even with more minimal datasets – save the heavy lifting until the final render.

The first thing I did with the trimmed-down data was to construct ‘call histories’ for each user in the set. I rendered out these histories as stacked bars of individual calls, which could then be placed into a histogram. Here’s a graph of about 10,000 users, sorted by their total time spent on the phone :

Here we see a very obvious power law distribution, with a few people talking a lot (really, a lot – 28.3 hours a week), and most callers talking relatively little (these is also a tail of text-only users at the very end). The problem here, of course, is that on a computer screen – or even in print – it’s hard to get into the data to learn anything useful. When I zoom into the graph, we can start to see the individual call histories (I’ve enlarged a few columns for detail). Here, long calls are rendered yellow, short calls are rendered red, and text messages are flat blue rectangles:

I took the same graph as above, and added another set of columns extending below – here the white bars show us how many ‘friends’ the individual callers had – ie. how many people they are regularly talking to over the week:

If I sort this graph by number of friends (rather than total call time), we can see that the two measures (talkativeness, and number of friends) don’t seem to be strongly correlated:

It’s interesting to note here as well, that the data set includes linkage information – so I can also visualize who is calling who within our group of individuals:

There is some interesting information to be dug up in here, but the long aspect of the graph and the general over-detail involved makes it not very usable – particularly for a magazine piece.

Ooh, and then Aaah.

The Infoporn section in Wired is a two page spread;  I always think of it as needing to serve two separate purposes for two different kinds of readers. First, it needs to be visually pleasing. I want people to say ‘Oooh…!’ when they turn the page to it. Once they’re hooked, though, I want them to learn something – the ‘Aaah!’ moment.

The data used in the graphs above seemed too complex to do anything truly revealing with – so perhaps it could be built into something sexy enough to draw an ‘Oooh!’ or two? In order to fit the long tails of these graphs onto the page, I wondered if I could add a bit of a curl to them. To make this structural change evident, I turned the graphs on a slight angle and rendered them in 3D. Here, we see five of these graphs, totaling about a million individual users, arranged into a single, tower-like shape:

While these structures took a little while to render, I could quite easily generate a unique set of them, which I assembled as a line trailing off to the page edge on the left:

Getting Personal

So far, the visuals for this project only tell a part of the story: that our individual calling habits fall into predictable patterns when placed with the larger whole (some excellent text from Michael Dumiak helps clarify this in the final piece). There’s another crucial piece, though. Cel phone usage data is inherently locative, since our provider always knows from which of their cel towers we are placing the call.

This is where the fun starts – we can use this locative data to track the mobility patterns of individual people (it’s worth saying here that all of the data the I worked with was anonymized). To do this, I created a tool (again, in Processing) to make ‘mobility cubes’ – which show a history of an individual’s movements over time:

The individual above, for example, travels around an area less than a square kilometer over a period of just under three days. If I flatten this graph, we can see that this person travels mostly between two locations:

From the data, we can identify a lot of individuals like this – commuters – who travel short distances between two places (home, and work). We can also find travelers (people who cover a long distance in a short period of time):

And others who seem to follow more elaborate (but often still regular) mobility patterns:

We can assemble a ‘mobility cube’ for each individual in the database – and very quickly gain a mechanism for recognizing patterns amongst these people:

Which brings us to the underlying point of the piece – we are all leaving digital trails behind us, as we make our way around our individual lives. These trails are largely considered individual – even ethereal – yet technology is making these trails more visible and more readable everyday.

Of course, to see the final piece – the polished assembly of some of the drafts and artifacts you’ve seen in this post – you’ll have to buy the magazine. Wired UK is available on newsstands in the UK, and to all of our clever subscribers.

If you want to read more about this – and you should – I’d highly recommend Albert-László Barabási’s Bursts, which goes into much more detail about human mobility & predictability.

Finally, huge thanks have to go out to László and his team at the lab, without whom this piece would have never made it to print!

I’ve spent the last 8 weeks or so traveling around, doing a lot of speaking and teaching. Below is an overview of some of the places I’ve been and things I’ve been doing.

First stop was the Alberta College of Art and Design, where Adam Tindale arranged for me to teach a workshop on Processing. Adam has a good thing going in Alberta – his students were great, and despite it being at the very end of the term we had a good turn out and I think people learned a lot. While I was there, I had the chance to see Adam perform live, which was a definite highlight – he mixed generative music with sets of abstract geometric forms live-coded in Processing.

I left Alberta and headed east for FITC in Toronto, where I gave a presentation titled ‘Hacking the Newsroom’. I also had the opportunity to sit on a panel organized by Tali Krakowsky, called ‘Storytelling: Absorbed, Obsessed, and Immersed’. I was humbled to be included in this group with John Underkoffler, Alex McDowell and Ben Kreukniet. You can read a good overview of the panel and our discussions here.

From Toronto, to Prague, where I was part of Transistor 2010, an event themed on digital archival run by a Czech new media organization called CIANT. The event was hosted at Prague’s famous FAMU school – many thanks to Eric Rosenzveig who invited me to take part and to Barbora and the staff at CIANT for taking care of me!

MultiMania is a free one-day conference that is organized in Kortrijk, Belgium by the lovely and talented Koen de Wegghelaire. Yes, you read that right – free. Not only that, it’s a huge event, held in a sparkling new conference centre, and packed with excellent presenters. I was really, really impressed. Again, I talked about data visualization, with a focus on some of the projects I have done with the NYTimes APIs.

Those of you who have read my blog for any period of time will know that I am a big fan of Daniel Shiffman, who teaches at New York’s ITP. Daniel’s book, Learning Processing, is not only the best Processing text out there, it’s the best ‘learning to program’ book that I have ever encountered. I was thrilled, then, to be asked last minute to join Daniel for a ‘Processing Salon’ in Amsterdam, at Mediamatic. The event turned out to be very popular – 160+ people crowded into the space to hear the two of us talk about our work in Processing. This was the first – and probably only – time I’ve spoken in a space where a plywood bike track snaked its way through the crowd!

Finally, I made it to Newcastle last week for Thinking Digital. Thinking Digital is a three-day event held at The Sage Gateshead, a very distinctive building just across the equally distinctive Millenium Bridge from Newcastle. Thinking Digital is loosely styled after events like TED and PopTech, bringing together speakers from a variety of disciplines – the TDC10 lineup included an origami expert, a mathematician, a comedian, along with the usual mix of ‘social media experts’ and ‘branding experts’. It was a really solid event – organizers looking for a model of a well-run conference should look to Herb Kim and his Codeworks team.

I am in the process of getting the three (four?) versions of my presentation up on SlideShare. In the meantime, if you attended any of these events and have questions or feedback, please feel free to leave a comment or to get in touch via e-mail.

My whirlwind of speaking behind me, I’ll be spending another week in the UK before heading back to North America for FlashBelt. I’ll be in London from the 9th to the 14th – if anyone fancies a pint while I’m there, let me know!

This post, then, is a first sketch of what a lesson plan for teaching Processing and data visualization might look like. I’m going to start from scratch, work through some examples, and (hopefully) make some interesting stuff. One of the nice things, I think, about this process, is that we’re going to start with fresh, new data – I’m not sure what kind of things we’re going to find once we start to get our hands dirty. This is what is really exciting about data visualization; the chance to find answers to your own, possibly novel questions.

Let’s Start With the Data

We’re not going to work with an old, dusty data set here. Nor are we going to attempt to bash our heads against an unnecessarily complex pile of numbers. Instead, we’re going to start with a data set that I made up – with the help of a couple of hundred of my Twitter followers. Yesterday morning, I posted this request:

Even on a Saturday, a lot of helpful folks pitched in, and I ended up with about 225 numbers. And so, we have the easiest possible dataset to work with – a single list of whole numbers. I’m hoping that, as well as being simple, this dataset will turn out to be quite interesting – maybe telling us something about how the human brain thinks about numbers.

I wrote a quick Processing sketch to scrape out the numbers from the post, and then to put them into a Google Spreadsheet. You can see the whole dataset here: http://spreadsheets.google.com/pub?key=t6mq_WLV5c5uj6mUNSryBIA&output=html

I chose to start from a Google Spreadsheet in this tutorial, because I wanted people to be able to generate their own datasets to work with. Teachers – you can set up a spreadsheet of your own, and get your students to collect numbers by any means you’d like. The ‘User’ and ‘Tweet’ columns are not necessary; you just need to have a column called ‘Number’.

It’s about time to get down to some coding. The only tricky part in this whole process will be connecting to the Google Spreadsheet. Rather than bog down the tutorial with a lot of confusing semi-advanced code, I’ll let you download this sample sketch which has the Google Spreadsheet machinery in place.

Got it? Great. Open that sketch in Processing, and let’s get started. Just to make sure we’re all in the same place, you should see a screen that looks like this:

At the top of the sketch, you’ll see three String values that you can change. You’ll definitely have to enter your own Google username and password. If you have your own spreadsheet of number data, you can enter in the key for your spreadsheet as well. You can find the key right in the URL of any spreadsheet.

The first thing we’ll do is change the size of our sketch to give us some room to move, set the background color, and turn on smoothing to make things pretty. We do all of this in the setup enclosure:

void setup() {
  //This code happens once, right when our sketch is launched
 size(800,800);
 background(0);
 smooth();
};

Now we need to get our data from the spreadsheet. One of the advantages of accessing the data from a shared remote file is that the remote data can change and we don’t have to worry about replacing files or changing our code.

We’re going to ask for a list of the ‘random’ numbers that are stored in the spreadsheet. The most easy way to store lists of things in Processing is in an Array. In this case, we’re looking for an array of whole numbers – integers. I’ve written a function that gets an integer array from Google – you can take a look at the code on the ‘GoogleCode’ tab if you’d like to see how that is done. What we need to know here is that this function – called getNumbers – will return, or send us back, a list of whole numbers. Let’s ask for that list:

void setup() {
  //This code happens once, right when our sketch is launched
 size(800,800);
 background(0);
 smooth();

 //Ask for the list of numbers
 int[] numbers = getNumbers();
};

OK.

World’s easiest data visualization!

 fill(255,40);
 noStroke();
 for (int i = 0; i < numbers.length; i++) {
   ellipse(numbers[i] * 8, width/2, 8,8);
 };

What this does is to draw a row of dots across the screen, one for each number that occurs in our Google list. The dots are drawn with a low alpha (40/255 or about 16%), so when numbers are picked more than once, they get brighter. The result is a strip of dots across the screen that looks like this:

Right away, we can see a couple of things about the distribution of our ‘random’ numbers. First, there are two or three very bright spots where numbers get picked several times. Also, there are some pretty evident gaps (one right in the middle) where certain numbers don’t get picked at all.

This could be normal though, right? To see if this distribution is typical, let’s draw a line of ‘real’ random numbers below our line, and see if we can notice a difference:

fill(255,40);
 noStroke();
 //Our line of Google numbers
 for (int i = 0; i < numbers.length; i++) {
   ellipse(numbers[i] * 8, height/2, 8,8);
 };
 //A line of random numbers
 for (int i = 0; i < numbers.length; i++) {
   ellipse(ceil(random(0,99)) * 8, height/2 + 20, 8,8);
 };

Now we see the two compared:

The bottom, random line doesn’t seem to have as many bright spots or as evident of gaps as our human-picked line. Still, the difference isn’t that evident. Can you tell right away which line is our line from the group below?

OK. I’ll admit it – I was hoping that the human-picked number set would be more obviously divergent from the sets of numbers that were generated by a computer. It’s possible that humans are better at picking random numbers than I had thought. Or, our sample set is too small to see any kind of real difference. It’s also possible that this quick visualization method isn’t doing the trick. Let’s stay on the track of number distribution for a few minutes and see if we can find out any more.

Our system of dots was easy, and readable, but not very useful for empirical comparisons. For the next step, let’s stick with the classics and

Build a bar graph.

Right now, we have a list of numbers. Ours range from 1-99, but let’s imagine for a second that we had a set of numbers that ranged from 0-10:

[5,8,5,2,4,1,6,3,9,0,1,3,5,7]

What we need to build a bar graph for these numbers is a list of counts – how many times each number occurs:

[1,2,1,2,1,3,1,1,1,1]

We can look at this list above, and see that there were two 1s, and three 5s.

Let’s do the same thing with our big list of numbers – we’re going to generate a list 99 numbers long that holds the counts for each of the possible numbers in our set. But, we’re going to be a bit smarter about it this time around and package our code into a function – so that we can use it again and again without having to re-write it. In this case the function will (eventually) draw a bar graph – so we’ll call it (cleverly) barGraph:

void barGraph( int[] nums ) {
  //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 1; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };
};

This function constructs an array of counts from whatever list of numbers we pass into it (that list is a list of integers, and we refer to it within the function as ‘nums’, a name which I made up). Now, let’s add the code to draw the graph (I’ve added another parameter to go along with the numbers – the y position of the graph):

void barGraph(int[] nums, float y) {
  //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 1; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };

 //Draw the bar graph
 for (int i = 0; i < counts.length; i++) {
   rect(i * 8, y, 8, -counts[i] * 10);
 };
};

We’ve added a function – a set of instructions – to our file, which we can use to draw a bar graph from a set of numbers. To actually draw the graph, we need to call the function, which we can do in the setup enclosure. Here’s the code, all together:

/*

 #myrandomnumber Tutorial
 blprnt@blprnt.com
 April, 2010

 */

//This is the Google spreadsheet manager and the id of the spreadsheet that we want to populate, along with our Google username & password
SimpleSpreadsheetManager sm;
String sUrl = "t6mq_WLV5c5uj6mUNSryBIA";
String googleUser = "YOUR USERNAME";
String googlePass = "YOUR PASSWORD";

void setup() {
  //This code happens once, right when our sketch is launched
 size(800,800);
 background(0);
 smooth();

 //Ask for the list of numbers
 int[] numbers = getNumbers();
 //Draw the graph
 barGraph(numbers, 400);
};

void barGraph(int[] nums, float y) {
  //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 1; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };

 //Draw the bar graph
 for (int i = 0; i < counts.length; i++) {
   rect(i * 8, y, 8, -counts[i] * 10);
 };
};

void draw() {
  //This code happens once every frame.
};

If you run your code, you should get a nice minimal bar graph which looks like this:

We can help distinguish the very high values (and the very low ones) by adding some color to the graph. In Processing’s standard RGB color mode, we can change one of our color channels (in this case, green) with our count values to give the bars some differentiation:

 //Draw the bar graph
 for (int i = 0; i < counts.length; i++) {
   fill(255, counts[i] * 30, 0);
   rect(i * 8, y, 8, -counts[i] * 10);
 };

Which gives us this:

Or, we could switch to Hue/Saturation/Brightness mode, and use our count values to cycle through the available hues:

//Draw the bar graph
 for (int i = 0; i < counts.length; i++) {
   colorMode(HSB);
   fill(counts[i] * 30, 255, 255);
   rect(i * 8, y, 8, -counts[i] * 10);
 };

Which gives us this graph:

Now would be a good time to do some comparisons to a real random sample again, to see if the new coloring makes a difference. Because we defined our bar graph instructions as a function, we can do this fairly easily (I built in an easy function to generate a random list of integers called getRandomNumbers – you can see the code on the ‘GoogleCode’ tab):

void setup() {
  //This code happens once, right when our sketch is launched
 size(800,800);
 background(0);
 smooth();

 //Ask for the list of numbers
 int[] numbers = getNumbers();
 //Draw the graph
 barGraph(numbers, 100);

 for (int i = 1; i < 7; i++) {
 int[] randoms = getRandomNumbers(225);
 barGraph(randoms, 100 + (i * 130));
 };
};

I know, I know. Bar graphs. Yay. Looking at the graphic above, though, we can see more clearly that our humanoid number set is unlike the machine-generated sets. However, I actually think that the color is more valuable than the dimensions of the bars. Since we’re dealing with 99 numbers, maybe we can display these colours in a grid and see if any patterns emerge? A really important thing to be able to do with data visualization is to

Look at datasets from multiple angles.

Let’s see if the grid gets us anywhere. Luckily, a function to make a grid is pretty much the same as the one to make a graph (I’m adding two more parameters – an x position for the grid, and a size for the individual blocks):

void colorGrid(int[] nums, float x, float y, float s) {
 //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 0; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };

//Move the drawing coordinates to the x,y position specified in the parameters
 pushMatrix();
 translate(x,y);
 //Draw the grid
 for (int i = 0; i < counts.length; i++) {
   colorMode(HSB);
   fill(counts[i] * 30, 255, 255, counts[i] * 30);
   rect((i % 10) * s, floor(i/10) * s, s, s);

 };
 popMatrix();
};

We can now do this to draw a nice big grid:

 //Ask for the list of numbers
 int[] numbers = getNumbers();
 //Draw the graph
 colorGrid(numbers, 50, 50, 70);

I can see some definite patterns in this grid – so let’s bring the actual numbers back into play so that we can talk about what seems to be going on. Here’s the full code, one last time:

/*

 #myrandomnumber Tutorial
 blprnt@blprnt.com
 April, 2010

 */

//This is the Google spreadsheet manager and the id of the spreadsheet that we want to populate, along with our Google username & password
SimpleSpreadsheetManager sm;
String sUrl = "t6mq_WLV5c5uj6mUNSryBIA";
String googleUser = "YOUR USERNAME";
String googlePass = "YOUR PASSWORD";

//This is the font object
PFont label;

void setup() {
  //This code happens once, right when our sketch is launched
 size(800,800);
 background(0);
 smooth();

 //Create the font object to make text with
 label = createFont("Helvetica", 24);

 //Ask for the list of numbers
 int[] numbers = getNumbers();
 //Draw the graph
 colorGrid(numbers, 50, 50, 70);
};

void barGraph(int[] nums, float y) {
  //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 1; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };

 //Draw the bar graph
 for (int i = 0; i < counts.length; i++) {
   colorMode(HSB);
   fill(counts[i] * 30, 255, 255);
   rect(i * 8, y, 8, -counts[i] * 10);
 };
};

void colorGrid(int[] nums, float x, float y, float s) {
 //Make a list of number counts
 int[] counts = new int[100];
 //Fill it with zeros
 for (int i = 0; i < 100; i++) {
   counts[i] = 0;
 };
 //Tally the counts
 for (int i = 0; i < nums.length; i++) {
   counts[nums[i]] ++;
 };

 pushMatrix();
 translate(x,y);
 //Draw the grid
 for (int i = 0; i < counts.length; i++) {
   colorMode(HSB);
   fill(counts[i] * 30, 255, 255, counts[i] * 30);
   textAlign(CENTER);
   textFont(label);
   textSize(s/2);
   text(i, (i % 10) * s, floor(i/10) * s);
 };
 popMatrix();
};

void draw() {
  //This code happens once every frame.

};

And, our nice looking number grid:

BINGO!

No, really. If this was a bingo card, and I was a 70-year old, I’d be rich. Look at that nice line going down the X7 column – 17, 27, 37, 47, 57, 67, 77, 87, and 97 are all appearing with good frequency. If we rule out the Douglas Adams effect on 42, it is likely that most of the top 10 most-frequently occurring numbers would have a 7 on the end. Do numbers ending with 7s ‘feel’ more random to us? Or is there something about the number 7 that we just plain like?

Contrasting to that, if I had played the x0 row, I’d be out of luck. It seems that numbers ending with a zero don’t feel very random to us at all. This could also explain the black hole around the number 50 – which, in a range from 0-100, appears to be the ‘least random’ of all.

Well, there we have it. A start-to finish example of how we can use Processing to visualize simple data, with a goal to expose underlying patterns and anomalies. The techniques that we used in this project were fairly simple – but they are useful tools that can be used in a huge variety of data situations (I use them myself, all the time).

Hopefully this tutorial is (was?) useful for some of you. And, if there are any teachers out there who would like to try this out with their classrooms, I’d love to hear how it goes.

I am going to be traveling for a few months starting at the end of April, and I have a pile of prints here that will be sitting idle if they don’t get shipped out the door before then. So, how ’bout a sale?

Single prints: 15% off

2 prints: 25% off

3 prints: Third (least expensive) print is free!

Free print: I’ll ship a free 18″x10″ flower.trees print to a random person who Tweets about this sale, using the hashtag #blprnts. I’ll draw this winner at 10PM PST, on Tuesday April 6th.

You can check out what I have in stock at my Etsy store. If you purchase via Etsy (this is easiest), I’ll refund you the discount/free print. Otherwise, get in touch and we can sort things out. All prints are on Hahnemühle photo-rag paper, and are printed with archival-quality inks. They are shipped flat. Some prints are editions (signed and numbered) – all prints come signed.

I’ll be shipping prints on Tuesday, April 13th. Please get orders in to me by Sunday, April 11th.

I’ve lived in Vancouver for 17 years (29, if you count time served in the suburbs). It’s a beautiful city, with a lot to offer. Today’s bright spring day is about as good of a demonstration of its benefits as is possible. It’s a city full of great people, doing interesting things, and as such has a long history of cultural and creative production.

All of that said, I’m leaving.

For personal reasons, I’m going to find myself without fixed address as of April 1st. After thinking it over for a little while, I’ve decided to take this opportunity to pull up roots and re-locate. I haven’t really felt at home in Vancouver over the last few months and the political situation in this province certainly doesn’t help matters. All in all, I’ll be happy to get away for a while.

I’m not sure where I will end up. I’m going to spend most of April visiting my family on Vancouver Island, and in May & June I’ll be traveling in Europe and the US to do some speaking and teaching. The plan is to give a new location serious consideration starting in July. I’ve begun to generate a short list of creative cities where I could easily live (with a dog) – if you have any suggestions, I’d love to hear them.

In the meantime, please expect some sporadicity in posts as I bounce around.

(top photo by Nick Dobbing)

At the end of April, I’ll be in Toronto for FITC, where I’ll be talking about data visualization, the NYTimes API, and showing a nice set of work that nobody has seen, yet. I’ll also be speaking on a panel called ‘Storytelling: Absorbed, Obsessed And Immersed‘ with Tali Krakoswky, John Underkoffler, Ben Kreukniet and Alex McDowell. FITC – April 24th to April 27th, Toronto, Canada

At the end of May I’ll be Newcastle to speak at Thinking Digital. This event brings together a group of interesting people from different disciplines – the venue and speaker lineup look fantastic. Again, I’ll be talking about data visualization, with an emphasis on deconstructing some of the work that I’ve done for Wired UK. Thinking Digital – May 25th to 27th, Newcastle upon Tyne, UK

A couple of weeks later I’ll be in Minneapolis for my favourite event of the year – Flashbelt. I’ll be giving a completely new talk, which I will share details about over the next couple of days. If you’ve been thinking about coming to Flashbelt in previous years but have never taken the leap – this year is the year to do it. Flashbelt – June 13th to 16th, Minneapolis, USA

The most valuable thing I take out of being part of these events is meeting interesting people. If you are going to be around for any of these events and would like to meet for a beer – please don’t hesitate to get in touch.

A few people asked me to expand on this, so let’s give it a try:

We are facing a very different data-related problem today than we were facing only a few years ago. Back then, the call was solely for more information. Since then, corporations and governments have started to answer this call and the result has been a flood of data of all shapes and sizes. While it’s important to remain on track with the goal of making data available, we are now faced with a parallel and perhaps more perplexing problem: What do we do with it all?

Of course, an industry has developed around all of this data; start-ups around the world are coming up with new ideas and data-related products every day. At the same time, open-sourcers are releasing helpful tools and clever apps by the dozen. Still, in a large part these groups are looking at the data with fiscal utility in mind. It seems to me that if we are going to make the most of this information resource, it’s important to bring more people in on the game – and to do that requires education.

At the post-secondary level, efforts should be made to educate academics for whom this new pile of data could be useful: journalists, social scientists, historians, contemporary artists, archivists, etc. I could imagine cross-disciplinary workshops teaching the basics:

1. A survey of what kind of data is available, and how to find it.
2. A brief overview of common data formats (CSV, JSON, XML, etc).
3. An introduction to user-friendly exploration tools like ManyEyes & Tableau
4. A primer in Processing and how it can be used to quickly prototype and build specialized visualization tools.

The last step seems particularly important to me, as it encourages people to think about new ways to engage with information. In many cases, datasets that are becoming available are novel in their content, structure, and complexity – encouraging innovation in an academic framework is essential. Yes, we do need to teach people how to make bar graphs and scatter charts; but let’s also facilitate exploration and experimentation.

Why workshops? While this type of teaching could certainly be done through tutorials, or with a well-written text book, it’s my experience that teaching these subjects is much more effective one-on-one. This is particularly true for students who come at data from a non-scientific perspective (and these people are the ones that we need the most).

The long-term goal of such an initiative would be to increase data-literacy. In a perfect world, this would occur even earlier – at the highschool level. Here’s where I put on my utopian hat: teaching data literacy to young people would mean that they could find answers to their own questions, rather than waiting for the media to answer those questions for them. It also teaches them, in a practical way, about transparency and accountability in government. The education system is already producing a generation of bloggers and citizen journalists – let’s make sure they have the skills they need to be dangerous. Veering a bit to the right, these are hugely valuable skills for workers in an ‘idea economy’ – a nation with a data-literate workforce is a force to be reckoned with.

Ideally this educational component would be build in to government projects like data.gov or data.hmg.gov.uk (are you listening, Canada?). More than that, it would be woven into the education mandate of governments at federal and local levels. Of course, I’m not holding my breath.

Instead, I’ve started to plan a bit of a project for the summer. Like last year, I taught a series of workshops at my studio in Vancouver, which were open to people of all skill levels. This year, I’m going to extend my reach a bit and offer a couple of free, online presentations covering some of the things that I’ve talked about in this post. One of these workshops will be specifically targeted to youth. At the same time, I’ll be publishing course outlines and sample materials for my sessions so that others can host similar events.

Stay tuned for details – and if you have any questions or would like to lend a hand, feel free to leave a comment or get in touch.

I was asked at the end of last week to produce a graphic for the Opinion page today – the idea was to compare the texts of various ‘state of the union’ addresses from around the world. The final result (pictured above) is not extraordinarily data-heavy. It worked quite nicely in the printed layout, where the individual ‘tentacles’ trailed to the text of the speeches that they index.

The process behind this piece was relatively simple. Each speech was indexed using a Processing application that I wrote which counts the frequency of individual names (the program ignores commonly used or unimportant words). The words for each speech were then ranked by mentions per thousand words (you can see a version of the piece with numbers here)

Almost every project I work on involves a period of ‘data exploration’ in which I try shake as many interesting things out of the information as I can. Even though this piece had a short turn-around, I did a fair amount of poking around, generating some simple bar graphs:

Another avenue I explored was to use the word weights to determine a ‘score’ for each sentence. By doing this, I can try to find the ‘kernel’ of the speech – the sentence that sums up the entire text in the most succinct way. This, I think was fairly successful. Here are the ‘power sentences’ for the UK:

The Netherlands:

And Botswana:

Which brings us to tonight’s State of the Union Address by Barack Obama. What was the ‘power sentence’ from this speech? I ran the weighting algorithm on the address and this is what it came up with:

Clarify some misunderstandings

I published a post last week comparing Haiti aid per capita to Avatar ticket prices. The post got a lot of attention, and the figures and general concept were cross-posted and re-hashed in many places. Some people seemed to have misunderstood the post, though, and thought that I was comparing the contributions of individual governments to the production costs of Avatar. This is not what I did.

To get my figures for ‘Avatar minutes’ I started with the total aid contribution for a country, and divided it by that country’s population to get a per-capita aid figure. I then calculated how many minutes of Avatar that per-person contribution would pay for, using a ticket price of $8.50 (with a running time of 162 minutes, an ‘Avatar minute’ is about 5.25 cents). So, with Canada’s aid contribution of $5.5M, and a population of 33.3M, the per-person donation is about 3 Avatar minutes. Now, before any of you angry Canadians start frothing at the mouth, let me

Update some figures

When I published by post last week, I used the data that was then available. Many people commented about my use of the figure $5.5M for Canada, since very shortly after the post it was announced that the Canadian government was drastically increasing their Haiti aid contributions, and at the same time stated that they would match Canadian citizen’s contributions dollar-for-dollar, with no capping amount. I highlighted Canada in my post not to shame the government, but because I live in Canada. Again, I used the data available. I promised at the time to update the figures as more information became available, so, without further ado:

• Canada: 74 minutes
• Sweden: 47 minutes
• Norway: 41 minutes
• Denmark: 39 minutes
• Luxembourg: 28 minutes
• Finland: 27 minutes
• Guyana: 25 minutes
• Spain: 19 minutes
• Estonia: 14 minutes
• Australia: 12 minutes
• Ireland: 12 minutes
• Switzerland: 11 minutes
• USA: 10 minutes
• France: 9.5 minutes
• Germany: 5 minutes
• Netherlands: 5 minutes
• Italy: 3 minutes
• Japan: 1 minute

The contributions pledged by the Canadian government are impressive. But the point of the original post was not to single out any individual country for either congratulation or condemnation. Instead, it was to take the figures and put them into some kind of context.

$130,733,775 is a lot of money. Really. But our measure of amounts always depends on what context we put the numbers in. $130 million is a lot of money when compared to my yearly income. But it’s not that much money compared to the 2010 olympic budget – $1,700 million for a two-week sporting event. It’s just under half of the estimated production costs of Avatar ($280M). It’s less than 4% of Canada’s foreign aid budget.

If we add up ALL of the contributions to Haiti Aid, we get an even bigger amount of money – $1.75 billion dollars. A huge amount, to be sure, but again, a number that needs to be looked at in context. $1.75B is just a little bit less than Avatar has made in global ticket sales. It’s about 50% of Canada’s foreign aid budget, and 0.25% of last year’s monstrous US financial bailout. It is, repeating myself from the last paragraph, pretty much exactly what Vancouver is spending on next month’s winter games.

All of this mention of Canada and foreign aid may have already have tipped you off that I’d like to

Talk about Canadian Foreign Policy

Canada’s foreign aid budget is $3.45B, or about 0.25% of Canada’s GDP. Compare that to the Danes, who spend 0.83% of their GDP on aid (up this year from 0.82%, despite a record forecast deficit), or to the Swedes who spend about 0.92%. Canadians like to believe that we are a shining example of global citizenry, but largely this is an artifact of the pre-Mulroney governments of the 1970s and 1980s. The Center for Glocal Development ranked Canada 11th in their Commitment to Development Index from 2009, behind countries like Sweden, Denmark, Netherlands, Ireland, Spain, and Australia.

This index includes factors like aid, trade, investment, and migration. As the report notes, our migration levels of unskilled immigrants from developing countries has changed very little since the 1980s (we rank 11th on the list for migration). 

Like many other Canadians I grew up feeling proud about my country and about our role in the world. Unfortunately, the more I look into the actual figures, I realize that we have in many ways failed to maintain these ideals in the last 30 years.

I hope that the Canada’s actions on Haiti mark a change for our government (and not, say, a convenient way to buy some much-needed PR). I would like nothing more than to see Canada return to the role of the good global citizen. In the meantime, I will continue watching the government’s record with a deserved amount of criticality.

Speaking of criticality, let me finish this post by taking a moment to

Respond to some criticism

Jen Stirrup wrote a nicely detailed blog post in response to my Avatar/Haiti piece, in which she argues that the visualization puts beauty in advance of clarity. If we take the images that I used in the post as examples of data visualizations, I can’t help but agree. However, these images weren’t intended to be stand-alone graphics. Instead, they are screenshots of an animated, interactive visualization tool that I built to explore the data. As is very often the case when I work with data, I wrote a little program using Processing which was constructed specifically to deal with this data. I use the term ‘little’ here to emphasize the fact that it was a quick project – from the time that I had the idea to the time when I pressed ‘publish’ last Sunday was about 4 hours.

I would love to develop a workflow to take these interactive visualization tools to a stage where they can be shared more easily – at this point they tend to sit around while I harbour the best intentions to clean up the code enough for a proper release. In the meantime I can say that if you ask nicely, I’m usually willing to share my messy pre-release code. I will also be posting a brief video which might give you a better feel for how the project behaves – which, for the sake of continuity, I’ll title ‘Remove a rock from a horse’s hoof’