A few people asked me to expand on this, so let’s give it a try:

We are facing a very different data-related problem today than we were facing only a few years ago. Back then, the call was solely for more information. Since then, corporations and governments have started to answer this call and the result has been a flood of data of all shapes and sizes. While it’s important to remain on track with the goal of making data available, we are now faced with a parallel and perhaps more perplexing problem: What do we do with it all?

Of course, an industry has developed around all of this data; start-ups around the world are coming up with new ideas and data-related products every day. At the same time, open-sourcers are releasing helpful tools and clever apps by the dozen. Still, in a large part these groups are looking at the data with fiscal utility in mind. It seems to me that if we are going to make the most of this information resource, it’s important to bring more people in on the game – and to do that requires education.

At the post-secondary level, efforts should be made to educate academics for whom this new pile of data could be useful: journalists, social scientists, historians, contemporary artists, archivists, etc. I could imagine cross-disciplinary workshops teaching the basics:

1. A survey of what kind of data is available, and how to find it.
2. A brief overview of common data formats (CSV, JSON, XML, etc).
3. An introduction to user-friendly exploration tools like ManyEyes & Tableau
4. A primer in Processing and how it can be used to quickly prototype and build specialized visualization tools.

The last step seems particularly important to me, as it encourages people to think about new ways to engage with information. In many cases, datasets that are becoming available are novel in their content, structure, and complexity – encouraging innovation in an academic framework is essential. Yes, we do need to teach people how to make bar graphs and scatter charts; but let’s also facilitate exploration and experimentation.

Why workshops? While this type of teaching could certainly be done through tutorials, or with a well-written text book, it’s my experience that teaching these subjects is much more effective one-on-one. This is particularly true for students who come at data from a non-scientific perspective (and these people are the ones that we need the most).

The long-term goal of such an initiative would be to increase data-literacy. In a perfect world, this would occur even earlier – at the highschool level. Here’s where I put on my utopian hat: teaching data literacy to young people would mean that they could find answers to their own questions, rather than waiting for the media to answer those questions for them. It also teaches them, in a practical way, about transparency and accountability in government. The education system is already producing a generation of bloggers and citizen journalists – let’s make sure they have the skills they need to be dangerous. Veering a bit to the right, these are hugely valuable skills for workers in an ‘idea economy’ – a nation with a data-literate workforce is a force to be reckoned with.

Ideally this educational component would be build in to government projects like data.gov or data.hmg.gov.uk (are you listening, Canada?). More than that, it would be woven into the education mandate of governments at federal and local levels. Of course, I’m not holding my breath.

Instead, I’ve started to plan a bit of a project for the summer. Like last year, I taught a series of workshops at my studio in Vancouver, which were open to people of all skill levels. This year, I’m going to extend my reach a bit and offer a couple of free, online presentations covering some of the things that I’ve talked about in this post. One of these workshops will be specifically targeted to youth. At the same time, I’ll be publishing course outlines and sample materials for my sessions so that others can host similar events.

Stay tuned for details – and if you have any questions or would like to lend a hand, feel free to leave a comment or get in touch.

I was asked at the end of last week to produce a graphic for the Opinion page today – the idea was to compare the texts of various ’state of the union’ addresses from around the world. The final result (pictured above) is not extraordinarily data-heavy. It worked quite nicely in the printed layout, where the individual ‘tentacles’ trailed to the text of the speeches that they index.

The process behind this piece was relatively simple. Each speech was indexed using a Processing application that I wrote which counts the frequency of individual names (the program ignores commonly used or unimportant words). The words for each speech were then ranked by mentions per thousand words (you can see a version of the piece with numbers here)

Almost every project I work on involves a period of ‘data exploration’ in which I try shake as many interesting things out of the information as I can. Even though this piece had a short turn-around, I did a fair amount of poking around, generating some simple bar graphs:

Another avenue I explored was to use the word weights to determine a ’score’ for each sentence. By doing this, I can try to find the ‘kernel’ of the speech – the sentence that sums up the entire text in the most succinct way. This, I think was fairly successful. Here are the ‘power sentences’ for the UK:

The Netherlands:

And Botswana:

Which brings us to tonight’s State of the Union Address by Barack Obama. What was the ‘power sentence’ from this speech? I ran the weighting algorithm on the address and this is what it came up with:

Clarify some misunderstandings

I published a post last week comparing Haiti aid per capita to Avatar ticket prices. The post got a lot of attention, and the figures and general concept were cross-posted and re-hashed in many places. Some people seemed to have misunderstood the post, though, and thought that I was comparing the contributions of individual governments to the production costs of Avatar. This is not what I did.

To get my figures for ‘Avatar minutes’ I started with the total aid contribution for a country, and divided it by that country’s population to get a per-capita aid figure. I then calculated how many minutes of Avatar that per-person contribution would pay for, using a ticket price of $8.50 (with a running time of 162 minutes, an ‘Avatar minute’ is about 5.25 cents). So, with Canada’s aid contribution of $5.5M, and a population of 33.3M, the per-person donation is about 3 Avatar minutes. Now, before any of you angry Canadians start frothing at the mouth, let me

Update some figures

When I published by post last week, I used the data that was then available. Many people commented about my use of the figure $5.5M for Canada, since very shortly after the post it was announced that the Canadian government was drastically increasing their Haiti aid contributions, and at the same time stated that they would match Canadian citizen’s contributions dollar-for-dollar, with no capping amount. I highlighted Canada in my post not to shame the government, but because I live in Canada. Again, I used the data available. I promised at the time to update the figures as more information became available, so, without further ado:

• Canada: 74 minutes
• Sweden: 47 minutes
• Norway: 41 minutes
• Denmark: 39 minutes
• Luxembourg: 28 minutes
• Finland: 27 minutes
• Guyana: 25 minutes
• Spain: 19 minutes
• Estonia: 14 minutes
• Australia: 12 minutes
• Ireland: 12 minutes
• Switzerland: 11 minutes
• USA: 10 minutes
• France: 9.5 minutes
• Germany: 5 minutes
• Netherlands: 5 minutes
• Italy: 3 minutes
• Japan: 1 minute

The contributions pledged by the Canadian government are impressive. But the point of the original post was not to single out any individual country for either congratulation or condemnation. Instead, it was to take the figures and put them into some kind of context.

$130,733,775 is a lot of money. Really. But our measure of amounts always depends on what context we put the numbers in. $130 million is a lot of money when compared to my yearly income. But it’s not that much money compared to the 2010 olympic budget – $1,700 million for a two-week sporting event. It’s just under half of the estimated production costs of Avatar ($280M). It’s less than 4% of Canada’s foreign aid budget.

If we add up ALL of the contributions to Haiti Aid, we get an even bigger amount of money – $1.75 billion dollars. A huge amount, to be sure, but again, a number that needs to be looked at in context. $1.75B is just a little bit less than Avatar has made in global ticket sales. It’s about 50% of Canada’s foreign aid budget, and 0.25% of last year’s monstrous US financial bailout. It is, repeating myself from the last paragraph, pretty much exactly what Vancouver is spending on next month’s winter games.

All of this mention of Canada and foreign aid may have already have tipped you off that I’d like to

Talk about Canadian Foreign Policy

Canada’s foreign aid budget is $3.45B, or about 0.25% of Canada’s GDP. Compare that to the Danes, who spend 0.83% of their GDP on aid (up this year from 0.82%, despite a record forecast deficit), or to the Swedes who spend about 0.92%. Canadians like to believe that we are a shining example of global citizenry, but largely this is an artifact of the pre-Mulroney governments of the 1970s and 1980s. The Center for Glocal Development ranked Canada 11th in their Commitment to Development Index from 2009, behind countries like Sweden, Denmark, Netherlands, Ireland, Spain, and Australia.

This index includes factors like aid, trade, investment, and migration. As the report notes, our migration levels of unskilled immigrants from developing countries has changed very little since the 1980s (we rank 11th on the list for migration). 

Like many other Canadians I grew up feeling proud about my country and about our role in the world. Unfortunately, the more I look into the actual figures, I realize that we have in many ways failed to maintain these ideals in the last 30 years.

I hope that the Canada’s actions on Haiti mark a change for our government (and not, say, a convenient way to buy some much-needed PR). I would like nothing more than to see Canada return to the role of the good global citizen. In the meantime, I will continue watching the government’s record with a deserved amount of criticality.

Speaking of criticality, let me finish this post by taking a moment to

Respond to some criticism

Jen Stirrup wrote a nicely detailed blog post in response to my Avatar/Haiti piece, in which she argues that the visualization puts beauty in advance of clarity. If we take the images that I used in the post as examples of data visualizations, I can’t help but agree. However, these images weren’t intended to be stand-alone graphics. Instead, they are screenshots of an animated, interactive visualization tool that I built to explore the data. As is very often the case when I work with data, I wrote a little program using Processing which was constructed specifically to deal with this data. I use the term ‘little’ here to emphasize the fact that it was a quick project – from the time that I had the idea to the time when I pressed ‘publish’ last Sunday was about 4 hours.

I would love to develop a workflow to take these interactive visualization tools to a stage where they can be shared more easily – at this point they tend to sit around while I harbour the best intentions to clean up the code enough for a proper release. In the meantime I can say that if you ask nicely, I’m usually willing to share my messy pre-release code. I will also be posting a brief video which might give you a better feel for how the project behaves – which, for the sake of continuity, I’ll title ‘Remove a rock from a horse’s hoof’

We’ve heard a lot this week about earthquake aid for Haiti. As is always the case when large numbers are bandied about in the news media, it’s hard to get a feeling of scale. For example, Canada has, at the time of writing, pledged to donate nearly 5.5M dollars to the aid effort. What does this number really mean? Well, considering Canada’s population of 33.3M, the aid works out to about 16 cents per Canadian citizen. 16 cents doesn’t buy you much these days. A sip of coffee, or – say – 3.14 minutes of Avatar; barely enough to get through the credits.

How many Avatar minutes are governments around the world pledging? Sweden leads the way, with almost 38 minutes per citizen – almost a quarter of the movie. Other Scandinavian countries round out the top 6, along with Luxembourg, Guyana, and Estonia.

Here are the times for some other countries:

• Sweden: 38 minutes
• Luxembourg: 28 minutes
• Denmark: 26 minutes
• Guyana: 25 minutes
• Norway: 20 minutes
• Estonia: 14 minutes
• Australia: 8 minutes
• Finland: 6 minutes
• United States: 6 minutes
• Switzerland: 5 minutes
• New Zealand: 4 minutes
• Netherlands: 3 minutes
• United Kingdom: 3 minutes
• Canada: 3 minutes
• Spain: 2 minutes
• Brazil: 2 minutes
• Germany: 1 minute
• Japan: 1 minute
• Morocco: 1 minute
• Poland: 1 minute
• Italy: 1 minute

The images in this post are exports from a Processing tool that I built to manage the data and to render the film strips. The application reads data from a Google spreadsheet – the original data was published by the always excellent Guardian Data Blog. If there’s enough interest, I will post the tool and the source later this week.

I was very much moved by Maggie Steber’s photo essay in The New York Times, titled ‘No End of Trouble. Ever.‘

The essay talks about Haiti’s violent history, and of the countries incredible tendency towards misfortune:

“How can nature or God or the fates or the universe do this to a country that has borne far too much sadness? An earthquake has now devastated the capital; claiming lives, hopes and the pitifully small dreams that people have held on to, despite political violence, unimaginable poverty, disease, corruption, dictators and nature’s full force of four hurricanes in a row.”

I built this very quick visualization to explore this topic a little further. Specifically, I wanted to compare Haiti to its Caribbean neighbours to see if the country is indeed as unlucky as it seems.

This visualization compares Haiti to 12 other Caribbean nations. It looks at articles published in the New York Times mentioning those countries between 1981 and 2010, and measures the occurence of specific words in those articles.

The pie charts in each row show the percentage of total articles on each country which contain the words in question. For example, we can see that about 25% of articles published about Haiti mention the word ‘violence’ – twice the frequency of any other country on the list.

Haiti has the highest frequency of the words ‘coup’, ‘violence’, ‘disease’, and ’strife’. It is second or third in mentions of ‘death’, ‘unrest’ and ‘famine’.

Likely this week’s events will lead to many more mentions of these words. As you’re likely aware, many NGOs small and large are organizing to help Haitians – both through emergency assistance and through long-term rebuilding. If you want to donate, I’d highly recommend considering Architecture for Humanity (for long-term projects) or Partners in Health (for emergency assistance). Both organizations are accepting donations through their websites.

Here is the data currently available, representing decades of data from over 1,500 land stations. As you can read on the linked page, the Met is at work to get more of this data released as soon as possible. There is some urgency here – the hope is that hard, un-deniable numbers might finally put some of the ‘debate’ surrounding the issue to rest.

Manuel Lima from VisualComplexity wrote a convincing blog post today, suggesting that the data community (how’s that for a general grouping?) can offer a lot to this cause. I couldn’t agree more. The general public certainly won’t gain much from this pile of strangely formatted text files – but they might be swayed by some well built, innovative visualizations that communicate and convince. Certainly, we can do better than the current graphics:

In order for this to be effective, I’d suggest three things are necessary:

1) Easy Access. I would love to see the data set placed into some format which is easily accessible, to save the work of everyone having to parse the data individually. Google Spreadsheets? MySQL tables? JSON? All of the above? Edit – mySQL tables are now available, along with a Perl parsing script, in the climate data forum (http://climatedata.blprnt.com)

2) Coordination. It would be useful to have a central place for people working with the data to ask questions and to share results. Ideally, a repository of graphics and interactive tools could be made available to the public and to the press.

3) Dialogue with Climate Scientists. The IPCC has more than 2500 expert reviewers, 800 contributing authors, and 450 lead authors. These people know what information needs to be shown, and what stories need to be told. Any effective effort to produce visualizations from this data would benefit from their input.

How does this start? As a quick measure to help with suggestion #2, I’ve created an open forum where we can start a dialogue, discuss some of these questions, and hopefully come up with some answers. For now, you can access the forum here:

http://climatedata.blprnt.com

Please pass on this invitation to any data-folks you might now – and of course any climate scientists, journalists, or other curious types who might want to get involved.

I spent a little bit of time today working on my text comparison tool, which I built last weekend to satisfy my curiosity about the similarities and differences between two very similar articles published on head injuries in the NFL (you can read the post here). I wanted to test out the tool with a different kind of content, and settled on something more political: two high profile foreign policy speeches by US President Barack Obama.

The first speech is Obama’s famous open address to the Muslim world, given in July at the University of Cairo. The second is much more recent – yesterday’s speech delivered at the Suntory Hall in Tokyo. As you might expect, the two speeches share a lot of common language. Here is the big picture, showing the top 100 words:

The shared words – ‘america’,'world’,'common’,'human’,'responsibility’, ect don’t offer much in the way of analysis. Things start to get interesting, though, when we look towards the edges (click on the images to see larger versions):

At the far extremes, the speech in Cairo was about Islam, about Palestinians, about peace, faith, and communities. The Tokyo address was about China, North Korea, security, agreement and growth. If we look at some of the common words that were used in both speeches, we can see some more interesting patterns emerge.

It seems, for instance that the Egyptian address was more about people, whereas the speech in Tokyo was directed towards nations:

Obama makes many more mentions about peace in Cairo (in Japan, this word seems to have been replaced by ’security’), and far more mentions of prosperity in Tokyo:

There was a lot of speculation prior to Obama’s speech in Asia about how much focus the President would put on human rights. In the speech, Obama mentions ‘rights’ only five times – once at the beginning of the speech and four times near the end. This weighting is interesting when we compare it to Obama’s reference to China during the speech, which is heavily concentrated at the beginning (China is not mentioned at all past the half-way point of the speech):

Though one of the five occurrences of ‘rights’ is in reference to China, it appears from this analysis that there may have been a deliberate plan to keep the ‘human rights part’ of the speech separated from the ‘China part’.

There likely many interesting things in this data set, a lot of which are open to interpretation. While it’s doubtful that one can steer entirely clear of political biases during this kind of comparison, the quantitative nature of the data makes it a little bit easier to make an attempt at nonpartisan analysis. I will be including these speeches as sample texts when I release the tool to the public (hopefully next week).

In October, I read a fascinating article on GQ.com about head injuries among former NFL players. Written by Jeanne Marie Laskas, the article was a forensic detective story, documenting a little known doctor’s efforts to bring the brain trauma issue to the attention of the medical community, the NFL, and the general public. It is a great read – an in-depth investigative piece with engaging personalities and plenty of intrigue.

A few weeks later, I picked up a copy of The New Yorker on my way home from Pittsburgh. I was surprised to see, on the cover, a promo for an article by Malcolm Gladwell about – you guessed it – brain trauma and the NFL. After having read both articles, I was surprised by how much these two investigative pieces differed. At the time I thought about doing a visualization to investigate, but somehow the idea slipped out of my head.

Until this weekend. I spent a few (okay, more like eight) hours putting together a tool with Processing that would examine some of the similarities and differences between the two articles. The most interesting data ended up coming from word usage analysis (I looked at sentences and phrases as well, but with not much luck). The base interface for the tool is a XY chart of the words – they are positioned vertically by their average position in the articles, and horizontally by which article they occur in more. The words in the centre are shared by both articles. Total usage affects the scale of the words, so we can see quite quickly which words are used most, and in which articles.

By focusing our attention on the big words which lie more or less in the center, we can see what the two articles have in common: brains, football, dementia, and a disease called CTE. What is perhaps more interesting is what lies on the outer edges; the subjects and topics that were covered by one author and not by the other.

Laskas’ article is about Dr. Bennet Omalu, dead NFL players (Mike Webster), Omalu’s colleagues (Dr. Julian Bailes & Bob Fitzsimmons) and the NFL (click on the images to see bigger versions):

Gladwell’s article, on the other hand, focuses partly on another scientist, Dr. Ann McKee, the sport of football in general, as well as s central metaphor in his piece – a comparison between football and dogfighting (the bridge between the two is Michael Vick):

The gulf between the two main scientific personalities profiled in the articles is interesting. Omalu and McKee are both experts in chronic traumatic encephalopathy (CTE) so it makes sense that they each appear in both articles (Omalu was the first to describe the condition; McKee. However, we see when we isolate these names that Laskas references Dr. Omalu almost exclusively (Omalu is mentioned 96 times by Laskas and only 6 times by Gladwell)* – it’s worth noting here that the Laskas article is 11.4% longer than the Gladwell piece – JT:

In contrast, Laskas only refers to McKee once (Dr. McKee is mentioned by Gladwell 21 times):

What is the relationship between Dr. McKee and Dr. Omalu? McKee is on the advisory board for the Sports Legacy Institute, a group which studies the results of brain trauma on athletes. SLI was founded by four individuals, including Bennet Omalu and the group’s current head, Chris Nowinski, a former professional wrestler. Omalu and the other three founders of SLI have now left the group, but it apparently continues to be a high-profile presence in the CTE field. Laskas writes:

“Indeed, the casual observer who wants to learn more about CTE will be easily led to SLI and the Boston group. There’s an SLI Twitter link, an SLI awards banquet, an SLI Web site with photos of Nowinski and links to videos of him on TV and in the newspapers. Gradually, Omalu’s name slips out of the stories, and Bailes slips out, and Fitzsimmons, and their good fight. As it happens in stories, the telling and retelling simplify and reduce.”

I wonder how much the path of an journalistic piece is affected by who you talk to first? If I had to guess, I’d say Gladwell started with the SLI, whereas Laskas seemed to have began from Dr. Omalu. This single decision could account for many of the differences between the two articles.

Other word-use choices might also give insight into editorial positions. Laskas, for example, uses the term NFL (below, at left) a lot – 57 times to Gladwell’s 11. Gladwell, on the other hand, talks more about the sport in general, using the word ‘football’ (below, at right)  40 times to Laskas’ 23:

According to Laskas, Dr. Omalu has been roundly shunned by the NFL – they have attempted to discredit his research on many occasions (attention that has not been so pointedly focused on Dr. McKee and the SLI). Though both articles are critical of the League, it seems clear both from the article and the data that Laskas and GQ have taken a more severe stance – the addresses the NFL much more often, and with more disdain.

This exercise of quantitatively analyzing a pair of articles may seem like a strange way to spend a weekend, but it helped me to more clearly understand the differences between the two stories and to consider my reactions to each. I uncovered a few things that I hadn’t picked up at first, and at the same time was able to reinforce some of the feelings that I had after reading the two articles.

It was also another opportunity to build a quick, lightweight visualization tool dedicated to a fairly specific topic (though in this case the tool could be used to compare any two bodies of text). This strategy holds a lot of appeal to me and I think deserves attention alongside the generalist approach that we tend to see a lot of on the web and in data visualization. It seems to me that this type of investigative technique could be useful for researchers of various stripes.

I will be releasing source code for this project as well as compiled applications for Mac, Linux & Windows. In the meantime, here’s a short video of how the interface behaves:

Two Sides of the Same Story: Laskas & Gladwell on CTE & the NFL from blprnt on Vimeo.

Photo: Trevor Mills, Vancouver Art Gallery

Last month, Manuel Lima caused a bit of a stir in the data visualization world when he proposed a divide between Information Visualization and Information Art. This kind of phylogenic approach aside, how does data fit into a contemporary art context? To answer this question, it might be helpful to start with an investigation into contemporary artists who use or have used data within their practice. A few months ago, I featured an article on Mark Lombardi, whose data-centric illustrations have found their way into the collections of many major art galleries & museums. This time I’ll move away from 2-d form and look at a piece by Canadian artist Brian Jungen.

Brian Jungen’s 2001 piece Isolated Depiction of the Passage of Time consists of hundreds of plastic cafeteria trays, stacked neatly on a wooden shipping pallet. As is often the case with Jungen’s work, this piece is more complex than it may at first appear. Start with the wooden pallet, which has been hand-carved from red cedar. Above it, the trays themselves are also carefully considered: the number and colour of the trays correspond to the population of Aboriginal males incarcerated in Canada’s prisons. This block of trays looks solid, but it is in fact a hollow container; in the center a TV plays, showing a looping series of daytime television programs. Finally, the entire work references a historical event, in which an inmate at Ontario’s Millhaven Institution escaped inside a makeshift structure of stacked trays.

I think it’s possible to understand Isolated Depiction of the Passage of Time, at least in part, as a data visualization. In general, Jungen’s practice resolves around disassembly and re-assembly – both key parts of any visualization project – so it is not too much of a surprise to see him treading into this territory. Though some of his later work involves data in more abstract ways (his famous Prototypes for New Understanding were made in a strictly numbered edition corresponding to Michael Jordan’s number; other works involve the construction of new maps from boundary data), Passage of Time remains his most data-centric work.

A major exhibition of Brian Jungen’s work is currently on show at the National Museum of the American Indian in Washington. An excellent introduction to Jungen’s work can be found on the NMAI’s website, including a brilliant essay by museum curator Paul Chaat Smith.

Project: Variance
Date: January, 2007
Language: ActionScript 2
Key Concepts: Genetic Algorithms

Overview:

I’ll admit – this is some fairly dusty code. But I wanted to end this 7 (or 25) days of source release with this project for a reason: I think it’s a pretty good idea. I took it to a prototype stage but not much further – maybe someone will find the time to move it forward into something more functional.

Variance is a tool that lets you evolve graphics, from pre-built ’sets’ of elements and typefaces. It combines some simple editing tools (you can manually change colour, position, rotation, etc.) with some evolutionary magic – you can select two different compositions and hybridize them to get a new set of graphics that share some of the properties of their parents. It works fairly well; usually within about 6 generations you can get a graphic that starts to look pretty good (starting from some jumbled random compositions).

When I was building this tool, I imagined how nice it would be if this kind of functionality was built into a ‘traditional’ tool like Illustrator. Working on a logotype, and are feeling a bit stuck? Have the application generate 9 mutations of your current design, then evolve those for a few generations until you get a result you like. Just getting started on a poster design? Pick some assets and have the Variance plug-in generate some possibilities. In both of these cases, the designer is still being a designer – picking the best results, tweaking layouts – but they are letting the genetic algorithms do the grunt work of coming up with possible variations. At the same time, they are allowing for mutations; often modifications that they may not have tried themselves. This is a very direct application of an Evolutionary Creative Process (ECP), which I’ve talked about before on this blog.

Utility aside, it’s fun to play with.

Getting Started:

You can see Variance in action here: http://www.blprnt.com/variance

To read more about the project and to get a walk-through, you can read this post (you may want to do this before you try the application)

You’ll need Flash to get into the source and to get working with it.

Download: VarianceSource(4.12 MB)

This software is licensed under the CC-GNU GPL version 2.0 or later.