blprnt:blpg

The Missing Piece of the OpenData / OpenGov Puzzle: Education

By Jer | March 7, 2010

Yesterday, I tweeted a quick thought that I had, while walking the dog:

A few people asked me to expand on this, so let’s give it a try:

We are facing a very different data-related problem today than we were facing only a few years ago. Back then, the call was solely for more information. Since then, corporations and governments have started to answer this call and the result has been a flood of data of all shapes and sizes. While it’s important to remain on track with the goal of making data available, we are now faced with a parallel and perhaps more perplexing problem: What do we do with it all?

Of course, an industry has developed around all of this data; start-ups around the world are coming up with new ideas and data-related products every day. At the same time, open-sourcers are releasing helpful tools and clever apps by the dozen. Still, in a large part these groups are looking at the data with fiscal utility in mind. It seems to me that if we are going to make the most of this information resource, it’s important to bring more people in on the game – and to do that requires education.

At the post-secondary level, efforts should be made to educate academics for whom this new pile of data could be useful: journalists, social scientists, historians, contemporary artists, archivists, etc. I could imagine cross-disciplinary workshops teaching the basics:

A survey of what kind of data is available, and how to find it.
A brief overview of common data formats (CSV, JSON, XML, etc).
An introduction to user-friendly exploration tools like ManyEyes & Tableau
A primer in Processing and how it can be used to quickly prototype and build specialized visualization tools.

The last step seems particularly important to me, as it encourages people to think about new ways to engage with information. In many cases, datasets that are becoming available are novel in their content, structure, and complexity – encouraging innovation in an academic framework is essential. Yes, we do need to teach people how to make bar graphs and scatter charts; but let’s also facilitate exploration and experimentation.

Why workshops? While this type of teaching could certainly be done through tutorials, or with a well-written text book, it’s my experience that teaching these subjects is much more effective one-on-one. This is particularly true for students who come at data from a non-scientific perspective (and these people are the ones that we need the most).

The long-term goal of such an initiative would be to increase data-literacy. In a perfect world, this would occur even earlier – at the highschool level. Here’s where I put on my utopian hat: teaching data literacy to young people would mean that they could find answers to their own questions, rather than waiting for the media to answer those questions for them. It also teaches them, in a practical way, about transparency and accountability in government. The education system is already producing a generation of bloggers and citizen journalists – let’s make sure they have the skills they need to be dangerous. Veering a bit to the right, these are hugely valuable skills for workers in an ‘idea economy’ – a nation with a data-literate workforce is a force to be reckoned with.

Ideally this educational component would be build in to government projects like data.gov or data.hmg.gov.uk (are you listening, Canada?). More than that, it would be woven into the education mandate of governments at federal and local levels. Of course, I’m not holding my breath.

Instead, I’ve started to plan a bit of a project for the summer. Like last year, I taught a series of workshops at my studio in Vancouver, which were open to people of all skill levels. This year, I’m going to extend my reach a bit and offer a couple of free, online presentations covering some of the things that I’ve talked about in this post. One of these workshops will be specifically targeted to youth. At the same time, I’ll be publishing course outlines and sample materials for my sessions so that others can host similar events.

Stay tuned for details – and if you have any questions or would like to lend a hand, feel free to leave a comment or get in touch.

Share:

3 Comments

State of the Union(s)

By Jer | January 28, 2010

I was asked at the end of last week to produce a graphic for the Opinion page today – the idea was to compare the texts of various ’state of the union’ addresses from around the world. The final result (pictured above) is not extraordinarily data-heavy. It worked quite nicely in the printed layout, where the individual ‘tentacles’ trailed to the text of the speeches that they index.

The process behind this piece was relatively simple. Each speech was indexed using a Processing application that I wrote which counts the frequency of individual names (the program ignores commonly used or unimportant words). The words for each speech were then ranked by mentions per thousand words (you can see a version of the piece with numbers here)

Almost every project I work on involves a period of ‘data exploration’ in which I try shake as many interesting things out of the information as I can. Even though this piece had a short turn-around, I did a fair amount of poking around, generating some simple bar graphs:

Another avenue I explored was to use the word weights to determine a ’score’ for each sentence. By doing this, I can try to find the ‘kernel’ of the speech – the sentence that sums up the entire text in the most succinct way. This, I think was fairly successful. Here are the ‘power sentences’ for the UK:

The Netherlands:

And Botswana:

Which brings us to tonight’s State of the Union Address by Barack Obama. What was the ‘power sentence’ from this speech? I ran the weighting algorithm on the address and this is what it came up with:

Share:

4 Comments

Haiti & Avatar – updates.

By Jer | January 26, 2010

This post is a bit of a swiss-army knife. Without being too long-winded, I’m going to clarify some misunderstandings, update some figures, talk about Canadian foreign policy, respond to some criticism and remove a rock from a horse’s hoof. To start, then, let’s

Clarify some misunderstandings

I published a post last week comparing Haiti aid per capita to Avatar ticket prices. The post got a lot of attention, and the figures and general concept were cross-posted and re-hashed in many places. Some people seemed to have misunderstood the post, though, and thought that I was comparing the contributions of individual governments to the production costs of Avatar. This is not what I did.

To get my figures for ‘Avatar minutes’ I started with the total aid contribution for a country, and divided it by that country’s population to get a per-capita aid figure. I then calculated how many minutes of Avatar that per-person contribution would pay for, using a ticket price of $8.50 (with a running time of 162 minutes, an ‘Avatar minute’ is about 5.25 cents). So, with Canada’s aid contribution of $5.5M, and a population of 33.3M, the per-person donation is about 3 Avatar minutes. Now, before any of you angry Canadians start frothing at the mouth, let me

Update some figures

When I published by post last week, I used the data that was then available. Many people commented about my use of the figure $5.5M for Canada, since very shortly after the post it was announced that the Canadian government was drastically increasing their Haiti aid contributions, and at the same time stated that they would match Canadian citizen’s contributions dollar-for-dollar, with no capping amount. I highlighted Canada in my post not to shame the government, but because I live in Canada. Again, I used the data available. I promised at the time to update the figures as more information became available, so, without further ado:

Canada: 74 minutes
Sweden: 47 minutes
Norway: 41 minutes
Denmark: 39 minutes
Luxembourg: 28 minutes
Finland: 27 minutes
Guyana: 25 minutes
Spain: 19 minutes
Estonia: 14 minutes
Australia: 12 minutes
Ireland: 12 minutes
Switzerland: 11 minutes
USA: 10 minutes
France: 9.5 minutes
Germany: 5 minutes
Netherlands: 5 minutes
Italy: 3 minutes
Japan: 1 minute

The contributions pledged by the Canadian government are impressive. But the point of the original post was not to single out any individual country for either congratulation or condemnation. Instead, it was to take the figures and put them into some kind of context.

$130,733,775 is a lot of money. Really. But our measure of amounts always depends on what context we put the numbers in. $130 million is a lot of money when compared to my yearly income. But it’s not that much money compared to the 2010 olympic budget – $1,700 million for a two-week sporting event. It’s just under half of the estimated production costs of Avatar ($280M). It’s less than 4% of Canada’s foreign aid budget.

If we add up ALL of the contributions to Haiti Aid, we get an even bigger amount of money – $1.75 billion dollars. A huge amount, to be sure, but again, a number that needs to be looked at in context. $1.75B is just a little bit less than Avatar has made in global ticket sales. It’s about 50% of Canada’s foreign aid budget, and 0.25% of last year’s monstrous US financial bailout. It is, repeating myself from the last paragraph, pretty much exactly what Vancouver is spending on next month’s winter games.

All of this mention of Canada and foreign aid may have already have tipped you off that I’d like to

Talk about Canadian Foreign Policy

Canada’s foreign aid budget is $3.45B, or about 0.25% of Canada’s GDP. Compare that to the Danes, who spend 0.83% of their GDP on aid (up this year from 0.82%, despite a record forecast deficit), or to the Swedes who spend about 0.92%. Canadians like to believe that we are a shining example of global citizenry, but largely this is an artifact of the pre-Mulroney governments of the 1970s and 1980s. The Center for Glocal Development ranked Canada 11th in their Commitment to Development Index from 2009, behind countries like Sweden, Denmark, Netherlands, Ireland, Spain, and Australia.

This index includes factors like aid, trade, investment, and migration. As the report notes, our migration levels of unskilled immigrants from developing countries has changed very little since the 1980s (we rank 11th on the list for migration).

Like many other Canadians I grew up feeling proud about my country and about our role in the world. Unfortunately, the more I look into the actual figures, I realize that we have in many ways failed to maintain these ideals in the last 30 years.

I hope that the Canada’s actions on Haiti mark a change for our government (and not, say, a convenient way to buy some much-needed PR). I would like nothing more than to see Canada return to the role of the good global citizen. In the meantime, I will continue watching the government’s record with a deserved amount of criticality.

Speaking of criticality, let me finish this post by taking a moment to

Respond to some criticism

Jen Stirrup wrote a nicely detailed blog post in response to my Avatar/Haiti piece, in which she argues that the visualization puts beauty in advance of clarity. If we take the images that I used in the post as examples of data visualizations, I can’t help but agree. However, these images weren’t intended to be stand-alone graphics. Instead, they are screenshots of an animated, interactive visualization tool that I built to explore the data. As is very often the case when I work with data, I wrote a little program using Processing which was constructed specifically to deal with this data. I use the term ‘little’ here to emphasize the fact that it was a quick project – from the time that I had the idea to the time when I pressed ‘publish’ last Sunday was about 4 hours.

I would love to develop a workflow to take these interactive visualization tools to a stage where they can be shared more easily – at this point they tend to sit around while I harbour the best intentions to clean up the code enough for a proper release. In the meantime I can say that if you ask nicely, I’m usually willing to share my messy pre-release code. I will also be posting a brief video which might give you a better feel for how the project behaves – which, for the sake of continuity, I’ll title ‘Remove a rock from a horse’s hoof’

Share:

3 Comments

Finding Perspective: Haiti Earthquake Aid in Avatar Minutes

By Jer | January 18, 2010

We’ve heard a lot this week about earthquake aid for Haiti. As is always the case when large numbers are bandied about in the news media, it’s hard to get a feeling of scale. For example, Canada has, at the time of writing, pledged to donate nearly 5.5M dollars to the aid effort. What does this number really mean? Well, considering Canada’s population of 33.3M, the aid works out to about 16 cents per Canadian citizen. 16 cents doesn’t buy you much these days. A sip of coffee, or – say – 3.14 minutes of Avatar; barely enough to get through the credits.

How many Avatar minutes are governments around the world pledging? Sweden leads the way, with almost 38 minutes per citizen – almost a quarter of the movie. Other Scandinavian countries round out the top 6, along with Luxembourg, Guyana, and Estonia.

Here are the times for some other countries:

Sweden: 38 minutes
Luxembourg: 28 minutes
Denmark: 26 minutes
Guyana: 25 minutes
Norway: 20 minutes
Estonia: 14 minutes
Australia: 8 minutes
Finland: 6 minutes
United States: 6 minutes
Switzerland: 5 minutes
New Zealand: 4 minutes
Netherlands: 3 minutes
United Kingdom: 3 minutes
Canada: 3 minutes
Spain: 2 minutes
Brazil: 2 minutes
Germany: 1 minute
Japan: 1 minute
Morocco: 1 minute
Poland: 1 minute
Italy: 1 minute

The images in this post are exports from a Processing tool that I built to manage the data and to render the film strips. The application reads data from a Google spreadsheet – the original data was published by the always excellent Guardian Data Blog. If there’s enough interest, I will post the tool and the source later this week.

Sweden: 38 seconds

Luxembourg: 28 seconds

Denmark: 26 seconds

Guyana: 25 seconds

Norway: 20 seconds

Estonia: 14 seconds

Australia: 8 seconds

Finland: 6 seconds

United States: 6 seconds

Switzerland: 5 seconds

New Zealand: 4 seconds

Netherlands: 3 seconds

United Kingdom: 3 seconds

Canada: 3 seconds

Spain: 2 seconds

Brazil: 2 seconds

Germany: 1 seconds

Japan: 1 seconds

Morocco: 1 seconds

Poland: 1 seconds

Italy: 1 seconds

Share:

39 Comments

Unlucky Haiti (1981-2009)

By Jer | January 14, 2010

I was very much moved by Maggie Steber’s photo essay in The New York Times, titled ‘No End of Trouble. Ever.‘

The essay talks about Haiti’s violent history, and of the countries incredible tendency towards misfortune:

“How can nature or God or the fates or the universe do this to a country that has borne far too much sadness? An earthquake has now devastated the capital; claiming lives, hopes and the pitifully small dreams that people have held on to, despite political violence, unimaginable poverty, disease, corruption, dictators and nature’s full force of four hurricanes in a row.”

I built this very quick visualization to explore this topic a little further. Specifically, I wanted to compare Haiti to its Caribbean neighbours to see if the country is indeed as unlucky as it seems.

This visualization compares Haiti to 12 other Caribbean nations. It looks at articles published in the New York Times mentioning those countries between 1981 and 2010, and measures the occurence of specific words in those articles.

The pie charts in each row show the percentage of total articles on each country which contain the words in question. For example, we can see that about 25% of articles published about Haiti mention the word ‘violence’ – twice the frequency of any other country on the list.

Haiti has the highest frequency of the words ‘coup’, ‘violence’, ‘disease’, and ’strife’. It is second or third in mentions of ‘death’, ‘unrest’ and ‘famine’.

Likely this week’s events will lead to many more mentions of these words. As you’re likely aware, many NGOs small and large are organizing to help Haitians – both through emergency assistance and through long-term rebuilding. If you want to donate, I’d highly recommend considering Architecture for Humanity (for long-term projects) or Partners in Health (for emergency assistance). Both organizations are accepting donations through their websites.

Share: