“We will unleash a tsunami of data”. So said our Downing Street contact in the weeks after the election.
And so it’s proven to be, from the numbers of civil servants and what they get paid (http://www.guardian.co.uk/news/datablog/2010/jun/18/civil-service-statistics-headcount) to the salaries of Westminster advisors (http://www.guardian.co.uk/news/datablog/2010/jun/13/government-special-advisers-list). Those would have been enough to keep us happy, but then we got COINS – the Treasury’s full record of public spending. These enormous CSV files would crash most machines, so our developers were busy full time to turn them into something you can navigate – you can see the COINS explorer here. http://www.guardian.co.uk/news/datablog/2010/jun/04/coins-database-search
When we launched the Datablog (http://www.guardian.co.uk/datablog) a year ago, we had no idea who would be interested in raw data, statistics and visualisations. As someone in my office said: “why would anyone want that?”
Well, the last year has answered that question for us. It has been an incredible one for public data. Obama opened up the US government’s data faults as his first legislative act (http://www.data.gov/), followed by government data sites around the world – Australia (http://data.australia.gov.au/), New Zealand http://www.data.govt.nz/, the British government’s Data.gov.uk and of course the London datastore.
It’s also a year in which data has made the news contstantly. We’ve had the MPs expenses scandal – Britain’s most unexpected piece of datajournalism – the resulting fallout of which has meant Westminster is now committed to releasing huge amounts of data every year.
We have had a general election where each of the main political parties was committed to data transparency, opening our own data vaults to the world.
And, as spending cuts start to bite, the role of using statistics to show what’s really going on around the country in peoples’ jobs and homes has never been more important.
At the same time, as the web pumps out more and more data, readers from around the world are more interested in the raw facts behind the news than ever before. When we launched the Datablog, we thought the audiences would be developers building applications. In fact, it’s people wanting to know our carbon emissions or Eastern European immigration to the UK or the breakdown of deaths in Afghanistan – or even the number of times the Beatles used the word Love in their songs (613).
It’s not that long ago that journalists were the gatekeepers to official data. We would write stories about the numbers and release them to a grateful public, who – by and large – were not interested in the raw statistics. The idea of us allowing our raw information into our newspapers was anathema.
Now that dynamic has changed beyond recognition. Our role is becoming interpreters, helping people understand the data – and even just publishing it because it’s interesting in itself.
But numbers without analysis are just numbers – which is going to be where we – with your help – fit in.
As the demand for us to use the data increases, so will the demand for people who have the skills to ensure that we are all analysing that data properly. We will need people to keep us on track.
In the very first issue of the Manchester Guardian, Saturday 5 May, 1821, the news was on the back page, like all papers of the day. First item on the front page was an ad for a missing Labrador.
And, amid the stories and poetry excerpts, a third of that back page is taken up with, well, facts. A comprehensive table of the costs of schools in the area never before “laid before the public”, writes “NH”.
NH wanted his data published because otherwise the facts would be left to untrained clergymen to report. His motivation was that: “Such information as it contains is valuable; because, without knowing the extent to which education … prevails, the best opinions which can be formed of the condition and future progress of society must be necessarily incorrect.” In other words, if the people don’t know what’s going on, how can society get any better?
I can’t think of a better rationale now for what we’re trying to do.