The City Intelligence Unit has published findings from research investigating the potential of new digital data sources to inform social policy analysis at City Hall.

We are undertaking a programme of work known as the Social Evidence Base to inform policy on issues such as social integration, economic fairness and social mobility – areas which are under-evidenced within our city. Too often, data available from existing sources have too small a sample size or are too out-of-date to understand a city with the superdiversity and complexity of London.

As part of this work we have been exploring how we can balance traditional sources of evidence with more innovative and sustainable forms. As citizens, we increasingly leave a trail of online ‘digital exhaust’ from our daily activities, such as using social media. We wanted to explore the potential of such data to provide new insight, and whether we could even reduce our dependence on more expensive forms of data collection. This work has similar aims to that of the ONS Data Science campus.

Despite the potential, there are serious challenges to using such data for policy making. As well as issues with the accessibility and usability of data, there are ethical issues in using data collected for one purpose for another, and these go beyond issues of individual privacy.

To understand how and where best to focus, we commissioned Demos to explore the potential of different sources to inform our work and to create a framework to help us understand the potential of each source to provide high quality evidence. The report, published today, shows the results of this work. It focuses on data available from new online platforms and services such as Foursquare and Trustpilot, excluding those where the GLA already has a good level of expertise (such as government administrative data).

The good news from the research is that there are sources of data that we can harness to add to our evidence base. These will provide a qualitative understanding of the issues faced by Londoners.

Following this report, we are undertaking three more experimental pieces of work using sources identified in the research:
• Exploring how social media data (in this case, twitter) can help our understanding of social integration. This project has been commissioned to IPSOS and is currently underway
• Understanding experiences of customer discrimination from rating websites
• Understanding Londoners’ experience of public and community spaces from reviews on Google Maps

We also are exploring the extent to which we can replicate some of the interesting academic work in this area. One study used review data from the business directory Yelp to understand consumer segregation. Another used google street view to map the spatial distribution of citizens to understand use of public space.

Somewhat less good news, but not a surprise, is that these sources are not yet mature enough to replace traditional sources. They are not representative and many marginalised population groups are excluded. In their report, Demos are clear that we still need traditional research for the time-being to benchmark findings and bring in a wider range of views, especially from those who don’t or can’t engage online. Our forthcoming Survey of Londoners, due to be published in June, will provide comprehensive quantitative data on the lived experience of a representative sample of 6,500 Londoners.

However, we are encouraged by the early signs of what digital data can offer and look forward to developing this work in future.