Understanding online job postings data

Introduction to the data

The data we are using is sourced from Lightcast (formerly Emsi Burning Glass), a leading supplier of real-time labour market information. Lightcast is a US-based company with a strong presence in the UK.

Lightcast’s online job postings data are collected by web-scraping job postings created by employers, including online job boards and company websites. The information retrieved goes through a process that standardises and classifies postings according to a number of data elements including location, occupation classification, and qualifications.

Prior to our analysis, Lightcast also carries out data cleaning such as identifying and accounting for duplicate job postings (e.g. because of multiple recruiters advertising at the same time) and applying minimum quality thresholds on some data fields.

Back to top

Strengths and limitations

The quantity of vacancy data collected via web-scraping is increasing as more job openings are posted on the internet. As a source of data, this has several key advantages. For example, the data from Lightcast are updated frequently, include a large volume of data, and cover a wide range of information besides occupation (or job title), including desired skills, location, and salary information.

This near real-time data can be used to track and assess the demand for labour in London, complementing more traditional sources of labour market information (LMI).

However, there are also challenges with using postings data from web-scraping. The Lightcast data should be treated with awareness of the following caveats:

• The number of job adverts being posted online is not a direct measure of labour demand: the number of online postings could respond to other changes such as how positions are recruited for (e.g. more jobs being recruited for online over time).

• Job adverts may not be removed from online vacancy boards immediately when the position is filled so the data may not fully reflect employers who have halted active recruitment; our analysis focuses on new postings within a given timeframe.

• The scope of online job postings does not fully capture the scope of economic activity in London; not all jobs are posted online – some jobs are only advertised internally, while others are advertised by word-of-mouth or in shop windows.

• The presentation and content of information listed in job postings is inconsistent and often hard to compare – some employers provide an extensive wish-list of skills for a given job role, while others list only those essential to performing in the role.

• In some cases, the location of a job posting is too high-level to be matched to a specific location; for example, many jobs are advertised at a ‘London’ level but cannot be assigned to a specific local authority area within the capital.

Back to top

Comparison with UK data

The ONS publishes official estimates of the stock of UK job vacancies based on a survey of UK employers, which is different to the method of collecting online job postings data performed by Lightcast.

• The ONS measure showed a steady increase in UK vacancies from 2012 until the beginning of the COVID-19 pandemic with only a minor decrease from early 2019 to 2020.

• By comparison, the Lightcast data on online job postings has been more volatile with notable dips from 2013 to 2015 and from 2018 to 2019.

• Both measures show steep falls after the introduction of lockdown measures in March 2020, followed by large increases from 2021 onwards.

Back to top

Index of dissimilarity

A dissimilarity index, as presented in this IFS report, shows how the composition of job postings by occupation has changed over time. The share of job postings is indexed to the average occupational share in the whole of 2019 and the index value ranges between 0 (no change) and 100 (complete change)*.

The occupational mix saw an abrupt change with the onset of the COVID-19 pandemic. The lockdowns likely lead to a fall in demand, and therefore job postings, for workers in certain industries which meant the mix changed significantly. The occupational mix has returned closer to the long-term trend in recent years, though it remains somewhat different from the 2019 average.

Back to top

*For more information, see the methodology section.

Data and methodology