Companies collect inordinate volumes of data in an ever-more connected global economy, where technology is facilitating the accumulation of information regarding individuals, and data is readily available from other sources that give an even richer picture into peoples lives. But, the hidden gems within this big data remain hidden unless the talents of a select group of individuals can unearth them. Hal Varian himself said that this “essentially free and ubiquitous data” needs data scientists to “extract value from it”. A relatively scarce supply of people with these skills exist, and that has created a significant gap in demand for the data scientist in today’s job market. It is little wonder then, that ‘Data Mining’ was ranked as LinkedIn’s “hottest skill of 2014”.
There is often a reference to the ever-widening “talent gap” between the demand for big data experts, and those in the labour market who can offer such skills to employers. McKinsey report that by 2018, the US economy will face a shortage of 140,000 to 190,000 people who have the skills to perform “deep analytics” of data; furthermore, the US will lack “1.5million managers and analysts with the know-how to use the analysis of big data to make effective decisions”. This shortage is beginning to visibly manifest itself in several indicators.