The Big Data talent hunt
When it comes to attracting skilled technology workers, Big Data can seem like a Big Game Hunt. In a survey conducted in August 2012 by the 1105 Government Information Group, more than half of nearly 200 government agencies expressed difficulty finding and keeping knowledge workers and data scientists for Big Data efforts (see fig.1).
Granted, hiring is a challenge in other technology areas as well. But Big Data presents special issues because the process of using Big Data requires significantly different skills than traditional data projects. It’s not a case of building databases around queries to answer common questions. Big Data is about having the creativity and savvy to react when you don’t know what the questions will be.
“Most of these tasks are unfamiliar to everyone,” says Thomas Redman, a consultant and author of "Data Driven: Profiting from Your Most Important Business Asset." “You will need different people who understand the implications of discovery. The business of getting the right kind of people throughout the whole process will be the hardest part of Big Data. You will need managers who can keep the analysts happy and pointed in the right direction.”
Over the next two years, 36 percent of agencies will increase hiring for Big Data projects. The 1105 Government Information Group survey found that agencies will allocate their spending differently, which may affect their hiring needs. For example, Department of Defense (DOD) projects will focus spending on agile and flexible networking for high-velocity data, while state and local governments will invest in additional server hardware.
The larger and more diverse datasets, which require these different management and analytical skills, will foment demand for data scientists and data architects, according to the survey (see figure 2). The Harvard Business Review recently declared that “data scientist” will be “the sexiest job title of the 21st century.”
“These people are hard to find because their skill sets require the intersection of several large domains, like statistics, computer science and machine learning,” says Michael Daconta, an author and former metadata program manager for the Homeland Security Department.
Data scientists often have advanced degrees in computational physics or biology, certain social sciences, or computer science. But they also need the business savvy to interact with non-technologists from other departments. The role requires, in the words of HBR, a delicate mix of “training and curiosity.”
At the same time, industry observers note that Big Data is so new that many agencies may not even realize all the new types of skills they will require. Eric Sweden, program director for enterprise architecture and governance for the National Association of State Chief Information Officers (NASCIO), foresees needing people with skills in monitoring cloud service level agreements since cloud computing is often used in tandem with Big Data projects. “You need to make sure you are not overpaying,” he says. “We are getting really good in using predictive analytics to reduce overcharging.”
In addition, government agencies may need some skills that private sector doesn’t require. “There are a lot of challenges in the government around the appropriate use of data to make sure privacy is protected,” says Bob Gourley, former CTO of the Defense Intelligence Agency (DIA) and founder of Crucial Point LLC, a technology research and advisory firm. “The ethics of data will require skills and approaches in government that a lot of industries don’t have and don’t care about.”