Big Data

Exploiting the data revolution

abstract head representing big data


Digital technologies have irrevocably changed business in recent years, and life in the era of big data inherently signifies that business decisions may never again lack for data.

But this big-data age, where crunching petabytes isn’t uncommon and compute can be purchased on-demand, is just "the warm-up act," according to Andrew McAfee, co-author of "The Second Machine Age: Work, Progress and Prosperity in the Age of Machines.”" 

"We’re nowhere near done with innovation and transformation, and I firmly believe during our lifetimes, we’ll witness the biggest transformation since the Industrial Revolution,” said McAfee, speaking at VMWare’s Government Innovation Summit on March 6. McAfee is a principal research scientist at the MIT Sloan School of Management.

"The Industrial Revolution helped us overcome the limitations of our muscles. We are now overcoming the limitations of our minds," McAfee said. "Over the next 50 years economies will go through the biggest change since the Industrial Revolution."

Having ridden in Google’s autonomous car and hung out with Baxter, a robot billed to have behavior-based common sense, McAfee said advances in digital technology, robotics and automation are beginning to change the world.

The changes have been rapid, limited only it seems by Moore's Law, money and imagination.

Jeopardy champion Ken Jennings was defeated by IBM's Watson less than two decades after former chess champion Garry Kasparov was defeated by IBM's Deep Blue computer, highlighting an important point: humans can now build machines that are better at tasks outside of arenas like number-crunching. Machines have always owned number-crunching. Now  they’re getting better than humans at other activities.

Machines can produce and sift through increasingly deep seas of data as fast as people can blink, leading to all sorts of "weird performance changes" that apply across the private and public sectors.

But the biggest changes occur not from machines or people, McAfee said, but from the combination of people and technology. McAfee used freestyle chess as an example. While the fastest chess-playing supercomputer can defeat the best human player, that supercomputer is dominated by a team of chess players and a slower computer.  

"[Machines and people] don’t have to be head to head. When we bring people and technology together correctly, the results are amazing," McAfee said.

The most obvious example in IT of teamwork between humans and machines is the use of computer algorithms to make predictions and data-driven decisions.

McAfee referenced Nate Silver, formerly of the New York Times, whose data-driven model predicted the presidential race correctly in all 50 states in 2012. Machines are great at producing data, compiling it and storing it, but there still is no substitute for human ingenuity in finding what best to do with data. And increasingly, organizations are releasing their big datasets to "the geeks of the world" for help. Allstate, for example, ran a 12-week contest for data nerds in hopes of improving its liability predictive prowess. In three months, the data teams had improved Allstate’s predictive baseline by 340 percent.

"In the era of big data that we're in, data is the lifeblood of science. It is how you get smarter about the world," McAfee said.

Because of this influx of technology and data, organizations and federal agencies face enormous opportunities. At the same time, they face daunting challenges handling the kind of disruption that’s only going to make the IT world seem more chaotic. Government sometimes struggles with innovation because of its traditionally slow-moving procurement process, while technology seems to evolve every few months.

"How we react to [the disruption] will be the defining issue for businesses, organizations, the economy and society for the rest of our lives," McAfee said. "Let's not screw this up, shall we?"


About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.


  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group