Big Data

Exploiting the data revolution

abstract head representing big data


Digital technologies have irrevocably changed business in recent years, and life in the era of big data inherently signifies that business decisions may never again lack for data.

But this big-data age, where crunching petabytes isn’t uncommon and compute can be purchased on-demand, is just "the warm-up act," according to Andrew McAfee, co-author of "The Second Machine Age: Work, Progress and Prosperity in the Age of Machines.”" 

"We’re nowhere near done with innovation and transformation, and I firmly believe during our lifetimes, we’ll witness the biggest transformation since the Industrial Revolution,” said McAfee, speaking at VMWare’s Government Innovation Summit on March 6. McAfee is a principal research scientist at the MIT Sloan School of Management.

"The Industrial Revolution helped us overcome the limitations of our muscles. We are now overcoming the limitations of our minds," McAfee said. "Over the next 50 years economies will go through the biggest change since the Industrial Revolution."

Having ridden in Google’s autonomous car and hung out with Baxter, a robot billed to have behavior-based common sense, McAfee said advances in digital technology, robotics and automation are beginning to change the world.

The changes have been rapid, limited only it seems by Moore's Law, money and imagination.

Jeopardy champion Ken Jennings was defeated by IBM's Watson less than two decades after former chess champion Garry Kasparov was defeated by IBM's Deep Blue computer, highlighting an important point: humans can now build machines that are better at tasks outside of arenas like number-crunching. Machines have always owned number-crunching. Now  they’re getting better than humans at other activities.

Machines can produce and sift through increasingly deep seas of data as fast as people can blink, leading to all sorts of "weird performance changes" that apply across the private and public sectors.

But the biggest changes occur not from machines or people, McAfee said, but from the combination of people and technology. McAfee used freestyle chess as an example. While the fastest chess-playing supercomputer can defeat the best human player, that supercomputer is dominated by a team of chess players and a slower computer.  

"[Machines and people] don’t have to be head to head. When we bring people and technology together correctly, the results are amazing," McAfee said.

The most obvious example in IT of teamwork between humans and machines is the use of computer algorithms to make predictions and data-driven decisions.

McAfee referenced Nate Silver, formerly of the New York Times, whose data-driven model predicted the presidential race correctly in all 50 states in 2012. Machines are great at producing data, compiling it and storing it, but there still is no substitute for human ingenuity in finding what best to do with data. And increasingly, organizations are releasing their big datasets to "the geeks of the world" for help. Allstate, for example, ran a 12-week contest for data nerds in hopes of improving its liability predictive prowess. In three months, the data teams had improved Allstate’s predictive baseline by 340 percent.

"In the era of big data that we're in, data is the lifeblood of science. It is how you get smarter about the world," McAfee said.

Because of this influx of technology and data, organizations and federal agencies face enormous opportunities. At the same time, they face daunting challenges handling the kind of disruption that’s only going to make the IT world seem more chaotic. Government sometimes struggles with innovation because of its traditionally slow-moving procurement process, while technology seems to evolve every few months.

"How we react to [the disruption] will be the defining issue for businesses, organizations, the economy and society for the rest of our lives," McAfee said. "Let's not screw this up, shall we?"


About the Author

Frank Konkel is a former staff writer for FCW.


  • Defense
    The U.S. Army Corps of Engineers and the National Geospatial-Intelligence Agency (NGA) reveal concept renderings for the Next NGA West (N2W) campus from the design-build team McCarthy HITT winning proposal. The entirety of the campus is anticipated to be operational in 2025.

    How NGA is tackling interoperability challenges

    Mark Munsell, the National Geospatial-Intelligence Agency’s CTO, talks about talent shortages and how the agency is working to get more unclassified data.

  • Veterans Affairs
    Veterans Affairs CIO Jim Gfrerer speaks at an Oct. 10 FCW event (Photo credit: Troy K. Schneider)

    VA's pivot to agile

    With 10 months on the job, Veterans Affairs CIO Jim Gfrerer is pushing his organization toward a culture of constant delivery.

Stay Connected


Sign up for our newsletter.

I agree to this site's Privacy Policy.