What will big data disrupt in five years?

Examples of big data being harnessed at the federal level abound.

Every piece of mail sent in America is photographed, monitored and analyzed by supercomputers; the Centers for Disease Control and Prevention mines social media data to track and predict disease outbreaks in real time, and perhaps the most complex big data operation on Earth, maintained by the National Security Agency, analyzes vast amounts of Internet data and phone records for foreign signals intelligence.

Yet we are squarely in the early stages of the big data bang, and developments today offer a glimpse of how ingrained in federal operations big data might be in the next five to 10 years, according to several industry and government experts speaking during Meritalk's Big Data Forecast webinar Aug. 13.

Dr. Sasi Pillay, chief technology officer in NASA's Office of the CIO, said the space agency is beginning to switch from a reactive model to a predictive one regarding big data—or in an analogy he offered drawing from science, moving from "earthquake reactionary" to  "tornado watch." Under the new model, algorithms, modeling and simulations – complete with advanced visualization tools – are helping the agency reduce testing of aeronautics equipment, potentially saving billions of dollars.

New aeronautics equipment comes with modern instrumentation that collects data multiple times per second, which ultimately helps NASA build better machines. Big data innovations are seeping into NASA's IT security division as well. The agency is "feeding in what is happening on social media" and applying that data stream to others, enriching information collected from isolated data points.

In addition, Pillay said, NASA is looking to big data to improve efficiency of airline traffic, comparing metrics such as airline routes, the density of flight patterns and fuel consumption. In some instances, NASA also makes use of large three-dimensional data visualizations, creating an "immersive" environment scientists "can walk around in."

Other fields are going to get a boost from big data, too.

Rich Campbell, chief technologist at EMC Federal, said big data will dramatically change how cyber-security is implemented and monitored as well as how digital forensics are carried out. In the broad scheme of federal agencies, Campbell said, efficiencies and processes will be streamlined through vast number crunching and smart algorithms.

Big data allows insights at the granular level, Campbell said, explaining how disparate data sets from a call center – how many customers called, how many claims were processed and how fast, and net results from calls – could be compared and used to make decisions.

Campbell also said the importance of metadata – commonly referred to as the "data about data" – will continue to grow. Appropriately tagged data could be shared across agencies, with organizations outside government or with the public itself. He called metadata a "big future trend."

"It's not just how we put our hands around it, but how we tag it appropriately" so it can be used, Campbell said.

The leading innovators of big data continue to be in the intelligence community, and there is little reason to doubt that advancements in cyber warfare will come through big data.

Sam Lee, data center practice director at Force 3, said cyber warfare efforts fit well in big data's space. The ability to store, sift and analyze huge amounts of Internet traffic at the point of attack is decidedly a big data effort, and could play an especially important role in the near future given the complexity of today's cyberattacks.

Big data is only going to get bigger in the future, too.

Pillay provided an outline of the Square Kilometer Array (SKA) project, which will create the world's largest telescope. When the telescope is operational – approximately a decade from now – it will create 1 million terabytes (one exabyte) of data per day, or double the amount of total worldwide Internet traffic each day. Managing that data won't be NASA's problem – the project doesn't involve the United States – but it will be as large a big data challenge as there's ever been.