Making Sense of the Data Deluge

Analytics puts agencies ahead of threats, informs decision-making

That security camera connected to your network? It’s a rich source of data. That email log? Another great data source. The same is true of transactions, social media posts, VPN logs, routers and switches, posts, mobile location data, equipment diagnostic records, machine logs, documents and much more.

This torrent of bits and bytes, commonly referred to as big data, is overwhelming federal agencies. But if harnessed and properly analyzed, it can help government to create efficiencies; improve processes; boost productivity; improve cybersecurity; increase citizen and employee engagement; reduce fraud, waste and abuse; enhance supply chain and asset management; and improve financial accountability.

The challenge is finding a way to make sense of the massive data flow that continuously surges into agencies’ systems. Without the right processes and technology, big data is unmanageable. According to a recent survey, almost half of federal employees say they struggle to manage big data – analyzing only 12 percent of what’s available – and use it to make decisions.

Harnessing the power of data requires the right technology to automate the collection and organization of data; continuously monitor events, conditions, users and important indicators; and provide real-time notification that important events are occurring or are about to occur. The most advanced systems use artificial intelligence and machine learning to detect unusual activity – and predictive analytics to determine potential outcomes based on a set of predetermined indicators.

Cybersecurity is an example of how agencies have taken advantage of these capabilities and adopted tools for specific uses.

What’s often missing, says Jim Smid, chief technology officer of Iron Bow Technologies, an IT solution provider specializing in government, is a way to quickly aggregate and correlate security data from multiple sources and stop problems before they escalate. Without these capabilities, it’s hard to analyze factors such as the time, location and context of data to determine the details of a cyberattack. Rather than reactively remediating incidents after receiving an alert, agencies can proactively protect systems and data by collecting and analyzing information from firewalls, Web proxies, VPNs, email, intrusion detection systems, host data from workstations and servers, malware scanners, databases, SIEM tools and other sources.

“Every agency wants to be able to compile, analyze and act on threats in real time – not simply to do the forensics after the fact, but that’s often what ends up happening,” Smid says. 

Gleaning data for insights also is a great way to ensure smooth system operations and to pinpoint potential system issues before they mushroom into bigger problems. With automated collection and monitoring of assets such as weblogs and application logs, agencies can gain the visibility they need to improve system efficiency and ensure compliance.

Data analytics can also make a difference as agencies strive to make public websites more reliable, efficient and user-friendly. Continuously analyzing server logs and user activity, for example, helps agencies to improve citizen satisfaction.

Agencies are striving to get better at managing big data at a time when the challenge is rapidly expanding. Most estimates put worldwide data growth at 44 zettabytes by 2020, driven in part by the proliferation of internet-attached sensors, mobile devices and other types of structured and unstructured data.

The federal government, like other big data users, can prepare now or catch up later.