Big Data

Finding structure for unstructured data

Big Data word graphic

If hype equaled performance, big data and cloud computing would already be unmatched cost-cutting, efficiency-increasing, bottom line-building tools in the federal space.

Those end results are not yet reality, but innovative techniques are already being put to use by certain agencies as they take advantage of the capabilities of big data in a cloud environment to monitor fraud and search for terrorists.

Intelligence analysts are using one such approach, called the Cloud Analytics Reference Architecture, to compile all available intelligence data into easily analyzed pool of data, according to Booz Allen Hamilton, which collaborated with the government to develop the technique.

“Reference architecture is simply a new way of looking at data, but one that revolutionizes our ability to gain knowledge and insight,” said Mark Jacobsohn, senior vice president at Booz Allen Hamilton in a new paper titled “Delivering on the Promise of Big Data and the Cloud.


Read Booz Allen Hamilton's new reports on big data and the cloud.

Delivering on the Promise of Big Data and the Cloud

Building a Business Case for Cloud

The paper was one of two cloud-related papers the firm released on Jan. 9.

Jacobsohn said reference architecture does away with conventional data and analytics silos, consolidating all information into a single medium designed to foster connections called a “data lake,” which reduces complexity and creates efficiencies that improve data visualization to allow for easier insights by analysts.

Reference architecture can be deployed into a cloud environment, ingesting petabytes of raw data and organizing it in a single structure that is then open for analyzing.

“Imagine the data lake as the largest spreadsheet you ever saw, with billions and billions of cells,” said Booz Allen Hamilton executive vice president Mark Herman. “This isn’t a cloud solution that hasn’t been implemented or 'should' work. The most important thing about reference architecture is that it is being used in operational systems, we’ve already used it successfully for government clients and we know it works.”

Mark Herman

Mark Herman

Without giving too many specifics, Herman said an intelligence agency has implemented the approach to search for terrorist threats, while the Federal Deposit Insurance Corporation, the Securities and Exchange Commission and the Internal Revenue Service could benefit from using similar models for fraud detection. Meanwhile, the National Security Agency is using similar technology to to improve performance and mission effectiveness.

Jacobsohn claimed the visualization capabilities that a reference architecture provides could be as important to decision-making as pie charts and bar graphs were in the 1950s and 1960s.  “Reference architecture will do the same – but this time with big data,” he said.

Yet as attractive as cloud-based big data solutions might seem, Herman offered caution to federal agencies looking to implement them, particularly those that have recently made large investments in legacy IT systems.

“You want to look at where you’re at in the life-cycle of your IT,” Herman said. “If you’ve just made a huge investment in something like a really expensive computer system, going to the cloud at that point is not going to save you money. But if you’re an agency at the end of your IT life-cycle, now you can invest and gain efficiencies. It all depends on IT liquidity.”

About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.


  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Wed, Jan 9, 2013 ColleenKellie Plettl-degowske

Good intro-interesting article. Security concerns On cloud access?

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group