Big Data

Finding structure for unstructured data

Big Data word graphic

If hype equaled performance, big data and cloud computing would already be unmatched cost-cutting, efficiency-increasing, bottom line-building tools in the federal space.

Those end results are not yet reality, but innovative techniques are already being put to use by certain agencies as they take advantage of the capabilities of big data in a cloud environment to monitor fraud and search for terrorists.

Intelligence analysts are using one such approach, called the Cloud Analytics Reference Architecture, to compile all available intelligence data into easily analyzed pool of data, according to Booz Allen Hamilton, which collaborated with the government to develop the technique.

“Reference architecture is simply a new way of looking at data, but one that revolutionizes our ability to gain knowledge and insight,” said Mark Jacobsohn, senior vice president at Booz Allen Hamilton in a new paper titled “Delivering on the Promise of Big Data and the Cloud.


Read Booz Allen Hamilton's new reports on big data and the cloud.

Delivering on the Promise of Big Data and the Cloud

Building a Business Case for Cloud

The paper was one of two cloud-related papers the firm released on Jan. 9.

Jacobsohn said reference architecture does away with conventional data and analytics silos, consolidating all information into a single medium designed to foster connections called a “data lake,” which reduces complexity and creates efficiencies that improve data visualization to allow for easier insights by analysts.

Reference architecture can be deployed into a cloud environment, ingesting petabytes of raw data and organizing it in a single structure that is then open for analyzing.

“Imagine the data lake as the largest spreadsheet you ever saw, with billions and billions of cells,” said Booz Allen Hamilton executive vice president Mark Herman. “This isn’t a cloud solution that hasn’t been implemented or 'should' work. The most important thing about reference architecture is that it is being used in operational systems, we’ve already used it successfully for government clients and we know it works.”

Mark Herman

Mark Herman

Without giving too many specifics, Herman said an intelligence agency has implemented the approach to search for terrorist threats, while the Federal Deposit Insurance Corporation, the Securities and Exchange Commission and the Internal Revenue Service could benefit from using similar models for fraud detection. Meanwhile, the National Security Agency is using similar technology to to improve performance and mission effectiveness.

Jacobsohn claimed the visualization capabilities that a reference architecture provides could be as important to decision-making as pie charts and bar graphs were in the 1950s and 1960s.  “Reference architecture will do the same – but this time with big data,” he said.

Yet as attractive as cloud-based big data solutions might seem, Herman offered caution to federal agencies looking to implement them, particularly those that have recently made large investments in legacy IT systems.

“You want to look at where you’re at in the life-cycle of your IT,” Herman said. “If you’ve just made a huge investment in something like a really expensive computer system, going to the cloud at that point is not going to save you money. But if you’re an agency at the end of your IT life-cycle, now you can invest and gain efficiencies. It all depends on IT liquidity.”

About the Author

Frank Konkel is a former staff writer for FCW.


  • Contracting
    8 prototypes of the border walls as tweeted by CBP San Diego

    DHS contractors face protests – on the streets

    Tech companies are facing protests internally from workers and externally from activists about doing for government amid controversial policies like "zero tolerance" for illegal immigration.

  • Workforce
    By Mark Van Scyoc Royalty-free stock photo ID: 285175268

    At OPM, Weichert pushes direct hire, pay agent changes

    Margaret Weichert, now acting director of the Office of Personnel Management, is clearing agencies to make direct hires in IT, cyber and other tech fields and is changing pay for specialized occupations.

  • Cloud
    Shutterstock ID ID: 222190471 By wk1003mike

    IBM protests JEDI cloud deal

    As the deadline to submit bids on the Pentagon's $10 billion, 10-year warfighter cloud deal draws near, IBM announced a legal protest.

Stay Connected

FCW Update

Sign up for our newsletter.

I agree to this site's Privacy Policy.