Big Data

Beyond the buzz of big data

Big Data word graphic

A lot of big-data buzz centers on potential, hinting at how federal agencies and organizations might someday make sense of their information or find the oft-mentioned needle in the haystack of data.

Meanwhile, some agencies are actually doing big data to great effect.

The Securities and Exchange Commission, for example, collects about 1 billion records every day with its Market Information Data Analytics System (MIDAS) platform.

The trading data comes from more than 10 different market exchange feeds and totals some 23 petabytes per year, meaning analysis for anomalies like the tell-tale signs of insider trading would be impossible for even a large staff of people to detect, according to SEC CIO Thomas Bayer.

MIDAS makes the analysis unfathomably easier and faster – in near real-time – allowing for both a detailed understanding of the current market and long-term trends.

"When you look at what we’ve been able to do analyzing that data, it's taken a lot of mystery out of what the market is perceived to be doing," said Bayer, speaking at the Symantec Government Symposium on March 11.

The SEC's needs continue to grow. As an example, Bayer said, the agency dissects 9 million page reports, impossible without automated analysis allowed in the big data age. Soon, the agency will collect even more, on the order of 2 petabytes of market data per day. Humans only handle that kind of data after advanced analytics carve it down. If sketchy trends – or needles in the haystacks – are found, "you can turn to humans to do further investigation and examination"Bayer said.

At the symposium, Lieut. Col. William Saxon, division chief and program director for the Army’s Enterprise Management Decision Support System Program, described the Army’s foray into big data. Today, the Army collects a wide assortment of information on "people, training, equipment and installations," and runs metrics against that data to produce another data store for its "readiness" effort.

The big data challenge for the Army was always in the "access and discovery of it," Saxon said, mostly because the Army has 3,600 different systems on which it stores information. In recent years, "it was a challenge to bring all that data to one place so that a handful of action officers in a dark room somewhere" could analyze it. Thanks to a focused evolution of its big data systems, Saxon said, that handful of action officers "can turn answers quickly"and utilize the existing system to present their answers.”

Ironically, while technology continues to evolve at a rapid pace, Saxon said the Army’s data problems now are "political."

"Someone had data but didn’t want to share," Saxon said. "Or we're getting the data out of legacy systems."

About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Ellen Lord - Textron DOD ATL USD

    Lord tapped to lead DOD acquisition

    The Trump administration has nominated Ellen Lord, president and CEO of defense contractor Textron Systems, to serve as undersecretary for Acquisition, Technology and Logistics.

  • Soraya Correa, DHS Chief Procurement Officer

    Confronting the culture of fear in government

    Steve Kelman gives kudos to DHS' Soraya Correa for facing the FLASH cancellation head-on.

  • DHS: Russia tried to hack voting systems in 21 states

    DHS officials confirmed for the first time that Russian hackers tried to penetrate voting systems in 21 different states in the run-up to the 2016 election, but said the hacking did not affect election results.

  • VA Secretary Dr. David Shulkin speaking at a June 20, 2017 Monitor Breakfast. Photo credit: Michael Bonfigli/The Christian Science Monitor

    VA expects to add an integrator to health record mix

    After coming to terms with Cerner on a price for its electronic health record system, VA expects to pivot to finding an integrator to handle legacy interoperability and change management.

  • Soraya Correa, DHS Chief Procurement Officer

    DHS execs own FLASH fail

    The department's failure to launch an agile services contract can serve as a teachable moment, according to DHS procurement officials.

  • Is it time to rethink the TIC?

    Current restrictions on internet gateways complicate agencies' move to the cloud, so the Office of Management and Budget is exploring new security architectures.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group