Big Data

Beyond the buzz of big data

Big Data word graphic

A lot of big-data buzz centers on potential, hinting at how federal agencies and organizations might someday make sense of their information or find the oft-mentioned needle in the haystack of data.

Meanwhile, some agencies are actually doing big data to great effect.

The Securities and Exchange Commission, for example, collects about 1 billion records every day with its Market Information Data Analytics System (MIDAS) platform.

The trading data comes from more than 10 different market exchange feeds and totals some 23 petabytes per year, meaning analysis for anomalies like the tell-tale signs of insider trading would be impossible for even a large staff of people to detect, according to SEC CIO Thomas Bayer.

MIDAS makes the analysis unfathomably easier and faster – in near real-time – allowing for both a detailed understanding of the current market and long-term trends.

"When you look at what we’ve been able to do analyzing that data, it's taken a lot of mystery out of what the market is perceived to be doing," said Bayer, speaking at the Symantec Government Symposium on March 11.

The SEC's needs continue to grow. As an example, Bayer said, the agency dissects 9 million page reports, impossible without automated analysis allowed in the big data age. Soon, the agency will collect even more, on the order of 2 petabytes of market data per day. Humans only handle that kind of data after advanced analytics carve it down. If sketchy trends – or needles in the haystacks – are found, "you can turn to humans to do further investigation and examination"Bayer said.

At the symposium, Lieut. Col. William Saxon, division chief and program director for the Army’s Enterprise Management Decision Support System Program, described the Army’s foray into big data. Today, the Army collects a wide assortment of information on "people, training, equipment and installations," and runs metrics against that data to produce another data store for its "readiness" effort.

The big data challenge for the Army was always in the "access and discovery of it," Saxon said, mostly because the Army has 3,600 different systems on which it stores information. In recent years, "it was a challenge to bring all that data to one place so that a handful of action officers in a dark room somewhere" could analyze it. Thanks to a focused evolution of its big data systems, Saxon said, that handful of action officers "can turn answers quickly"and utilize the existing system to present their answers.”

Ironically, while technology continues to evolve at a rapid pace, Saxon said the Army’s data problems now are "political."

"Someone had data but didn’t want to share," Saxon said. "Or we're getting the data out of legacy systems."

About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Shutterstock image (by wk1003mike): cloud system fracture.

    Does the IRS have a cloud strategy?

    Congress and watchdog agencies have dinged the IRS for lacking an enterprise cloud strategy seven years after it became the official policy of the U.S. government.

  • Shutterstock image: illuminated connections between devices.

    Who won what in EIS

    The General Services Administration posted detailed data on how the $50 billion Enterprise Infrastructure Solutions contract might be divvied up.

  • Wikimedia Image: U.S. Cyber Command logo.

    Trump elevates CyberCom to combatant command status

    The White House announced a long-planned move to elevate Cyber Command to the status of a full combatant command.

  • Photo credit: John Roman Images / Shutterstock.com

    Verizon plans FirstNet rival

    Verizon says it will carve a dedicated network out of its extensive national 4G LTE network for first responders, in competition with FirstNet.

  • AI concept art

    Can AI tools replace feds?

    The Heritage Foundation is recommending that hundreds of thousands of federal jobs be replaced by automation as part of a larger government reorganization strategy.

  • DOD Common Access Cards

    DOD pushes toward CAC replacement

    Defense officials hope the Common Access Card's days are numbered as they continue to test new identity management solutions.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group