Big Data

Beyond the buzz of big data

Big Data word graphic

A lot of big-data buzz centers on potential, hinting at how federal agencies and organizations might someday make sense of their information or find the oft-mentioned needle in the haystack of data.

Meanwhile, some agencies are actually doing big data to great effect.

The Securities and Exchange Commission, for example, collects about 1 billion records every day with its Market Information Data Analytics System (MIDAS) platform.

The trading data comes from more than 10 different market exchange feeds and totals some 23 petabytes per year, meaning analysis for anomalies like the tell-tale signs of insider trading would be impossible for even a large staff of people to detect, according to SEC CIO Thomas Bayer.

MIDAS makes the analysis unfathomably easier and faster – in near real-time – allowing for both a detailed understanding of the current market and long-term trends.

"When you look at what we’ve been able to do analyzing that data, it's taken a lot of mystery out of what the market is perceived to be doing," said Bayer, speaking at the Symantec Government Symposium on March 11.

The SEC's needs continue to grow. As an example, Bayer said, the agency dissects 9 million page reports, impossible without automated analysis allowed in the big data age. Soon, the agency will collect even more, on the order of 2 petabytes of market data per day. Humans only handle that kind of data after advanced analytics carve it down. If sketchy trends – or needles in the haystacks – are found, "you can turn to humans to do further investigation and examination"Bayer said.

At the symposium, Lieut. Col. William Saxon, division chief and program director for the Army’s Enterprise Management Decision Support System Program, described the Army’s foray into big data. Today, the Army collects a wide assortment of information on "people, training, equipment and installations," and runs metrics against that data to produce another data store for its "readiness" effort.

The big data challenge for the Army was always in the "access and discovery of it," Saxon said, mostly because the Army has 3,600 different systems on which it stores information. In recent years, "it was a challenge to bring all that data to one place so that a handful of action officers in a dark room somewhere" could analyze it. Thanks to a focused evolution of its big data systems, Saxon said, that handful of action officers "can turn answers quickly"and utilize the existing system to present their answers.”

Ironically, while technology continues to evolve at a rapid pace, Saxon said the Army’s data problems now are "political."

"Someone had data but didn’t want to share," Saxon said. "Or we're getting the data out of legacy systems."

About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group