Big Data

Government can borrow from private sector’s big-data playbook

analytics concept art

What do Web giants like Facebook and Google have in common with advertising technology companies?

They’re both leaders in big data, with easily scalable IT infrastructures that allow for the collection of real-time systems data of disparate natures and sources, and in in-memory computing power to run big jobs fast.

This leads to faster actionable insights with all sorts of applications in civilian agencies, according to Carl Wright, executive vice president at MemSQL, a company that specializes in in-memory SQL databases.

These kind of next-generation infrastructures, predicated on commoditized scalable architectures, fill buildings with distributed frameworks of servers that cut down significantly on latency in big batch projects. Such systems process in minutes what legacy enterprise systems – those common at many government agencies – could take hours or days to do, Wright said. These systems are the reason a user’s Internet search term can quickly lead to user-specific ads.

“You have the ability to use every core and server to facilitate a load, so it creates a true high-performance computing environment,” Wright said.

Real-time systems data already plays a major role in how defense and intelligence agencies carry out their missions and daily operations. As some of the National Security Agency’s surveillance capabilities and data ingest techniques have come to light in recent months, it is clear that the intelligence community is far ahead in the big-data game.

Yet Wright said real-time systems have many potential benefits on the civilian side of government. The ability to compute more data at faster intervals gives an agency a better handle on its decision making, allowing it to use significantly more data over longer periods of time to arrive at conclusions. In arenas such as fraud detection and disease control, Wright said, various collections of data – whether unstructured, structured or semi-structured -- can be processed for real-time reporting.

Depending on the vendor, in-memory computing used in distributed frameworks can vary significantly in price, Wright said, but the fact that most government agencies are facing budget shortfalls might actually be a good thing for innovation. And some real-time systems can at least run on the front-end of existing legacy systems. The real revolution, Wright said, is on the back-end.

“The good news for change is that it means agencies can’t keep doing business as usual. They have to look for more economic systems and can’t keep going back to the same vendors locked in bad cost structures,” Wright said.

About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group