COMMENTARY

Data overload threatens with rise of smart tech and real-time sensors

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm that specializes in IT, biotechnology, nanotechnology and clean technology.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal business intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement and public safety roles.

Sensors and monitoring applications — whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine — produce streaming data that is live, continuously changing, structurally invariant and voluminous.


Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.


The military, homeland security, financial, manufacturing and energy sectors increasingly rely on streaming data. Collection devices are appearing in the health care, transportation and public safety sectors at astounding rates.

“Smart” will soon be a staple of the daily lexicon. Smart buildings are being outfitted with sensors to gauge energy use and carbon emissions, and smart hospitals track medication and supply inventories.

For government, the challenge in adapting to this new reality will be to unlearn what it has been taught about business intelligence. Professionals in that field will still need to design and implement data warehouses and integrate the client-side environment. But if they apply long-established methods, they might experience spectacular failure because streaming data is fundamentally different from traditional business intelligence in three important ways.

  1. The extraction, transformation and load process is no longer needed. Streaming data is loading constantly, so there are no updates, only inserts. Sources are widely distributed — there could be tens of thousands of them — and the target schema is usually simple.
  2. Traditional business intelligence tools are not used with streaming data. Instead, analytics must be performed in or close to the database programmatically rather than relying on ad hoc query and reporting tools. The primary mission of streaming data analysis is to spot anomalies and outliers for further investigation, then perform a more iterative analysis to determine the root cause of the abnormality, what it means and what actions are needed.
  3. Data visualization must be dynamic. The sliding time windows and multiple data dimensions required for effective sensor-based business intelligence aren’t well illustrated by static pie charts and visualization tools in spreadsheets. Heat-map and whisker diagrams that change with incoming data are needed.

The government business intelligence community still has time to adapt to this new reality while the technology industry works to create more powerful database management systems and a networking infrastructure with the processing capacity, flexibility and scalability required to effectively analyze massive amounts of data in real time.

Business intelligence professionals should begin now to educate themselves and visit colleagues who focus on applications for the military, energy management, and other agency programs that are ahead of the curve in using sensors and streaming data — or even consider switching jobs to one of those environments.

As is often the case, there will likely be cultural resistance to this change. However, if business intelligence professionals view streaming data as an exotic type that must be dealt with only now and then, they risk being drowned in what the National Science Foundation has characterized as a coming tsunami of sensor-based data.

Worse, failure to respond effectively to the new technical challenges posed by streaming data could render federal government agencies ineffective or even irrelevant in fulfilling their growing functions to regulate and oversee markets and ensure the well-being of their constituents.

Every area of our lives is preparing to go real time in its day-to-day functions. The federal government and its business intelligence community need to go real-time, too. And the time to prepare is now.

Editor’s note: Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.

About the Author

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm specializing in IT, biotechnology, nanotechnology and clean technology. You can contact him at marc@noumenal.com.

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Fri, Mar 4, 2011 Dennis Mobile, AL

The author assumes that the technology will only result in the well being of the people. Like most technocrats, he fails to consider that the technology will be used to exploit and control the people, which in it's most dramatic applications it surely will.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group