COMMENTARY

Data overload threatens with rise of smart tech and real-time sensors

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm that specializes in IT, biotechnology, nanotechnology and clean technology.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal business intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement and public safety roles.

Sensors and monitoring applications — whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine — produce streaming data that is live, continuously changing, structurally invariant and voluminous.


Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.


The military, homeland security, financial, manufacturing and energy sectors increasingly rely on streaming data. Collection devices are appearing in the health care, transportation and public safety sectors at astounding rates.

“Smart” will soon be a staple of the daily lexicon. Smart buildings are being outfitted with sensors to gauge energy use and carbon emissions, and smart hospitals track medication and supply inventories.

For government, the challenge in adapting to this new reality will be to unlearn what it has been taught about business intelligence. Professionals in that field will still need to design and implement data warehouses and integrate the client-side environment. But if they apply long-established methods, they might experience spectacular failure because streaming data is fundamentally different from traditional business intelligence in three important ways.

  1. The extraction, transformation and load process is no longer needed. Streaming data is loading constantly, so there are no updates, only inserts. Sources are widely distributed — there could be tens of thousands of them — and the target schema is usually simple.
  2. Traditional business intelligence tools are not used with streaming data. Instead, analytics must be performed in or close to the database programmatically rather than relying on ad hoc query and reporting tools. The primary mission of streaming data analysis is to spot anomalies and outliers for further investigation, then perform a more iterative analysis to determine the root cause of the abnormality, what it means and what actions are needed.
  3. Data visualization must be dynamic. The sliding time windows and multiple data dimensions required for effective sensor-based business intelligence aren’t well illustrated by static pie charts and visualization tools in spreadsheets. Heat-map and whisker diagrams that change with incoming data are needed.

The government business intelligence community still has time to adapt to this new reality while the technology industry works to create more powerful database management systems and a networking infrastructure with the processing capacity, flexibility and scalability required to effectively analyze massive amounts of data in real time.

Business intelligence professionals should begin now to educate themselves and visit colleagues who focus on applications for the military, energy management, and other agency programs that are ahead of the curve in using sensors and streaming data — or even consider switching jobs to one of those environments.

As is often the case, there will likely be cultural resistance to this change. However, if business intelligence professionals view streaming data as an exotic type that must be dealt with only now and then, they risk being drowned in what the National Science Foundation has characterized as a coming tsunami of sensor-based data.

Worse, failure to respond effectively to the new technical challenges posed by streaming data could render federal government agencies ineffective or even irrelevant in fulfilling their growing functions to regulate and oversee markets and ensure the well-being of their constituents.

Every area of our lives is preparing to go real time in its day-to-day functions. The federal government and its business intelligence community need to go real-time, too. And the time to prepare is now.

Editor’s note: Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.

About the Author

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm specializing in IT, biotechnology, nanotechnology and clean technology. You can contact him at marc@noumenal.com.

The 2015 Federal 100

Meet 100 women and men who are doing great things in federal IT.

Featured

  • Shutterstock image (by venimo): e-learning concept image, digital content and online webinar icons.

    Can MOOCs make the grade for federal training?

    Massive open online courses can offer specialized IT instruction on a flexible schedule and on the cheap. That may not always mesh with government's preference for structure and certification, however.

  • Shutterstock image (by edel): graduation cap and diploma.

    Cybersecurity: 6 schools with the right stuff

    The federal government craves more cybersecurity professionals. These six schools are helping meet that demand.

  • Rick Holgate

    Holgate to depart ATF

    Former ACT president will take a job with Gartner, follow his spouse to Vienna, Austria.

  • Are VA techies slacking off on Yammer?

    A new IG report cites security and productivity concerns associated with employees' use of the popular online collaboration tool.

  • Shutterstock image: digital fingerprint, cyber crime.

    Exclusive: The OPM breach details you haven't seen

    An official timeline of the Office of Personnel Management breach obtained by FCW pinpoints the hackers’ calibrated extraction of data, and the government's step-by-step response.

  • Stephen Warren

    Deputy CIO Warren exits VA

    The onetime acting CIO at Veterans Affairs will be taking over CIO duties at the Office of the Comptroller of the Currency.

  • Shutterstock image: monitoring factors of healthcare.

    DOD awards massive health records contract

    Leidos, Accenture and Cerner pull off an unexpected win of the multi-billion-dollar Defense Healthcare Management System Modernization contract, beating out the presumptive health-records leader.

  • Sweating the OPM data breach -- Illustration by Dragutin Cvijanovic

    Sweating the stolen data

    Millions of background-check records were compromised, OPM now says. Here's the jaw-dropping range of personal data that was exposed.

  • FCW magazine

    Let's talk about Alliant 2

    The General Services Administration is going to great lengths to gather feedback on its IT services GWAC. Will it make for a better acquisition vehicle?

Reader comments

Fri, Mar 4, 2011 Dennis Mobile, AL

The author assumes that the technology will only result in the well being of the people. Like most technocrats, he fails to consider that the technology will be used to exploit and control the people, which in it's most dramatic applications it surely will.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above