COMMENTARY

Data overload threatens with rise of smart tech and real-time sensors

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm that specializes in IT, biotechnology, nanotechnology and clean technology.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal business intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement and public safety roles.

Sensors and monitoring applications — whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine — produce streaming data that is live, continuously changing, structurally invariant and voluminous.


Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.


The military, homeland security, financial, manufacturing and energy sectors increasingly rely on streaming data. Collection devices are appearing in the health care, transportation and public safety sectors at astounding rates.

“Smart” will soon be a staple of the daily lexicon. Smart buildings are being outfitted with sensors to gauge energy use and carbon emissions, and smart hospitals track medication and supply inventories.

For government, the challenge in adapting to this new reality will be to unlearn what it has been taught about business intelligence. Professionals in that field will still need to design and implement data warehouses and integrate the client-side environment. But if they apply long-established methods, they might experience spectacular failure because streaming data is fundamentally different from traditional business intelligence in three important ways.

  1. The extraction, transformation and load process is no longer needed. Streaming data is loading constantly, so there are no updates, only inserts. Sources are widely distributed — there could be tens of thousands of them — and the target schema is usually simple.
  2. Traditional business intelligence tools are not used with streaming data. Instead, analytics must be performed in or close to the database programmatically rather than relying on ad hoc query and reporting tools. The primary mission of streaming data analysis is to spot anomalies and outliers for further investigation, then perform a more iterative analysis to determine the root cause of the abnormality, what it means and what actions are needed.
  3. Data visualization must be dynamic. The sliding time windows and multiple data dimensions required for effective sensor-based business intelligence aren’t well illustrated by static pie charts and visualization tools in spreadsheets. Heat-map and whisker diagrams that change with incoming data are needed.

The government business intelligence community still has time to adapt to this new reality while the technology industry works to create more powerful database management systems and a networking infrastructure with the processing capacity, flexibility and scalability required to effectively analyze massive amounts of data in real time.

Business intelligence professionals should begin now to educate themselves and visit colleagues who focus on applications for the military, energy management, and other agency programs that are ahead of the curve in using sensors and streaming data — or even consider switching jobs to one of those environments.

As is often the case, there will likely be cultural resistance to this change. However, if business intelligence professionals view streaming data as an exotic type that must be dealt with only now and then, they risk being drowned in what the National Science Foundation has characterized as a coming tsunami of sensor-based data.

Worse, failure to respond effectively to the new technical challenges posed by streaming data could render federal government agencies ineffective or even irrelevant in fulfilling their growing functions to regulate and oversee markets and ensure the well-being of their constituents.

Every area of our lives is preparing to go real time in its day-to-day functions. The federal government and its business intelligence community need to go real-time, too. And the time to prepare is now.

Editor’s note: Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.

About the Author

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm specializing in IT, biotechnology, nanotechnology and clean technology. You can contact him at marc@noumenal.com.

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.

Featured

  • FCW @ 30 GPS

    FCW @ 30

    Since 1986, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

  • Shutterstock image.

    Merged IT modernization bill punts on funding

    A House panel approved a new IT modernization bill that appears poised to pass, but key funding questions are left for appropriators.

  • General Frost

    Army wants cyber capability everywhere

    The Army's cyber director said cyber, electronic warfare and information operations must be integrated into warfighters' doctrine and training.

  • Rising Star 2013

    Meet the 2016 Rising Stars

    FCW honors 30 early-career leaders in federal IT.

Reader comments

Fri, Mar 4, 2011 Dennis Mobile, AL

The author assumes that the technology will only result in the well being of the people. Like most technocrats, he fails to consider that the technology will be used to exploit and control the people, which in it's most dramatic applications it surely will.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group