COMMENTARY

Data overload threatens with rise of smart tech and real-time sensors

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm that specializes in IT, biotechnology, nanotechnology and clean technology.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal business intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement and public safety roles.

Sensors and monitoring applications — whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine — produce streaming data that is live, continuously changing, structurally invariant and voluminous.


Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.


The military, homeland security, financial, manufacturing and energy sectors increasingly rely on streaming data. Collection devices are appearing in the health care, transportation and public safety sectors at astounding rates.

“Smart” will soon be a staple of the daily lexicon. Smart buildings are being outfitted with sensors to gauge energy use and carbon emissions, and smart hospitals track medication and supply inventories.

For government, the challenge in adapting to this new reality will be to unlearn what it has been taught about business intelligence. Professionals in that field will still need to design and implement data warehouses and integrate the client-side environment. But if they apply long-established methods, they might experience spectacular failure because streaming data is fundamentally different from traditional business intelligence in three important ways.

  1. The extraction, transformation and load process is no longer needed. Streaming data is loading constantly, so there are no updates, only inserts. Sources are widely distributed — there could be tens of thousands of them — and the target schema is usually simple.
  2. Traditional business intelligence tools are not used with streaming data. Instead, analytics must be performed in or close to the database programmatically rather than relying on ad hoc query and reporting tools. The primary mission of streaming data analysis is to spot anomalies and outliers for further investigation, then perform a more iterative analysis to determine the root cause of the abnormality, what it means and what actions are needed.
  3. Data visualization must be dynamic. The sliding time windows and multiple data dimensions required for effective sensor-based business intelligence aren’t well illustrated by static pie charts and visualization tools in spreadsheets. Heat-map and whisker diagrams that change with incoming data are needed.

The government business intelligence community still has time to adapt to this new reality while the technology industry works to create more powerful database management systems and a networking infrastructure with the processing capacity, flexibility and scalability required to effectively analyze massive amounts of data in real time.

Business intelligence professionals should begin now to educate themselves and visit colleagues who focus on applications for the military, energy management, and other agency programs that are ahead of the curve in using sensors and streaming data — or even consider switching jobs to one of those environments.

As is often the case, there will likely be cultural resistance to this change. However, if business intelligence professionals view streaming data as an exotic type that must be dealt with only now and then, they risk being drowned in what the National Science Foundation has characterized as a coming tsunami of sensor-based data.

Worse, failure to respond effectively to the new technical challenges posed by streaming data could render federal government agencies ineffective or even irrelevant in fulfilling their growing functions to regulate and oversee markets and ensure the well-being of their constituents.

Every area of our lives is preparing to go real time in its day-to-day functions. The federal government and its business intelligence community need to go real-time, too. And the time to prepare is now.

Editor’s note: Marc Demarest will discuss the promises and perils of the streaming data revolution and its influence on performance management at the TDWI Government Summit April 5 in Arlington, Va.

About the Author

Marc Demarest is CEO and principal at Noumenal, a principals-only private intellectual capital consulting firm specializing in IT, biotechnology, nanotechnology and clean technology. You can contact him at marc@noumenal.com.

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.

Featured

  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from Shutterstock.com

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Fri, Mar 4, 2011 Dennis Mobile, AL

The author assumes that the technology will only result in the well being of the people. Like most technocrats, he fails to consider that the technology will be used to exploit and control the people, which in it's most dramatic applications it surely will.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group