Data Analytics

Nate Silver: Big data's value is not in the data

Big Data

Big data isn't the deep well of critical information it's cracked up to be if the person or systems peering into it don't know what they're looking for, according to one of the world's leading number-crunchers.

Nate Silver, who successfully predicted outcome of the presidential election in 49 of 50 states in 2008 and all 50 in 2012 with his close analysis of statistics and polls, said the vast pool of information their agencies have amassed, or inherited, is only as good as the bucket they use to draw it out.

"Large volumes of data can make people biased," he told attended at the 2013 SAS Government Leadership Summit, held in Washington, D.C. on May 21.

There is a vast and steadily growing amount of data these days. He noted that when people are hit with a lot of data, they tend to polarize their views or stance and don't look beyond the immediate.

The onslaught of data isn't letting up. "Ninety percent of all data has been created in the last two years," he said. He likened the spike in data creation to the paradigm shift that split governments and society after the printing press was invented in the 15th century.

"There are huge gains to be had, but there is a gap between what you know and what you think you know," he said.

Analyzing big data, such as weather forecast information, financial information or other vast reservoirs of information has predictable pitfalls that can be avoided, according to Silver. He warned in particular against seeing things in data that aren't really there. "We're wired to detect patterns," he said, but a pattern may not be exactly what it appears to be at first glance.

He also warned that data can offer seductive, but possibly dangerous illusions if not addressed honestly. "Describing data is important, but it isn't a prediction," he said.

Silver noted that Japanese earthquake researchers had looked at data from the Fukushima area before a nuclear reactor was built there. The researchers used historical data that went back 45 years and found no earthquakes of more than 8.0 on the Richter scale. They didn't look further back to find data that showed the area had, indeed, been hit by a larger quake. They subsequently assumed that a 9.0 quake wasn't possible in the area and constructed the reactor to withstand an 8.5 quake. The 9.0 earthquake that shook Japan and set off a historic tsunami in April 2011 also ruptured reactors at the Fukushima plant.

To avoid such pitfalls in analyzing data, Silver recommended thinking "probabilistically," incorporating potential problems into the analysis.

About the Author

Mark Rockwell is a staff writer at FCW.

Before joining FCW, Rockwell was Washington correspondent for Government Security News, where he covered all aspects of homeland security from IT to detection dogs and border security. Over the last 25 years in Washington as a reporter, editor and correspondent, he has covered an increasingly wide array of high-tech issues for publications like Communications Week, Internet Week, Fiber Optics News, tele.com magazine and Wireless Week.

Rockwell received a Jesse H. Neal Award for his work covering telecommunications issues, and is a graduate of James Madison University.

Click here for previous articles by Rockwell. Contact him at mrockwell@fcw.com or follow him on Twitter at @MRockwell4.


The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.

Featured

  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from Shutterstock.com

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group