Nate Silver on big data's future: It's about attitude

In his latest book, statistician and predictive analytics expert Nate Silver describes his approach to forming forecasts out of data.

signal noise book cover

"The Signal and the Noise," by Nate Silver, discusses various ways to use data in forecasting.

When statistician and predictive analytics expert Nate Silver speaks, people pay attention.

After he successfully predicted the outcome of the presidential election in 49 of 50 states in 2008 and all 50 in 2012, his star has grown almost as fast the unread messages in his e-mail inbox -- more than 99,000 in a recent image he posted to Twitter.

Media outlets crowned Silver the real winner in November, with the triumphant success of his data-based predictive model in the face of critical go-by-the-gut pundits that he had claimed all along weren't much more accurate than a coin-flip.

Silver, an East Lansing, Mich., native, made the same assertion in his latest book, "The Signal and the Noise: Why so Many Predictions Fail – But Some Don't", which quickly became a New York Times bestseller after its September 2012 publication.

In it, he describes a range of forecasters, including enigmatic poker players and the supercomputer-based big-data models used by the National Oceanic and Atmospheric Administration that recently nailed the path of Hurricane Sandy five days before it struck land.

Somewhere along the way, Silver has helped make data cool. Political junkies hailed it, news outlets lauded it and some pundits weren't too happy about it. Yet data is everywhere – IBM says 2.5 quintillion bytes of it are created every day – even if corporations and federal agencies are still in the early stages of making sense of "big data," the massive data sets too large to process via traditional methods.

Hidden in these vast data sets are insights that could help agencies solve big problems, but in an interview with FCW, Silver said the "big data" era will only be successful if the government is willing to evolve with it.

"In some cases, the government has the best data in the world, but not always the ability to use it," Silver said, speaking with FCW prior to delivering a keynote speech at the Feb. 12 Adobe Government Assembly.

"The governments that are willing to evolve with it will benefit, certainly," Silver said. "But there is no end-point to big data."

If big data is to become a standard tool by which the government operates, aging legacy IT systems will have to modernize. This is no small feat, given that 70 percent of the federal government's $79 billion IT budget in 2011 was spent on maintaining existing systems. Many federal systems still run Common Business-Oriented Language (COBOL), a programming language developed in 1959, ten years before NASA put a man on the moon.

But perhaps the biggest obstacle required before big data becomes a big staple in the government's IT arsenal is peoples' "goals and attitudes," Silver said.

Old-hat policies need to change – Silver later told feds at the conference that "bureaucracy is the enemy of imagination," to smiles and knowing glances around the room. Changing policies means changing people's beliefs about what big data is and what it might do.

Big data is not a cure-all, and it is inherently filled with noise and uncertainty, but it does have tremendous potential if people approach if the right way. "The world is not lacking for techniques, it's more about the right goals and right attitudes," Silver said.

Data-based statistical prediction has been changing the way people think since Bayes' Theorem – named after English mathematician Thomas Bayes – was published in the early 1800s. As Silver notes in his book, the expression forever linked science, probability and prediction. It basically states the world follows predictable laws and that the more data you have, the closer you are to a full picture of reality – and better predictions – which contrasted with most people's views of divinity at the time.

But since we can collect more data than ever before, shouldn't big data reduce uncertainty?

Not likely, Silver said.

Quotable

There isn't any more truth in the world than there was before the Internet or the printing press. Most of the data is just noise, as most of the universe is filled with empty space. -- Nate Silver in "The Signal and the Noise."

Generating trillions of relationships between variables does not necessarily generate any more meaningful relationships between them. If you flush your toilet in Washington, D.C. and it rains in Australia, you can use supercomputers to look for connections and you might even find some, but it doesn't mean those events are linked in any meaningful way.

"It's okay to fail," said Silver, who recently flopped himself, erroneously picking the San Francisco 49ers as the winners over the Baltimore Ravens in the 2013 Super Bowl. "It's okay to fail as long as you learn."

Big data predictions may be more apt to fail than those based on smaller data sets, but just like hitting the lottery, the payoffs can be enormous.

In November, NOAA's big-data based supercomputers took in data from polar-orbiting satellites, weather buoys and Gulfstream-IV and P-3 jets to churn out high-resolution computer models of Hurricane Sandy's path every six hours for several days before the storm landed. Residents in the storm's path had ample warning, and though more than 100 people died, without big data the net loss of life would have been far higher.

"If Hurricane Sandy happened 20 years back, it would almost certainly have been a disaster without much warning," Dr. Sundararaman Gopalakrishnan, a senior meteorologist at NOAA's Atlantic Oceanographic and Meteorological Laboratory in Miami, told FCW at the time.

Even in its infancy, big data is changing the way agencies perform. The Department of Health and Human Services is changing the way it responds to disasters and the Department of Defense's global shared service center has implemented a big-dated based business activity monitoring software tool that has so far detected $4 billion in improper payments.

Silver's rising star as a data wiz has helped show the importance of data in predicting and shaping the future to the masses. While he is likely to continue political predictions at his New York Times-affiliated FiveThirtyEight.com through the 2016 election, Silver said he plans to expand its focus to other categories.

Sports for sure, he said – Silver first made a name for himself developing a sabermetric system for monitoring Major League Baseball player performance in the early 2000s. (A sabermetric system is an analytics tool that measures the performance of baseball players' statistics.) He'd also like to "broaden the scope a bit" by delving into topics like economics, and perhaps health and education. Even entertainment is fair game -- on Feb. 22 Silver made predictions for the Oscar awards that were announced Feb. 24, and went four-for-six on his picks.

Where the federal government goes with big data is perhaps less clear, but tough choices will have to be made. There will never be a time, Silver said, when "Pressing a button solves all our problems."

"One thing data doesn't do is tell you what your goals ought to be," Silver said. "You still have to think about what to accomplish. Data is not a substitute for judgments you have to make."

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.