Analytics

IARPA seeks to predict the unpredictable

Shutterstock image: illuminated crystal ball. 

Imagine being able to accurately forecast pandemics, cyberattacks and geopolitical breakdowns before they occur by aggregating data and discerning historical trends. That’s exactly what the Intelligence Advanced Research Projects Activity is trying to do -- and the agency is getting better at doing.

The intelligence community's research arm is looking to "forecasting tournaments" that "invite teams from industry, academia and hobbyists to come and forecast real-world events in real-time," said IARPA director Dr. Jason Matheny in an Oct. 18 presentation at the Predictive Analytics World for Government conference in Washington, D.C.

"We keep score [for accuracy], so we sort of act as referees in a giant match. And it turned out being the world's largest set of forecasting tournaments," he said.

Matheny said the result was tens of thousands of participants providing millions of judgments. And with that data, he said, "we know a lot more about what kinds of events are, in principle, predictable and what kinds aren't… and we also know something about the methods that have been more accurate and less accurate."

Matheny said the key to forecasting events is looking for indicators or patterns that tend to surface before they even take place. "Using a variety of these signals, you can predict… with relatively short lead time" disease outbreaks, foreign election outcomes and weapons tests, among other events, he added.

For instance, when it comes to projecting cyberattacks, IARPA has stood up its Cyber-attack Automated Unconventional Sensor Environment program, which searches for “indicators in the wild whether or not a cyberattack is being planned,” said Matheny. "That includes indicators like chatter and hacker forums, web search query trends that suggests groups are mapping IP networks, or changes in the black market prices of malware."

Due to the massive amounts of data needed to search through to conduct analyses, IARPA employs text mining and machine learning techniques to "process volumes and velocities of data that humans can't," Matheny said.

For classified data, IARPA has initiated project Mercury, which seeks to determine the highest-value situations in which to leverage classified information in an appropriate way. Matheny said this program also can address indicators of terrorist planning, but because it was created recently, "it will be a few months before we have the first results."

Although many of IARPA's tools are used to forecast negative events, Matheny believes most IARPA assets are "transferrable" throughout government and industry, and can influence data-driven decision-making for policy-makers, government agencies and companies.

"One of the lessons is if you're going to make an important decision, don’t take it into the boardroom," said Matheny. "Instead, poll your entire workforce ... ask everybody to contribute."

About the Author

Chase Gunter is a staff writer covering civilian agencies, workforce issues, health IT, open data and innovation.

Prior to joining FCW, Gunter reported for the C-Ville Weekly in Charlottesville, Va., and served as a college sports beat writer for the South Boston (Va.) News and Record. He started at FCW as an editorial fellow before joining the team full-time as a reporter.

Gunter is a graduate of the University of Virginia, where his emphases were English, history and media studies.

Click here for previous articles by Gunter, or connect with him on Twitter: @WChaseGunter

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group