Analytics

IARPA seeks to predict the unpredictable

Shutterstock image: illuminated crystal ball. 

Imagine being able to accurately forecast pandemics, cyberattacks and geopolitical breakdowns before they occur by aggregating data and discerning historical trends. That’s exactly what the Intelligence Advanced Research Projects Activity is trying to do -- and the agency is getting better at doing.

The intelligence community's research arm is looking to "forecasting tournaments" that "invite teams from industry, academia and hobbyists to come and forecast real-world events in real-time," said IARPA director Dr. Jason Matheny in an Oct. 18 presentation at the Predictive Analytics World for Government conference in Washington, D.C.

"We keep score [for accuracy], so we sort of act as referees in a giant match. And it turned out being the world's largest set of forecasting tournaments," he said.

Matheny said the result was tens of thousands of participants providing millions of judgments. And with that data, he said, "we know a lot more about what kinds of events are, in principle, predictable and what kinds aren't… and we also know something about the methods that have been more accurate and less accurate."

Matheny said the key to forecasting events is looking for indicators or patterns that tend to surface before they even take place. "Using a variety of these signals, you can predict… with relatively short lead time" disease outbreaks, foreign election outcomes and weapons tests, among other events, he added.

For instance, when it comes to projecting cyberattacks, IARPA has stood up its Cyber-attack Automated Unconventional Sensor Environment program, which searches for “indicators in the wild whether or not a cyberattack is being planned,” said Matheny. "That includes indicators like chatter and hacker forums, web search query trends that suggests groups are mapping IP networks, or changes in the black market prices of malware."

Due to the massive amounts of data needed to search through to conduct analyses, IARPA employs text mining and machine learning techniques to "process volumes and velocities of data that humans can't," Matheny said.

For classified data, IARPA has initiated project Mercury, which seeks to determine the highest-value situations in which to leverage classified information in an appropriate way. Matheny said this program also can address indicators of terrorist planning, but because it was created recently, "it will be a few months before we have the first results."

Although many of IARPA's tools are used to forecast negative events, Matheny believes most IARPA assets are "transferrable" throughout government and industry, and can influence data-driven decision-making for policy-makers, government agencies and companies.

"One of the lessons is if you're going to make an important decision, don’t take it into the boardroom," said Matheny. "Instead, poll your entire workforce ... ask everybody to contribute."

About the Author

Chase Gunter is a staff writer covering civilian agencies, workforce issues, health IT, open data and innovation.

Prior to joining FCW, Gunter reported for the C-Ville Weekly in Charlottesville, Va., and served as a college sports beat writer for the South Boston (Va.) News and Record. He started at FCW as an editorial fellow before joining the team full-time as a reporter.

Gunter is a graduate of the University of Virginia, where his emphases were English, history and media studies.

Click here for previous articles by Gunter, or connect with him on Twitter: @WChaseGunter

Featured

  • Cybersecurity

    DHS floats 'collective defense' model for cybersecurity

    Homeland Security Secretary Kirstjen Nielsen wants her department to have a more direct role in defending the private sector and critical infrastructure entities from cyberthreats.

  • Defense
    Defense Secretary James Mattis testifies at an April 12 hearing of the House Armed Services Committee.

    Mattis: Cloud deal not tailored for Amazon

    On Capitol Hill, Defense Secretary Jim Mattis sought to quell "rumors" that the Pentagon's planned single-award cloud acquisition was designed with Amazon Web Services in mind.

  • Census
    shutterstock image

    2020 Census to include citizenship question

    The Department of Commerce is breaking with recent practice and restoring a question about respondent citizenship last used in 1950, despite being urged not to by former Census directors and outside experts.

Stay Connected

FCW Update

Sign up for our newsletter.

I agree to this site's Privacy Policy.