Big data

DARPA seeks innovative ways to make sense of data

Placeholder Image for Article Template

Big data is everywhere, and although the collection of huge datasets from sensors, machines and device-wielding humans has become increasingly automated, understanding causes and connections is still left to people.

The Defense Advanced Research Projects Agency wants to change that.

DARPA's Big Mechanism program will provide up to $45 million in contracts, grants and cooperative agreements for as many as 12 award winners that demonstrate "innovative approaches that enable revolutionary advances in science, devices or systems," according to a newly released solicitation.

In other words, if big data highlights the search for a needle in a haystack of data, DARPA aims to develop technology that explains why the needle is there.

"Big Mechanisms are causal, explanatory models of complicated systems in which interactions have important causal effects," the solicitation states. "The collection of Big Data is increasingly automated, but the creation of Big Mechanisms remains a human endeavor made increasingly difficult by the fragmentation and distribution of knowledge. To the extent that we can automate the construction of Big Mechanisms, we can change how science is done."

DARPA said new research will be necessary in numerous fields related to how machines and humans process massive amounts of data, including statistical and knowledge-based natural language processing, curation and ontology, systems and mathematical biology, and representation and reasoning.

As the solicitation points out, some of the systems that matter most to the Defense Department are complex, but they are studied in a fragmented, inconsistent fashion that makes it difficult to build complete, accurate models.

For its first project, DARPA plans to use the Big Mechanism program to address cancer pathways, which are molecular interactions that can cause some cells to turn cancerous.

Officials are seeking an automated way to extract useful information and causal mechanisms from abstracts and research papers about cancer biology. Computers should then assemble the fragments of data into "complete pathways of unprecedented scale and accuracy" to determine cause-and-effect relationships and how those relationships could be altered or manipulated to prevent or control the disease.

"The language of molecular biology and the cancer literature emphasize mechanisms," said Paul Cohen, a DARPA program manager, in a press release. "Papers describe how proteins affect the expression of other proteins and how these effects have biological consequences. Computers should be able to identify causes and effects in cancer biology papers more easily than in, say, the literatures of sociology or economics."

He added: "Unfortunately, what we know about big mechanisms is contained in enormous, fragmentary and sometimes contradictory literatures and databases, so no single human can understand a really complicated system in its entirety. Computers must help us."

If DARPA's effort with cancer research shows promise, the implications would be far-reaching. Any article or bit of published research about any topic of interest could become part of a computer-maintained and computer-examined causal model of a larger system -- what DARPA calls the Big Mechanism.

"Causal models are needed to predict how systems will respond to interventions -- how a patient or an economy will respond to a drug or a new tax -- and to understand why systems behave as they do," Cohen said. "By emphasizing causal models and explanation, Big Mechanism may be the future of science."

About the Author

Frank Konkel is a former staff writer for FCW.

The 2015 Federal 100

Meet 100 women and men who are doing great things in federal IT.


  • Shutterstock image (by venimo): e-learning concept image, digital content and online webinar icons.

    Can MOOCs make the grade for federal training?

    Massive open online courses can offer specialized IT instruction on a flexible schedule and on the cheap. That may not always mesh with government's preference for structure and certification, however.

  • Shutterstock image (by edel): graduation cap and diploma.

    Cybersecurity: 6 schools with the right stuff

    The federal government craves more cybersecurity professionals. These six schools are helping meet that demand.

  • Rick Holgate

    Holgate to depart ATF

    Former ACT president will take a job with Gartner, follow his spouse to Vienna, Austria.

  • Are VA techies slacking off on Yammer?

    A new IG report cites security and productivity concerns associated with employees' use of the popular online collaboration tool.

  • Shutterstock image: digital fingerprint, cyber crime.

    Exclusive: The OPM breach details you haven't seen

    An official timeline of the Office of Personnel Management breach obtained by FCW pinpoints the hackers’ calibrated extraction of data, and the government's step-by-step response.

  • Stephen Warren

    Deputy CIO Warren exits VA

    The onetime acting CIO at Veterans Affairs will be taking over CIO duties at the Office of the Comptroller of the Currency.

  • Shutterstock image: monitoring factors of healthcare.

    DOD awards massive health records contract

    Leidos, Accenture and Cerner pull off an unexpected win of the multi-billion-dollar Defense Healthcare Management System Modernization contract, beating out the presumptive health-records leader.

  • Sweating the OPM data breach -- Illustration by Dragutin Cvijanovic

    Sweating the stolen data

    Millions of background-check records were compromised, OPM now says. Here's the jaw-dropping range of personal data that was exposed.

  • FCW magazine

    Let's talk about Alliant 2

    The General Services Administration is going to great lengths to gather feedback on its IT services GWAC. Will it make for a better acquisition vehicle?

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above