Big Data

DARPA wants better ways to search the web

Big Data

The Defense Advanced Research Projects Agency is looking for a new way for defense and other government agencies -- and eventually the public -- to search the ocean of Internet data more deeply.

The research agency said it wants ways to perform user-defined, domain-specific searches of public information, instead of the current commercial one-size-fits-all approach that uses the same set of search tools for all inquiries.

In a Feb. 9 statement, DARPA said that while that commercial approach has been wildly successful, it doesn't work as well for some government applications.

For example, commonly used search systems remain mostly manual processes that don't save sessions, require almost exact input with one-at-a-time entry, and don't organize or aggregate results beyond a list of links. They also miss information that could be embedded deeper in the web that aren't indexed by standard search engines, and skip over information that might be shared across web pages.

DARPA's statement said the initial application for its deep-web "Memex" program would enhance the Defense Department's efforts to combat human trafficking, a scourge that has a significant web presence. Using a more flexible search technology, DARPA contends the DoD could look deeper into online forums, chats, advertisements, job postings, hidden services and other out-of-the-way sites that enable the growing industry of modern slavery. An index curated for the counter-trafficking domain, along with configurable interfaces for search and analysis, would unlock new opportunities to uncover and derail trafficking enterprises, the DARPA statement said.

DARPA launched the Memex program looking to develop a next-generation search engine that will allow users to extend the reach of current search capabilities and quickly and thoroughly organize subsets of information based on individual interests. The program, the agency said, also aims to produce search results that are more immediately useful to specific domains and tasks, and to improve the ability of military, government and commercial enterprises to find and organize mission-critical publicly available information on the Internet.

"We're envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around," Chris White, DARPA program manager, said in the statement. "By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualize access to information. Ease of use for non-programmers is essential."

Through its solicitation, DARPA said, the program would explore three technical areas of interest: domain-specific indexing, domain-specific search, and DoD-specified applications.

DARPA noted that it is not interested in proposals for attributing anonymous services, deanonymizing or attributing identity to servers or IP addresses, or accessing information not intended to be publicly available. The agency said the program plans to use commodity hardware and emphasize creating and leveraging open source technology and architecture.

The agency plans a proposer's day on Feb 18 in Arlington, Va., to familiarize interested parties with the program.

About the Author

Mark Rockwell is a staff writer at FCW.

Before joining FCW, Rockwell was Washington correspondent for Government Security News, where he covered all aspects of homeland security from IT to detection dogs and border security. Over the last 25 years in Washington as a reporter, editor and correspondent, he has covered an increasingly wide array of high-tech issues for publications like Communications Week, Internet Week, Fiber Optics News, magazine and Wireless Week.

Rockwell received a Jesse H. Neal Award for his work covering telecommunications issues, and is a graduate of James Madison University.

Click here for previous articles by Rockwell. Contact him at or follow him on Twitter at @MRockwell4.

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Shutterstock image: looking for code.

    How DOD embraced bug bounties -- and how your agency can, too

    Hack the Pentagon proved to Defense Department officials that outside hackers can be assets, not adversaries.

  • Shutterstock image: cyber defense.

    Why PPD-41 is evolutionary, not revolutionary

    Government cybersecurity officials say the presidential policy directive codifies cyber incident response protocols but doesn't radically change what's been in practice in recent years.

  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group