Software conducts Deep Web searches

Officials at BrightPlanet Corp. believe the three-year-old company has what the government ordered: technology that will help agencies locate information stored on Web servers that is not found by traditional search engines or Web crawlers.

Technologies that provide access to data stored on the so-called Deep Web and help agencies manage and store digital data should serve agencies well in the era of information overload, said Michael Bergman, chief technology officer of the Sioux Falls, S.D.-based company.

"What's easy to find is the doorstep into these sites," he said. "What's impossible to find is their content." The amount of content on the Deep Web is 500 times greater than that visible to conventional search engines, he wrote in a September 2001 white paper.

Bergman also reported that the 60 largest known Deep Web sites contain about 750 terabytes of data. Many of those are government sites, including Government Printing Office Access, the National Climatic Data Center at the National Oceanic and Atmospheric Administration, and NASA's Earth Observing System Data and Information System.

BrightPlanet has two main capabilities designed to help users locate and manage this data. Deep Query Manager 2 enables agencies to find content on Web sites, including Deep Web and government sites, and store and manage it as files that can then be archived, organized, shared and compared.

The Automated Information Portal (AIP), geared toward existing Web portals and offered as a hosted service, organizes Web documents, Deep Web databases and existing Web content into a searchable portal.

BrightPlanet opened a sales office in Washington, D.C., in March to target Fortune 100 customers and federal agencies, said John Fry, director of strategic account sales at that office.

The company has federal and state customers of its own — South Dakota Live, the official site for all South Dakota government documents, uses AIP — and is in discussions with others, including the Homeland Security Department, but is interested in partnering.

Lt. Cmdr. Andrew Chester, NATO open source intelligence program coordinator, said the alliance's intelligence organization has been using Deep Query Manager for about two years.

NATO likes Deep Query Manager because it resides on the server. "Our analysts can get access to it anywhere on Earth," Chester said. It also allows a distributed workforce to work collaboratively, he said. "I can save results in Norfolk [Va.] that can be viewed by colleagues in Brussels," Belgium.

***

Needle in the online haystack

Deep Web sites contain dynamic content served up in real time from a database in response to a direct query. In contrast, "surface" Web pages are static and linked to other pages.

BrightPlanet Corp. has technology that can help agencies find information stored on Deep Web sites. Government sites that are considered Deep Web include the National Climatic Data Center at the National Oceanic and Atmospheric Administration and Government Printing Office Access.

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.

Featured

  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from Shutterstock.com

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group