Schwartz: Focusing on searchability

When it comes to online searches, people are concerned with what they aren't finding.

In March 2007, I turned to Google and Yahoo search engines to find a proposed government rule on whether polar bears should be on the endangered species list. Although I found old news releases from major environmental groups, I was surprised to find no hits that led me to a U.S. government Web site.  

Those familiar with e-government would certainly expect a hit on Alas, nothing.

At least I should have found a hit on the Fish and Wildlife Service’s Web site leading me to the proposed rule. Yet, nothing.

Sadly, my experience isn’t a fluke.

Google estimates that more than 2,000 U.S. government data sources are invisible to users of its search engine. The Pew Internet Project showed that commercial search engines are by far the most popular means of finding government information.

Because of this finding, my organization and colleagues at OMB Watch felt the missing data situation highlighted critical gaps in online access to government information.

Our organizations released a study, “Hiding in Plain Sight,”which found that vast amounts of government information are invisible to the industry’s major search engines. The amount of hidden information is as troubling as the quality of hidden information.

For example, we found that the following agency resources had information obscured:

  • Federal Emergency Management Agency databases. This includes a Flood Map Modernization project at FEMA, which shows flood hazards.

  • Other Homeland Security Department databases. This includes topics such as environmental radiation monitoring.

  • Federal Business Opportunities Web site database. This list has about 200 government business opportunities in the field of telecommunications.

  • Central Contractor Registration database. The database lists who does business and receives money from the federal government.

  • Federal Procurement Data Services database. This has data on all government contracts, including all telecom contracts.

  • Smithsonian Institution resources. This includes many online content collections, including the Smithsonian Institution Research Information System.

We believe that most of this information is not available because of relatively minor technical obstacles that the agencies could — and should — quickly remedy.

In particular, these sites are either not site mapping their data using the industry standard Extensible Markup Language protocol or they are putting it behind directories listed in robots.txt files, which instruct search engines to voluntarily ignore certain areas of the site.

It is unclear whether these agencies know that their information is not publicly searchable and have not taken the adequate steps to change their practices or if the agencies do not know that the search engines are not indexing important information.

For agencies to solve the problem, they must first acknowledge it, and in this case, making information available is the only way to do that.

Schwartz ( is the deputy director of the Center for Democracy and Technology. 

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group