Schwartz: Focusing on searchability

When it comes to online searches, people are concerned with what they aren't finding.

In March 2007, I turned to Google and Yahoo search engines to find a proposed government rule on whether polar bears should be on the endangered species list. Although I found old news releases from major environmental groups, I was surprised to find no hits that led me to a U.S. government Web site.  

Those familiar with e-government would certainly expect a hit on Regulations.gov. Alas, nothing.

At least I should have found a hit on the Fish and Wildlife Service’s Web site leading me to the proposed rule. Yet, nothing.

Sadly, my experience isn’t a fluke.

Google estimates that more than 2,000 U.S. government data sources are invisible to users of its search engine. The Pew Internet Project showed that commercial search engines are by far the most popular means of finding government information.

Because of this finding, my organization and colleagues at OMB Watch felt the missing data situation highlighted critical gaps in online access to government information.

Our organizations released a study, “Hiding in Plain Sight,”which found that vast amounts of government information are invisible to the industry’s major search engines. The amount of hidden information is as troubling as the quality of hidden information.

For example, we found that the following agency resources had information obscured:



  • Federal Emergency Management Agency databases. This includes a Flood Map Modernization project at FEMA, which shows flood hazards.

  • Other Homeland Security Department databases. This includes topics such as environmental radiation monitoring.

  • Federal Business Opportunities Web site database. This list has about 200 government business opportunities in the field of telecommunications.

  • Central Contractor Registration database. The database lists who does business and receives money from the federal government.

  • Federal Procurement Data Services database. This has data on all government contracts, including all telecom contracts.

  • Smithsonian Institution resources. This includes many online content collections, including the Smithsonian Institution Research Information System.



We believe that most of this information is not available because of relatively minor technical obstacles that the agencies could — and should — quickly remedy.

In particular, these sites are either not site mapping their data using the industry standard Extensible Markup Language protocol or they are putting it behind directories listed in robots.txt files, which instruct search engines to voluntarily ignore certain areas of the site.

It is unclear whether these agencies know that their information is not publicly searchable and have not taken the adequate steps to change their practices or if the agencies do not know that the search engines are not indexing important information.

For agencies to solve the problem, they must first acknowledge it, and in this case, making information available is the only way to do that.

Schwartz (ari@cdt.org) is the deputy director of the Center for Democracy and Technology. 

The Fed 100

Read the profiles of all this year's winners.

Featured

  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group