Schwartz: Focusing on searchability

When it comes to online searches, people are concerned with what they aren't finding.

In March 2007, I turned to Google and Yahoo search engines to find a proposed government rule on whether polar bears should be on the endangered species list. Although I found old news releases from major environmental groups, I was surprised to find no hits that led me to a U.S. government Web site.  

Those familiar with e-government would certainly expect a hit on Regulations.gov. Alas, nothing.

At least I should have found a hit on the Fish and Wildlife Service’s Web site leading me to the proposed rule. Yet, nothing.

Sadly, my experience isn’t a fluke.

Google estimates that more than 2,000 U.S. government data sources are invisible to users of its search engine. The Pew Internet Project showed that commercial search engines are by far the most popular means of finding government information.

Because of this finding, my organization and colleagues at OMB Watch felt the missing data situation highlighted critical gaps in online access to government information.

Our organizations released a study, “Hiding in Plain Sight,”which found that vast amounts of government information are invisible to the industry’s major search engines. The amount of hidden information is as troubling as the quality of hidden information.

For example, we found that the following agency resources had information obscured:



  • Federal Emergency Management Agency databases. This includes a Flood Map Modernization project at FEMA, which shows flood hazards.

  • Other Homeland Security Department databases. This includes topics such as environmental radiation monitoring.

  • Federal Business Opportunities Web site database. This list has about 200 government business opportunities in the field of telecommunications.

  • Central Contractor Registration database. The database lists who does business and receives money from the federal government.

  • Federal Procurement Data Services database. This has data on all government contracts, including all telecom contracts.

  • Smithsonian Institution resources. This includes many online content collections, including the Smithsonian Institution Research Information System.



We believe that most of this information is not available because of relatively minor technical obstacles that the agencies could — and should — quickly remedy.

In particular, these sites are either not site mapping their data using the industry standard Extensible Markup Language protocol or they are putting it behind directories listed in robots.txt files, which instruct search engines to voluntarily ignore certain areas of the site.

It is unclear whether these agencies know that their information is not publicly searchable and have not taken the adequate steps to change their practices or if the agencies do not know that the search engines are not indexing important information.

For agencies to solve the problem, they must first acknowledge it, and in this case, making information available is the only way to do that.

Schwartz (ari@cdt.org) is the deputy director of the Center for Democracy and Technology. 

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.

Featured

  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from Shutterstock.com

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group