Web mashups put transparency to the test

Web mashups are expected to become a valuable tool for sharing and analyzing government data, but mashup creators say government could do more to make their data more accessible.

A cornerstone of the Obama administration’s vision for a more open and accountable government rests on making government data easily accessible to the public.

However, that’s only the first step. From there, people and interest groups must be able to analyze or package the data in ways that make it more useful to others.

Web mashups — sites that present data from multiple sources in one place — are expected to become a valuable tool for sharing and analyzing government data. There are already several examples of mashups that provide more convenient access to public information and services or produce innovative insights about the economy. Visitors could not necessarily get that information from one agency’s site alone.

Architects of three early efforts say that even though agency data is increasingly available in standard Web formats such as Extensible Markup Language, building a mashup is still harder than it needs to be, and the government could do more to help.

“For mashups, it’s now dependent on the implementer to reconcile the data in order to use them, and that can often be an arduous task,” said Ruhollah Farchtchi, an associate at Booz Allen Hamilton.

That work is expected to lighten once Semantic Web, or Web 3.0, technologies become more available. Whereas Web 2.0 is about connecting people through social-networking applications such as Facebook and Twitter, Web 3.0 will be about connecting information in more convenient, meaningful and dynamic ways.

To lay the groundwork for Web 3.0, government and other data providers will use Semantic Web standards, such as the Resource Description Framework (RDF) and Web Ontology Language (OWL), to add context and meaning to the data they post online.

Sites such as the Office of Management and Budget’s Data.gov might end up doing the Semantic Web work for agencies by converting their data to the required standards and providing the application programming interfaces (APIs) that mashup developers will need to access the data.

In the meantime, three mashup examples illustrate the possibilities these composite Web sites can offer and the challenges their creators face.

This We Know

This We Know was built as a Semantic Web mashup from the beginning. The site takes data that agencies have uploaded to Data.gov and converts it into an RDF database, which is used to organize and present the data according to geographic communities. For example, site visitors can choose a town and find data about its population, unemployment rate and cancer incidence, among other facts.

The site won a competition launched last year by the nonprofit Sunlight Foundation, which honored the best application developed from data on Data.gov.

The site’s developers organized the site based on geographic places to make it more compelling to visitors, said Michael Knapp, managing director and partner of GreenRiver.org, a Vermont-based Web application development and data visualization company.

Place is also a convenient foundation for developers because many government databases already include location data.

GreenRiver.org partnered on the mashup with Sway Design, a Web design studio, and Semantic Web database company Intellidimension.

The site represents about 350 hours of labor. Most of that was spent on developing the interface, though the companies also spent considerable time cleaning up the data because of the nonstandard way it was described in each of the databases.

Knapp said agencies should improve data standards because mashup developers need to understand what the various fields and values mean in the databases. Agencies could significantly help developers by creating and following a standard data dictionary.

“If you have that, it takes 75 percent of the work out of putting these mashups together,” Knapp said.

FedViewer.com

FedViewer.com presents a host of national economic data from the U.S. Federal Reserve as a visual temperature map — darkening shades of red indicate trouble, lighter colors point to positive trends. The site shows the performance over time of a range of economic indicators, along with contextual facts, such as when a recession or a government bailout program began.

The site is the brainchild of Will Riley, who wanted to examine the Federal Reserve's data as part of an investment analysis project. Traditional tools such as spreadsheets and graphs did not work well.

“Representing 10, 20 or several hundred variants simultaneously forces one to consider [other] solutions,” said Riley, who runs Real World Analytics, an Internet consulting firm. “The FedViewer.com moving average display attempts to manage such combinations in a comfortable and, hopefully, intuitive interface.”

The Federal Reserve offers some of its statistical data in comma-separated values and XML formats, which developers can incorporate automatically into a mashup. But other important databases require techniques such as screen-scraping, in which developers write a custom program to read and copy data presented on a Web page rather than accessing the original database.

But screen-scraping is a brittle technique and depends on page content retaining a consistent form, Riley said. A change in a single character in the values is enough to break the extraction process. Also, screen-scraping can be time-consuming, so it isn’t practical for pulling data from lots of different Web sites.

Time equals cost, so datasets that can’t be handled automatically likely won't be part of the information that FedViewer.com seeks to present.

Riley said he looks forward to when automated queries that use SPARQL, a software query language for RDF databases, can pull timeline data by topic from federal and municipal sites, Wikipedia, and elsewhere.

“The scope of official [government] activity could be rebiased toward the provision of public APIs rather than the work of building and maintaining data browsing sites, such as we have now,” he said.

For that to happen, he said he believes a single federal entity should develop data-mastering and publication standards. In addition, an agency should create guidelines for handling data at the municipal level.

Riley said he believes that President Barack Obama’s Open Government Directive, which calls on agencies to release more accessible and user-friendly datasets, is a historic move and step in the right direction.

Neighborhood Knowledge California

Neighborhood Knowledge California provides tools that allow potential homeowners and community grass-roots organizations throughout the state to design maps, collate information and publish their own research, using data from private and public resources available on NKCA.

The goal is to increase understanding of issues such as fair housing and fair lending laws in areas where there are usually few resources that people can tap to find that information. The University of California at Los Angeles, which runs NKCA, worked with more than 40 organizations in five metropolitan areas in the state to build the site.

NKCA builds on the success of Neighborhood Knowledge Los Angeles, the first site of its kind. Another project based on the NKLA example is Living Independently in Los Angeles, which was created by local residents with disabilities. The site uses their expert knowledge to identify and map independent living resources.

The tools used with the newer sites have come a long way since NKLA debuted in the mid-1990s, when the goal was the simple dissemination of information, said Charanjeet Singh, NKCA’s program manager.

In the mid-1990s, users in search of particular data, such as building permits or code complaints, needed to separately query each applicable database to get the information. Now the sites include an integrated database in which users can view all of the property data at one time.

Mapping tools also have advanced. The earliest sites used spreadsheets that automatically sorted data by geographic fields, which users then extracted to use with geographic information system mapping software. That gave a decent, though limited, 2-D representation, but the goal is for a more immersive system.

NKCA now uses Google Maps with Street View, which allows users to automatically provide maps where people can zoom in to see photographs of locations and properties. The goal is to cut down on the flood of available data by making it more instantly meaningful, which is where the Semantic Web will help.

“It means you don’t have to go through the data hassles to make it more accessible,” Singh said. “You can bring in more data from different sources and give it different contexts.”

He said he believes government agencies are doing a better job of understanding what type of data is important to the public. The missing link is OWL, RDF and other Semantic Web APIs that agencies need to build to make their data more easily accessible to users, many of whom are building their sites on-the-fly.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.