Know your data

A top challenge for the government's efforts to wring more value from the data it collects — through better analysis and sharing — is ensuring that information in agency repositories is consistent, current and comprehensible to users and programmers.

A growing number of agencies are finding that one way to attain this goal is by tapping the benefits of "metadata." This data about data contains valuable information about each piece of information in an agency's data holdings, such as its source and format and the changes it has undergone since entering the system.

When separated from the primary data and collected in a metadata repository, this information gives systems administrators the means to organize their data holdings similar to a card catalog in a library, said Wayne Eckerson, director of education and research at the Data Warehousing

Institute.

This enables administrators to

introduce greater efficiency and accuracy into their data operations, eliminating inconsistencies, redundancies and irrelevant information. In turn, data growth and change can be more easily managed because new information can be filed using the metadata index.

Metadata also provides a foundation for greater data sharing and more advanced and potentially more lucrative information analysis when

using data mining applications, for example.

But metadata solutions also can pose significant cultural, technical and financial challenges, and "many people do as little as possible," Eckerson said. "You can build a data warehouse with minimal metadata and get by."

Resistance to data warehousing may be reflected in the marketplace, where metadata repository sales — chiefly by Computer Associates International Inc. and ASG (formerly Allen Systems Group Inc.) — total about $250 million per year, according to Michael Blechar, a vice president and research director at market research firm Gartner Inc.

The market is not very big or growing, he said, adding that Gartner's tally does not include sales of services and numerous general-purpose tools used by organizations to build metadata solutions.

Interest in metadata management "is growing in some places more than others," Blechar said. For

example, reflecting the increased popularity of Extensible Markup Language, more organizations are using metadata systems to provide their programmers with well-defined catalogs of their XML services and components.

Organizations are also using metadata to help add sophistication to data warehousing programs, such as advanced search and analysis

capabilities.

Federal agency efforts in this area generally have been hobbled by the lack of "metadata management policies or practices