Measuring agency performance: Out of reach
- By William Matthews
- Apr 03, 2000
It is clear to Dean Mesterharm, the Social Security Administration's deputy
commissioner for systems, that investing in information technology is essential
To improve delivery of $416 billion in benefits to 51 million
Americans each year, SSA spent more than $1 billion to put a PC on every
worker's desk. Now employees have at their fingertips reference manuals,
electronic claim forms, decision support software, a case control system,
quality assurance and other information. Automation enables SSA to serve
an ever-increasing number of Americans with 25 percent fewer employees than
it had in 1985.
"That couldn't happen without automation," Mesterharm said.
But trying to prove it is another matter.
"There's not a simple mathematical equation that compares before and
after," Mesterharm said. "It's complex because a lot more things are changing
than just the technology part." Processes, laws and workloads, for example,
are in a state of flux, he said.
Like other agencies, SSA has sent Congress its first performance report
required by the Government Performance and Results Act. And like other agencies
that struggled to meet the March 31 deadline, SSA has had a hard time providing
a solid cost-benefit analysis for its spending on information technology.
GPRA reports are supposed to provide taxpayers a businesslike accounting
of how agencies spend money on programs, including IT, and what the spending
But senior government officials, including agency chief information
officers, concede that the first report cards generally fall short of providing
the clear and useful picture of spending and results envisioned in 1993
when Congress passed GPRA.
"No one should pretend that we don't have a very long way to go," said
Joshua Gotbaum, controller and associate director of the Office of Management
IT has proven to be one of the hardest functions of agency operations
to fit into a corporate accounting archetype. Together, government agencies
spend more than $30 billion a year buying and maintaining computer systems
and software. But senior agency executives say they are often unable to
pinpoint — unable sometimes to even estimate — the value derived from that
"It's a fair question" when Congress asks agencies to quantify IT spending
results, said Daryl White, chief information officer at the Interior Department
and co-chairman of the CIO Council's committee on capital planning and IT
management. In exchange for handing agencies billions of dollars, lawmakers
are right to want to know what the money is buying. "We can't just tell
them they're getting policy," he said.
But enumerating the returns for IT investments "has been troublesome
for CIOs," White said. "And it will continue to be so until we can decide
on a common metric." The problem is that neither Congress nor anyone else
has devised a workable way to perform reliable cost-benefit analysis for
government spending on IT.
The Problem of What to Measure
With GPRA, Congress intended to make agencies operate more like businesses,
OMB's Gotbaum said. "It forces us to focus on the bottom line." That's easy
in business, where the bottom line is "either you do or don't make a profit,"
Gotbaum said. "Government is much messier."
No agencies exist primarily to make money, he said. They exist for national
defense, to help the elderly and the poor, to protect the environment and
for dozens of other purposes. But the lack of a single, simple measure such
as profitability makes it more difficult to gauge performance.
For some agencies, performance can be measured by the number of people
served, the number of crimes solved, improvements in health or the decreases
in complaints. But assessing the value of the various contributors to those
achievements is problematic.
No one would question that the Environmental Protection Agency had made
a valuable contribution if its policies reduced the number of deaths due
to pollution. But it would be difficult to attach a precise dollar amount
to those accomplishments.
It would be even more difficult to calculate the contribution IT spending
made to those outcomes, said Paul Wohlleben, former deputy CIO at the EPA.
Yet information systems are woven throughout EPA activities. Without them,
the agency would be crippled.
Even when technology is tied directly to improved agency performance,
it can be difficult to weigh costs against benefits, said Patrick Plunkett,
a program analyst for the General Services Administration's Office of Governmentwide Policy. The National Weather Service, for example, has won praise
for providing faster warnings of tornadoes, thunderstorms and hurricanes.
Better radar, supercomputers and other advanced technology made better forecasting
possible. But the agency has not made a clear assessment of the value of
IT's contribution to that outcome, he said.
Often, the best measure may not be dollars, Plunkett said. "Congress wants
to see that the money they spend on information technology has some kind
of return." Showing that service has improved, taxpayer satisfaction has
increased or costs have been avoided may be sufficient, he said.
Trying to Solve Computer Chaos
If the accounting Congress wants is difficult, the reason for it is
GPRA was passed when polls showed that the public believed the government
wasted 48 cents of every tax dollar it collected. Information technology
was one of the notorious fiscal black holes.
Major federal agencies, from the Internal Revenue Service to the Federal
Aviation Administration, and programs such as Social Security and Medicare
were haunted by computer horror stories.
In 1994, then-Sen. William Cohen (R-Maine) released a report, "Computer
Chaos: Billions Wasted Buying Federal Computer Systems." It attacked agencies
for failing to perform better cost-benefit analyses on their computer programs
and led to passage of the Clinger-Cohen Act, which requires such analyses.
But the act lacked deadlines and penalties and provided no money or
strategy for agencies to do a better job of developing and maintaining IT,
said Joan Steyaert, deputy associate administrator of GSA's Office of Information
By 1996, cost overruns on government computer projects had exceeded
$200 billion, Steyaert said. Deadlines regularly slipped, programs were
unstable and the worth of many projects was questionable, she said.
One of the biggest problems — then and now — is a lack of data on true costs,
true benefits and program risks, said David McClure, an IT analyst at the
General Accounting Office. "There's no baseline," and it will take years
to collect such data, he said.
The dearth of data makes it harder for agencies to develop sound IT
performance plans and reliable performance reports. And the deficiency has
been noted on Capitol Hill.
A review last summer of GPRA performance plans submitted by the 24 largest
federal agencies showed that many did not include performance goals for
IT, said an aide to Sen. Fred Thompson (R-Tenn.).
The Customs Service, despite planning a $1.2 billion computer modernization
project, did not write IT performance goals, the aide said. Neither did
the Agriculture Department's Bureau of Land Management for its automated
land and mineral records system project, which was under development for
15 years and had a budget that nearly doubled from its original $240 million
price tag before BLM killed it last year.
GAO in March lodged similar criticism against the Department of Housing
and Urban Development, the Department of Veterans Affairs, the National
Weather Service and SSA.
"From our perspective, information technology is a major challenge for
every federal agency," Thompson's aide said. Thompson, chairman of the Senate
Governmental Affairs Committee, is a bulldog on government computer reform.
But there have been some GPRA successes as well.
Gotbaum cites the Health Care Financing Administration as having had
some success in trying to improve IT development. The agency is recovering
from a computer disaster by following GPRA guidelines.
In 1994, HCFA began to work on a massive $151 million Medicare Transaction
System that was supposed to take over the work of nine older systems and
process $280 billion worth of Medicare bills each year. But by 1997, the
agency had spent $43 million and was farther than ever from owning a workable
Medicare billing system. Far behind schedule, the estimated cost had ballooned
in price to nearly $1 billion. All the agency could do was pull the plug.
"HCFA's computers were a mess," Gotbaum told the House Rules Committee March
22. The agency regrouped and submitted a new modernization plan to OMB,
but it, too, was unworkable. HCFA had failed to follow the basic guidelines
of GPRA, he said.
"We went back to first principles," Gotbaum said. HCFA set IT performance
goals and then developed an IT architecture around them. Last year, OMB
sent HCFA's plan to Congress, which approved funding this year and is expected
to approve funding for 2001, Gotbaum said.
GAO also has been trying to impose GPRA-style discipline on agency IT
projects. Its most significant action so far has been to correct "serious
management and technical weaknesses with the IRS' multibillion-dollar effort
to modernize its tax systems," GAO reported in March.
In 1996, Congress cut off most of the funding for the IRS' Tax System
Modernization program, charging that the agency had little to show for the
$3.3 billion it already had spent on system upgrades. The IRS then developed
a new multibillion-dollar modernization project called the Prime Systems
Integration Services Contract, in which it plans to follow more businesslike
IT development processes.
At GAO's direction, the IRS established a facility to test new computer
systems before installing them. The tax agency also agreed to hire only
software contractors with "mature software development capabilities."
GAO last year persuaded the IRS to adopt an incremental approach to
the multiyear IT modernization program, trimming the IRS' budget request
from $500 million to $323 million.
'Slower Than a Turtle's Crawl'
If figuring out how to comply with GPRA has been frustrating for agency
managers, waiting for more businesslike performance from agencies has been
equally frustrating for Congress.
"The government has made progress toward this goal, but at a pace slower
than a turtle's crawl," said Rep. Stephen Horn (R-Calif.). "So far, only
a handful of agencies have had limited success" at linking their program
plans to their budget requests, as GPRA requires, Horn said. That linkage
"is supposed to form the basis of their budget requests," he said.
Unresponsive agencies aren't the only problem, said Horn, who is chairman
of the House Government Management, Information and Technology Subcommittee.
"Widespread, enthusiastic leadership is missing at all levels of government,
from Congress and the OMB to each federal department and agency."
"It irks me not to see a lot of people show up at these meetings," he
groused during a Rules Committee hearing on GPRA March 22. Only four of
13 committee members attended, and only Rep. John Linder (R-Ga.) stayed
for the whole hearing.
Linder said he may propose a change in House rules that would force
committees to pay closer attention to agency compliance with GPRA during
Closer oversight by Congress would help, agreed GAO comptroller David
Walker. And tying GPRA compliance to budgets would be a good way to make
agencies try harder to develop useful performance goals and measurements,
Better oversight could begin this spring. The incoming GPRA performance
reports should be adequate enough to give Congress some insight into agency
goals and performance in time for budget hearings.
Among the items lawmakers should look for in the reports is "the status
of the agency's efforts to use information technology to achieve results,"
Going Beyond 'Gotcha'
Agency officials familiar with the politics of the annual budget exercise
remain wary of the direction GPRA-based budget hearings might take.
"Performance measurements are supposed to tell how well programs were
doing," said GSA's Plunkett, but "the Washington environment is about "gotcha.'"
There is widespread concern among agency officials that when performance
reports reveal deficiencies, they will be used to hammer agency heads and
be turned into ammunition in budget battles and wider political brawls.
"Agencies are afraid to be too honest," he said. "This is a political
environment, and decisions are made on a political basis."
Gotbaum has the same concern. "When an agency makes a good faith effort
but doesn't meet their goal, or if the measures need to be revised, we need
to be careful," he told the Rules Committee. If reporting poor performance
leads to public criticism, agencies will conclude that the reward for honest
analysis will be punishment. "If agencies come to that conclusion, GPRA
will never work," he said.
Gotbaum urged Congress to use "a combination of public encouragement
and private criticism" to elicit honest assessments of program costs and
payoffs from agencies.
When GPRA was passed, former OMB Director Franklin Raines foresaw the
problem in slightly different terms. If agencies' budgets are based on their
performance reports, he said, the only performance reported will be good
If agencies are to actually perform better, GPRA has to be more than
a report, said Interior's White. Agencies have to change the way they work.
"You need a cultural change," he said.
Agencies have traditionally measured success by the amount of money
they have spent on programs, the number of staff members they employ and
the number of tasks they have completed, GAO's Walker said. Such measures
miss the point. Agencies should be measuring whether their programs contribute
to the quality and security of citizens, he said.
GPRA was intended to force a fundamental shift in the way agencies think
about carrying out their missions. Instead of focusing on "activities,"
agencies are supposed to focus on "outcomes."
For IT managers, that means no longer calling it a success when new
computer systems are installed and IT budgets are fully spent. Success must
be measured by what IT contributes to the agency's mission.
But the cultural change needs to go further still, White said. "It is
laudable that someone wants to hold the government accountable for the funding
given to us by the taxpayers, but if they're going to hold me accountable,
they need to give me the authority and resources to succeed."
Although Interior's budget includes about $500 million for IT, White
said much of it is outside his control. Instead, much of his responsibility
is for "support functions" — such as ensuring that the agency payroll system
works — and not for functions that directly affect the department's primary
mission of managing national resources, he said.
"Chief information officers and chief financial officers are not getting
sufficient support from the people they report to," said Jerome Climer,
president of the Congressional Institute. "You're not seeing the political
appointees asking a lot of performance results questions yet."
The task agencies face in developing meaningful IT analyses remains
a monumental one. But one, OMB's Gotbaum believes, that is doable as agencies
learn what is expected. "These changes are neither quick nor easy," Gotbaum
said. "They take time.
"The vision created by GPRA is a challenging one, but it is important
and, with effort, achievable," he said. "Government, like everyone else,
learns by doing."