Agencies seek security metrics
- By Diane Frank
- Jun 19, 2000
As if creating information security programs is not hard enough, most government
agencies are now realizing that they have no way to measure the effectiveness
of those programs.
The key, they say, is defining metrics. Across government, agencies are
trying to develop yardsticks against which to measure the success of their
programs. But while many agencies are trying to get a handle on these issues,
none are working together.
"There are lots of players out there, but there is no real rule book, and
there seems to be very little sharing," said Fran Nielsen, a computer scientist
at the National Institute of Standards and Technology's Computer Security
The NIST Computer System Security and Privacy Advisory Board (CSSPAB)
sponsored a workshop last week about security metrics, trying to determine
what solutions are available to federal agencies and what work needs to
The issue is multifaceted. Agencies need to figure out how to measure the
level of risk to a system — to know what security to put in place — the
security capability and awareness of employees, and the improvement from
one measurement to the next.
The biggest problem is determining what needs to be measured, workshop participants
agreed. "Measurement is fine, but measurement that does not link to action
does no good," said James Craft, information system security officer at
the U.S. Agency for International Development.
And an agency should not just perform measurements and find vulnerabilities
without measuring whether they are fixing those vulnerabilities and improving
their security, said Bill Hadesty, associate chief information officer for
cybersecurity at the Agriculture Department. "You've got to understand whether
you're solving the problem," he said.
It appears that agencies have a lot of tools with which to work. For
example, plenty of metrics exist for individual security products, including
the National Information Assurance Partnership's Common Criteria Evaluation,
since it is fairly easy to measure whether a product does what a vendor
But agencies have no clear way to measure the effectiveness of those products
when they are put together into a network. And it is even harder to measure
the effectiveness of security awareness and training programs, which aim
to reduce the number of vulnerabilities created by human error.
Meanwhile, a joint public/private- sector organization has developed
the Systems Security Engineering Capability Maturity Model (SSE-CMM), based
on the Carnegie Mellon University CMM system to measure the maturity of
an organization's processes.
Also, the CIO Council's Security Committee is developing an Information
Technology Security Assessment Framework based on the CMM system, the Office
of Management and Budget's Circular A-130 Appendix III and other federal
The General Accounting Office provides its Federal Information System
Controls Audit Manual to agencies and inspectors general to use as security
audit metrics. Internally, GAO auditors use a five-level system that measures
the effectiveness of agencies' security. The Defense Department is also
developing its own metrics system, the Information Assurance Readiness Assessment.
Still, the plethora of possible solutions leaves most agencies trying
to figure out which way to go.
A governmentwide standard could help everyone get on the same page,
said Franklin Reeder, chairman of the CSSPAB. Because of the relative immaturity
of this area, the board hopes to be able to foster the continued development
of these models and systems. "We hope to come out of this with the basis
for a conversation about what we do next," he said.