Agencies seek security metrics

As if creating information security programs is not hard enough, most government

agencies are now realizing that they have no way to measure the effectiveness

of those programs.

The key, they say, is defining metrics. Across government, agencies are

trying to develop yardsticks against which to measure the success of their

programs. But while many agencies are trying to get a handle on these issues,

none are working together.

"There are lots of players out there, but there is no real rule book, and

there seems to be very little sharing," said Fran Nielsen, a computer scientist

at the National Institute of Standards and Technology's Computer Security


The NIST Computer System Security and Privacy Advisory Board (CSSPAB)

sponsored a workshop last week about security metrics, trying to determine

what solutions are available to federal agencies and what work needs to

be done.

The issue is multifaceted. Agencies need to figure out how to measure the

level of risk to a system — to know what security to put in place — the

security capability and awareness of employees, and the improvement from

one measurement to the next.

The biggest problem is determining what needs to be measured, workshop participants

agreed. "Measurement is fine, but measurement that does not link to action

does no good," said James Craft, information system security officer at

the U.S. Agency for International Development.

And an agency should not just perform measurements and find vulnerabilities

without measuring whether they are fixing those vulnerabilities and improving

their security, said Bill Hadesty, associate chief information officer for

cybersecurity at the Agriculture Department. "You've got to understand whether

you're solving the problem," he said.

It appears that agencies have a lot of tools with which to work. For

example, plenty of metrics exist for individual security products, including

the National Information Assurance Partnership's Common Criteria Evaluation,

since it is fairly easy to measure whether a product does what a vendor


But agencies have no clear way to measure the effectiveness of those products

when they are put together into a network. And it is even harder to measure

the effectiveness of security awareness and training programs, which aim

to reduce the number of vulnerabilities created by human error.

Meanwhile, a joint public/private- sector organization has developed

the Systems Security Engineering Capability Maturity Model (SSE-CMM), based

on the Carnegie Mellon University CMM system to measure the maturity of

an organization's processes.

Also, the CIO Council's Security Committee is developing an Information

Technology Security Assessment Framework based on the CMM system, the Office

of Management and Budget's Circular A-130 Appendix III and other federal


The General Accounting Office provides its Federal Information System

Controls Audit Manual to agencies and inspectors general to use as security

audit metrics. Internally, GAO auditors use a five-level system that measures

the effectiveness of agencies' security. The Defense Department is also

developing its own metrics system, the Information Assurance Readiness Assessment.

Still, the plethora of possible solutions leaves most agencies trying

to figure out which way to go.

A governmentwide standard could help everyone get on the same page,

said Franklin Reeder, chairman of the CSSPAB. Because of the relative immaturity

of this area, the board hopes to be able to foster the continued development

of these models and systems. "We hope to come out of this with the basis

for a conversation about what we do next," he said.


  • Defense
    Ryan D. McCarthy being sworn in as Army Secretary Oct. 10, 2019. (Photo credit: Sgt. Dana Clarke/U.S. Army)

    Army wants to spend nearly $1B on cloud, data by 2025

    Army Secretary Ryan McCarthy said lack of funding or a potential delay in the JEDI cloud bid "strikes to the heart of our concern."

  • Congress
    Rep. Jim Langevin (D-R.I.) at the Hack the Capitol conference Sept. 20, 2018

    Jim Langevin's view from the Hill

    As chairman of of the Intelligence and Emerging Threats and Capabilities subcommittee of the House Armed Services Committe and a member of the House Homeland Security Committee, Rhode Island Democrat Jim Langevin is one of the most influential voices on cybersecurity in Congress.

Stay Connected


Sign up for our newsletter.

I agree to this site's Privacy Policy.