Agile at DOD -- going beyond sticky notes and standing meetings

Steve Kelman reports on how the military is applying agile and DevOps to Pentagon-scale projects.

Agile development graphic

“In DOD now, everybody says they’re doing agile. But when they say they are doing agile, they mean you stand instead of sitting down at meetings, and that you use sticky notes to track tasks. If you do those things, you’re an agile god. They are still doing lengthy requirements development, and they are still designing to exact requirements. But they do this in two-week sprints.”

The speaker is above is Col. Enrique Oti, who was the main presenter at the recent Code for America summit, which I blogged about recently in the context of civic tech involvement with the Defense Department. Oti is a 21-year Air Force veteran who spent many years working on cybersecurity and developed an interest in how Silicon Valley does software development while on a stint at the Hoover Institution at Stanford, where the Air Force sends a few officers each year. He was leaving just as Secretary Ash Carter was standing up the Defense Innovation Unit Experimental (DIUx), and he was one of the initial people assigned there.

The project about which Oti presented was to redesign a system at the Air Operations Center in Qatar, from which planes fly on missions against ISIS. One of the center’s many tasks was aircraft refueling planning.

With many different planes in the air at many different locations, the center needed to figure out each day when each one would need refueling, where it would be when refueling was needed, and what refueling aircraft should meet it, given the schedules and locations of those aircraft. The system the center used to record the wheres and whens of the daily planning schedule was a big whiteboard where people wrote information using markers.

This is a complex task. The input data was a series of math calculations that had been developed for Excel. Because developing a plan took so long, the planners did not have enough time to consider alternative plans that might be more efficient or effective. Also, if a change happened after they had already finished the plan, it was very difficult to update that plan. Instead, they would simply launch another tanker aircraft, which created extra fuel costs.

DOD had a big contract to redesign the entire Air Operations Center, with its tens of applications, using traditional software development methods involving detailed requirements development (very complex for a complex system), coding, and then, when everything was done, testing/retesting and rollout. After 10 years and $750 million spent, no new operating software had yet been delivered.

DIUx’s work in Qatar grew out of a visit from the Defense Innovation Board, which visits DOD installations to learn about how better to use technology (Raj Shah, the first head of DIUx, sits on the Board, as does Jen Pahlka, the founder of Code for America). Shah decided he wanted to take on the project as an example of how DOD could develop software faster and cheaper. Oti signed onto the project as an opportunity to use real agile at DOD. The first minimum viable product coming out of the effort was fielded after four months.

The effort avoided the endless requirements process for DOD software development. “Our requirements were that Raj told me we needed to make life better for users," Oti said. "We began by talking with users about their pain points, and our approach was then to build a minimal viable product around that pain point.” That minimum viable product involved integrating the planning with the calculations so that any decision made by the planner was immediately validated or invalidated, which rapidly sped up the process.

Oti described the big cultural change to allow agile to happen is allowing work to be done by a smaller number of lower-level people, rather than huge, ponderous staff committees that consider everything to death. He feels you need senior leadership who will buy into that approach to make it possible.

Listening to Oti describe what his team did, my reaction was that this was all pretty basic stuff, sort of agile 101. I hardly regard myself as an agile maven, but not much sounded new to me. However, Oti noted that these practices are still quite unusual at DOD, though there are small pockets of use. (He said the defense organization that has most strongly embraced agile is the National Geospatial Intelligence Agency, which Oti consulted for suggestions.)

For me the most interesting parts of Oti’s account were his discussions of product testing, cyber compliance, and personnel security clearances. The basic message was don’t wait until the end. Oti said there are COTS tools very widely used in industry to make these processes automated and continuous. Using these tools, new code is fed for testing as soon as it is ready, then quickly tested and put into use.

“It is a rigorous process, we are proving every day that it meets operating requirements,” he said. Continuous testing of increments of software is crucial to speeding up deployment, and prevents the do-loop of software not being tested till the end, problems discovered, and a cycle of fixes and re-tests. It is never perfect the first time around, but changes can be made based on feedback from initial use. The same approach is used for cyber compliance, using tools that permit continuous updating of cyber compliance throughout the process, rather than waiting to the end.

To deal with the molasses-slow security clearance process for new contractors, DIUx used a little-known provision of the existing regulations that allow for a "24-hour clearance," under which somebody can gain relatively rapid approval for short-term access to attend an activity that deals with classified material but doesn’t allow the person to have a full clearance or persistent access. – a good way to get commercial folks quickly involved in classified DOD areas on a one-off basis to ensure they have context on the problem.

Development work on the new process was done mostly by six in-house airmen, trained in agile by four people from a small agile vendor called Pivotal Labs, rather than the incumbent working on the big system redesign contract. The total cost of developing the new tool was $2 million. The new tool reduces the time to schedule refueling each day from 60 person-hours to four or five. Given that the operations center had been scheduling 1-3 additional sorties a day because of errors and last-minute changes, at $250,000 fuel cost a pop, the new tool paid for itself in a week.

DOD has a great success story here. After spending $750 million and 10 years on a traditional waterfall systems redesign big bang, the operations center had nothing to show. Agile allowed them actually to deliver an initial working capability in one area after a few months.

I will confess I was disappointed that, by Oti’s own account, DOD has not moved, after a fair number of years -- and two decades after it seriously began to spread in the private sector -- further towards agile. Will this be a turning point for agile in DOD?

The Air Force has announced they will now use agile for the big systems redesign for air operations centers. The tanker story got some considerable attention even before Code for America summit – Oti had been speaking about it so often that if anything when we talked he appeared tired of the topic. The Qatar whiteboard has even been memorialized by being sent off to the Air Force Museum. A visible success counts far more than a lot of theorizing. Stay tuned.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.