Can IT procurement be saved?
The problems with large federal IT projects are deep, broad and mostly immune to quick fixes. So what's to be done?
If, as U.S. CIO Steven VanRoekel said, the launch of the HealthCare.gov site was a "teachable moment," it's only fair to ask: Is it a learnable moment?
Can the tens of thousands of people involved in government procurement — employees and contractors alike — absorb any life lessons for navigating the often bumpy road of large government IT projects?
Yes, say contractors, project managers and government watchdogs — or at least "yes, but."
High-profile government procurement projects sometimes take a wrong turn or crash spectacularly into technological or logistical ditches. Sometimes they can be resurrected or salvaged, and sometimes they are scrapped. The smoldering remains can be attributed to the myriad miscues, oversights or missteps in a hugely complex system.
In other words, bungled launches didn't start with HealthCare.gov, and the Obama administration's Department of Health and Human Services is hardly alone in its stumbling.
In 2006, the Department of Homeland Security picked Boeing to oversee its $1.9 billion program dubbed SBInet, which sought to revolutionize border security by integrating infrastructure, technology and border security agents.
Unfortunately, it didn't work.
After an initial deployment showed that the technology was far from integrated, DHS pulled the plug on the showcase program in 2011 and distributed the funds to more conventional border security programs.
Other examples abound, but there is never a single reason for big projects to wash out, said Scott Amey, general counsel at the Project on Government Oversight.
Complex federal procurement rules can contribute to the failure of advanced IT systems such as HealthCare.gov or SBInet, but Amey contends that even a wholesale overhaul of those rules probably wouldn't help much. Others say spending cuts and budget uncertainty have sped up an erosion of the federal procurement workforce. Still others blame rules they say place a crippling emphasis on getting the lowest price at the expense of what might work best.
Ultimately, said Roger Waldron, president of the Coalition for Government Procurement, successful federal contracts happen when the government understands what it wants and what it is asking of bidders, and bidders understand how to work with the government. It's not just about getting to yes — it's about getting to yes, I know exactly what I want, and here's how I want you to give it to me.
The people
In federal procurement, getting what you want starts with people who know how to ask the right questions. A risk-averse workforce that is behind on training can drag down the entire enterprise.
Unfortunately, training that keeps employees up-to-date on changing IT technology has been one of the first victims of sequestration cuts, said Dan Gordon, former administrator of the Office of Federal Procurement Policy and now an associate dean at George Washington University Law School.
"The loss of training makes our people less current in understanding how to run their procurements," Gordon said. "It makes them less current in their knowledge of the industries. It makes them less comfortable in dealing with technological innovation. It damages the procurement system in many ways, which are unfortunately probably too subtle for the public to appreciate. But from the point of view of sophisticated interaction with vendors, it has real impact."
The dearth of training shows itself in bloated contracts, the purchase of outdated equipment and amateur negotiations with the private sector, Gordon said. If you don't know or understand the latest technology, you're going to get left behind. Without adequate training, "you can't run a fair-opportunity competition to award a task order worth hundreds of millions of dollars, which is part of our business."
The resulting imbalance gives the advantage to contractors who have more advanced training in negotiations and procurement and who know their products better. "We pay a price as taxpayers when we don't let our acquisition workforce get the same top-level training that private industry's people have," Gordon said.
When he talked about government contracts at an annual procurement conference earlier this year, all the attendees were from the private-sector side of the business.
"Traditionally, the audience is a good mix of government employees and contractors, all of whom work in the federal procurement arena," Gordon said. "This year, I was disappointed that there were almost no federal attendees. One result was that the open lines of communication that a conference can foster couldn't function because the government people were absent."
Rob Frazzini, leader of Deloitte Consulting's federal digital practice, pointed to another weakness for the federal acquisition workforce: too much of an incentive to avoid risk, which makes employees embrace the legacy waterfall acquisition model instead of an agile model. He said the more fluid agile approach results in products that coincide with current technology.
"These are not uneducated people. These are not uncaring people," Frazzini said. "These are quality, talented individuals who I believe lack the support of the organization and lack the tools necessary to allow them to change into a more agile environment."
He added that a lack of training contributes to a growing gap between the private and public sectors.
Agencies "may not be as particularly nimble as the commercial market because of the size and scope, but they need to stay up with that commercial market," he said. "So I don't believe ignoring agile or a more enlightened approach to development is possible. What I believe needs to happen is a process of looking at how to give a procurement officer the tools necessary and educate the customers effectively so that they can indeed manage risk while at the same time producing a higher-quality outcome."
The rules
Depending on which federal procurement expert you talk to, HealthCare.gov's meltdown either shows that the government needs a completely new way to buy high-tech services and gear, or it is a picture-perfect example of how not to use a mostly adequate set of procurement rules.
"It's a teaching moment for procurement," said Waldron, echoing VanRoekel.
Forget policy differences, technical glitches and customer overload. At the heart of the failure, some say, is a mess of a procurement system.
HealthCare.gov was not assembled under a competitive, overarching development and integration contract like those used on many big federal IT projects. Instead, it was created with task orders placed on indefinite delivery, indefinite quantity (IDIQ) contracts, which can be seen as something of a shortcut for large, complex projects. SBInet was based on IDIQs, and its failure profoundly shifted DHS' internal acquisition strategy away from technically complicated made-to-order systems and toward proven off-the-shelf capabilities.
Some procurement experts say the government has long needed to focus on the contracting basics, beginning with the description of what it wants to accomplish with the purchased good or service.
"Requirements development has been an Achilles' heel for 20 years now," Waldron said. "The best, most effective way for the government to acquire high-level, best-value contractor performance in support of agency missions is through improved requirements development."
"The government has to be a little smarter when trying to buy things quickly," Amey said. For the most part, IDIQs work, he added, except when they're used to buy more innovative, cutting-edge technology and services.
Industry representatives have a different view, however. "The government has to undertake a wholesale review of the way it buys IT products and services in the wake of the meltdown of Health and Human Services' HealthCare.gov website," said Trey Hodgkins, a longtime TechAmerica executive who recently became the Information Technology Industry Council's senior vice president for the public sector.
The federal budgeting process that fuels procurement is also broken, he said. "We're using horse-and-buggy-era funding processes to fund leading-edge technology. It can't keep pace. It's arcane and has a three- to five-year lead time."
Hodgkins said it is time for a new set of rules to replace regulations developed in the 1980s and early 1990s, such as the 1996 Clinger-Cohen Act that mandated that agencies have CIOs. And indeed, Congress is considering some IT acquisition reform. The Federal IT Acquisition Reform Act (FITARA), which passed the House as part of the Defense Authorization bill in June, would establish a single, presidentially appointed CIO at each federal agency and consolidate IT budget authority in that office.
Past attempts at procurement reform
Mouse over a bill name for more information.
- Federal Acquisition
Streamlining Act 1994 - Clinger-Cohen Act
1996 - Services Acquisition
Reform Act 2003 - General Services Administration
Reorganization Act 2005 - Fiscal 2009 National Defense
Authorization Act 2009 - Federal Information Technology
Acquisition Reform Act 2013
However, others were less sure that a complete overhaul was necessary. Waldron said a review of existing procurement rules is due, but it should be tempered with additional guidelines developed by industry and government. The Acquisition Advisory Panel, mandated by the Services Acquisition Reform Act of 2003, conducted a review in 2007 and recommended a greater emphasis on defining requirements, structuring solicitations to facilitate competition and fixed-price offers, and monitoring contract performance.
"It's been seven years," Waldron said. "It might be time for another panel."
President Barack Obama seems to agree. HealthCare.gov's problems have been "inexcusable," the president said on Nov. 4. "And there are a whole range of things that we're going to need to do once we get this fixed — to talk about federal procurement when it comes to IT and how that's organized."
The money
As government budgets continue to stagnate, agencies are increasingly turning to lowest price, technically acceptable (LPTA) evaluation criteria to select between competing bids for IT projects.
Critics argue that the LPTA approach stifles innovation among contractors and sacrifices long-term value for agencies, but some government procurement experts believe LPTA contracting can benefit feds in certain situations.
"The LPTA environment is appropriate for status augmentation type procurements," said Lynne Gummo, who leads the Government Contracting Industry Group at SC&H, an audit, tax and consulting firm.
She said LPTA contracts can work for projects that do not require leading-edge IT solutions and where price matters more than anything else. When budgets are constricted, however, it's better to pinch pennies on your help desk rather than on contracts related to cybersecurity or cloud computing, where performance issues can drastically impact the mission.
LPTA procurement is a "natural response to constrained resources," said Dan Chenok, executive director of the IBM Center for the Business of Government, but should be avoided on complex IT-related services or projects that call for innovation.
"It tends to be a more upfront way to reduce costs, but it isn't a best-value or even a most cost-effective approach in the long run," Chenok said.
He offered several alternatives to the traditional waterfall approach for complex services or mission-critical IT projects. The agile approach involves ongoing discussions between agencies and industry throughout a solution's development. Some agile methodologies break contracts into increments based on sets of negotiated deliverables, which reduces upfront costs and increases transparency because decisions must be made at each step to move the contract forward. However, agile methods are not well-suited to large-scale projects and mission-critical systems for which failure is not an option.
In addition, Chenok said some agencies might find it beneficial to conduct a "technology bake-off" among industry partners for certain kinds of IT solutions. Agencies supply a small amount of money to encourage companies to participate by creating prototypes and choose the best prototype to be scaled up.
Chenok said cash-strapped agencies could use a challenge system to crowdsource requirements "to get the best-value approach." That strategy, which would require limited seed money, works for agencies that know their problems but are unsure of an exact solution.
The legacy issue
Although HealthCare.gov has been pilloried even by backers of the Affordable Care Act, VanRoekel went out of his way to praise the site's architecture at a recent event.
"We should all be proud that something this complex, this integrated to legacy systems — and there are mainframes out there that this thing hooks to — was done at Internet scale and taken online in this way," VanRoekel said.
Integrating state-of-the-art technology with legacy systems is a fact of life in federal IT. However, David Cutler, a former health care adviser to Obama, voiced concerns a few years ago that the IT staff and systems at the Centers for Medicare and Medicaid Services were not up to the task of building HealthCare.gov. In a 2010 memo to Larry Summers, who was the top White House economic adviser at the time, Cutler wrote that "IT services [at CMS] are antiquated" and a restructuring of the IT department would be required to implement the health care law.
But it does not appear that legacy systems alone are the problem. The component of HealthCare.gov that required the most interaction with other systems performed the best. The data hub was designed to tap into systems at DHS, the Defense Department, the Social Security Administration, the Internal Revenue Service, the Department of Veterans Affairs and others so the site could determine an applicant's eligibility for insurance under the Affordable Care Act. The hub functioned smoothly during the launch of HealthCare.gov until a failure at cloud provider Verizon Terremark bumped it offline for a few days in late October.
The data hub was the most extensively tested component of HealthCare.gov, with functional trials going back to October 2012, according to CMS Administrator Marilyn Tavenner.
In addition to testing, predesign decisions and planning are essential to successfully integrating legacy systems, said Jay Shah, executive vice president of Octo Consulting Group. One key, he said, is not to rush the procurement process. Budget pressures are forcing agencies with legacy systems to "think incrementally and not transformationally" when it comes to implementing new systems, he added, but agile development is not a silver bullet.
"While most government agencies love the idea of agile, the [federal] procurement process and capital planning [are] counter to what agile espouses," Shah said.
Congress will have to get involved if there are going to be large-scale changes to the federal IT procurement process. The House-approved FITARA puts more emphasis on workforce training and assigns more accountability to agency CIOs, but in the wake of HealthCare.gov's difficulties, some lawmakers are looking for ways to go further.
Rep. Anna Eshoo (D-Calif.), who represents a swath of Silicon Valley, wants to make the system more open and accessible. "My sense is that there are inside-the-Beltway contractors that know the current system very, very well, and they are the ones that are awarded the contracts," she said.
Eshoo was among the more aggressive members of the House Energy and Commerce Committee who questioned executives from CGI Federal and QSSI in an Oct. 24 hearing about HealthCare.gov's problems, and she was not pleased with the answers she got.
"I think the major contractors failed us," she said, adding that she plans to look for partners in both political parties who will help her press for contracting reform.