Can IT procurement be saved?

Fixing IT

If, as U.S. CIO Steven VanRoekel said, the launch of the site was a "teachable moment," it's only fair to ask: Is it a learnable moment?

Can the tens of thousands of people involved in government procurement — employees and contractors alike — absorb any life lessons for navigating the often bumpy road of large government IT projects?

Yes, say contractors, project managers and government watchdogs — or at least "yes, but."

High-profile government procurement projects sometimes take a wrong turn or crash spectacularly into technological or logistical ditches. Sometimes they can be resurrected or salvaged, and sometimes they are scrapped. The smoldering remains can be attributed to the myriad miscues, oversights or missteps in a hugely complex system.

In other words, bungled launches didn't start with, and the Obama administration's Department of Health and Human Services is hardly alone in its stumbling.

In 2006, the Department of Homeland Security picked Boeing to oversee its $1.9 billion program dubbed SBInet, which sought to revolutionize border security by integrating infrastructure, technology and border security agents.

Unfortunately, it didn't work.

After an initial deployment showed that the technology was far from integrated, DHS pulled the plug on the showcase program in 2011 and distributed the funds to more conventional border security programs.

Other examples abound, but there is never a single reason for big projects to wash out, said Scott Amey, general counsel at the Project on Government Oversight.

Complex federal procurement rules can contribute to the failure of advanced IT systems such as or SBInet, but Amey contends that even a wholesale overhaul of those rules probably wouldn't help much. Others say spending cuts and budget uncertainty have sped up an erosion of the federal procurement workforce. Still others blame rules they say place a crippling emphasis on getting the lowest price at the expense of what might work best.

Ultimately, said Roger Waldron, president of the Coalition for Government Procurement, successful federal contracts happen when the government understands what it wants and what it is asking of bidders, and bidders understand how to work with the government. It's not just about getting to yes — it's about getting to yes, I know exactly what I want, and here's how I want you to give it to me.

The people

In federal procurement, getting what you want starts with people who know how to ask the right questions. A risk-averse workforce that is behind on training can drag down the entire enterprise.

Unfortunately, training that keeps employees up-to-date on changing IT technology has been one of the first victims of sequestration cuts, said Dan Gordon, former administrator of the Office of Federal Procurement Policy and now an associate dean at George Washington University Law School.

"The loss of training makes our people less current in understanding how to run their procurements," Gordon said. "It makes them less current in their knowledge of the industries. It makes them less comfortable in dealing with technological innovation. It damages the procurement system in many ways, which are unfortunately probably too subtle for the public to appreciate. But from the point of view of sophisticated interaction with vendors, it has real impact."

The dearth of training shows itself in bloated contracts, the purchase of outdated equipment and amateur negotiations with the private sector, Gordon said. If you don't know or understand the latest technology, you're going to get left behind. Without adequate training, "you can't run a fair-opportunity competition to award a task order worth hundreds of millions of dollars, which is part of our business."

The resulting imbalance gives the advantage to contractors who have more advanced training in negotiations and procurement and who know their products better. "We pay a price as taxpayers when we don't let our acquisition workforce get the same top-level training that private industry's people have," Gordon said.

When he talked about government contracts at an annual procurement conference earlier this year, all the attendees were from the private-sector side of the business.

"Traditionally, the audience is a good mix of government employees and contractors, all of whom work in the federal procurement arena," Gordon said. "This year, I was disappointed that there were almost no federal attendees. One result was that the open lines of communication that a conference can foster couldn't function because the government people were absent."

Rob Frazzini, leader of Deloitte Consulting's federal digital practice, pointed to another weakness for the federal acquisition workforce: too much of an incentive to avoid risk, which makes employees embrace the legacy waterfall acquisition model instead of an agile model. He said the more fluid agile approach results in products that coincide with current technology.

"These are not uneducated people. These are not uncaring people," Frazzini said. "These are quality, talented individuals who I believe lack the support of the organization and lack the tools necessary to allow them to change into a more agile environment."

He added that a lack of training contributes to a growing gap between the private and public sectors.

Agencies "may not be as particularly nimble as the commercial market because of the size and scope, but they need to stay up with that commercial market," he said. "So I don't believe ignoring agile or a more enlightened approach to development is possible. What I believe needs to happen is a process of looking at how to give a procurement officer the tools necessary and educate the customers effectively so that they can indeed manage risk while at the same time producing a higher-quality outcome."

The rules

Depending on which federal procurement expert you talk to,'s meltdown either shows that the government needs a completely new way to buy high-tech services and gear, or it is a picture-perfect example of how not to use a mostly adequate set of procurement rules.

"It's a teaching moment for procurement," said Waldron, echoing VanRoekel.

Forget policy differences, technical glitches and customer overload. At the heart of the failure, some say, is a mess of a procurement system. was not assembled under a competitive, overarching development and integration contract like those used on many big federal IT projects. Instead, it was created with task orders placed on indefinite delivery, indefinite quantity (IDIQ) contracts, which can be seen as something of a shortcut for large, complex projects. SBInet was based on IDIQs, and its failure profoundly shifted DHS' internal acquisition strategy away from technically complicated made-to-order systems and toward proven off-the-shelf capabilities.

Some procurement experts say the government has long needed to focus on the contracting basics, beginning with the description of what it wants to accomplish with the purchased good or service.

"Requirements development has been an Achilles' heel for 20 years now," Waldron said. "The best, most effective way for the government to acquire high-level, best-value contractor performance in support of agency missions is through improved requirements development."

"The government has to be a little smarter when trying to buy things quickly," Amey said. For the most part, IDIQs work, he added, except when they're used to buy more innovative, cutting-edge technology and services.

Industry representatives have a different view, however. "The government has to undertake a wholesale review of the way it buys IT products and services in the wake of the meltdown of Health and Human Services' website," said Trey Hodgkins, a longtime TechAmerica executive who recently became the Information Technology Industry Council's senior vice president for the public sector.

The federal budgeting process that fuels procurement is also broken, he said. "We're using horse-and-buggy-era funding processes to fund leading-edge technology. It can't keep pace. It's arcane and has a three- to five-year lead time."

Hodgkins said it is time for a new set of rules to replace regulations developed in the 1980s and early 1990s, such as the 1996 Clinger-Cohen Act that mandated that agencies have CIOs. And indeed, Congress is considering some IT acquisition reform. The Federal IT Acquisition Reform Act (FITARA), which passed the House as part of the Defense Authorization bill in June, would establish a single, presidentially appointed CIO at each federal agency and consolidate IT budget authority in that office.

Past attempts at procurement reform

Mouse over a bill name for more information.

  • Federal Acquisition
    Streamlining Act
  • Clinger-Cohen Act
  • Services Acquisition
    Reform Act
  • General Services Administration
    Reorganization Act
  • Fiscal 2009 National Defense
    Authorization Act
  • Federal Information Technology
    Acquisition Reform Act

However, others were less sure that a complete overhaul was necessary. Waldron said a review of existing procurement rules is due, but it should be tempered with additional guidelines developed by industry and government. The Acquisition Advisory Panel, mandated by the Services Acquisition Reform Act of 2003, conducted a review in 2007 and recommended a greater emphasis on defining requirements, structuring solicitations to facilitate competition and fixed-price offers, and monitoring contract performance.

"It's been seven years," Waldron said. "It might be time for another panel."

President Barack Obama seems to agree.'s problems have been "inexcusable," the president said on Nov. 4. "And there are a whole range of things that we're going to need to do once we get this fixed — to talk about federal procurement when it comes to IT and how that's organized."

The money

As government budgets continue to stagnate, agencies are increasingly turning to lowest price, technically acceptable (LPTA) evaluation criteria to select between competing bids for IT projects.

Critics argue that the LPTA approach stifles innovation among contractors and sacrifices long-term value for agencies, but some government procurement experts believe LPTA contracting can benefit feds in certain situations.

"The LPTA environment is appropriate for status augmentation type procurements," said Lynne Gummo, who leads the Government Contracting Industry Group at SC&H, an audit, tax and consulting firm.

She said LPTA contracts can work for projects that do not require leading-edge IT solutions and where price matters more than anything else. When budgets are constricted, however, it's better to pinch pennies on your help desk rather than on contracts related to cybersecurity or cloud computing, where performance issues can drastically impact the mission.

LPTA procurement is a "natural response to constrained resources," said Dan Chenok, executive director of the IBM Center for the Business of Government, but should be avoided on complex IT-related services or projects that call for innovation.

"It tends to be a more upfront way to reduce costs, but it isn't a best-value or even a most cost-effective approach in the long run," Chenok said.

He offered several alternatives to the traditional waterfall approach for complex services or mission-critical IT projects. The agile approach involves ongoing discussions between agencies and industry throughout a solution's development. Some agile methodologies break contracts into increments based on sets of negotiated deliverables, which reduces upfront costs and increases transparency because decisions must be made at each step to move the contract forward. However, agile methods are not well-suited to large-scale projects and mission-critical systems for which failure is not an option.

In addition, Chenok said some agencies might find it beneficial to conduct a "technology bake-off" among industry partners for certain kinds of IT solutions. Agencies supply a small amount of money to encourage companies to participate by creating prototypes and choose the best prototype to be scaled up.

Chenok said cash-strapped agencies could use a challenge system to crowdsource requirements "to get the best-value approach." That strategy, which would require limited seed money, works for agencies that know their problems but are unsure of an exact solution.

The legacy issue

Although has been pilloried even by backers of the Affordable Care Act, VanRoekel went out of his way to praise the site's architecture at a recent event.

"We should all be proud that something this complex, this integrated to legacy systems — and there are mainframes out there that this thing hooks to — was done at Internet scale and taken online in this way," VanRoekel said.

Integrating state-of-the-art technology with legacy systems is a fact of life in federal IT. However, David Cutler, a former health care adviser to Obama, voiced concerns a few years ago that the IT staff and systems at the Centers for Medicare and Medicaid Services were not up to the task of building In a 2010 memo to Larry Summers, who was the top White House economic adviser at the time, Cutler wrote that "IT services [at CMS] are antiquated" and a restructuring of the IT department would be required to implement the health care law.

Next steps

Although there is considerable disagreement among acquisition officials, contractors and lawmakers about how to go about overhauling the procurement system -- or even whether an overhaul is necessary — some consensus exists on steps that could be taken in the short term to make the process run more smoothly:

  • Do a better job of defining desired outcomes upfront.
  • Improve the training options for the federal acquisition workforce to put them on an even footing with vendors.
  • Give agency CIOs more budget authority.
  • Avoid lowest price, technically acceptable contracts on large, innovation-heavy projects.
  • Use agile development strategically and mainly when a project does not require a lot of interaction with legacy systems.

But it does not appear that legacy systems alone are the problem. The component of that required the most interaction with other systems performed the best. The data hub was designed to tap into systems at DHS, the Defense Department, the Social Security Administration, the Internal Revenue Service, the Department of Veterans Affairs and others so the site could determine an applicant's eligibility for insurance under the Affordable Care Act. The hub functioned smoothly during the launch of until a failure at cloud provider Verizon Terremark bumped it offline for a few days in late October.

The data hub was the most extensively tested component of, with functional trials going back to October 2012, according to CMS Administrator Marilyn Tavenner.

In addition to testing, predesign decisions and planning are essential to successfully integrating legacy systems, said Jay Shah, executive vice president of Octo Consulting Group. One key, he said, is not to rush the procurement process. Budget pressures are forcing agencies with legacy systems to "think incrementally and not transformationally" when it comes to implementing new systems, he added, but agile development is not a silver bullet.

"While most government agencies love the idea of agile, the [federal] procurement process and capital planning [are] counter to what agile espouses," Shah said.

Congress will have to get involved if there are going to be large-scale changes to the federal IT procurement process. The House-approved FITARA puts more emphasis on workforce training and assigns more accountability to agency CIOs, but in the wake of's difficulties, some lawmakers are looking for ways to go further.

Rep. Anna Eshoo (D-Calif.), who represents a swath of Silicon Valley, wants to make the system more open and accessible. "My sense is that there are inside-the-Beltway contractors that know the current system very, very well, and they are the ones that are awarded the contracts," she said.

Eshoo was among the more aggressive members of the House Energy and Commerce Committee who questioned executives from CGI Federal and QSSI in an Oct. 24 hearing about's problems, and she was not pleased with the answers she got.

"I think the major contractors failed us," she said, adding that she plans to look for partners in both political parties who will help her press for contracting reform.

Cyber. Covered.

Government Cyber Insider tracks the technologies, policies, threats and emerging solutions that shape the cybersecurity landscape.


Reader comments

Mon, Dec 2, 2013 Mike Honcho Herndon, VA

Should have outsourced it to a company like Facebook. (Or some massive online system expert) Procurement is not the problem in my opinion. Its the unrealistic leaders who thought they could develop such a solution in a short period of time by the lowest bidder. (or 2nd lowest in this case)

Sat, Nov 30, 2013 John Weiler, Alexandria VA

Excellent article. There are solutions to these problems, and not sure new legislation with make a difference given the lack of traction with ClingerCohen Act. OMB needs to start holding agencies accountable for outcomes, not compliance. DoD is even worse, where those who succeed by avoiding their DOD5000 are penalized. Congress could put in place both incentives and penalties for inaction as was attempted with the National Performance Review which tied bonuses and promotions to measurable outcomes. Seems like we are promoting those who make no decisions or focus on compliance vs outcomes.

Sat, Nov 23, 2013 Lisa Mascolo, CEO Optimos

Thanks to the authors, who did a great job researching and articulating the top reasons why some large government IT programs fail. We've been studying the top-10 list for a long time - well before Letterman! Unlike Letterman's, though, this list is largely unchanged over time. Yet we continue to have epic failures in federal government IT. It’s time to shift our focus. I think of it in terms of our approach to healthcare. We spend huge amounts of money studying and curing disease - and I'm not saying we shouldn't. But what if we spent half of that money preventing disease? There are plenty of big mission-critical program successes. In the name of prevention, why don't we – government and contractors alike - analyze these in detail to learn why they’re successful, rather than concentrate on the failures? If we understand the key characteristics of successful programs and ensure their inclusion in all programs, I’ve no doubt that we'll increase the success rate.

Thu, Nov 21, 2013 Al

Full disclosure: I'm a procurement person. If the most tested part of (the data hub) was the most successful, then isn't that the answer to this puzzle? As a previous commenter notes, IT explosions are present in non-government organizations as well as the Federal Government. The Federal Procurement System can be improved, but in this case the problem is the project itself. Also an observation: if an Agency is trying to issue a $XX,000,000.00 fair opportunity task order (with fixed rates), this means the Agency needs to hire more people to procure that service separately.

Thu, Nov 21, 2013

One has to accept that the “politicians” don’t get it when dealing with the complexities of large scale systems. They get shown really neat web systems, see that things like “agile” promise and can often deliver something that seems to meet the needs, OF ONE PERSON at a time data processing. Unfortunately that single thread demo doesn’t scale into a major system that can supply bulletproof processing of 50,000 or more users concurrently on a 7/24 schedule. The scale up takes more than just spinning up 50,000 virtual instances with a cloud provider.

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group