'Show me, don't tell me' enters the government contracting mainstream
Contracting source selection has traditionally revolved around the evaluation of detailed proposals produced in response to requests for proposals. Proposal writing was an arcane skill, serving as a barrier to entry for firms new to the government marketplace and taking lots of both time and resources for contractors and government evaluators alike.
At the extreme, there were stories out of proposals being delivered to the government by forklift -- something that probably occurred only for major weapons systems procurement, but that expressed vividly the excesses of the system. Beyond that, many critics, such as Ralph Nash of the George Washington University Law School, heaped scorn on written proposals as an “essay writing contest,” where bidders would talk about their skills and (normally without contractually binding commitments) what they were going to do. It has sometimes been suggested, with exaggeration but not completely inaccurately, that the core competence of traditional government contractors has been proposal writing more than producing good work.
There have been a fair number of efforts to reform the system to deal with these criticisms and produce some version of show don’t tell, dating back to the 1980s. Perhaps the first significant example was the use of competitive prototyping in weapons systems acquisition, often known under the rubric “fly before you buy.” It was used sometimes, but never got a huge amount of traction because of the cost of supporting parallel development of prototypes -- this involved so much work that no contractor would do it for free in the hopes of getting a production contract. (Most prototypes today are developed under DoD’s other transactions authority, but these don’t produce competing prototypes.)
One notable non-DoD example of competitive prototyping from the 1990s was the Federal Aviation Administration’s acquisition for a system to improve communication between air traffic controllers and aircraft. Two prototypes were developed, and they were tested under real conditions in the air traffic control system, with users judging how well a system worked. In the IT arena in the late 1980s, the government occasionally demanded “live test demonstrations” by bidders, where firms had to bring their gear in and show that it worked. Unfortunately, bidders often brought in jury-rigged solutions that had been carefully tuned to work at least once.
Starting in the 1990s, there were also significant efforts to introduce page limits for proposals, which didn’t take on the system’s reliance on evaluating written proposals, but tried to streamline proposals themselves.
Perhaps the most far-reaching effort to reduce reliance on written proposals in the 1990s was the introduction of oral presentations as part of source selection. The idea was that the government should have a chance to actually interact in person with people who would be leading the work if the company is awarded the contract, and to ask them questions and as part of source selection to grade responses. The idea was that this would give the government more information about the actual skills of the company, as opposed to what they wrote down on a piece of paper. (I spent considerable time promoting this idea while in government, and many assumed it came from the White House. In fact, the initiative came from career contracting professionals, as best as I remember initially from Mike Del-Colle, the progressive head of contracting at the Department of Interior’s Minerals Management Service.)
More recently there has been a surge in what are often called “tech demos.” These represent a radical re-imagination of source selection in the direction of “show me don’t tell me.” The basic idea is that those wishing to bid on work send company representatives to a government location for a fixed amount of time -- ranging from a half day to several days – to work together on developing a solution to an IT software problem, which may or may not be the same problem as the government seeks to solve with the procurement. At the end of the period, participating firms submit what they have developed, and this is evaluated as part, or most, of source selection. Some of the early uses of tech demos have been at the Department of Homeland Security, Centers for Medicare and Medicaid Services, the Department of Veterans Affairs and then General Services Administration. A while back I blogged about tech demos, calling them “the next big thing” in government contracting.
There are numerous attractive features of the use of tech demos. The most obvious is that the information the government gets from a tech demo can be much more useful than information in a proposal -- as the name indicates, in a tech demo bidders show what they can do rather than just making representations.
The second advantage is that the tech demo dramatically streamlines source selection, saving time both in developing and evaluating proposals. One needn’t be an expert on government contracting to participate in a tech demo – simply an expert on software development. For this reason, tech demos have been especially attractive to new, nontraditional contractors in the federal marketplace, who don’t know the ins and outs of the traditional system very well. Note that the main function of most earlier efforts to reduce reliance on written proposals helped with getting the government better information but not with streamlining the process or encouraging the entry of new vendors.
Now it seems as if -- in response to growing government interest in “show me, don’t tell me” --some traditional contractors are developing a capability to deliver demos. A recent piece in Federal Times discussed how SAIC, a big IT and security contractor, had “changed to meet new IT contract styles.” SAIC CTO Charles Onstott was quoted as saying, “We’re increasingly seeing requests for proposals (that ask for) instead of a written response, show up with a team and deploy a product by the end of the day.” In October 2018 the company set up an Innovation Factory, “a component of the company that relies on innovative and fast working teams of IT professionals.”
In response to my sending him the article, my friend Stan Soloway, formerly at DOD and now head of Celero Strategies, wrote back to note that “there's a whole bunch [of contractors] that have invested significantly in innovation, like Deloitte, CGI, and Accenture, to name just a few.” (His examples were interesting: Deloitte and Accenture are not just government contractors but also have a significant commercial market presence, and CGI had the searing experience of the Healthcare.gov fiasco.)
Will this produce real change? The procurement system has shown an uncanny ability to assimilate source selection reforms with notably less actual change than had been originally hoped. There are certainly oral presentations even today that correspond more or less of the original vision with interactive conversation around specific questions the customer asks. Yet I suspect that most oral presentations today are tightly orchestrated and pre-rehearsed presentation of slide decks with little interaction. (A similar observation about reality falling short of hopes can be made about another 1990s innovation, using past performance in source selection.)
My worries with tech demos are that participants come to be contractor tech demo staff, who do these all the time and will not actually be working on the job being procured, and that tech demos will be added onto the existing system rather than replacing significant aspects of it, such that the tech demo makes the existing process even more complicated. The Federal Times piece, which said that SAIC would deploy “expert teams of developers that stay together across projects,” is very different from the idea that participants in a tech demo would be people who would be assigned to the project if the team won.
Another question is the extent to which these new efforts will produce phony tech demos, as many believe has happened with contractors calling what they do agile even if it has little in common with actual agile practice. The difference here is that, just because tech demos embody “show me, don’t tell me,” phoniness is not an option in the same way. It’s not enough to talk a good tech demo game -- it is what is produced that counts.
Finally, in my view tech demos are appropriate mostly or only for small projects. If the government will need a hundred developers, they are not all going to be at a tech demo, so the information the tech demo provides becomes less valuable. In my view, if the project is larger and the government wants a don’t tell me show me approach, it is likely to be better to go with competitive prototypes for source selection -- I hope we will start seeing more of these.
I think such efforts will need to be organized as two-stage downselects, with perhaps two finalists be chosen based on past performance and some brief discussion of how one intends to try to solve the problem. Pricing needs to be considered before final source selection (actually the procurement regulations require this), but an advantage of prototyping over traditional source selection is that it is likely to be more practical for bidders to bid a fixed price once the prototype has been developed.
I have been around long enough to realize there are no contracting panaceas. But, despite some caution about tech demos, “don’t tell me show me” represents a promising development in the contracting system.
Posted by Steve Kelman on Mar 19, 2019 at 5:27 PM