Cloud Computing

Moving to the cloud? Learn from CBP's mistakes

CBP's Wolf Tombe

Customs and Border Protection Chief Technology Officer Wolf Tombe, shown here at a 2010 Government Technology Research Alliance presentation, shared his agency's hard-won lessons on early cloud migrations at a Jan. 29 FCW Executive Briefing.

Customs and Border Protection was one of the first federal agencies to adopt cloud computing, and the Homeland Security component responsible for protecting U.S. borders has plenty of lessons to share with the rest of government.

With the hype about the cloud and the types of services it offers -- infrastructure-, software-, storage-, and platform-as-a-service, to name a few -- increasing by the week, federal customers may attempt to deploy expensive, high-profile services right away simply because they can.

That's what CBP did a few years ago, making email-as-a-service for its 60,000 employees one of its first major cloud projects. According to Wolf Tombe, the agency's chief technology officer, it was a huge mistake.

Tombe, speaking Jan. 29 at an executive briefing in Washington, D.C., said the agency did not specify with the vendor how the migration to cloud email would occur, nor did it contractually demand visibility into the vendor's cloud infrastructure.

Upon signing the contract, Tombe said, the agency learned the vendor would initially be able to migrate only about 100 users per week to the cloud. A server blade failure soon after led to a total system outage, getting CBP'S email-as-a-service offering off to a terrible start.

"We should have known we were in for trouble," Tombe said. "It wasn't what we signed up for, and we're still not seeing the cost realization you'd expect for cloud. It was a custom infrastructure built for us -- not a managed service."

Tough lessons like that have hardened CBP'S cloud acquisition strategy, and their lessons serve well the rest of the federal community. According to Wolfe, they include:

  • Start with small projects. "Low-profile, low-visibility services" are a much smarter to migrate to the cloud than big applications that might affect hundreds or thousands of end users. At the same time, Tombe said, set "reasonable expectations for cost, savings, and performance." In other words, start small and don't expect to change the world right away.

  • Get mission owners on board. "Talk with the people who own the applications or data, that is not IT, that is the mission," Tombe said. Moving to cloud is more than an IT decision, it's a whole-of-business decision.

  • Demand expected investment returns. "Ask for proof of cost savings up front. Show us what your cost benefit is up front. If you can't do that, we're going to be having a very different conversation," said Tombe. Agencies should demand "usage-based licensing" -- the pay-by-the-drink approach to purchasing IT services -- to reduce infrastructure spending and decrease operational expenses. In fact, Tombe said, cloud computing should lead to a 70 percent reduction in costs associated with managing IT systems. Whatever that number may be, Tombe said, it's imperative that it is part of the conversation.

  • Require visibility into the IT system. "At CBP, we mandate [vendors] provide us that kind of access," Tombe said. "There's been a few that say no. We tell them to have a nice day. We demand absolute visibility into the core infrastructure, and we do audit that." Such transparency is key, Tombe said, and not just for security. CBP demands robust utilization reports to ensure it is getting the most bang for its taxpayer bucks, and also requires "full cooperation" for forensics and investigations.

  • Use open standards by default. Tombe said CBP demands open standards in its cloud acquisitions. Agency personnel must make a special request if they wish to purchase a proprietary solution. Those requests go to Tombe, and he said he hasn't approved one yet, though he likely will at some point to maintain the mainframe systems the agency is beginning to phase out. Open standards make for smooth migrations, which are "imperative" for CBP.

    "We still have proprietary capabilities, but how do we put open standards in front of that?" Tombe said. "Our first principle of IT is that you must use open standards by default. [A vendor] can be proprietary on your side as long as it is all available to us in open standards."

  • All contracts must address data ownership and exit strategy. Agencies must contractually ensure that they -- and not vendors -- own the information contained within the cloud infrastructure. It might seem like a no-brainer, but data ownership is of major importance.

    What happens to data on old drives when they are removed and upgraded within a data center? Most early cloud adopters, including CBP, didn't specify that in contracts, though the agency now contractually demands data be expunged on old drives to National Security Agency standards. Tombe said CBP is looking to go as far as encrypting data on its end and "not giving the encryption keys to vendors," meaning even with access, a vendor couldn't view potentially sensitive data. Agencies should also contractually ensure data from a previous clowd owner is not retained if it is moved off cloud.

  • Demand high availability. Tombe said agencies should demand 99.999 percent -- sometimes called the five nines -- and should subsequently demand not to pay extra for it. CBP does, however, pay "a little extra" for priority service options and a dedicated account representative to ensure they receive the best possible customer service.

About the Author

Frank Konkel is a former staff writer for FCW.

Cyber. Covered.

Government Cyber Insider tracks the technologies, policies, threats and emerging solutions that shape the cybersecurity landscape.


Reader comments

Thu, Jan 29, 2015 Morris Segal Clifton, VA

Just came across this. Good article. In one respect the "cloud" is a pricing model, much like "time sharing" was in the '70s. But, managing todays diverse technologies is magnitudes more complex than those days. Luckily, todays underlying technologies can help by providing core architectural controls. This should not be seen as an opportunity to turn a blind eye to the management infrastructure that supports your business. Basic concerns that drive solid business, engineering and security practices must remain under the customers control. The vendor's job is to provide service and technologies that support those concerns. It can be a good partnership when managed well.

Thu, Mar 20, 2014

Tombe wants 99.999% availability, that means 5 minutes of downtime per year. It costs 200 time more than 99.5% availability, which allows 3 hours and 39 minutes of downtime per month. Why would anyone expect the higher level of availability to be free? Who will bear the costs? Shouldn't the customer who expects less - say 99.5% availability - pay less as a result?

Mon, Feb 3, 2014 Tempest In a Teapot

"Tombe said agencies should demand 99.999 percent -- sometimes called the five nines -- and should subsequently demand not to pay extra for it. " Really??? How does that work? Each "9" is an order of magnitude more effort to deliver, and that entails additional cost. Someone's gotta pay for it. Why not just demand ten 9s?

Fri, Jan 31, 2014

Mr. Tombe's comments are entirely accurate.

Thu, Jan 30, 2014

"Tombe said CBP demands open standards in its cloud acquisitions. Agency personnel must make a special request if they wish to purchase a proprietary solution." What sort of "special request" was made for the Oracle Exalogic purchases?

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group