Workforce, infrastructure and avoiding future crashes: Q&A with Greg Ambrose

FCW sat down recently for an exclusive interview with Greg Ambrose, the State Department's director of consular systems and technology, who talked about how his office is trying to prevent a repeat of the malfunction its visa and passport database experienced July 20.

Shutterstock image: data center lights and hardware.

In July 2014, the State Department's Consular Consolidated Database -- a system of 12 databases that processes U.S. visa and passport requests -- went offline and created a backlog in visa processing that disrupted travel for thousands of people around the world.

In part 2 of this exclusive interview with FCW, State Department Director of Consular Systems and Technology Greg Ambrose discusses his agency's broader IT and workforce challenges, along with steps the Bureau of Consular Affairs is taking to prevent a future CCD crash.  (Part 1, published on Oct. 20, details the technical failings that led to the crash and how the Bureau of Consular Affairs responded.)

The Q&A was edited for clarity. The reporter’s asides, explaining technical details of the CCD or other context, are labeled as such and in italic.

FCW: How much attention did the CCD failure get from senior State Department leadership?

Ambrose: All the way to the top and even the White House was aware of it. [There was a] focus on getting us whatever resources we needed government-wide.

Greg Ambrose

Greg Ambrose

FCW: Were you on a daily or weekly basis reporting to Secretary of State John Kerry, for example?

Ambrose: We had twice-a-day meetings with acting Assistant Secretary for Consular Affairs Michele Thoren Bond and she met, I believe, daily with Undersecretary for Management Patrick Kennedy. And Undersecretary Kennedy kept Secretary Kerry [informed]. We did have interest from the White House. We also had on our calls Deputy Secretary Heather Higginbottom and her staff.

FCW: If you could go back and change one thing in the six months leading up to the database difficulties, what would it be?

Ambrose: It was clear that there was not a lot of infrastructure investment put in early enough to keep pace with the tremendous growth of the [Consular Affairs] mission. And so we knew that we had to move to 11G and Linux and more powerful servers to have a more reliable, highly available, high-performing system.

REPORTER'S NOTE: The CCD was running on a Windows server and Oracle's 10G database; Ambrose's team had planned a migration to both a Linux platform and a newer version of Oracle's database. See part 1 of this interview for additional details.

We believe we had, based on the way the system was functioning, some time before the end of this calendar year to get that done. And because of the growth and the unforeseen events that that patch introduced, it accelerated. It required us to do an additional action.

So one of the things that I look at is working with my peers, understanding the growth of Consular Affairs worldwide, so that we ensure that we’ve got investments in the right areas so we don’t have this problem again.

A silver lining to this CCD problem that we had is we now have government people who understand more about the CCD than we ever did. Historically, it was a black box.

FCW: Does that mean having closer contact with embassies?

Ambrose: In many cases we send people there. There’s a very different view overseas than from a cubicle or office in northwest D.C. So understanding that part of the mission helps us make decisions that directly affect the people on the visa lines or who are issuing passports to American citizens.

We have, in some cases, more than 2,000 non-immigrant visa interviews per day. And you can’t tell the people to come back tomorrow because there are 2,000 more coming tomorrow.

FCW: As a result of these technical difficulties, are you going to send larger and more frequent IT teams to consulates and embassies around the world?

Ambrose: We send 'refresh teams' to consulates and embassies to help with installations of new servers, to help with installation of the new workstations. We have refresh teams that help with training of the locally employed staff on new system enhancements.

FCW: How big are the 'refresh teams'?

Ambrose: It depends on the mission, but it's usually three to five people.

FCW: Are they a combination of Consular Affairs staffers, Foreign Service officers and contractors?

Ambrose: It’s a combination; [it's] mostly contractors, but a government person will go along with them.

FCW: Do you think the Foreign Service values and cultivates IT skills enough?

Ambrose: Well, I’m biased as a techie, so I always believe there’s a benefit to anybody having technical skills. There are information technology specialists in the Foreign Service, and those are the folks who usually run the staff at post who do a lot of the on-the-ground, day-to-day maintenance for us.

You had people focused on the day-to-day operations and that’s why, in some areas, the modernization suffered -- they were dealing with ‘the today’ and they didn’t have a whole lot of cycles left to deal with ‘the tomorrow.’

FCW: Do you think there are enough of those people?

Ambrose: I know that there’s a shortage of those people. And I know that the bureaus in the department are prioritizing what positions need to be filled because there aren’t enough people to fill those positions.

FCW: Were the CCD's recent struggles a wake-up call to a need for more IT skills at the State Department?

Ambrose: IT has priority in Consular Affairs.

We have technology throughout CA, so I know it’s a priority within CA. One of the things that I continue to push is the need to have technical leaders within the government. And we certainly hire contract organizations to provide services for us and they're integral to what we do. But we need to make sure that we have enough government technical leaders giving the direction to those contractors so that we’re all moving together in unison to a common goal.

A silver lining to this CCD problem that we had is we now have government people who understand more about the CCD than we ever did. Historically, it was a black box that was maintained by a small business contractor, and now we have government leaders who understand how it’s architected, understand the places we need to untangle, and it makes the conversation much easier at the higher levels when we talk about the investments that are needed, the focus that’s needed and the prioritization.

FCW: Were you unhappy at all with how the previous contractor, Creative Information Technology Inc. (CITI), handled its work?

Ambrose: CITI has some very dedicated, talented people. My surprise when I came in -- I mentioned the government staff was under-staffed, lot of vacancies -- I was surprised to see how few people we had in some of our key contracts to manage this worldwide operation. So you had people focused on the day-to-day operations and that’s why, in some areas, the modernization suffered, because they were dealing with 'the today' and they didn’t have a whole lot of cycles left to deal with 'the tomorrow.'

FCW: How confident are you that the database won’t fail again?

Ambrose: I’m confident that we are focused on ensuring that we don’t have that problem again.

FCW: When the database went down, did you reach out to anyone in the federal IT community for advice? How much feedback or criticism did you get?

Ambrose: We had a lot of people who offered help, and where they had some advice that helped us solve the problem that we were focused on at the time, we accepted that.

We did have a discussion with [then-U.S. CIO] Steve VanRoekel and Lisa Schlosser from OMB. They asked who I had engaged, and I told them who I had engaged, and they concurred that we had the right people focused on helping solve the problem. And we kept them informed. Microsoft and Oracle [and EMC] came with their senior folks and pledged their complete support.

We need to make sure that we have enough government technical leaders giving the direction to those contractors so that we’re all moving together in unison to a common goal.

FCW: Were you aggressive enough in getting the CCD fully functioning, given how many travelers were affected? How did you balance the need to act aggressively with the need to not do further damage to the system?

Ambrose: I believe we got tremendous support from the department and the bureau to do the things we wanted to do, and we were aggressive in managing [that]. The first priority was to get the system functioning, and we did that.

One of the things that we did do is we turned off, temporarily, our internal reporting capabilities, our ancillary dashboard functions. We still pulled reports for ad hoc reasons, to keep our oversight of operations, but we have a lot of ancillary reports that we temporarily disabled to make sure that we could get the functions of processing transactions operational.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.