Workforce, infrastructure and avoiding future crashes: Q&A with Greg Ambrose
FCW sat down recently for an exclusive interview with Greg Ambrose, the State Department's director of consular systems and technology, who talked about how his office is trying to prevent a repeat of the malfunction its visa and passport database experienced July 20.
In July 2014, the State Department's Consular Consolidated Database -- a system of 12 databases that processes U.S. visa and passport requests -- went offline and created a backlog in visa processing that disrupted travel for thousands of people around the world.
In part 2 of this exclusive interview with FCW, State Department Director of Consular Systems and Technology Greg Ambrose discusses his agency's broader IT and workforce challenges, along with steps the Bureau of Consular Affairs is taking to prevent a future CCD crash. (Part 1, published on Oct. 20, details the technical failings that led to the crash and how the Bureau of Consular Affairs responded.)
The Q&A was edited for clarity. The reporter’s asides, explaining technical details of the CCD or other context, are labeled as such and in italic.
FCW: How much attention did the CCD failure get from senior State Department leadership?
Ambrose: All the way to the top and even the White House was aware of it. [There was a] focus on getting us whatever resources we needed government-wide.
FCW: Were you on a daily or weekly basis reporting to Secretary of State John Kerry, for example?
Ambrose: We had twice-a-day meetings with acting Assistant Secretary for Consular Affairs Michele Thoren Bond and she met, I believe, daily with Undersecretary for Management Patrick Kennedy. And Undersecretary Kennedy kept Secretary Kerry [informed]. We did have interest from the White House. We also had on our calls Deputy Secretary Heather Higginbottom and her staff.
FCW: If you could go back and change one thing in the six months leading up to the database difficulties, what would it be?
Ambrose: It was clear that there was not a lot of infrastructure investment put in early enough to keep pace with the tremendous growth of the [Consular Affairs] mission. And so we knew that we had to move to 11G and Linux and more powerful servers to have a more reliable, highly available, high-performing system.
REPORTER'S NOTE: The CCD was running on a Windows server and Oracle's 10G database; Ambrose's team had planned a migration to both a Linux platform and a newer version of Oracle's database. See part 1 of this interview for additional details.
We believe we had, based on the way the system was functioning, some time before the end of this calendar year to get that done. And because of the growth and the unforeseen events that that patch introduced, it accelerated. It required us to do an additional action.
So one of the things that I look at is working with my peers, understanding the growth of Consular Affairs worldwide, so that we ensure that we’ve got investments in the right areas so we don’t have this problem again.
A silver lining to this CCD problem that we had is we now have government people who understand more about the CCD than we ever did. Historically, it was a black box.
FCW: Does that mean having closer contact with embassies?
Ambrose: In many cases we send people there. There’s a very different view overseas than from a cubicle or office in northwest D.C. So understanding that part of the mission helps us make decisions that directly affect the people on the visa lines or who are issuing passports to American citizens.
We have, in some cases, more than 2,000 non-immigrant visa interviews per day. And you can’t tell the people to come back tomorrow because there are 2,000 more coming tomorrow.
FCW: As a result of these technical difficulties, are you going to send larger and more frequent IT teams to consulates and embassies around the world?
Ambrose: We send 'refresh teams' to consulates and embassies to help with installations of new servers, to help with installation of the new workstations. We have refresh teams that help with training of the locally employed staff on new system enhancements.
FCW: How big are the 'refresh teams'?
Ambrose: It depends on the mission, but it's usually three to five people.
FCW: Are they a combination of Consular Affairs staffers, Foreign Service officers and contractors?
Ambrose: It’s a combination; [it's] mostly contractors, but a government person will go along with them.
FCW: Do you think the Foreign Service values and cultivates IT skills enough?
Ambrose: Well, I’m biased as a techie, so I always believe there’s a benefit to anybody having technical skills. There are information technology specialists in the Foreign Service, and those are the folks who usually run the staff at post who do a lot of the on-the-ground, day-to-day maintenance for us.
You had people focused on the day-to-day operations and that’s why, in some areas, the modernization suffered -- they were dealing with ‘the today’ and they didn’t have a whole lot of cycles left to deal with ‘the tomorrow.’
FCW: Do you think there are enough of those people?
Ambrose: I know that there’s a shortage of those people. And I know that the bureaus in the department are prioritizing what positions need to be filled because there aren’t enough people to fill those positions.
FCW: Were the CCD's recent struggles a wake-up call to a need for more IT skills at the State Department?
Ambrose: IT has priority in Consular Affairs.
We have technology throughout CA, so I know it’s a priority within CA. One of the things that I continue to push is the need to have technical leaders within the government. And we certainly hire contract organizations to provide services for us and they're integral to what we do. But we need to make sure that we have enough government technical leaders giving the direction to those contractors so that we’re all moving together in unison to a common goal.
A silver lining to this CCD problem that we had is we now have government people who understand more about the CCD than we ever did. Historically, it was a black box that was maintained by a small business contractor, and now we have government leaders who understand how it’s architected, understand the places we need to untangle, and it makes the conversation much easier at the higher levels when we talk about the investments that are needed, the focus that’s needed and the prioritization.
FCW: Were you unhappy at all with how the previous contractor, Creative Information Technology Inc. (CITI), handled its work?
Ambrose: CITI has some very dedicated, talented people. My surprise when I came in -- I mentioned the government staff was under-staffed, lot of vacancies -- I was surprised to see how few people we had in some of our key contracts to manage this worldwide operation. So you had people focused on the day-to-day operations and that’s why, in some areas, the modernization suffered, because they were dealing with 'the today' and they didn’t have a whole lot of cycles left to deal with 'the tomorrow.'
FCW: How confident are you that the database won’t fail again?
Ambrose: I’m confident that we are focused on ensuring that we don’t have that problem again.
FCW: When the database went down, did you reach out to anyone in the federal IT community for advice? How much feedback or criticism did you get?
Ambrose: We had a lot of people who offered help, and where they had some advice that helped us solve the problem that we were focused on at the time, we accepted that.
We did have a discussion with [then-U.S. CIO] Steve VanRoekel and Lisa Schlosser from OMB. They asked who I had engaged, and I told them who I had engaged, and they concurred that we had the right people focused on helping solve the problem. And we kept them informed. Microsoft and Oracle [and EMC] came with their senior folks and pledged their complete support.
We need to make sure that we have enough government technical leaders giving the direction to those contractors so that we’re all moving together in unison to a common goal.
FCW: Were you aggressive enough in getting the CCD fully functioning, given how many travelers were affected? How did you balance the need to act aggressively with the need to not do further damage to the system?
Ambrose: I believe we got tremendous support from the department and the bureau to do the things we wanted to do, and we were aggressive in managing [that]. The first priority was to get the system functioning, and we did that.
One of the things that we did do is we turned off, temporarily, our internal reporting capabilities, our ancillary dashboard functions. We still pulled reports for ad hoc reasons, to keep our oversight of operations, but we have a lot of ancillary reports that we temporarily disabled to make sure that we could get the functions of processing transactions operational.