Data management

State's passport and visa system crashes

Shutterstock image: United States passport with an unidentifiable credit card resting underneath it.

Passport verifications are one of several functions to be compromised by the Consular Consolidated Database crash.

The colossal data warehouse that supports the State Department's worldwide visa and passport verification operations has crashed, potentially stranding thousands of people waiting for the documents around the world.

The Consular Consolidated Database (CCD) at the State Bureau of Consular Affairs "is currently experiencing technical problems with our passport/visa system," said Marie Harf, deputy spokesperson for the State Department in a July 24 statement to FCW. The Associated Press first reported the CCD problems on July 23.

Harf said the CCD, the agency's system of record, has had "significant performance issues, including outages, since Saturday, July 19." The issues, she said, have resulted in significant backlogs. Visas are approved, recorded and printed through the CCD. "Until the system comes back online, we are unable to print visas. We apologize to applicants and recognize this may cause hardship to applicants waiting on visas and passports. We are working to correct the issue as quickly as possible," she said.

Ongoing problems with the CCD have been no secret, an industry source familiar with the system told FCW. The database crashed earlier this year -- without the visible disruption of the July 19 incident, but seriously enough that Bureau of Consular Affairs CIO Greg Ambrose sent an email to State Department and industry stakeholders about the problem.

The CCD system is massive, containing millions of visa cases and photographs, with links to other federal agency security databases, including the FBI's Integrated Automated Fingerprint Identification System (IAFIS) and the Department of Homeland Security's Automated Biometric Identification System (IDENT). It is also the gateway to the Department of State Facial Recognition system and the NameCheck system.

According to a State Department privacy impact assessment completed in 2010, the CCD is one of the largest Oracle-based data warehouses in the world. As of December 2009, according to the PIA, the database contained more than 100 million visa cases and 75 million photographs, used billions of rows of data, and had a growth rate of about 35,000 visa cases every day.

The PIA said the database was created to provide Consular Affairs a near real-time aggregate of the consular transaction activity collected domestically and at consular post databases worldwide. "The CCD is the IT implementation that provides for a set of centralized visa and American citizen services supporting consular posts and back office functions," the assessment found.

Three of CCD's primary functions include supporting data delivery to approved applications via industry-standard Web Service queries, providing users with easy-to-use data entry interfaces to CCD, and allowing emergency recovery of consular post databases. Additionally, according to the PIA, authorized State Department and other government agency personnel can use the CCD Portal to view centralized data and access other applications.

Harf said the issue is worldwide and not specific to any particular country, citizenship document or visa category. "We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon."

Reader comments

Thu, Aug 14, 2014 sandra new york

Bad bad news;) . Could you telle us when this experiencing technical problems will ba make up please ? Is it ok right now ? Thank you

Mon, Aug 4, 2014

This work is currently being conducted by an incumbent contractor. ActioNet will be assuming this work in October.

Mon, Jul 28, 2014 DOS Insider

This has been in the making for much longer than two years. Nothing happens fast in the government. According to those in the know, the database is built using outdated technology. The new technology leadership in that bureau is working to implement enhancements to bring their technology into the current century. Let's hope they get the support needed to be successful. Episodes like this are reminders that regular technology upgrades are needed to support our country's growing support to its citizens.

Sun, Jul 27, 2014

ActioNet and CITI... The former was selcted over the latter (incumbent) on price, big surprise!! :)

Fri, Jul 25, 2014 Michelle

Database as "The Weakest Link" The State Department is only one of the more visible instances of an organization suffering a service outage at the hands of a database. A quick search on "database outage" yields a throng of mea culpa blog entries of companies detailing how a "seemingly simple" database change or upgrade resulted in an outage of minutes, hours, or - as the poor State Department is enduring - many days. Web load balancers deliver uptime for web servers, network redundancy and routing algorithms deliver uptime for networks - what protects the database? Sadly, typically nothing. Most often, apps are directly written and connected to the database - leaving them vulnerable to database infrastructure changes. A new approach - front ending databases with database traffic management - is emerging, with all sorts of benefits in zero downtime maintenance, auto failover without app errors, and no app modifications needed to leverage database failover and scalability. Check out more ideas on moving to zero downtime for apps here: http://www.scalearc.com/blog/architecting-for-uptime-avoiding-app-interruptions-during-database-failover/

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above