Watchdog dings contractors for IRS tax day outage

An unpatched firmware bug, contractor errors and a lack of established communication protocols by top IT officials were responsible for an outage that shut down the Internal Revenue Service's e-file system and 58 other IRS processing systems for 11 hours on tax day, according to an Inspector General report.

The online e-file system for the IRS suffered an embarrassing Tax Day outage on April 17. The agency got the system back up and running within 11 hours, but wound up extending the tax filing deadline by another day.

Errors by two contractors, IBM and Unisys, were cited as contributing factors. When one of the high-availability storage arrays at IRS began detecting errors early in the morning on tax day, it sent an alert to the vendor, IBM. However, the company reportedly designated the error with a lower-level alert classification that only merited a response by the end of the next business day.

IBM actually identified the bug in 2017 and developed an updated version that fixed the flaw, but both IBM and IRS officials decided to go with an older version for the remainder of the filing season after determining it would be "more stable." Another client suffered a similar issue in January 2018, but IBM reportedly did not notify IRS or offer the preventative script developed to remediate the error.

The report also criticized Unisys because three of the company's storage arrays were not replicating all application and systems data from the main computing center in Memphis, creating "a single point of failure for tax processing operations." The IG also criticized Unisys for taking more than 20 hours to acknowledge and fix the problem – a process that was supposed to take at most 6.5 hours per the Unisys-IRS contract.

Auditors were largely contented with IRS' internal efforts to remedy problems. But the report notes the associate and deputy associate CIO were not notified until they arrived at work four hours after the problems began. Auditors advised the tax agency to document lessons learned during the process to incorporate for similar breakdowns in the future, formalize monthly meetings with Unisys and IBM to discuss microcode issues and build out more layers of approval for software updates.

They also recommended that IRS seek liquidated damages from Unisys for failing to meet performance levels during the outage. The IRS concurred and said it is seeking an unspecified amount and is seeking to modify several aspects of its contract to prevent similar failures.

In a response letter, IRS CIO Gina Garza agreed with remaining recommendations while noting that IRS believes its response to the outage "demonstrated our improved ability to rapidly and successfully respond to major incidents." She also noted that IRS had experienced a similar issue in 2016 and had cut down its response time from 30 hours to 11.

"This is significant and demonstrates our commitment to process improvements," said Garza. "We continue to develop and implement new technologies, refine our processes and conduct robust testing."

FCW has reached out to Unisys and IBM for comment.