Oversight

Watchdog dings contractors for IRS tax day outage

Shutterstock photo ID: photo ID: 245503636 By Mark Van Scyoc Sign outside the Internal Revenue Service building in downtown Washington, DC on December 26, 2014. 

An unpatched firmware bug, contractor errors and a lack of established communication protocols by top IT officials were responsible for an outage that shut down the Internal Revenue Service's e-file system and 58 other IRS processing systems for 11 hours on tax day, according to an Inspector General report.

The online e-file system for the IRS suffered an embarrassing Tax Day outage on April 17. The agency got the system back up and running within 11 hours, but wound up extending the tax filing deadline by another day.

Errors by two contractors, IBM and Unisys, were cited as contributing factors. When one of the high-availability storage arrays at IRS began detecting errors early in the morning on tax day, it sent an alert to the vendor, IBM. However, the company reportedly designated the error with a lower-level alert classification that only merited a response by the end of the next business day.

IBM actually identified the bug in 2017 and developed an updated version that fixed the flaw, but both IBM and IRS officials decided to go with an older version for the remainder of the filing season after determining it would be "more stable." Another client suffered a similar issue in January 2018, but IBM reportedly did not notify IRS or offer the preventative script developed to remediate the error.

The report also criticized Unisys because three of the company's storage arrays were not replicating all application and systems data from the main computing center in Memphis, creating "a single point of failure for tax processing operations." The IG also criticized Unisys for taking more than 20 hours to acknowledge and fix the problem – a process that was supposed to take at most 6.5 hours per the Unisys-IRS contract.

Auditors were largely contented with IRS' internal efforts to remedy problems. But the report notes the associate and deputy associate CIO were not notified until they arrived at work four hours after the problems began. Auditors advised the tax agency to document lessons learned during the process to incorporate for similar breakdowns in the future, formalize monthly meetings with Unisys and IBM to discuss microcode issues and build out more layers of approval for software updates.

They also recommended that IRS seek liquidated damages from Unisys for failing to meet performance levels during the outage. The IRS concurred and said it is seeking an unspecified amount and is seeking to modify several aspects of its contract to prevent similar failures.

In a response letter, IRS CIO Gina Garza agreed with remaining recommendations while noting that IRS believes its response to the outage "demonstrated our improved ability to rapidly and successfully respond to major incidents." She also noted that IRS had experienced a similar issue in 2016 and had cut down its response time from 30 hours to 11.

"This is significant and demonstrates our commitment to process improvements," said Garza. "We continue to develop and implement new technologies, refine our processes and conduct robust testing."

FCW has reached out to Unisys and IBM for comment.

About the Author

Derek B. Johnson is a senior staff writer at FCW, covering governmentwide IT policy, cybersecurity and a range of other federal technology issues.

Prior to joining FCW, Johnson was a freelance technology journalist. His work has appeared in The Washington Post, GoodCall News, Foreign Policy Journal, Washington Technology, Elevation DC, Connection Newspapers and The Maryland Gazette.

Johnson has a Bachelor's degree in journalism from Hofstra University and a Master's degree in public policy from George Mason University. He can be contacted at djohnson@fcw.com, or follow him on Twitter @derekdoestech.

Click here for previous articles by Johnson.


Featured

  • Contracting
    8 prototypes of the border walls as tweeted by CBP San Diego

    DHS contractors face protests – on the streets

    Tech companies are facing protests internally from workers and externally from activists about doing for government amid controversial policies like "zero tolerance" for illegal immigration.

  • Workforce
    By Mark Van Scyoc Royalty-free stock photo ID: 285175268

    At OPM, Weichert pushes direct hire, pay agent changes

    Margaret Weichert, now acting director of the Office of Personnel Management, is clearing agencies to make direct hires in IT, cyber and other tech fields and is changing pay for specialized occupations.

  • Cloud
    Shutterstock ID ID: 222190471 By wk1003mike

    IBM protests JEDI cloud deal

    As the deadline to submit bids on the Pentagon's $10 billion, 10-year warfighter cloud deal draws near, IBM announced a legal protest.

Stay Connected

FCW Update

Sign up for our newsletter.

I agree to this site's Privacy Policy.