Program Management

Charting agency big data progress

Federal agencies, led and inspired by an Obama administration initiative, are making some progress in using big data.

Although much of the federal government has lagged behind local governments and the private sector in pursuing big data solutions in earnest, a number of agencies have made headway. Here are some examples.

1. Internal Revenue Service

Using big data analytics, the IRS was able to fix errors early in the tax-filing process for 2012, saving $100 million in erroneous claims. The agency also has a research lab developing data models for detecting fraud.

2. Bureau of Indian Affairs

Until recently, the agency’s law enforcement efforts on Indian reservations involved responding to crimes and other incidents on an individual basis, with the result that an agencywide picture was murky at best and only available from data given to the FBI. But a recently introduced pilot program connected four reservations and allowed them to share crime data, including historical and current reports. The entire agency has since adopted a similar model.

3. NASA: Big Data Challenge

In October, NASA launched something besides rockets: an online competition that asks participants to come up with novel ideas to make use of the large, unwieldy heterogeneous datasets kept by various federal agencies in the areas of health, energy and earth science. The submission period closed Oct. 13, and NASA is due to award three grand prizes of $750 each for the best ideas, with a 30 percent bonus for ideas that make use of live streaming data.

4. NASA: Johnson Space Center

NASA’s Johnson Space Center in Houston is the storehouse for one of the world’s largest still imagery and audiovisual collections. It consists of 4 million still shots and more than 81,000 hours of video, and the collection is growing at an exponential rate. The agency is able to keep track of it all through analog-to-digital migration and file compression and storage processes.

Main story: Managing big data

Related: How big data fights crime

5. National Institute of Biomedical Imaging and Bioengineering

Big data aided in an innovative risk management approach at the institute that involves using analytics software to assess entire research portfolios rather than individual grants. The approach helps determine the success of projects as measured by such salient features as number of publications and licensing agreements, to name a few, giving the institute an idea of where best to allocate research dollars.

6. Department of Veterans Affairs

VA maintains one of the largest datasets in the federal government; as of mid-2012, the agency had more than 175 data centers. The Veterans Health Administration’s Office of Analytics and Business Intelligence keeps track in real time of homeless veterans nationwide and shares that information with the Department of Housing and Urban Development in its efforts to combat homelessness.

7. Centers for Medicare and Medicaid Services

CMS has increasingly been using big data to help keep track of trends in its patients’ health status and to improve care.

8. National Archives and Records Administration

As of the beginning of 2012, NARA was in charge of 142 terabytes of data and 4 billion cubic feet of analog archives, including 400 million pages of classified information. To keep track of it all, the agency has constructed its Electronic Records Archives to be able to quickly adapt to users’ changing needs, resulting in a more rapid capability to archive and categorize new records.

About the Author

Carl Franzen is a technology reporter based in Brooklyn, N.Y. He writes for TPM IdeaLab. Follow him on Twitter at: @carlfranzen

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.


  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group