Big Data

Can privacy exist in a big data world?

abstract head representing big data

The federal government is the biggest data collector in the world, housing increasingly enormous databases on everything from tax records and census information to signals intelligence and scientific research.

Many federal agencies, especially in the defense and intelligence communities, have begun to make use of big data tools and techniques to sift through those gobs of information looking for key insights they might otherwise miss.

In some cases, especially in light of the National Security Agency’s vast surveillance efforts revealed in leaks by former contractor Edward Snowden, big data have generated big privacy concerns in the public. By simply using a government system – or, in the NSA example, even using the Internet or a telephone – users are offering up information that is logged and can be stored indefinitely.

The Department of Homeland Security, for example, continuously monitors the incoming flow of goods, services and people from outside U.S. borders against no-fly and watch lists to ascertain risk and determine threats. According to Alan Bersin, assistant secretary of international affairs and chief diplomatic officer for DHS, U.S. Customers and Border Patrol “has data from every person that crossed over [the border] in the last 10 years or more.”

Users may not sign up to share that information, but it is shared and stored regardless.

“We have kept a bargain with the American people that the data is used only for the purpose with which it was gathered,” said Bersin, speaking at a recent Bipartisan Policy Center event on big data. “Rather than deny the existence of big data, we can institute it in a regime consistent with American values.”

The Privacy Act of 1974 bars the government to using collected data for any purpose beyond that for which it was intended.  Mary Ellen Callahan, former chief privacy officer for DHS, said concerns grow at the federal level when agencies look to wield the shiniest new technology without knowing exactly what data they want and what they plan to do with it. Now a partner at Jenner & Block, Callahan’s job at DHS was to ensure agency policies protected people’s privacy.

“From the big data perspective, what makes me nervous is using it with uncertainty of how to do anything, not finding what you want to do with the data before you get it,” Callahan said. “Big data needs a lot more definition, more parameters.”

Privacy advocates argue against the government’s collection of data simply because the technology and cheaper cost of data storage allow for it.

Greg Nojeim, senior counsel for the Center for Democracy and Technology, said there is “not a logical end” to collecting increasingly large amounts of data. He said privacy should be addressed on the front-end of policies that allow for large-scale data collection; otherwise they are likely to be overrun in the name of collecting data.

“The world we’re moving toward is one of ‘collect everything, put use restrictions on the back end, and that will fundamentally change the relationship we have with state,’” Nojeim said. “It puts the state in a better control position. We get that big data is here and we adapt to it – to things like use restrictions. But I don’t think we have to have a world where every data set that might contain a piece of data ought to be collected.”

Such restrictions on data collection are simply denying the inevitable, argued Stewart Baker, former assistant secretary for policy at DHS.  “It’s striking to me the extent at which we’re fighting the future,” Baker said.

Baker said “sooner or later, we’ll all get used to the idea” of different notions of privacy, and suggested large-scale data collection could not be stopped even if the government tried. Baker argued against “trying to get people to adopt dumb laws aimed at fighting off the future” as a waste of resources and time, and instead said the government should “try to maintain limitations on use and mission.”

It is technology that allows for the collection, analyzing of and storage of vast amounts of data, but technology may also be what ultimately enables privacy and civil rights to exist in the big data era.

Christina Norwich, an engineer for Silicon Valley software company Palantir, said big data efforts and evolving technologies need not come at the expense of privacy.

“Technology can enable these programs to exist and have proper oversight exist,” Norwich said, noting that some innovative private sector companies have installed internal compliance teams with access to “immutable audit logs.” In such an environment, someone is “watching the watchers,” and every action a person has taken on data is viewable.

“From a technical perspective, I don’t think it has to be either/or,” Norwich said.


About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.


  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Thu, Nov 21, 2013

Big Brother is a usual product of Big Government. ObamaCare is a prime example of an expanded government trying to run all aspects of people's lives - and it is made possible by Big Data. That is not to say that Big Data is bad, but it is a tool that can be used for bad as well as good purposes. As such, it is reasonable for people to be suspicious of any big data project that is controlled by politicians. The IRS database has been recently abused for political purposes and many believe that the ObamaCare site will become a prime target for future abuses. There are several other big databases that are also being proposed that can also be used politically against the people they are supposed to be helping. Many believe that the intentions of some supporters of these proposed databases is for these databases to be used for unethical political advancement - and they have plenty of evidence to support that theory. As such, until there are strong enough limitations on the government, Big Data will be seen as a threat to many law abiding people who do not bow down to and follow the political elite.

Wed, Nov 20, 2013 big data training

nice article, big data is the future

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group