Big Data

Can privacy exist in a big data world?

abstract head representing big data

The federal government is the biggest data collector in the world, housing increasingly enormous databases on everything from tax records and census information to signals intelligence and scientific research.

Many federal agencies, especially in the defense and intelligence communities, have begun to make use of big data tools and techniques to sift through those gobs of information looking for key insights they might otherwise miss.

In some cases, especially in light of the National Security Agency’s vast surveillance efforts revealed in leaks by former contractor Edward Snowden, big data have generated big privacy concerns in the public. By simply using a government system – or, in the NSA example, even using the Internet or a telephone – users are offering up information that is logged and can be stored indefinitely.

The Department of Homeland Security, for example, continuously monitors the incoming flow of goods, services and people from outside U.S. borders against no-fly and watch lists to ascertain risk and determine threats. According to Alan Bersin, assistant secretary of international affairs and chief diplomatic officer for DHS, U.S. Customers and Border Patrol “has data from every person that crossed over [the border] in the last 10 years or more.”

Users may not sign up to share that information, but it is shared and stored regardless.

“We have kept a bargain with the American people that the data is used only for the purpose with which it was gathered,” said Bersin, speaking at a recent Bipartisan Policy Center event on big data. “Rather than deny the existence of big data, we can institute it in a regime consistent with American values.”

The Privacy Act of 1974 bars the government to using collected data for any purpose beyond that for which it was intended.  Mary Ellen Callahan, former chief privacy officer for DHS, said concerns grow at the federal level when agencies look to wield the shiniest new technology without knowing exactly what data they want and what they plan to do with it. Now a partner at Jenner & Block, Callahan’s job at DHS was to ensure agency policies protected people’s privacy.

“From the big data perspective, what makes me nervous is using it with uncertainty of how to do anything, not finding what you want to do with the data before you get it,” Callahan said. “Big data needs a lot more definition, more parameters.”

Privacy advocates argue against the government’s collection of data simply because the technology and cheaper cost of data storage allow for it.

Greg Nojeim, senior counsel for the Center for Democracy and Technology, said there is “not a logical end” to collecting increasingly large amounts of data. He said privacy should be addressed on the front-end of policies that allow for large-scale data collection; otherwise they are likely to be overrun in the name of collecting data.

“The world we’re moving toward is one of ‘collect everything, put use restrictions on the back end, and that will fundamentally change the relationship we have with state,’” Nojeim said. “It puts the state in a better control position. We get that big data is here and we adapt to it – to things like use restrictions. But I don’t think we have to have a world where every data set that might contain a piece of data ought to be collected.”

Such restrictions on data collection are simply denying the inevitable, argued Stewart Baker, former assistant secretary for policy at DHS.  “It’s striking to me the extent at which we’re fighting the future,” Baker said.

Baker said “sooner or later, we’ll all get used to the idea” of different notions of privacy, and suggested large-scale data collection could not be stopped even if the government tried. Baker argued against “trying to get people to adopt dumb laws aimed at fighting off the future” as a waste of resources and time, and instead said the government should “try to maintain limitations on use and mission.”

It is technology that allows for the collection, analyzing of and storage of vast amounts of data, but technology may also be what ultimately enables privacy and civil rights to exist in the big data era.

Christina Norwich, an engineer for Silicon Valley software company Palantir, said big data efforts and evolving technologies need not come at the expense of privacy.

“Technology can enable these programs to exist and have proper oversight exist,” Norwich said, noting that some innovative private sector companies have installed internal compliance teams with access to “immutable audit logs.” In such an environment, someone is “watching the watchers,” and every action a person has taken on data is viewable.

“From a technical perspective, I don’t think it has to be either/or,” Norwich said.


About the Author

Frank Konkel is a former staff writer for FCW.

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.


  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Thu, Nov 21, 2013

Big Brother is a usual product of Big Government. ObamaCare is a prime example of an expanded government trying to run all aspects of people's lives - and it is made possible by Big Data. That is not to say that Big Data is bad, but it is a tool that can be used for bad as well as good purposes. As such, it is reasonable for people to be suspicious of any big data project that is controlled by politicians. The IRS database has been recently abused for political purposes and many believe that the ObamaCare site will become a prime target for future abuses. There are several other big databases that are also being proposed that can also be used politically against the people they are supposed to be helping. Many believe that the intentions of some supporters of these proposed databases is for these databases to be used for unethical political advancement - and they have plenty of evidence to support that theory. As such, until there are strong enough limitations on the government, Big Data will be seen as a threat to many law abiding people who do not bow down to and follow the political elite.

Wed, Nov 20, 2013 big data training

nice article, big data is the future

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group