'To purge or not to purge'

man studying data

It's not easy to nail down the rules of the road followed by U.S. spy agencies when it comes to how they store the data they collect. At a recent hearing of the Privacy and Civil Liberties Review Board, top intelligence community lawyers tried to explain the nuances of how they store data collected from Internet services and intercepted as it travels across global networks.

"To purge or not to purge -- that is the question," summed up Rajesh De, general counsel of the National Security Agency. The answer, as it turns out, is not very clear.

The collection practices, revealed to the public by the Edward Snowden leaks, are authorized by section 702 of the Foreign Intelligence Surveillance Act. They are not designed to collect information on people protected by U.S. law, including U.S. citizens whether at home or abroad, and persons inside the U.S.

But there are fears that the collection procedures, which give analysts fairly wide latitude in intercepting and storing  all manner of Internet communications, including email, social media, video and voice calls, chat, and more, can be used to target subjects ostensibly protected by U.S. law. The intelligence community points to minimization procedures, periodic reviews and oversight from Congress and  agency inspectors general as checks against unlawful collection.

Those internal checks are meant to lead to the purging of information on U.S. persons – citizens and permanent residents -- that was incidental to other collection, collected inadvertently or is outside the boundaries of the authorizing statute. Given the secrecy of the programs and differences in the rules about different modes of collection, it can be difficult to determine when an intelligence agency is required to expunge data from its systems.

Records collected from Internet companies through the Prism program are retained for five years. Data siphoned directly from the Internet backbone, called Upstream collection, are supposed to be expunged after two years, because of a Foreign Intelligence Surveillance Court determination that such information is more likely to contain incidental or unauthorized communications.

Both methods authorized by 702 collect the entirety of communications -- the metadata and the underlying content. According to De, about 10 percent of NSA collections come from Upstream targets, and the rest comes from Prism. Neither method is considered bulk collection by the intelligence agencies -- information is targeted using "selectors" like an email address or a phone number. It's not clear what else might constitute a selector, but presumably information such as a device's IP address or a mobile device’s unique identifier could be included. Intelligence agencies can also monitor communications for references to a third-party selector. These so-called "about" collections are supposed to target selectors pertaining to non-U.S. persons.

Robert Litt, general counsel for the Office of the Director of National Intelligence, told the board that selectors are focused on devices or accounts, not broad populations. For instance, an area code, he said, would not qualify as a selector.

The declassified version of the NSA's minimization rules indicates that the agency deposits data from 702 collections into a "segregated repository" where it can be tagged by specialists as being relevant to NSA analysts. Data collected under Prism can be queried for information pertaining to U.S. persons, as long as it is "first approved in accordance with NSA procedures." The NSA is charged with keeping records on all such cases. Data intercepted via Upstream collection may not be queried for information designed to target U.S. persons. 

According to a redacted compliance report declassified by the DNI, problems arise in one half of one percent of selectors, including collecting data on U.S. persons or collections outside the scope of what is authorized.

Collections and targets that are out of compliance are handled in different ways, according to De.

Intelligence agencies are supposed to "detask" or stop collection on selectors that are determined to have been targeted my mistake, or outside the scope of FISA. Data collected on those targets is supposed to be removed from NSA systems in such a way that it cannot be used. It's not clear if this means that the data is deleted or that it is tagged in such a way that it cannot be queried. De indicated that NSA systems are designed to take into account the requirements of the oversight and minimization regimes.

While De said he couldn't articulate all the exceptions to the NSA's minimization procedures, he said that wholly domestic communications as a default rule must be purged. At the same time, there is an exception for information that is said to have foreign intelligence value, as determined by an analyst.

Information that implicates U.S. persons in potential crimes or terrorist networks is also retained. If a terrorist is calling a U.S. person in Minneapolis, "it's of high interest to us," said Brad Wiegmann, deputy assistant attorney general in the National Security Division of the Department of Justice.

The minimization procedures and the exceptions "give the government broad authority to collect purely domestic communications," said Jameel Jaffer, deputy legal director of the American Civil Liberties Union.

Despite widespread and highly vocal criticism, the spy agencies have no desire to eliminate or pare back the programs. While Litt and others acknowledged a need for more transparency, they said the programs have been useful and worth the expense.

"You can see time and again in important intelligence reports provided to policymakers that it is derived from section 702 collection," Litt said.

About the Author

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy, health IT and the Department of Veterans Affairs. Prior to joining FCW, Mr. Mazmanian was technology correspondent for National Journal and served in a variety of editorial at B2B news service SmartBrief. Mazmanian started his career as an arts reporter and critic, and has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, Architect magazine, and other publications. He was an editorial assistant and staff writer at the now-defunct New York Press and arts editor at the About.com online network in the 1990s, and was a weekly contributor of music and film reviews to the Washington Times from 2007 to 2014.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Shutterstock image: looking for code.

    How DOD embraced bug bounties -- and how your agency can, too

    Hack the Pentagon proved to Defense Department officials that outside hackers can be assets, not adversaries.

  • Shutterstock image: cyber defense.

    Why PPD-41 is evolutionary, not revolutionary

    Government cybersecurity officials say the presidential policy directive codifies cyber incident response protocols but doesn't radically change what's been in practice in recent years.

  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group