CFTC mulls retooling market surveillance for high-frequency trading

astonished man in data center

U.S. financial regulators are in the midst of upgrading their IT capabilities to monitor and police high speed computer trading. As documented in the recent Michael Lewis book "Flash Boys," high-speed traders go to considerable expense to co-locate their network infrastructure inside exchanges to gain a latency advantage over less nimble rivals that can be measured in microseconds. They hire top mathematicians to write algorithms to predict the movement of markets and trade accordingly. Events like the 2010 "Flash Crash" have roused regulators to the need to develop investigative capabilities that match the speed and dynamism of the market.

The consensus is that regulators are still a long way off.

The Commodity Futures Trading Commission is collecting more data than ever via new rules established under the authority of the Dodd-Frank legislation. Over the past year, the CFTC has seen a 166 percent increase in the kinds of data its system takes in, according to Jorge Herrada, associate director of the CFTC's Office of Data and Technology. The loading of structured data into CFTC systems requires standing up new architecture, customized loaders, and quality checks between the agency and designated swap data repositories.

"Each new data set is a major project unto itself, analogous to standing up a new factory production line," Herrada said at a June 3 meeting of the CFTC's Technology Advisory Committee.

The CFTC has collected about 160 terabytes of transaction data, covering billions of transactions. But there are still lingering issues about what the data represents.

There is room for interpretation about data like the time of the execution or modification of a trade, and until such definitions are harmonized across the industry, the ability to track the movements of trades from one SDR to another is limited.

"We have the raw materials for a great surveillance program, but we still have a lot to do," Herrada said.

Regulators are also limited by law to specific roles. But technically it could be feasible to design a system that provides regulators with a window into activity across different asset classes, whether stocks, bonds, futures or other categories.

Bad actors move across markets, said Steve Joachim, executive vice president of the Financial Industry Regulatory Authority, a self-regulatory organization. "Frankly, today the regulatory community does not treat the marketplace on a broad scale in one view," Joachim said. A 21st century system would give a more complete picture of the markets across the world and across sectors.

Pivoting to this new reality in government time might prove challenging. While the 160 terabyte dataset at CFTC is large by most government standards, it is dwarfed by the processing demands of the NSA, NASA, NOAA, and even some private-companies. Dave Lauer, president and managing partner at KOR Group, said there are open source platforms to manage more data than are generated in the financial services industry.

These big data issues "are intimidating, but they are problems that have been solved," Lauer said, adding that government regulators are behind the times when it comes to advanced analytics, machine learning, and other capabilities that could improve surveillance of financial markets.

The real issue is redeploying resources to attack the problem --the government IT workforce as currently constituted can't fix the problem, according to Lauer.

"You can't do this at GS-13 or GS-14. That's why a radical reallocation of budget resources is necessary. This is what regulation is going to be in the 21st century," he said.  

About the Author

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy, health IT and the Department of Veterans Affairs. Prior to joining FCW, Mr. Mazmanian was technology correspondent for National Journal and served in a variety of editorial at B2B news service SmartBrief. Mazmanian started his career as an arts reporter and critic, and has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, Architect magazine, and other publications. He was an editorial assistant and staff writer at the now-defunct New York Press and arts editor at the online network in the 1990s, and was a weekly contributor of music and film reviews to the Washington Times from 2007 to 2014.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.

The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.


  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group