Taking video analytics to the next level

The technology has become more common in the federal space and even in consumer settings, but it still has a long way to go.

Placeholder Image for Article Template

The science of pulling useful data out of digitized video -- or video analytics -- is an increasingly confounding task given the staggering amount of archived video and new footage captured every day from an expanding array of devices. It's not just coming from fixed and pan-tilt-zoom cameras any more: Lapel cameras, helmet-mounted cameras, consumer technologies such as Google Glass and the ubiquitous smartphone all contribute to the video stream.

Government uses tend to focus on areas such as perimeter security, where motion detection and object tracking are important applications of video analytics by agencies such as U.S. Customs and Border Protection, which started using the technology in the early 2000s. Government agencies also use video analytics to capture images of license plates and identify people.

Although the current applications have proven useful, today's technology has significant limitations. Government, academic and industry researchers confront a number of challenges that range from exploring ways to more efficiently comb through huge stores of video -- essentially a big-data problem -- to coping with cameras that move around as they record objects that are also in motion.

Furthermore, developments in software have failed to keep pace with the explosion in video camera hardware. Jie Yang, a program director at the National Science Foundation, said the field of computer vision, a key component of video analytics, has made progress in terms of object detection and tracking, facial detection, and license-plate recognition. But high-level video analysis, which would provide a more comprehensive understanding of the objects recorded in video and their relationships to one another, is far from mature.

"People are working hard, but we are not really there yet," said Yang, who is responsible for the Information and Intelligent Systems core programs and the National Robotics Initiative at NSF.

Why it matters

Video analytics tools are often associated with security. The technology can continuously monitor multiple video feeds for movement or other details that could escape the attention of a human observer. Agencies that protect government-owned buildings and infrastructure use the technology to get more out of their investments in video surveillance.

"We are seeing more agencies putting out more cameras," said Warren Brown, president of ObjectVideo, which licenses its video analytics technologies to IP video manufacturers. "Inevitably, there are not enough people to watch all of that video, and that is where the...push for analytics comes from."

Law enforcement agencies, meanwhile, increasingly rely on video analytics for facial recognition. The technology played a role in the investigation into the Boston Marathon bombing last year, and some federal agencies continue to explore that use of the technology. The FBI, for instance, has launched a study of video and digital image processing and analytics, issuing a request for information last year asking industry leaders to demonstrate capabilities in facial, vehicle and license-plate recognition.

In the RFI, officials said they would like to "identify current capabilities, assess gaps and develop a road map for the FBI's future video analytics architecture."

The technology, however, reaches beyond the realms of security and law enforcement. Yang cited the example of a research project at the Georgia Institute of Technology that uses video analytics to help screen children for autism. The research is funded by a $10 million grant from NSF's Expeditions in Computing program.

Georgia Tech's initiative involves using facial recognition and video analytics to detect anomalies in a child's eye contact with adults that could indicate autism. The automated approach eliminates hours of studying video frames to identify moments of eye contact, researchers said.

video still of child being assessed for autism

 

Video analytics technology is also finding a niche in humanities research. The Large-Scale Video Analytics (LSVA) research effort uses supercomputing power to explore video collections. The project brings together researchers from the University of Southern California (USC); the Institute for Computing in Humanities, Arts and Social Sciences; and the National Center for Supercomputing Applications, which is supported by grants from NSF and other federal agencies.

Virginia Kuhn, an associate professor in USC's School of Cinematic Arts, said it is hard to imagine an area in humanities research that would not benefit from improved video analytics.

"There [are] the 115 years of cinema [and] the massive amount of broadcast television and cable shows that are migrating to sites like Netflix, Amazon [Instant] Video and Hulu," she said. "All of these are important aspects of culture insofar as they impact our sense of identity and our knowledge of the world."

The fundamentals

Video analytics software that zeros in on the object or event of interest is part of a broader architecture that includes cameras, encoders, servers, storage and networks. The analytics capability might reside on servers, the cameras or the encoders, which convert video from analog cameras so the moving images can travel over IP networks.

Moving analytics to the edge -- on cameras or video encoders -- provides several advantages, according to Scott Dunn, director of business development at Axis Communications, a network video vendor.

"First, the camera can process all the video before it's sent over the network," he said. "So, for example, instead of streaming constant video to read license plates, the camera knows to send only the relevant five-second clip and process the plate number. This also means the video being analyzed is raw and uncompressed, as opposed to server-based intelligence that uses compressed video. The end result of edge processing is that video no longer must be sent to the centralized server, meaning you can dramatically decrease bandwidth and centralized computing power needs, increasing savings for the total system."

Indeed, storage is an important, if unsung, component of the architecture. In smaller-scale environments, videos might be housed on digital or network video recorders. Organizations that need the scalability to support vast amounts of video can invest in network-attached storage or storage-area networks.

The demand for such storage is growing rapidly. In February, MarketsandMarkets estimated that the global market for video surveillance storage will grow from $4.9 billion in 2013 to $10.4 billion in 2018, a compound annual growth rate of 16.3 percent.

Furthermore, the report notes that the prices for hard disk drives are going down. Consequently, the declining cost of storage could encourage the growth of the video analytics solutions that rely on them.

Camera prices are also dropping. Brown said thermal imaging cameras, which once cost tens of thousands of dollars, have declined sharply in the past 18 to 24 months, with the price range for some devices dipping to $1,000 to $2,000.

Brown said the lower prices make thermal imaging a more cost-effective option relative to other perimeter-protection tools, and those cameras can be enhanced by using video analytics.

The hurdles

The volume of available video -- and the time it takes to ingest and process it -- is an important limitation. "Working with a large number of images is a big-data problem," Yang said. "The majority of the images, we can't touch them."

"It's...time-consuming but also nowhere near an exact science," Kuhn said.

The future of video analytics

Experts say video analytics technology is likely to evolve in the following areas:

  • Unmanned aircraft systems. Applying analytics to UAS video has yet to emerge in a significant way, but it is on the horizon.
  • Better computer vision. Researchers continue to battle the fundamental problem of image quality, which is at the mercy of the camera's capabilities and lighting conditions.
  • Cloud computing. Products such as Dropcam's Wi-Fi home-and-business video-monitoring service, which offers the ability to store footage in the cloud, could point the way to a bigger future role for cloud computing in video analytics.

-- John Moore

The sheer amount of video complicates activities such as content tagging to facilitate searching. "Tagging is too labor-intensive for humans to do, and there are also problems with tags since a word can never adequately represent an image," Kuhn said.

The LSVA project seeks to harness the power of high-performance computing -- specifically the Gordon supercomputer at the San Diego Supercomputer Center. Gordon can analyze large video archives, but human experts supplement its work. For instance, when the computer searches a video archive, researchers verify the results.

Another ongoing challenge involves recognizing and matching objects recorded on video. Even facial recognition, which is considered fairly mature, still stumbles on occasion. For instance, recognition systems mainly focus on the front view of faces, but many of the faces recorded in surveillance video are in profile because people tend not to stare directly at wall- or ceiling-mounted cameras.

"The side of the face is much harder" to match, Yang said, noting that the profile recognition issue has yet to be fully resolved.

Matching also proves difficult in the case of transformable objects, said Gregory Pepus, managing partner and founder of Flex Analytics. Cars and buildings, for example, typically don't change shape, but that's not the case for people and animals. A person can stand upright or contort into a yoga position.

Flex Analytics taps technology from companies such as piXlogic to address that problem. Pepus said that with PiXlogic, "we are very far along" in solving the issue of matching transformed images. The piXlogic technology segments video into smaller pieces to identify more and more granularity, he added.

Analyzing video generated by mobile cameras is another challenge. Brown said most algorithms assume that the camera isn't moving, so video analytics providers must start from scratch on new algorithms.

Mobile cameras of all kinds, including those integrated into unmanned aircraft systems, are becoming an important new source of video, and Yang was optimistic about the evolution of video general. "It creates a lot of new challenges and opportunities," he said.

NEXT STORY: Exit interview: Joseph Klimavicz

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.