Surveillance Data Storage in the Age of AI

As connected digital devices replace standalone analog devices, they are generating vast amounts of data that, in turn, allow us to refine and improve systems and processes – both security and in general – in ways that couldn’t be imagined in the past. Eventually, Big Data and AI metadata will be as critical as the introduction of electricity– touching nearly every aspect of our lives and the consequences will be significant. By 2025, an average connected person anywhere in the world will interact with connected devices – computers, mobile devices, smart TVs and more – nearly 4,800 times a day. That’s essentially one interaction every 18 seconds. In the next few years, an additional 26 billion sensors will be installed worldwide to help manage, monitor and improve our daily lives.

In this increasingly connected world where access to real-time information has more or less become a necessity, our thirst for data has never been greater than it is today, and that thirst will only continue to grow. This is especially true in the surveillance space where data is being analyzed more than ever before.

Like the datasphere itself, data analysis is expected to grow substantially in the next few years. IDC estimates the amount of data generated worldwide that is subject to data analysis will grow by a factor of 50 to 5.2 zettabytes by 2025, with the amount of data that is “touched” by cognitive systems growing by a factor of 100 to 1.4 zettabytes in that same time.

The ultimate value of data generated by an increasingly wider variety of disparate security and non-security sensors, devices and systems is its ability to be analyzed and distilled into actionable intelligence that can be used to increase security and improve operations. This virtual flood of data and the growing requirement for analysis pave the way for a new set of technologies including machine learning, natural language processing and artificial intelligence, or AI. Collectively these terms are known as cognitive systems, and they are capable of facilitating the shift of data analysis from a relatively uncommon and after-the-fact practice to a more proactive driver of strategic action and decision-making. All this data can be analyzed and used to introduce unique user experiences and mitigate potential risks, opening up a whole new world of business opportunities.

With data from AI-enabled security systems, important tasks can be completed, such as seamless, efficient traffic flow management among connected vehicles to prioritize traffic protocols for emergency vehicles, detecting fraud in real time, or employing facial recognition to improve security at sporting venues or transportation facilities. With the data gathered from AI systems, manufacturers can recognize operational efficiencies in production lines and can make immediate adjustments. Medical staff at hospitals can recognize unusual trends in a patient’s health and take action to significantly reduce mortality. Retailers can better understand their customer’s behavior as well as peak hours for foot traffic.

From a security standpoint, AI opens the door for intelligent real-time video analysis that can transition today’s ultra-high-resolution video from a tool used primarily for reactionary, post-incident investigation to a more proactive tool that enables preemptive action. Instead of reviewing countless hours of typically uneventful video, like a car moving in the parking lot, AI allows security integrators and end users to identify specific events and triggers in the footage, such as, a blue bike riding south or a grey-haired man walking towards the building with a dog. The ability to capture scene footage with this knowledge and insight enables more accurate alerts and forensics, dramatically reducing the time to act and analyze the video manually.

New Frontiers in Surveillance Storage

From a video storage solutions perspective, these trends have instigated a shift from primarily write-only applications toward constant, ongoing deep learning and analysis that produce unstructured data. Until recently, these systems relied on the processing power of cloud data centers to manage this analysis and heavy workload, but that model was fraught with issues around latency.

Today, AI is being built into more video NVR systems to enable them to process, analyze and recognize patterns on-site in real time at the edge, rather than dealing with the latency associated with transferring data and video off-site for analysis. What has spurred this AI evolution, particularly in edge video surveillance applications, is significantly cheaper and faster GPU’s with practically limitless storage. Hard disk drives (HDD) must be capable of writing data at high speeds to keep up with these AI applications and simultaneously support both AI and video workloads.

Software-defined storage technology has enabled rapid creation and migration of edge storage environments at the point where live data and Big Data analytics intersect to meet the need of local and mobile analytic workloads. The growing amount of data creation across the IoT universe in a mobile, real-time world is a fundamental driver of edge storage.

Recognizing the benefits of edge-based processing and analysis, storage solutions providers are developing or currently offer drives that offer high throughput and enhanced caching to deliver low latency and excellent read performance to quickly locate and deliver video images and footage analysis that doesn’t require the power of an off-site datacenter.

Selecting the Right Hard Drive

The seemingly unlimited demand for data and the sheer amount of video being captured by surveillance systems are driving the need for more advanced and cost-effective storage options that are optimized for machine learning, deep learning, high-resolution video, advanced analytics streaming and much more. End users and systems integrators alike are looking for reliable, high-capacity drives that are purpose-built for surveillance that support multiple cameras, 24/7 availability and the capability to maximize streaming and frame rate performance.

Given the increasing criticality of video for security and other uses, deploying the right recording and storage solution for the right purposes is critical. Organizations simply can’t afford to lose any data or video integrity, which surveillance AI systems hinge their accuracy and predictive powers on. Additionally, storing more data for longer lengths of time is necessary for AI systems to become “smart,” increasing their predictive analytic capabilities. With all these factors in mind, there are a few best practices integrators can employ to ensure they are selecting the most appropriate hard disk drive for a particular customer application – and the workload associated with it – to ensure long-term success.

For starters, it’s important to select a solution that is purpose-built for the demands of video surveillance applications. Consider that a typical desktop PC hard drive which has no camera support and is built to work only eight hours a day, five days a week, or 2,000 hours over a full year. A surveillance hard drive gets no down time, working 24 hours a day, seven days a week –over four times as many hours as a desktop drive over the course of that same year. The differences in the way these two types of drives operate means they must be capable of performing under the extreme workload stresses to which they will be subjected in a deployment. Choosing anything less than a surveillance-optimized drive for video applications will result in a sub-optimal outcomes and even failures for video streaming quality, workload performance and analytical capabilities. In addition, drives that offer some sort of health monitoring should be considered to provide prevention, intervention and recovery from potential data loss related to unexpected physical damage as a result of vandalism or natural disasters.

However, it is important to remember that beyond picking the right drive, proper drive handling and installation in surveillance systems is critical to overall performance, reliability and TCO. Always keep in mind, the drive must be cooled properly and the proper tools must be used for drive installation to avoid multi-bay vibration issues.

Conclusion

The ever-growing datasphere has led to advanced and creative uses for the video and data generated by connected devices. Analyzing this virtual sea of data can generate actionable intelligence in real time to enable proactive actions that can mitigate or prevent a security issue from occurring, streamline operational processes, improve customers’ retail experience and much, much more. Among all this data, video plays a key role, and AI-enabled surveillance storage solutions enable the faster, more accurate analysis organizations need to quickly adapt to achieve their security and business objectives.

However, all storage hard drives are not created equal, and it is vital that integrators carefully evaluate solutions based on the criteria that will impact their performance for the workload specifications of each customer and application. Purpose-built drives with advanced technologies to allow both video streaming and AI analysis ensure the best possible drive reliability and performance, while delivering optimal TCO and the most valuable, actionable intelligence possible.