Generative AI is finally enabling the promise of big data

Those who have been following big tech industry trends for a while now will undoubtedly recall the concept of "big data.” The idea was that companies were going to pull together all the various data sources they had access to — traditional office documents and emails, business process data, sales results, customer databases, videos, chat logs and more — and then tap into all that data to glean meaningful insights to empower their organizations. 

In theory, the concept was sound and expectations around it were high. There were bound to be hidden nuggets of useful information and lots of unexpected insights that would start to appear as all the various data sources were combined into what proponents believed would be a powerful mélange of meaning. In practice, unfortunately, the results were far different. 

Early challenges for big data 

First, it turned out to be significantly harder to organize a company’s data into a structure that allowed the various sources to be combined or compared in a meaningful way. Not only were there issues with things like pulling together structured and unstructured data, but there were also difficulties with reformatting, importing, linking and performing other forms of data wrangling.

What proved to be even more challenging, however, was trying to do analysis on the data stores they did have access to. It turned out that only those who had very specialized training in advanced data analytics tools — i.e., SQL jockeys — could put together the very complex commands required to tap into this vast trove of data. Unfortunately, many of those people didn’t know what types of queries could generate the unexpected insights that big data promised. General businesspeople who did have a sense of those questions couldn’t easily generate the queries and so many efforts ended up essentially lost in translation between the two groups. 

Bringing the promise to life with GenAI 

With the growing widespread use of GenAI — which is extremely good at finding patterns and generating ideas off a huge base of data — the situation has begun to turn around. By feeding an organization’s data into an AI model — either by training a custom model or customizing an existing large language model (LLM) — organizations are now finally able to create the giant data store that was always intended to be at the heart of big data queries. Plus, the simple chatbot-style interfaces that tap into these models are now available to people at any level of an organization to easily use. The net result is that the original promise of big data is finally coming to life. From junior salespeople digging into a hunch about a trend they think they’re starting to see in the field, to C-level executives looking for big picture dashboards that combine certain key metrics, people across organizations are now able to leverage GenAI to get an enormous range of insights into businesses. 

Implications for data storage 

The implications of this on data storage within an organization are huge. While in the past some organizations might have discarded or taken certain data sources offline because of their limited perceived value, there’s growing recognition that any data source could end up helping in the discovery of new, unforeseen insights and trends. As a result, companies are not only ensuring that they’re keeping all the data they generate, but they’re making it all available as well.

One of the key enablers of this trend is good old traditional magnetic hard drives. Thanks to technology advancements such as Seagate Mozaic^TM, it’s now possible to fit 3TB of data on a single platter inside a hard drive. Scaling this up into a rack-style storage system in a corporate data center or co-location site converts to as much as 32PB of storage in a single 19-inch wide and 73-inch tall (42U) rack space. By enabling these types of storage capacities, organizations can very efficiently store vast amounts of data, allowing them to consolidate numerous lower capacity drives into smaller, more power-efficient systems and ensuring they have plenty of room for further growth.

Looking at the bigger picture, these types of high-capacity hard drives fit nicely into an overall storage architecture. Organizations will continue to use high-speed SSDs for storing the latest versions of their GenAI models and other applications where the importance of access speed to memory outweighs capacity demands. Similarly, other types of SSDs will likely be leveraged for things like AI chatbots, prompt query storage and other moderately demanding applications. For general purpose data storage of many of the sources that feed into these customized AI models, however, high-capacity hard drives provide an optimal set of characteristics that are very well suited to the application. 

Resurgence in building in-house AI infrastructure

Another critical factor is the location of these data storage devices. For cost and security reasons, most organizations keep much of their data behind their own firewall as opposed to in the cloud. This is particularly true for some of the less accessed data sources that can now be more easily integrated into AI models with new model training and customization tools. As organizations start to build their own AI models, there’s been a large resurgence in building out their own in-house AI infrastructure to train, customize and host some of those models. Companies like Dell, HPE, Lenovo and Cisco are seeing large jumps in demand for GPU-equipped servers designed for the enterprise, and Nvidia has been talking up the rise of enterprise AI factories for a while now. The result is a renewed interest in building up corporate data centers with all the compute, networking and storage resources that this entails. 

With all these hardware elements falling into place combined with the rapidly expanding capabilities and growing usage of GenAI models and tools, the potential for the kind of big data vision of meaningful insights we were originally promised is finally upon us. While not all efforts will necessarily lead to magical “a-ha” insights, it’s already clear that one of the most surprising and beneficial results of GenAI usage — the true democratization of data analysis — is here and starting to make its impact known. 

Looking to bring your big data vision to life? Talk to an expert to find out how.