Here at Seagate, our engineering teams and I regularly sit down with the world’s largest cloud and AI infrastructure builders.
As well as supplying them with exabytes of high-capacity hard drives; we work side by side to help shape their storage architectures.
Through these partnerships, I’ve had a front-row seat into how hyperscale storage decisions are made. The common thread is clear: economics, software orchestration, and hardware capabilities must align to maximize performance, efficiency, and the value of data.
That alignment has become even more important as AI workloads continue to increase dataset sizes, access frequency, context windows, parallelism, retention time, and the demands placed on shared storage systems.
These shifts in scale have fundamentally changed what “primary storage” means.
Historically, primary storage referred to tightly coupled block or file systems positioned close to compute. In cloud and AI environments, however, primary storage is increasingly defined by software-defined, globally distributed architectures that treat object storage as a persistent system of record that retains and serves massive volumes of data across workloads.
To better understand how this redefinition unfolded, let’s unpack the design principles that originally shaped enterprise storage.
How Scale Changed the Storage Paradigm
For decades, the ecosystem operated under a common standard: the Portable Operating System Interface (POSIX). Conceived in an era of more localized infrastructure, POSIX gave developers a predictable model for interacting with data.
It emphasized strong read-after-write consistency, synchronous file locking, and hierarchical directory structures. For a single machine or a localized cluster, it was highly effective, and remains critical for many enterprise and application environments today.
When the cloud model came along, however, the governing tradeoffs changed. Cloud-scale systems were built for a fundamentally different scale, distribution model, and cost structure than POSIX-first systems were originally designed to serve.
In a distributed environment, POSIX-style implementations can require significant orchestration across nodes to preserve directory semantics, file locking, and in-place updates.
Cloud platforms needed massive scale—eventually expanding to support tens to hundreds of exabytes—and in this environment, the coordination overhead of tightly coupled designs began to introduce latency and place practical limits on growth.
In modern AI workloads requiring even larger datasets, checkpointing, token processing, inference, and highly parallel data pipelines, those pressures only intensified.
Across the industry—from Google Cloud Storage (GCS) and Colossus to Microsoft Azure Blob, Amazon S3, and Meta’s Tectonic—cloud platforms adopted software-defined architectures purpose-built for globally distributed data and hyperscale workloads, and refined them over time as scale and requirements evolved.
In this new paradigm, software assumes more responsibility for orchestration, resiliency, and data flow so the underlying storage media can be used as efficiently as possible.
Hard Drives Anchor Storage at Scale
In cloud architectures like those I reference above, hard drives are the foundation for storing data at scale.
This reflects the enduring economics of capacity and the physics of high-density recording. Modern high-capacity hard drives incorporate technologies such as Shingled Magnetic Recording (SMR) and Heat-Assisted Magnetic Recording (HAMR) to keep increasing areal density and make exabyte-scale storage possible.
At this scale, fleets of hard drives serve as the system of record, delivering durability, cost efficiency, and volumetric density that alternative storage technologies simply cannot match.
There’s a reason that 87% of large data center exabytes are stored on hard drives1!
As cloud estates continue to expand and AI workloads consume, generate, retain, and reuse larger volumes of data, these advantages become even more consequential.
But they can only be fully realized if the software architecture is designed to align with the strengths of high-capacity disks.
Traditional POSIX access patterns—especially within tightly coupled distributed filesystem models that emphasize fragmented, random, in-place updates—are not always well aligned with those strengths at extreme scale.
Modern software-defined cloud platforms addressed this by designing their storage stacks around hard drives, allowing them to prioritize sequential, high-throughput data flows while supporting operational economics that scale.
In the case of Amazon S3, a service storing 500 trillion objects and serving 200 million requests per second—a recent AWS re:Invent keynote2 emphasized that the secret to cloud storage performance is writing software that optimizes for the capabilities of the hard drive—described in the presentation as an “engineering marvel.”
Rather than forcing the drive to conform to software abstractions designed for a different era, modern cloud architectures are engineered to complement the strengths of modern high-density hard drives.
How Cloud Architectures Unlock Hard Drive Efficiency
This engineering design took several forms, but across leading cloud platforms it generally reflects four architectural principles. Together, they show how cloud storage has become increasingly software-defined in the way it manages data flow, metadata, resiliency, and ingest behavior.
1. Object Immutability Favors Sequential Data Flows
Services like GCS and Amazon S3 are designed to support object immutability and versioned updates. Once data is written to an object store, updates are typically handled by writing a new version of the object rather than modifying the existing one in place.
By reducing the need for random, in-place binary writes, cloud-scale architectures shift more of the disk’s workload toward large, sequential data flows. That aligns better with how high-capacity drives deliver throughput and efficiency at scale. The benefit becomes even more important under AI workloads, where checkpointing, dataset movement, and parallel pipelines can generate sustained pressure on shared storage systems.
2. Metadata Increasingly Handled By Separate Or Distributed Control Services
In a traditional POSIX environment, storage systems often manage both file metadata and file payloads in closely coupled ways. Hyperscale cloud platforms changed this by separating metadata services from capacity storage, moving much of the tracking and coordination into faster, more scalable control layers rather than leaving that burden on the disks themselves.
Google Cloud’s Colossus shifts much of this work into memory-resident services, while Meta’s Tectonic separates metadata—within a distributed filesystem model—into stateless microservices running on a horizontally scalable key-value store. The result is less structural overhead on the underlying hard drives and more opportunity for them to deliver dense, efficient capacity at scale.
3. Erasure Coding Strengthens Distributed Resiliency
A third principle is the use of erasure coding and broad data distribution to make large-scale storage systems more resilient and efficient.
Cloud architectures reduce that sensitivity through erasure coding and broad data distribution. By spreading objects across many disks, these systems can isolate localized hotspots, continue serving data through brief latency spikes, and reconstruct as needed. This makes the storage layer more resilient and helps sustain throughput under mixed cloud and AI workloads.
4. The Multi-Tier Data Path
A fourth principle is how data is staged before it reaches capacity media.
To bridge the gap between unpredictable application traffic and the structured environment that high-density hard drives handle best, modern architectures use a multi-tiered data path, incorporating flash or memory to buffer ingest and optimize data placement.
A flash tier absorbs variable arrival rates from API traffic and application writes. It stages and organizes incoming data before it moves to capacity media, allowing background processes to flush it to hard drive arrays in long, sequential passes.
Under AI workloads, where ingest, checkpointing, and dataset movement can be especially bursty, this buffering role becomes even more important because it helps preserve both low-latency ingest and efficient hard drive utilization.
Fig. 1. A comparison of traditional POSIX vs cloud-native storage software in maximizing the benefits of hard drive centric storage architecture.
A New Model for Primary Storage
Together, these architectural shifts have reshaped how primary storage is defined. Historically, “primary storage” often referred to expensive, highly available block or file systems attached closely to compute. Object storage was more commonly treated as a lower-tier destination for archive, backup, or secondary data.
Today, many cloud-native architectures define primary storage more broadly: stateless compute paired with a global object store. Data lakes built on platforms such as S3, Azure, and GCS increasingly serve as a system of record for large-scale analytics, cloud applications, and AI workflows.
In this model, primary storage is increasingly software-defined, with object services, metadata layers, flash buffering, and high-capacity hard drives working together as a coordinated system.
Compute instances are often treated as more elastic and stateless, pulling data from the object layer, processing it, and writing results back into the same shared environment.
The Convergence of Object Storage and File Semantics
As object storage became more central to cloud architectures over the past decade and more recently to AI workflows another important trend emerged: high-performance, parallel file systems.
Systems such as Lustre, Weka, and VAST are designed to maximize performance for tightly coupled workloads, often exposing POSIX-compliant interfaces to support checkpointing, coordination, and high-throughput data access.
At the same time, object storage platforms have continued to evolve—optimizing for global scalability while advancing performance to support an expanding set of AI and data-intensive workloads.
In large-scale cloud and AI environments, these approaches are converging. High-performance file systems frequently layer on top of, or tier into, object storage backends, combining performance for the active working set with the scalability and economics of object storage as the system of record.
This convergence reflects a broader architectural shift: rather than choosing between file and object, modern systems combine them. It preserves the convenience of folders, namespaces, and familiar file behaviors without sacrificing the scale advantages of object storage.
Implications for Cloud and AI Infrastructure Builders
Viewed together, these shifts point to a broader conclusion: cloud and AI architectures required different software and system tradeoffs than POSIX-first models were originally designed to optimize for.
Those tradeoffs increased the importance of designing software to optimize use of the underlying hard drive fleets systems are built on. In that sense, cloud and AI workloads have not just altered storage architecture; they have redefined primary storage itself.
For infrastructure builders, the takeaway is clear: designing for modern systems means moving beyond the assumption that primary storage must map cleanly to a local operating-system file tree. It means choosing software and access models that align with the economics, physics, and workload realities of AI at scale.
Organizations that get this right will be better positioned to execute AI strategies efficiently, with higher GPU utilization, better inference economics, and fewer performance bottlenecks.
Learn more about hard drive innovation that powers primary storage for the world’s largest AI and cloud infrastructure builders.
Sources
1. IDC Datasphere and IDC Storagesphere
2. AWS re:Invent 2025, Andy Warfield keynote: S3 stores 500+ trillion objects, serves 200M requests/second, processes 1+ quadrillion requests/year
