Generative AI infrastructure discussions usually focus on model size, GPU clusters and the enormous amount of computing power required for training and inference.

Yet another challenge sits behind every interaction with ChatGPT: the platform must reliably retrieve the application data required to operate.

Account settings, preferences, application state and other operational information need to remain accessible even when many users interact with OpenAI products simultaneously.

Habitat is the company's internal platform for handling much of that online storage.

On October 23, OpenAI engineers Jon Lee, Hrachya Hakobyan and Chaomin Yu published a detailed account of the system's evolution.

The numbers illustrate its scale. Habitat manages over 500 petabytes of data and supports services used by more than one billion people across 40 geographic regions.

The reported storage footprint does not exclusively represent ChatGPT conversations. Habitat serves multiple products and workloads.

The challenge is not merely storing extraordinary amounts of information. It is making the infrastructure reliable, efficient and straightforward for application teams to use.

Habitat provides a common storage layer for OpenAI products

Habitat is not a consumer application.

It is an internal platform that provides storage access for services including ChatGPT, the OpenAI API, Codex and other company systems.

Instead of requiring every product team to operate its own complete database architecture, Habitat offers a shared interface.

Its responsibilities include routing, access controls, encryption, workload isolation, request prioritization and data placement.

The underlying infrastructure uses several technologies, including Azure Cosmos DB, Valkey caches and additional storage services.

The platform aims to hide much of this complexity from application developers.

Doing so becomes increasingly difficult as traffic, data volume and the number of supported features grow.

The original problem: noisy neighbors

OpenAI initially relied on a shared Azure Cosmos DB container for multiple workloads.

Sharing infrastructure offers a clear advantage.

Unused capacity from one application can help accommodate demand from another, reducing the amount of excess capacity that individual teams must reserve.

However, shared systems can suffer from noisy neighbors.

When one workload consumes an unusually large amount of resources, unrelated applications may experience delays.

OpenAI reports that some secondary workloads could interfere with critical login and signup operations.

A problem affecting one set of application data could therefore disrupt functions required by a much wider user population.

The challenge involved both database partitioning and the way different services competed for physical resources.

Logical and physical partitions created unexpected limits

Azure Cosmos DB distributes data using logical partition keys, which are mapped onto physical partitions.

This allows storage and throughput to be distributed across infrastructure resources.

However, each layer has practical limits.

OpenAI explains that different workloads frequently reused common partition keys such as user_id and workspace_id.

Those collisions could cause otherwise separate workloads to compete for the same throughput limits.

Physical partitions introduced another source of contention.

Several logical partitions can share a physical partition, which means heavy activity on one dataset can affect others located on the same underlying resource.

OpenAI also encountered weak partitions.

As a container grows, its physical partitions may split to accommodate additional storage.

If provisioned throughput remains unchanged, the capacity available to each partition can decrease.

A workload can therefore experience reduced performance even when its request rate remains stable.

This is a particularly challenging failure mode for systems whose storage volume grows faster than their interactive traffic.

Dedicated containers improved isolation but complicated operations

OpenAI's next approach involved using dedicated containers for important services.

Separating workloads reduced the possibility that one application would disrupt another.

It also limited the impact of certain failures.

However, every dedicated container required capacity planning.

Product teams needed to predict how much traffic a new feature might generate before its public launch.

A workload producing only a small number of requests during testing might suddenly require vastly greater throughput in production.

OpenAI describes situations in which teams launched features without adequately preparing their database capacity.

Infrastructure engineers then had to scale resources under pressure.

The dedicated model improved isolation but created additional operational complexity across the organization.

OpenAI returned to shared containers with better controls

The third phase of Habitat's storage evolution involved returning to shared containers by default.

That decision might appear contradictory given the problems encountered earlier.

The difference was a collection of new mechanisms developed in cooperation with Microsoft Azure.

Hot-key throttling prioritizes limiting the workloads responsible for disproportionately high demand.

Quality-of-service controls distinguish important interactive requests from less urgent background operations.

This allows background workloads to be throttled before they interfere with critical user-facing traffic.

Microsoft also developed an improved Azure Cosmos DB serverless mode that, according to OpenAI, allows physical partitions to scale to 10,000 request units per second.

Partition splitting can also respond to throughput usage instead of depending exclusively on storage growth.

These changes have reduced the amount of manual capacity planning required by product teams.

OpenAI reports experiencing no provisioning or noisy-neighbor incidents in the months since the improvements were introduced.

That is an internal operational report, not a guarantee against future outages.

An Azure outage exposed the limits of managed infrastructure

The technical report also revisits two incidents from late 2024.

A power failure affected the primary Azure region for one of OpenAI's largest Cosmos DB accounts.

During the resulting failover process, engineers discovered that moving the company's largest accounts to another region could require several hours.

Approximately one month later, the return to the original region triggered another outage.

A large number of clients attempted to reconnect simultaneously, overwhelming an Azure Cosmos DB gateway.

The behavior of the gateway-mode client caused connections and requests to return to the original region before all physical partitions had moved back.

These incidents demonstrated that managed cloud services do not eliminate the need for detailed infrastructure knowledge.

At exceptional scale, failure modes may emerge that were never encountered by smaller customers.

OpenAI and Microsoft now coordinate more closely

The Habitat and Azure engineering teams now share more operational processes.

OpenAI describes joint dashboards, production-scale testing environments and direct engineering communication.

The teams can also alert each other without going through conventional support escalation processes.

The goal is to identify and address problems before they affect production traffic.

According to OpenAI, its largest Azure Cosmos DB account stores tens of petabytes of data and handles more than 20 million queries per second.

At that scale, infrastructure modifications require careful testing and coordination.

The cloud provider becomes an active engineering partner rather than simply a vendor offering a standardized service.

Data locality matters for global performance

Improving reliability is only part of the problem.

Habitat also needs to reduce the time required to access data.

An application request processed in one geographic region may need information stored much farther away.

Even efficient software cannot eliminate the network delay introduced by physical distance.

One solution is to locate frequently accessed data closer to the services that use it.

OpenAI operates Azure Cosmos DB accounts in different regions, including deployments serving specific data-residency requirements.

However, placing all storage near every user is not practical.

Some regions lack sufficient data center capacity, power or cooling to accommodate the required storage footprint.

This creates a trade-off between allocating scarce infrastructure capacity to nearby storage or additional computing resources.

Global service performance therefore depends on physical infrastructure decisions as well as software design.

Replicate only the most latency-sensitive datasets broadly

OpenAI also uses selective replication.

Small datasets required by especially latency-sensitive operations can be copied across many regions.

Authentication metadata is one example.

Some Azure Cosmos DB accounts replicate data across all 40 regions where OpenAI serves traffic.

Others use less extensive replication, including arrangements with at least one region per continent.

The approach reduces network round trips for particularly important information.

However, broad replication of every large dataset would consume substantial capacity.

OpenAI therefore reserves the most extensive replication strategies for selected data.

This does not mean that all user data is copied to every region.

Placement depends on workload requirements, infrastructure constraints and applicable data-residency policies.

Valkey caches reduce expensive storage reads

More than 90% of Habitat requests are reads, according to OpenAI.

A relatively small working set accounts for a significant portion of those accesses.

That makes caching particularly valuable.

Habitat uses distributed Valkey caches positioned near application-serving infrastructure.

When recently accessed data is available in a local cache, the system can avoid a request to more distant storage.

The benefit becomes particularly significant when the database is located in another geographic region.

However, caching introduces a trade-off between performance and freshness.

Keeping information in the cache longer increases the likelihood of a cache hit but also raises the risk of serving outdated data.

Short expiration periods improve freshness while increasing database traffic.

OpenAI has therefore adopted a more advanced invalidation mechanism.

Change Data Capture helps keep caches fresh

Habitat uses Azure Cosmos DB's change feed to detect modifications to stored data.

Internal services propagate invalidation events to affected caches.

When an application subsequently requests the modified data, the stale cache entry is replaced by a fresh version.

OpenAI reports that these invalidations propagate globally within seconds.

This helps maintain high cache hit rates without relying exclusively on fixed expiration timers.

Change Data Capture also supports several other infrastructure functions.

These include secondary indexes, analytics and data movement pipelines.

OpenAI describes millions of captured changes per second, distributed through more than a hundred Kafka topics.

The system must recover from worker failures, regional disruptions and processing errors without blocking unrelated streams.

Reliable recovery and workload distribution are therefore essential parts of the architecture.

Rust contributes to a twelvefold reduction in median read latency

OpenAI also rewrote its Habitat service in Rust to reduce processing overhead associated with the earlier Python implementation.

The rewrite works alongside distributed caching and selective data replication.

Together, these changes reportedly reduced global median read latency from approximately 120 milliseconds to about 10 milliseconds.

That represents a twelvefold improvement in the company's internal measurements.

The distinction between storage latency and complete application response time is essential.

The reported figure describes Habitat reads, not the time required for ChatGPT to generate a complete answer.

Model inference can still involve substantial processing time independently of storage access.

Median latency also does not describe the slowest operations.

Cache misses may still require retrieving information from storage located in a distant region.

Nevertheless, reducing the latency of an operation performed at very high frequency can provide meaningful infrastructure benefits.

An exabyte-scale storage system is under development

OpenAI does not consider its current design a permanent endpoint.

The company is developing an internal storage system intended to support exabyte-scale workloads.

The project combines RocksDB Cloud technology from Rockset with replication based on Raft.

It implements a document model compatible with the one used by Azure Cosmos DB while adopting a different physical storage architecture.

Its design uses disaggregated storage and physical replication to reduce the amount of flash storage required.

The company acknowledges a trade-off: the system targets greater storage efficiency for bulky data types in exchange for lower peak query throughput.

This new architecture remains in development.

OpenAI has not announced a complete replacement of Azure Cosmos DB or a public schedule for migrating all Habitat workloads.

The project instead illustrates the challenges expected as storage demand continues to increase.

Scale changes the rules of cloud engineering

OpenAI's October 23 report provides a useful reminder that running global AI products requires far more than powerful models.

Infrastructure engineers must determine where data should live, which workloads deserve priority, when to replicate information and how to recover from regional failures.

Habitat's evolution also shows why there is no universally optimal infrastructure design.

Fully shared storage can expose services to noisy neighbors.

Fully isolated storage can introduce excessive planning and operational complexity.

OpenAI's current approach combines shared resources, workload prioritization, targeted replication and close cooperation with its cloud provider.

The reported latency improvement is notable, although it remains an internal measurement of a specific storage operation.

The broader lesson is that the responsiveness and reliability of AI services depend on a vast infrastructure beyond the models themselves.

Managing over 500 petabytes of data for products serving more than one billion users requires database engineering at an exceptional scale.