AI inference has a memory problem, and the pressure keeps growing. As agentic AI workloads use longer context windows, GPUs spend more time repeating work instead of generating tokens.

At AI Infrastructure Field Day 5, MinIO and Solidigm presented memKV, a distributed context memory store for AI inference. Dil Radhakrishnan explained how memKV reduces wasted GPU cycles by keeping the KV cache available across a GPU pod.

In short, memKV moves the KV cache to fast NVMe-backed shared flash. As a result, GPUs can spend more time decoding tokens and less time recomputing context.

KV cache acts as the working memory of AI inference. During token generation, a large language model (LLM) stores the keys and values from each attention layer. Then, it reuses that context for each new token.

However, a KV cache can grow quickly as context length and concurrency increase. Without a KV cache, the model would repeatedly recalculate previous token relationships.

For example, an NVIDIA H100 GPU includes 80 GB of high-bandwidth memory. The model weights occupy much of that memory even before inference begins, reducing the available memory.

MinIO stated that one 32k-token inference session can consume about 11 GB of KV cache. At 32 concurrent requests, this would approach 342 GB, which exceeds the capacity of a single GPU.

Consequently, the GPU evicts cache entries and later recomputes them. This eviction cycle increases time-to-first-token and reduces effective GPU utilization.

MinIO memKV targets this problem directly. It creates a purpose-built, distributed context memory store for inference workloads that need fast, shared access to KV cache.

Importantly, memKV is not object storage or a file system. It also does not repurpose MinIO AIStor for a workload it was not designed to serve.

Instead, MinIO built memKV as a memory-oriented storage engine. It uses direct NVMe I/O and avoids file system metadata, POSIX semantics, and durability overhead that a transient KV cache does not need.

MinIO places memKV in what it calls the G3.5 tier. This layer sits between local NVMe storage and shared network storage in the inference memory hierarchy.

The hierarchy starts with GPU HBM, then CPU DRAM, then local NVMe. Finally, shared network storage offers greater capacity but higher latency.

memKV fills the gap between those tiers. It provides pod-level shared flash with approximately 100-microsecond access and petabyte-scale capacity.

The AI inference memory bottleneck will not disappear as models and agents become more capable. In fact, longer context windows will make the issue more visible.

Therefore, infrastructure teams need to measure effective GPU utilization, not just raw GPU activity. A busy GPU does not always generate useful output.

memKV addresses the problem at the architecture level. Keeping shared context available reduces recomputation and helps GPUs focus on token generation.

The AI inference memory bottleneck will not disappear as models and agents become more capable. In fact, longer context windows will make the issue more visible.

Therefore, infrastructure teams need to measure effective GPU utilization, not just raw GPU activity. A busy GPU does not always generate useful output.

memKV addresses the problem at the architecture level. Keeping shared context available reduces recomputation and helps GPUs focus on token generation.

For more details, review the MinIO memKV product page at min.io/product/memkv, and the MinIO blog post at min.io/blog/introducing-minio-memkv,

For a broader context on KV cache bottlenecks, NVIDIA has published several blog posts on how KV cache offloading can reduce recomputation, improve concurrency, and lower inference costs in long-context workloads.

If your inference environment looks busy but still delivers slow responses, start by measuring KV cache behavior, TTFT, and effective GPU utilization. That data will show whether you have a compute problem, a memory problem, or both.

HighFens helps enterprises identify AI infrastructure bottlenecks and design scalable inference platforms. Contact us if you want a practical assessment of your GPU utilization and inference architecture.