Supermicro, IBM and Nvidia announced on November 5, 2026, a joint key-value cache platform designed for large-scale AI models. It keeps models’ working memory in shared flash storage to prevent GPUs from recomputing context they have already processed. On a validated reference architecture, time to first token was reduced by up to 56 times for contexts of 130,000 tokens, while concurrent request throughput rose from 0.19 to 4.26 requests per second. Total processing time for a batch of 200 requests fell by 95%, and throughput reached 3.6 requests per second in a noisy-neighbor load test. The platform combines Supermicro’s Petascale storage servers, IBM Storage Scale software, Nvidia Dynamo inference software and Nvidia Spectrum-X Ethernet switches.
Source: Read the original article
