A Comparative Study of KV-Cache Quantisation Methods for Large Language Model Inference

Abstract

A four-billion-parameter language model accumulates several gigabytes of key–value (KV) cache at a context of 32K tokens, often exceeding the size of the model weights. Quantising this cache reduces the memory footprint, but five published methods take different routes to that goal, with non-obvious trade-offs between them. This paper compares six KV-cache representations an uncompressed bf16 reference, KIVI, KVQuant, QuaRot, QJL, and Turbo Quant along five criteria including bit budget, calibration requirement, and decode-time key reconstruction. All numerical figures are taken from the corresponding source publications. Two findings follow. Reported compression ratios range from 2.6× (KIVI at 2-bit) to 6.4× (KVQuant at 2-bit non-uniform), and the gap between nominal bit budget and reported ratio reflects the cost of per-block metadata. QJL and TurboQuant share two properties absent from KIVI, KVQuant, and QuaRot: they require no calibration data, and they compute attention without reconstructing keys. This combination is most valuable in deployments where calibration prompts are unstable or unavailable, or where the same compressed cache spans multiple model variants.