# Kushkash: 4-bit KV cache plugin for vLLM

**URL:** <https://forum.modular.com/t/kushkash-4-bit-kv-cache-plugin-for-vllm/3615>\
**Category:** Community Showcase\
**Created:** [October 31, 2026, 8:30pm UTC](https://forum.modular.com/t/kushkash-4-bit-kv-cache-plugin-for-vllm/3615 "2026-10-31T20:30:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![altunyurt](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@altunyurt](https://forum.modular.com/u/altunyurt)\
**Post date:** [October 31, 2026, 8:30pm UTC](https://forum.modular.com/t/kushkash-4-bit-kv-cache-plugin-for-vllm/3615/1 "2026-10-31T20:30:24Z")

</div>

Hi!

I built kushkash, a performant compressed KV Cache compatible with latest vLLM and SGLang as plugins, with HF Transformens and plain tensors. It is highly performant (~3x tokens, %0.1 ppl) thanks to fused Mojo kernels. Even though it is mostly written by several AI agents, I don’t think it would be a fun journey dealing with plain CUDA if it wasn’t for mojo.

All the benchmarks, tests and how to replicate is in the readme.

> **[GitHub - altunyurt/kushkash: kushkash is a compressed KV cache for LLM...](https://github.com/altunyurt/kushkash)**
>
> kushkash is a compressed KV cache for LLM inference.

It showed pretty much stable in my tests and the ppl do not build with the load. As the load increases, output tok/s per request degrades pretty gracefully, still beating bf16.

Smoke test script can fetch and verify if a model / list of models could be run with kushkash.

Feedbacks are welcome.