Hi!
I built kushkash, a performant compressed KV Cache compatible with latest vLLM and SGLang as plugins, with HF Transformens and plain tensors. It is highly performant (~3x tokens, %0.1 ppl) thanks to fused Mojo kernels. Even though it is mostly written by several AI agents, I don’t think it would be a fun journey dealing with plain CUDA if it wasn’t for mojo.
All the benchmarks, tests and how to replicate is in the readme.
It showed pretty much stable in my tests and the ppl do not build with the load. As the load increases, output tok/s per request degrades pretty gracefully, still beating bf16.
Smoke test script can fetch and verify if a model / list of models could be run with kushkash.
Feedbacks are welcome.