Mammoth KVCache sharing

[toiletsandpaper (Ilya I. Lubenets)]

Does mammoth support or plans to support sharing KV cache between LLM replicas?

[deep (Deep Dhillon)]

Thanks for reaching out! We don’t currently support KV cache sharing between replicas, but it’s a feature we have on our roadmap and plan to add in the future.