# Where are the SOTA quantized models?

**URL:** <https://forum.modular.com/t/where-are-the-sota-quantized-models/2029>\
**Category:** General\
**Created:** [July 28, 2025, 10:02am UTC](https://forum.modular.com/t/where-are-the-sota-quantized-models/2029 "2025-07-28T10:02:39Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![BradLarson](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/bradlarson/32/62_2.png) [@BradLarson](https://forum.modular.com/u/BradLarson)\
**Post date:** [July 28, 2025, 3:00pm UTC](https://forum.modular.com/t/where-are-the-sota-quantized-models/2029/4 "2025-07-28T15:00:59Z")

</div>

There are multiple kinds of pre-quantized models supported in MAX ([documentation here](https://docs.modular.com/max/graph/quantize/)). The framework automatically detects the quantization used in the source weights, which can also vary from weight to weight.

For CPU-based models the following quantization types are supported:

- `q4_0`
- `q4_k`
- `q6_k`

And for GPU-accelerated models, we support:

- `QPTQ`
- `AWQ`

For pure numerical datatypes, float32 and bfloat16 are supported natively across GPUs and we are actively working on fp8 formats.

The limitations on what quantization types are supported on what platform are purely for which kernels have been brought up and optimized so far. Those to date have been driven by demand, with llama.cpp’s quantization formats being popular for small models on CPU and QPTQ being used for larger models on GPUs.

This is an area where we’d welcome community contributions, because all of our Mojo quantization-aware kernels are [in open source today](https://github.com/modular/modular/tree/main/max/kernels) and anyone can add to them and expand support in areas you want them to go (`q4_k` on GPU, lower-bit quantizations, etc.). As I mentioned in [a previous post](https://forum.modular.com/t/calling-all-amd-rdna-users-help-us-bring-full-max-support-to-your-gpus/2015) we even have some great items to give away to contributors that help expand our kernel support!

---

_[View the full topic](https://forum.modular.com/t/where-are-the-sota-quantized-models/2029)._