# MAX on CPU doubt and request

**URL:** <https://forum.modular.com/t/max-on-cpu-doubt-and-request/434>\
**Category:** MAX\
**Tags:** discussion, feature-request, 24\_6\
**Created:** [January 8, 2025, 8:04pm UTC](https://forum.modular.com/t/max-on-cpu-doubt-and-request/434 "2025-01-08T20:04:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dasor](https://avatars.discourse-cdn.com/v4/letter/d/22d042/32.png) [@Dasor](https://forum.modular.com/u/Dasor)\
**Post date:** [January 8, 2025, 8:04pm UTC](https://forum.modular.com/t/max-on-cpu-doubt-and-request/434/1 "2025-01-08T20:04:22Z")

</div>

As far as I understand (feel free to correct me) most AI inference workloads specially LLMS are memory bound so running inference on CPU can also be a great idea. So, I thought it would be great to also have a MAX server container for CPU.

I’ve also been trying to run some models with MAX using CPU using the modular/max-openai-api container and just removing the gpu option in the entrypoint.sh and although I’ve managed to run mixtral with it I can’t run the modularai/llama-3.1 model as in the GPU tutorial since once I make a request it crashes.

I don’t know if that is because the model only supports GPU (I’ve never heard of something like that) or if I may be doing something wrong.

---

<div class="post-metadata">

**Author:** ![BradLarson](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/bradlarson/32/62_2.png) [@BradLarson](https://forum.modular.com/u/BradLarson)\
**Post date:** [January 8, 2025, 8:45pm UTC](https://forum.modular.com/t/max-on-cpu-doubt-and-request/434/2 "2025-01-08T20:45:35Z")

</div>

If you have reproducible steps for the Llama 3.1 model crashing on CPU, would you mind sharing them here or in [a GitHub issue](https://github.com/modularml/max/issues)? MAX fully supports running efficiently on CPUs as well as GPUs, and there’s nothing GPU-specific about any of our models. I will say that bfloat16 weights will only work on GPU and on Intel CPUs, because ARM instructions are currently missing for bfloat16 operations.

When running on CPU, I highly recommend looking into using quantized weights. We support `q4_0`, `q4_k`, and `q6_k` quantization schemes, and MAX delivers state-of-the-art performance for many quantized models on CPU. Our server container should support this by removing the `--use-gpu` flag from the model and adding `--quantization-encoding q4_k` when pointing at `modularai/llama-3.1` or another Hugging Face repository that has quantized weights in GGUF format.

In addition to using our container, you can set up Magic on your local machine, clone our MAX repository, and [follow these steps](https://github.com/modularml/max/tree/main/pipelines/python) serve a model on CPU or GPU using Magic and MAX.

---

<div class="post-metadata">

**Author:** ![Dasor](https://avatars.discourse-cdn.com/v4/letter/d/22d042/32.png) [@Dasor](https://forum.modular.com/u/Dasor)\
**Post date:** [January 10, 2025, 7:59pm UTC](https://forum.modular.com/t/max-on-cpu-doubt-and-request/434/3 "2025-01-10T19:59:41Z")

</div>

Thanks for the recommendation, I will keep playing around with it!. I’ve also opened a [Github issue](https://github.com/modularml/max/issues/279) as you requested. I tried with both AMD and Intel chips to check if it was the problem but both gave me the same error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/2751e0fbdc595a99718b216730957e9db4448cfd.jpeg) [@system](https://forum.modular.com/u/system)\
**Post date:** [July 9, 2025, 7:59pm UTC](https://forum.modular.com/t/max-on-cpu-doubt-and-request/434/4 "2025-07-09T19:59:50Z")

</div>

Closed on July 9, 2025, 7:59pm UTC

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.