# MAX vs vLLM for HPC inference farm **URL:** \ **Category:** Serving\ **Tags:** discussion\ **Created:** [September 3, 2026, 11:37pm UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464 "2026-09-03T23:37:41Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Curious](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/curious/32/955_2.png) [@Curious](https://forum.modular.com/u/Curious)\ **Post date:** [September 3, 2026, 11:37pm UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/1 "2026-09-03T23:37:41Z")
I am looking to create an inference farm on a Docker HPC host with 4 nodes with 4 H100s each. I asked Gemini and it said MAX loses to vLLM for more users and is more geared towards local deployments. I took this personnally, but can anyone share any experience with using MAX to serve say 50-200 users accross compute nodes, please? ---
**Author:** ![Ehsan](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/ehsan/32/142_2.png) [@Ehsan](https://forum.modular.com/u/Ehsan)\ **Post date:** [September 4, 2026, 2:15am UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/2 "2026-09-04T02:15:34Z")
There’s a lot of factors involved. Our focus has been on optimizing for more recent GPU architectures from Blackwell and above so H100 isn’t included yet. It also depends on the models and the architecture we support. There’re many cases that MAX is faster than vLLM/SGLang and we published a few case studies: - [FLUX.2 Image Generation, Sub-Second, $0.009/Image | Modular](https://www.modular.com/solutions/flux2-image-generation) - [Modular: Hippocratic Case Study](https://www.modular.com/case-studies/hippocratic) - [Modular: Inworld Case Study](https://www.modular.com/case-studies/inworld) Also we should distinguish between using MAX as is or our Modular Cloud which has cluster optimizations etc. needed to get the best perf per dollar on multi-hardware scenario which translates to TCO. ---
**Author:** ![Curious](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/curious/32/955_2.png) [@Curious](https://forum.modular.com/u/Curious)\ **Post date:** [September 4, 2026, 2:55am UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/3 "2026-09-04T02:55:54Z")
Unfortunately, MAX Cloud is not an option for Governmental (foreign at that) work. I am just trying to figure out how I can use as much Modular tech as possible as the results of “rewriting geospatial science code in Mojo” have been magical, but my area of expertise is not LLM deployment. Any tips, tricks, violent insults, technical leads and reading on the subject is welcomed 🙂 ---
**Author:** ![0xamlab](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/0xamlab/32/1311_2.png) [@0xamlab](https://forum.modular.com/u/0xamlab)\ **Post date:** [September 30, 2026, 7:41pm UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/4 "2026-09-30T19:41:36Z")
Take a look here. I was able to beat vLLM on Gemma 3 27b different NVIDIA hardware. See benchmarks [here](https://docs.algenta.ai/benchmarks/nvidia-llm-throughput). It requires a very high fine tuning on different layers. Angel ---
**Author:** ![aicost\_tools](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/aicost_tools/32/1331_2.png) [@aicost\_tools](https://forum.modular.com/u/aicost_tools)\ **Post date:** [October 12, 2026, 9:22am UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/5 "2026-10-12T09:22:05Z")
For an H100 farm the serving stack choice (MAX vs vLLM) mostly shows up as tokens/sec and occupancy, not as a vendor invoice — the API-shaped cost only appears when you keep a cloud fallback for overflow or eval. As a rough external benchmark, a frontier chat card today is often around $2–$4 per million input and $6–$20 per million output depending on the model, so even a small percentage of traffic spilled to the API can dominate the GPU power bill if prompts are long. Worth deciding up front which requests may leave the cluster (and with which max\_tokens), and logging input vs output tokens on both the local path and the fallback so you can compare $/1M completed tokens instead of arguing from synthetic throughput alone. ---
**Author:** ![rachelwilson](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rachelwilson](https://forum.modular.com/u/rachelwilson)\ **Post date:** [October 13, 2026, 10:21am UTC](https://forum.modular.com/t/max-vs-vllm-for-hpc-inference-farm/3464/6 "2026-10-13T10:21:34Z")
for 50 to 200 users i would benchmark replicas before anything exotic. run one model replica per node with tensor parallel inside the node, put a simple load balancer in front, and measure time to first token and inter token latency at your real concurrency, not just tokens per sec alone. cross node tensor parallel usually hurts latency unless your interconnect is very fast, so replicas tend to scale cleaner for chat style traffic.