# Inference performance issue **URL:** \ **Category:** MAX\ **Created:** [October 28, 2025, 10:01pm UTC](https://forum.modular.com/t/inference-performance-issue/2408 "2025-10-28T22:01:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Mkhl](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Mkhl](https://forum.modular.com/u/Mkhl)\ **Post date:** [October 28, 2025, 10:01pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/1 "2025-10-28T22:01:33Z")
Hi there, I’m running inference on **Llama-3.1-8B-Instruct** using **Modular MAX 25.6** on an **H100**. I compared its performance with **vLLM** using **NVIDIA GenAI-Perf** and noticed a **2x difference** in speed, which makes me think I might be missing something. - **Average Time To First Token (TTFT):** 40 ms (vLLM ~20 ms) - **Output Token Throughput:** 800 tokens/sec (vLLM ~1324 tokens/sec) I’m using the **same hardware** and **default settings** , providing only the path to the model. Could you suggest what I might be missing or what could cause this difference? Any ideas would be appreciated. ---
**Author:** ![Ehsan](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/ehsan/32/142_2.png) [@Ehsan](https://forum.modular.com/u/Ehsan)\ **Post date:** [October 29, 2025, 2:57am UTC](https://forum.modular.com/t/inference-performance-issue/2408/2 "2025-10-29T02:57:11Z")
Thanks for trying MAX! [Here](https://docs.modular.com/max/deploy/benchmark) is our benchmark guide which should be compatible with how vLLM does benchmarking. I don’t recall using NVIDIA GenAI-Perf before so could provide its command you’re using to benchmark them? Another important difference is the last time I checked, MAX serve default settings was different from vLLM so could you verify than on your end and what vLLM version are you using? ---
**Author:** ![Mkhl](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Mkhl](https://forum.modular.com/u/Mkhl)\ **Post date:** [October 29, 2025, 12:29pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/3 "2025-10-29T12:29:29Z")
Hi Ehsan, Thank you for answering! My GenAI-Perf (from tritonserver-25.09-py3-sdk image ) config is: ```bash export INPUT_SEQUENCE_LENGTH=200 export INPUT_SEQUENCE_STD=0 export OUTPUT_SEQUENCE_LENGTH=200 export CONCURRENCY=10 export MODEL=/fs1/shared/model/llm/Meta-Llama-3.1-8B-Instruct/ genai-perf profile \ -m $MODEL \ --endpoint-type chat \ --streaming \ -u 127.0.0.1:8000 \ --warmup-request-count 20 \ --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \ --synthetic-input-tokens-stddev $INPUT_SEQUENCE_STD \ --concurrency $CONCURRENCY \ --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \ --extra-inputs max_tokens:$OUTPUT_SEQUENCE_LENGTH \ --extra-inputs min_tokens:$OUTPUT_SEQUENCE_LENGTH \ --extra-inputs ignore_eos:true \ --tokenizer $MODEL \ --num-requests 500 ``` I used vllm version v0.10.1.1 with default options: ```bash python3 -m vllm.entrypoints.openai.api_server --model /fs1/shared/model/llm/Llama-3.1-8B-Instruct/ ``` My results on H100: The first is MAX, the second is vllm. ![max_results](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/0b10a9b725b97933803cb4a0e9cad3e0cdefb3be.jpeg) ![vllm_results](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/ff2916f43742d3d4ea55b253a4a1007282de6e23.jpeg) ---
**Author:** ![Ehsan](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/ehsan/32/142_2.png) [@Ehsan](https://forum.modular.com/u/Ehsan)\ **Post date:** [October 29, 2025, 7:20pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/4 "2025-10-29T19:20:36Z")
Thanks! Let us reproduce it and will update you on this. ---
**Author:** ![rcmpge](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/rcmpge/32/849_2.png) [@rcmpge](https://forum.modular.com/u/rcmpge)\ **Post date:** [October 30, 2025, 4:51am UTC](https://forum.modular.com/t/inference-performance-issue/2408/5 "2025-10-30T04:51:59Z")
This looks interesting. I haven’t tested MAX or vLLM out to give precise comparisons. These metrics require rigorous testing. Initial conjecture - throughput metrics look interesting. Awesome benchmarking ---
**Author:** ![Mkhl](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Mkhl](https://forum.modular.com/u/Mkhl)\ **Post date:** [October 30, 2025, 12:02pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/6 "2025-10-30T12:02:08Z")
Yesterday, I had a great opportunity to discuss this issue with NVIDIA experts at the GTC conference. I captured a profile using Nsight Systems, which they reviewed, but no significant issues were identified at first glance. Next, I’ll capture a profile for VLLM so we can compare the results side by side. By the way, how can I verify if MAX is using FP16 rather than FP32? This could explain the twofold difference in performance. ---
**Author:** ![Ehsan](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/ehsan/32/142_2.png) [@Ehsan](https://forum.modular.com/u/Ehsan)\ **Post date:** [October 31, 2025, 6:36pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/7 "2025-10-31T18:36:38Z")
We could reproduce the results. MAX is respecting the dtype which is [bfloat16](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/config.json#L34). The genai-perf tool is new and we haven’t investigated it before so thank you for bring it up. One thing we noticed is the in MAX `--enable-prefix-caching` helps but still there’s a gap. Our internal benchmarking is based on vLLM benchmarking tool and we test it on more realistic datasets like SharedGPT or Arxiv summarization. I’ve made an internal ticket and we’ll do further investigation on this. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/2751e0fbdc595a99718b216730957e9db4448cfd.jpeg) [@system](https://forum.modular.com/u/system)\ **Post date:** [April 29, 2026, 6:37pm UTC](https://forum.modular.com/t/inference-performance-issue/2408/8 "2026-04-29T18:37:23Z")
Closed on April 29, 2026, 6:37pm UTC This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.