I wanted to run gemma 4 models using modular on a machine that has a h200 gpu and was not successful i tried various steps with copilot and a complete summary generated by copilot is as under please help in figuring out the issue i am very bullish on using modular for inference of gemma 4 models and this is my first attempt on cuda boxes would follow it up on apple silicon and amd also… my primary test now is text to text so i tried the assistants from modular sites and others all detailed below each time i run out of ram…
# MAX Serve Gemma 4 12B/26B/31B graph compilation consumes approximately 470 GiB host RAM on H200
Suggested category: **MAX**
Suggested tags: **debugging**, **gpu**
Disclosure: I used GitHub Copilot to organize and edit this post from command
logs and system measurements that I collected locally. I verified the commands,
versions, and measurements against the retained run artifacts.
## Summary
I cannot get `max serve` to reach readiness with any of these BF16 Gemma 4
models on an NVIDIA H200 NVL system:
- `google/gemma-4-12B-it`
- `google/gemma-4-26B-A4B-it`
- `google/gemma-4-31B-it`
MAX builds the vision + language graph even when I select
`–task text_generation`. During compilation, one Python compiler worker grows
to approximately 468-474 GiB RSS, consumes the remaining 503 GiB of host RAM and
all 8 GiB swap, and is terminated before the server becomes ready. GPU memory
remains around 1-1.6 GiB during this phase.
One 26B run has direct privileged `earlyoom` evidence showing that `earlyoom`
sent `SIGTERM` to the compiler worker. Other runs have the same memory trajectory
and MAX error but do not have enough privileged evidence to prove the signal
sender.
I would like to know whether this compiler working set is expected, whether a
text-only Gemma 4 path can avoid compiling the vision graph, and whether there
is a supported way to compile/export the graph on a larger builder and load it
on this H200.
## Minimal reproduction
Prerequisites: accept the corresponding Gemma license and export an authorized
`HF_TOKEN`. The following is the smallest configuration that reproduces the
problem for me:
```bash
export HF_TOKEN=‘set-outside-shell-history’
pixi run max serve \
–model-path google/gemma-4-12B-it \
--task text_generation \
–reasoning-parser none \
--no-device-graph-capture \
–port 8080 \
--max-length 8192 \
–max-batch-size 4 \
--device-memory-utilization 0.85
```
The log continues with one-minute heartbeats:
```text
INFO: Building, compiling, and initializing vision + language model…
INFO: Still compiling vision + language model (…s elapsed)
```
It eventually ends before readiness:
```text
INFO: Still compiling vision + language model (5039.1s elapsed)
ERROR: Worker crashed (Terminated), shutting down…
…/python3.14/multiprocessing/resource_tracker.py:396: UserWarning:
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
```
The semaphore warning appears after the worker dies and seems to be shutdown
fallout, not the initiating failure.
## Environment
No hostname or access credentials are included here.
| Component | Value |
| — | — |
| OS | Oracle Linux Server 9.7 |
| Kernel | `6.12.0-201.74.2.2.el9uek.x86_64` |
| CPU | 2 x Intel Xeon 6530P, 32 cores/socket, 2 threads/core, 128 logical CPUs |
| NUMA | 2 nodes, approximately 251 GiB each |
| Host RAM | `527439964 KiB` (approximately 503 GiB) |
| Swap | `8388604 KiB` (8 GiB) |
| GPU | NVIDIA H200 NVL, Hopper/SM90 |
| GPU memory | `143771 MiB` (approximately 140.4 GiB) |
| NVIDIA driver | `595.71.05` |
| CUDA version reported by `nvidia-smi` | `13.2` |
| Pixi | `0.67.1` |
| MAX | `26.5.0.dev2026072505` from the MAX nightly channel |
| Modular package | `26.5.0.dev2026072505` |
| Python | `3.14.4` |
| `vm.overcommit_memory` | `0` |
| Process virtual-memory limit | unlimited |
| THP during baseline runs | `always` |
| THP comparison | `madvise` |
| earlyoom | enabled and active |
The H200 exceeds the current Gemma 4 31B quickstart requirement of more than
96 GiB GPU RAM. The installed driver also exceeds MAX’s documented NVIDIA
minimum of 580. The observed limit is host memory during compilation, not H200
VRAM capacity.
### earlyoom configuration
```text
EARLYOOM_ARGS=“-r 0 -m 4 -M 2097152 --prefer ‘^(Web Content|Isolated Web Co)$’ --avoid ‘…’”
```
The relevant setting is `-m 4`: earlyoom may act when available memory falls
below 4%, approximately 20 GiB on this host. The swap was already fully consumed
in the retained preflight snapshots for these experiments.
## Results
RSS values below are converted from the retained `/proc//status` `VmRSS`
values, which are reported in KiB. This avoids treating a value such as
`496479896 KiB` as 496 GiB; it is approximately 473.5 GiB.
| Model and profile | Compile progress | Final retained compiler state | Result |
| — | — | — | — |
| 12B, 8K/4, explicit `text_generation` | heartbeat through 5,039 seconds; run duration about 86 minutes | `VmRSS 490815176 KiB` (468.1 GiB), 552 threads, `MemAvailable 6535504 KiB` (6.2 GiB), swap free 0, GPU 1,565 MiB | `Worker crashed (Terminated)`, exit 1, no readiness |
| 26B A4B, 8K/4, explicit `text_generation` | 5,277 seconds | approximately 470.5 GiB RSS, approximately 1.19 TiB virtual memory, 552 threads, approximately 3.5 GiB available, swap free 0, GPU approximately 1.5 GiB | same error, exit 1, no readiness |
| 26B A4B, 8K/4, automatic task selection | more than 6,100 seconds | approximately 493.7 GB RSS as recorded by the process sampler; `earlyoom` reported `VmRSS 483349 MiB`, approximately 2.46 GiB available, swap free 0 | confirmed earlyoom `SIGTERM`, then same MAX error |
| 26B A4B MTP target + assistant, 8K/4, THP `madvise` | heartbeat through 6,177 seconds; run duration about 104 minutes | `VmRSS 496479896 KiB` (473.5 GiB), approximately 1.11 TiB virtual memory, 517 threads, approximately 2.7 GiB available, swap free 0, GPU 1,565 MiB | same error, exit 1, no readiness |
| 31B, 8K/4, explicit `text_generation` | heartbeat through 4,918 seconds | rising host RSS; terminal sample retained | same error, exit 1, no readiness |
| 31B, 8K/4, automatic task selection | heartbeat through 6,118 seconds | rising host RSS; terminal sample retained | same error, exit 1, no readiness |
| 31B target + MTP assistant, 8K/4 | heartbeat through 4,678 seconds | approximately 496 GB RSS (about 473 GiB if the source unit is KiB), approximately 1.156 TiB virtual memory, 552 threads, approximately 2.5 GiB available, swap free 0, GPU approximately 1.5 GiB | same error, no readiness |
Memory returns immediately after the compiler worker exits, which explains why
`free -h` looks healthy after a failed run.
## Direct earlyoom evidence
For the 26B automatic-task run, a privileged earlyoom log recorded:
```text
sending SIGTERM to process … VmRSS 483349 MiB
```
At that point, the retained process sample showed approximately 2.46 GiB
`MemAvailable` and no free swap. Seven seconds later MAX logged:
```text
ERROR: Worker crashed (Terminated), shutting down…
```
This run confirms earlyoom as its immediate termination source. The generic MAX
message alone does not prove that every other run had the same signal source.
A separate CPython 3.14 kernel segfault was observed 67 seconds after an earlier
26B supervisor exit:
```text
Thread-1 (run_s[…]: segfault at 1cb8 … error 6
```
The faulting PID was not present in the retained process-group samples, so I
cannot prove it was the large compiler worker or that it caused the failure.
## Commands and variations tested
### 12B, 26B, and 31B text-generation profiles
I substituted each model below into the same command:
```text
google/gemma-4-12B-it
google/gemma-4-26B-A4B-it
google/gemma-4-31B-it
```
```bash
export MODULAR_DEVICE_CONTEXT_SYNC_MODE=true
export MODULAR_MAX_DEBUG=true
export MAX_DISABLE_CUDA_GRAPHS=1
pixi run max serve \
–model-path MODEL_ID \
--reasoning-parser none \
–no-device-graph-capture \
--force \
–port 8080 \
--max-length 8192 \
–max-batch-size 4 \
--enable-in-flight-batching \
–enable-prefix-caching \
--device-memory-utilization 0.85 \
–task text_generation
```
I also omitted `–task text_generation` to test MAX task auto-detection. MAX
still selected text generation and still compiled the vision + language graph.
### Direct 26B warm-cache
```bash
pixi run max warm-cache \
–model google/gemma-4-26B-A4B-it \
--devices gpu:0 \
–max-length 8192 \
--max-batch-size 4 \
–device-memory-utilization 0.85 \
--task text_generation \
–no-device-graph-capture
```
This entered the same graph-compilation path. I also tried the default command:
```bash
pixi run max warm-cache --model google/gemma-4-26B-A4B-it
```
MAX initially selected the model’s 262,144-token maximum and clamped it to
231,296 based on KV-cache capacity, but still entered the same growing host-RSS
compile path.
### CPU-affinity experiment
I wrapped the complete 26B warm-cache process tree with `taskset`:
```bash
taskset --cpu-list 0-15 \
pixi run max warm-cache \
--model google/gemma-4-26B-A4B-it \
--devices gpu:0 \\
--max-length 8192 \
--max-batch-size 4 \\
--device-memory-utilization 0.85 \
--task text_generation \\
--no-device-graph-capture
```
Affinity worked: compiler threads fell from approximately 517 to 69. It did not
reduce the final working set. After approximately 88 minutes, RSS was about
496.6 GB (about 473.6 GiB if sourced from KiB), `MemAvailable` was about 2.2
GiB, swap was exhausted, and GPU allocation was about 1 GiB. The supervised run
was then interrupted with `SIGTERM` and did not complete cache warming.
### MTP target + assistant recipes
For the installed recipes, I tested both the 26B A4B and 31B target/draft pairs:
```bash
RECIPE=‘.pixi/envs/default/lib/python3.14/site-packages/max/pipelines/architectures/gemma4/recipes/gemma4_26b_a4b_tuned.yaml’
pixi run max serve \
–config-file “$RECIPE” \
--port 8081 \
–max-length 8192 \
--max-batch-size 4 \
–reasoning-parser none \
--no-device-graph-capture \
–device-memory-utilization 0.85
```
For 31B, I used `gemma4_31b_tuned.yaml` with the same CLI settings. The recipes
use the matching MTP assistant and three speculative tokens. Adding the draft
model did not avoid target graph compilation or lower host RSS.
### Transparent Huge Pages
Baseline runs used:
```text
[always] madvise never
```
I repeated the 26B MTP run after setting:
```bash
sudo sh -c “printf ‘%s\n’ madvise > /sys/kernel/mm/transparent_hugepage/enabled”
cat /sys/kernel/mm/transparent_hugepage/enabled
# always [madvise] never
```
The `madvise` run still reached `496479896 KiB` RSS and failed before readiness.
THP allocation and compaction counters changed negligibly. A `never` mode was
prepared but has not been completed, so I am not presenting it as a tested
result.
### Other CLI settings checked
- `–max-length 8192` rather than the model maximum
- `–max-batch-size 4`
- `–device-memory-utilization 0.85`
- `–no-device-graph-capture` and `MAX_DISABLE_CUDA_GRAPHS=1`
- explicit `–task text_generation` versus task auto-detection
- `–reasoning-parser none`
- prefix caching and in-flight batching enabled for direct serve runs
- `–use-subgraphs` left at its enabled default
- no vision cache override in the core reproducer
These context, batch, and GPU-cache controls did not materially reduce the host
compiler working set. Device graph capture was already disabled. Subgraphs were
already enabled by default.
## What I have not claimed or tested
- I have not disabled earlyoom globally because that could leave the host with
no safety margin. The confirmed worker already reached approximately 472 GiB
RSS with only a few GiB available.
- I have not counted the prepared THP `never` profile as a completed result.
- I have not treated the delayed CPython segfault as the root cause because its
PID was not correlated with the compiler worker.
- Stable MAX 26.4 lists 31B but did not list the 26B A4B model in the local model
registry, so it was not a useful equivalent A4B comparison.
- The installed nightly exposes `–use-subgraphs` but does not expose the newer
`–export-mefs` or `–precompiled-mefs` flags shown in newer documentation.
## Questions for the Modular team
1. Is an approximately 468-474 GiB host-RSS working set expected when compiling
Gemma 4 12B, 26B A4B, or 31B with MAX on Hopper/SM90?
2. Why does `–task text_generation` still build the vision + language model
graph, and is there a supported text-only compile path?
3. Are there compiler flags or recipe settings that reduce host compilation RAM
without changing model correctness?
4. Is Python 3.14.4 supported/recommended for these Gemma 4 compilation paths,
or should I reproduce with another supported Python version?
5. Is H200 expected to work for the 31B quickstart when it has 143,771 MiB VRAM,
even though B200 is the NVIDIA GPU listed as tested for serving?
6. In which MAX build should `–export-mefs` and `–precompiled-mefs` be used,
and can an SM90 graph be compiled on a higher-RAM H200 builder and loaded on
this H200 with the same MAX/model/configuration?
7. What additional compiler diagnostics would be most useful: a bounded stack
trace, compiler debug log, memory allocation trace, or another artifact?
I can provide sanitized `server.log`, ten-second process/RSS samples,
preflight/postflight `/proc` snapshots, and a bounded privileged earlyoom/kernel
log for a requested reproduction.