Curious
July 24, 2026, 1:29am
1
Now that we are a Qualcomm community sort of speak, would issues like the one below (sm_90 segfault), be more likely to be prioritized, less so or about the same?
Stated in another way, is Modular’s focus shifting away from HPC/Datacenter GPUs to more edge devices and Qualcomm specific accelerators?
opened 05:28PM - 17 Jun 26 UTC
bug
crash
max
NV:H100/H200
MAX/Framework
### Bug description
## Bug description
On a Slurm/PBS Linux HPC node (RHEL 9 f… amily) with an NVIDIA H100 80GB (`sm_90a`), CUDA 13.3, and driver 610.43.x, **MAX Engine GPU graph compilation succeeds** but **`InferenceSession.load()` followed by `model.execute()` on GPU-resident buffers fails** during the native execute path.
**Observed failure modes (same minimal reproducer):**
| Mode | Symptom |
|------|---------|
| Default | **SIGSEGV (exit -11)** within a few seconds of `model.execute()` |
| `CUDA_LAUNCH_BLOCKING=1` | **Indefinite hang** at `model.execute()` (Ctrl+C required; SIGINT backtrace shows mutex wait) |
On another HPC node (H200, older CUDA stack), the same Engine GPU execute path has also produced hang, SIGABRT (-6), `RuntimeError: Invalid argument`, and SIGKILL (-9).
**What works on the same allocation / environment:**
- MAX driver GPU detection, H2D/D2H, and device-side buffer copy
- Standalone Mojo GPU kernels (`DeviceContext`)
- MAX Engine **CPU** `model.execute()` (built-in `ops.add`)
- MAX Engine **GPU compile** for built-in ops and custom Mojo ops (`ops.custom` + `@compiler.register`)
- `gpu-query` reporting CUDA API 13.3 (`api_version: 13030`)
**GDB backtraces (pixi env, symbols partially stripped):**
*A) Default — segfault at execute:*
```
[engine-gpu-exec] compile done in ~3s
[engine-gpu-exec] executing on GPU...
Thread 1 received signal SIGSEGV
#0 sem_post@GLIBC_2.2.5 () from /lib64/libc.so.6
#1-#6 ?? () from libmax.so
#7 M::ExecutableModel::execute(...) from libmax.so
#8 M::GraphCompiler::Driver::executeFromDriver(...) from libmax.so
#9-#12 max/_core.cpython-314-x86_64-linux-gnu.so
```
*B) `CUDA_LAUNCH_BLOCKING=1` — hang; SIGINT backtrace:*
```
[engine-gpu-exec] executing on GPU...
(hangs 1+ hour)
Thread 1 received signal SIGINT
#0 __lll_lock_wait () from /lib64/libc.so.6
#1 pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2 ?? () from libAsyncRTMojoBindings.so
#3-#8 ?? () from libmax.so
#9 M::ExecutableModel::execute(...) from libmax.so
#10 M::GraphCompiler::Driver::executeFromDriver(...) from libmax.so
#11-#14 max/_core.cpython-314-x86_64-linux-gnu.so
```
Both traces converge on `ExecutableModel::execute` / `executeFromDriver`. The default path crashes in `sem_post` (likely invalid/destroyed semaphore); with `CUDA_LAUNCH_BLOCKING=1` the main thread blocks in `pthread_mutex_lock` inside `libAsyncRTMojoBindings.so`. This points to a **threading/synchronization lifecycle bug in the Engine async runtime**, not user Python code or GPU kernel compilation.
**Expected:** `model.execute()` returns GPU output matching `a + b` (e.g. `[11., 22., 33., 44.]`).
**Actual:** Process segfaults or hangs on first GPU execute after successful compile. No Python exception on the segfault path.
**Impact:** Cannot run MAX graphs end-to-end on GPU in this HPC environment. Workaround: Mojo `DeviceContext` or MAX driver buffers for GPU compute; Engine limited to compile + CPU execute.
**Mitigations tried (no change):**
1. `load_devices([CPU, GPU])` before `InferenceSession`
2. Subprocess isolation with timeout
3. Contiguous NumPy inputs; `dev.synchronize()` before execute
4. Engine subprocess before other CUDA context use in parent
5. System CUDA 13.3 on `PATH` and `LD_LIBRARY_PATH` (`nvcc` resolves)
6. `CUDA_VISIBLE_DEVICES=0` (numeric) — still fails
**Suggested investigation areas:**
- `libmax.so` execute path (`ExecutableModel::execute`, MLRT async values)
- `libAsyncRTMojoBindings.so` mutex/semaphore coordination
- Slurm `CUDA_VISIBLE_DEVICES` UUID binding vs numeric index
- CUDA 13.3 + driver 610.x on `sm_90a` (compile vs execute code paths)
- pthread/glibc interaction under minimal `LD_LIBRARY_PATH` (`/lib64` only vs CUDA lib prepend — segfault persists after prepend)
---
### Steps to reproduce
## Steps to reproduce
**1. Environment**
Install MAX/Mojo nightly via pixi (or conda `max-nightly`). Run inside a **single-GPU Slurm/PBS allocation** on Linux with CUDA 13.3 and an H100-class GPU (`sm_90a`).
Optional (did not fix the issue, but reflects our HPC setup):
```bash
export CUDA_HOME=/etc/alternatives/cuda-13
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$(readlink -f $CUDA_HOME/lib64):${LD_LIBRARY_PATH:-}"
```
**2. Minimal reproducer** — save as `repro_gpu_engine_execute.py`:
```python
import numpy as np
from max.driver import Buffer, DeviceSpec, load_devices
from max.dtype import DType
from max.engine import InferenceSession
from max.graph import DeviceRef, Graph, TensorType, ops
gpu_id = 0
a_host = np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)
b_host = np.array([10.0, 20.0, 30.0, 40.0], dtype=np.float32)
n = len(a_host)
devices = load_devices([DeviceSpec.cpu(), DeviceSpec.accelerator(gpu_id)])
session = InferenceSession(devices=devices)
with Graph(
"gpu_vector_add",
input_types=[
TensorType(dtype=DType.float32, shape=[n], device=DeviceRef.GPU(gpu_id)),
TensorType(dtype=DType.float32, shape=[n], device=DeviceRef.GPU(gpu_id)),
],
) as graph:
lhs, rhs = graph.inputs
graph.output(ops.add(lhs, rhs))
model = session.load(graph) # succeeds in ~1–3 s
dev = model.input_devices[0]
a = Buffer.from_numpy(np.ascontiguousarray(a_host)).to(dev)
b = Buffer.from_numpy(np.ascontiguousarray(b_host)).to(dev)
dev.synchronize()
out = model.execute(a, b)[0] # SIGSEGV (-11) or hang
print(out.to_numpy())
```
**3. Run**
```bash
pixi run python -u repro_gpu_engine_execute.py
```
**Expected on bug:** segfault within seconds (exit -11), or hang at execute.
**4. Optional GDB (run through pixi so `max` is importable)**
```bash
pixi run gdb -batch -ex run -ex bt --args python -u repro_gpu_engine_execute.py
```
**5. Optional hang variant**
```bash
CUDA_LAUNCH_BLOCKING=1 pixi run python -u repro_gpu_engine_execute.py
# or same with gdb; interrupt after hang for mutex backtrace
```
**6. Confirm surrounding stack works (same node)**
```bash
pixi run gpu-query # CUDA 13.3, sm_90a
# Mojo GPU vector add via DeviceContext — PASS
# MAX driver H2D/D2H/copy — PASS
# Engine CPU execute + GPU compile only — PASS
# Engine GPU execute — FAIL (this issue)
```
Custom Mojo kernel in graph (`ops.custom` + `@compiler.register`) shows the same split: CPU execute PASS, GPU compile PASS, GPU execute FAIL.
---
If the template has only those two fields, put everything under **Bug description** and keep **Steps to reproduce** as section 2 above (items 1–6). I can also produce a single flat markdown file for a gist or attachment if you want that.
### System information
**System context (no site-specific paths):**
| Component | Detail |
|-----------|--------|
| OS | Linux x86_64, RHEL 9 family |
| Scheduler | Slurm/PBS GPU job |
| GPU | NVIDIA H100 80GB HBM3, CC 9.0, `sm_90a`, 132 SMs |
| Driver | 610.43.x |
| CUDA runtime API | 13.3 (`api_version: 13030` via `gpu-query`) |
| System CUDA toolkit | 13.3 (via `alternatives` symlink to `cuda-13.3`) |
| MAX | `26.4.0.dev2026060906`; also reproduced on `26.5.0.dev2026061706` |
| Mojo | `1.0.0b2.dev2026060906` / `1.0.0b3.dev2026061706` |
| Python | 3.14.3 |
| NumPy | 2.4.6 |
| Install | `pixi`, conda channel `max-nightly` |
| `nvidia-smi` | Not in PATH (typical HPC); `gpu-query` used |
| `CUDA_DEVICE_ORDER` | `PCI_BUS_ID` |
| `CUDA_VISIBLE_DEVICES` | Slurm UUID form (`GPU-<uuid>`) on primary node |
```
Kausch
July 25, 2026, 10:30am
2
I wouldn’t think the focus would shift away from Datacenter class compute at all. Even Qualcomm is joining the CPU (server versions of Oryon) and networking stack for AI for hyperscalers. Plus they are going heavy on ASICs as AI accelerators based on a scaled up Hexagon NPU architecture for inference.
Snapdragon X Elite inference too is based on CPU + NPU.
But it would be a valid concern regarding the extent of official support for what could now be considered as competitor hardware. I would only really bother about Jetson class chips though w.r.t Nvidia for physical AI. Many industries just buy Jetson as the only option. Modular has existing arrangements with AMD for MI 3xx class Instinct GPUs, and the workflow of many Mojo users (including Modular staff) would involve Apple Silicon MacBook Pros. So, MAX being as capable as possible on Apple Silicon, especially M5 and later, is perhaps something as native to Mojo as it could get. Whether future AMD Instinct and Radeon architectures get the same attention is to be seen, and I wonder if they might be left to the community after open sourcing more of Mojo and MAX. So, future multi-vendor support strategy and free tier is something that they would disclose in ModCon I guess!
We’re massively stretched with preparations for modcon, and the MAX team in particular has a lot of high priority work.
Although I can’t provide any guarentee here, the extra people that Qualcomm can provide, while they won’t shift the prioritization of this and issues like it, may give us the resources to address issue like this one alongside necessary feature work. For this in particular, there are a lot of things this could be since we don’t test H100s with that driver version since most DC deployments use an older driver for validation reasons.
It would also be rather silly of us to walk away from a market segment we’re the fastest in, so don’t expect us to walk away from HPC/DC GPUs.