Mojo compiler crashes in HPC environments with strict per-process virtual memory limits

Mojo
1

Hi,

On one HPC system I use (https://tacc.utexas.edu/), there are strict per-process virtual memory limits of 8 GiB. Unfortuantely, it is unlikely I can convince the sysadmins to raise this limit.

This causes the Mojo compiler to fail to run:

error: the Mojo compiler ran out of memory.
If your program has excessive compile-time computations or comptime recursion, try simplifying it.
If that does not help, this may be a Mojo compiler bug; please file a report at https://github.com/modular/modular/issues

Is there any way to work around this?

Thanks,
Ben

2

8 GiB is actually not a lot of memory for a modern compiler, especially one which is automatically parallel. You probably aren’t helped by tcmalloc (Mojo’s backing allocator for now) being eager to grab big piles of virtual memory.

Are you using MAX or just Mojo?

If you’re just using Mojo, can you compile locally before moving the binary over?

If you’re using MAX and that’s where it’s hitting issues, then I’d need to defer to @BradLarson on ways to AOT compile MAX graphs.

8 GB seems like a very small memory limit for an HPC system, especially considering that one would expect users to do LTO builds on the headnode.

3

I’m just using Mojo.

No disagreement on the memory limit, but it’s what I have to work with. I think LTO builds are still uncommon in HPC, at least for the vast majority of users on these smaller-scale HPC systems.

For now, I can start an interactive job and build there (although I can’t access the internet from compute nodes). The limits are on the login node.

4

@bwibking have you tried to limit number of parallel compilations using -j N option ? For example, using -j1 should disable multithreading.

5

It’s hitting the limit when running mojo package, but that doesn’t seem to accept the -j option:

$ mojo package mojo/amrex -o .pixi/envs/default/lib/mojo/amrex.mojopkg
error: the Mojo compiler ran out of memory.
If your program has excessive compile-time computations or comptime recursion, try simplifying it.
If that does not help, this may be a Mojo compiler bug; please file a report at https://github.com/modular/modular/issues

$ mojo package -j 1 mojo/amrex -o .pixi/envs/default/lib/mojo/amrex.mojopkg
mojo: error: unrecognized argument '-j'
6

mojo package creates something that is architecture agnostic, so you can build that locally and ship it over.

1 Like
7

Hello @bwibking ,
you might consider removing your login/credential from the terminal code snippet example.

1 Like
8

Thanks, removed.

9

8GB for mojo package seems quite a lot; that step should not be consuming an excessive amount of memory. If you are able to share your code, please create an issue in GitHub · Where software is built

1 Like