Suppose I qsub 5 compute nodes each having CPUs and GPUs, I now use MPI to distribute my job accross them but can custom ops and MAX do the same or is there a Mojo std.distributed or something of that sort?
You should be able to talk to MPI via C FFI. I’m not aware of any bindings to common implementations or to the general API (macros become C functions over FFI). Right now, I’d look at MAX more in the vein of OpenMP in terms of it being able to handle a full node but not really spanning beyond that.
We will eventually have something, but MPI as an API doesn’t really work for modern clusters due to its general lack of fault tolerance. Checkpoint and restore gets you a decent way, but at some point idling 10 MW of GPUs because one network card died and you need to move a checkpoint over to a hot spare node and restart MPI becomes wildly irresponsible from an energy use perspective and thus financially non-viable. As a result, we’ll have to do it the hard way and design for nodes failing in the middle of a job without taking the job down.
Got it. That sounds more of a Milestone 3 on the Roadmap though, right?
Good point. I was thinking of Mojo for OpenMP + MPI clusters specifically in the context of NextSilicon’s Maverick 2 and the dataflow architecture. It’s the product that has excited me the most for HPC hardware, as throwing MW at GPUs at exascale level, and moving data through DRAM was messy, compounds faults, and the whole NV Link situation isn’t great too. Maverick 2 makes the whole GPGPU thing for HPC scale look silly, and they scale to HPC clusters through high speed ethernet. Because Mojo is magic, who could showcase it best more than a maverick? 
My instinct is that the points of potential faults will be greatly reduced with Maverick 2 being the accelerator, with the dataflow engine, its efficiency, and greatly reduced thermal cycling. It offers dataflow-level checkpoint, selective rollback, and rack level spare nodes for redundancy planning designed to work with OpenMP and MPI. This is a natural case for Mojo imo, and rise of high performance RISC-V would also add to that.
The other hardware architecture I’m really keen on is Tenstorrent w.r.t AI, as they have a more direct mapping of hardware purpose built for dataflow and Tiles being a physical thing. It would be really cool to work with the Blackhole cards using Mojo, but that’s another topic. Mojo can effectively consolidate TT-Lang and TT-Metallium into one. My startup is being built on TT hardware though, and the folks we work with like it for efficiency as well as control over RISC-V ISA for European AI sovereignty goals.
The roadmap has it as a later milestone 3 item.
While Maverick is a very cool piece of silicon, dataflow hardware does a lot of data movement, and data movement makes heat.
That redundancy is great, but not having to roll back much at all is even better. Of there are spare nodes in the rack, bringing them online without stopping everything else should be the goal, since you can probably do at least some parts of the computation while you wait. If you have a system which doesn’t have to stall to add/remove nodes, or otherwise deal with failures, I think you’ll pretty much always come out ahead. Graph compilers can easily figure this out, which helps a ton.