Context: My background is in scientific computing so I started this as a personal project as a way to actually use MAX kernel APIs in a domain I know well and where MAX hasn’t been pushed into much yet. The idea is to catch issues early, esp. the UX ones.
This is very cool to see autodiff and Numpy and scipy all rolled into a single Mojo package.
However, it is a bit concerning that float64 special functions are not correct (to at least 1 ulp) in Mojo. I cannot imagine that users could adopt this for scientific computing workloads until that’s fixed.
Awesome to see, is there a reason why you did not go with NuMojo? Is there an equivalent to PyData or to Quansight to oversee the building blocks of the Mojo ML/Data analysis stack?
I already implemented a lot of libraries using Mojo as an accelerator for Xarray/Dask/Numpy datasets and ideally I would love to move my whole pipeline to Mojo but am missing
full numpy parity (NuMojo and now vMAX or zero copy numpy arrays with all the caveats it entails)
dask like scheduler (nothing yet)
xarray like organization (nothing yet)
parquet and delta lake/iceberg (marrow gets me part way there)
zarr (which I implemented myself to great effect for my use case, called it mozarr ™ )
The performances so far have been staggering but I cannot shake this feeling that the python interop and Dask scheduling and cross node out of memory communication is my one last 10x improvement.
Good question! It’s mainly about the focus. numax is MAX-first i.e. leveraging MAX and Mojo to a new domain whereas everything else I’ve seen doesn’t have this focus. I’m also particularly interested in various API explorations and take the feedback in and make MAX better.
Is it posible to make MAX cross compute node as well (attack OpenMPI) so that I can run accross qsubed nodes?
I am not sure what the extra challenge for the MAX graph builder to take into account multiple nodes though.
Also take a look at something like Polytope which I think is an awesome example why the memory model of a dataset matters a lot and then the array APIs are built on top of that.