I am looking to create an inference farm on a Docker HPC host with 4 nodes with 4 H100s each.
I asked Gemini and it said MAX loses to vLLM for more users and is more geared towards local deployments.
I took this personnally, but can anyone share any experience with using MAX to serve say 50-200 users accross compute nodes, please?
There’s a lot of factors involved. Our focus has been on optimizing for more recent GPU architectures from Blackwell and above so H100 isn’t included yet. It also depends on the models and the architecture we support. There’re many cases that MAX is faster than vLLM/SGLang and we published a few case studies:
Also we should distinguish between using MAX as is or our Modular Cloud which has cluster optimizations etc. needed to get the best perf per dollar on multi-hardware scenario which translates to TCO.
Unfortunately, MAX Cloud is not an option for Governmental (foreign at that) work. I am just trying to figure out how I can use as much Modular tech as possible as the results of “rewriting geospatial science code in Mojo” have been magical, but my area of expertise is not LLM deployment.
Any tips, tricks, violent insults, technical leads and reading on the subject is welcomed 