Clarification on which libraries are usable in the standard library

[owenhilyard]

Hello all,

Given that MAX and Mojo are moving closer together, I think the stdlib needs some clarification on what we are allowed to use. We currently have the following libraries shipped with the MAX SDK:

algorithm.mojopkg
autotune.mojopkg
benchmark.mojopkg
buffer.mojopkg
compile.mojopkg
compiler_internal.mojopkg
compiler.mojopkg
complex.mojopkg
gpu.mojopkg
kv_cache.mojopkg
layout.mojopkg
max.mojopkg
_mlir.mojopkg
register.mojopkg
runtime.mojopkg
stdlib.mojopkg
subprocess.mojopkg
tensor_internal.mojopkg
tensor.mojopkg

Of these, I think that _mlir, and anything with _internal is probably out, but others are probably still useful. Algorithm has vectorize, which can be useful in the stdlib, as well as reduce. buffer is useful for a lot of BLAS-like operations if we want something numpy-shaped to help people port python code. benchmark is used quite a bit in tests. gpu will probably want to at least be used for semaphores and memory barriers. autotune, once that is in a better state, will be useful for handling different platforms with different balances of operation costs to try to make code more portable. complex could very easily have a home in the math module and we probably want the math module to support it to some degree. subprocess also probably belongs in the stdlib, but I understand that threading in Mojo is currently a field of landmines so it may be better to keep it separate for now.

Tagging @joe for stdlib policy and @timdavis since I think this touches licensing.

I’m mostly looking for a list of what is allowed which I’ll put into the contributor docs.

[joe]

Thanks for asking the question! Looking at the left nav bar from the Mojo API docs should help clarify this a bit.

From a quick look at your list though:

algorithm.mojopkg
autotune.mojopkg
benchmark.mojopkg
buffer.mojopkg
compile.mojopkg
compiler_internal.mojopkg
compiler.mojopkg
complex.mojopkg
gpu.mojopkg
kv_cache.mojopkg # not stdlib
layout.mojopkg
max.mojopkg # not stdlib, just the main max package itself
_mlir.mojopkg
register.mojopkg # not part of stdlib, this is used for registering kernels in graph compiler
runtime.mojopkg
stdlib.mojopkg
subprocess.mojopkg # very minimal, but part of stdlib
tensor_internal.mojopkg # will get removed soon/consolidated back into tensor
tensor.mojopkg # not part of stdlib

Anything not annotated as “not stdlib” is part of the (current) closed source modules. The stdlib.mojopkg is a special collection of other (open) source packages which is what allows cycles and other things across packages to work transparently. The rest of these closed source packages have a 1:1 correspondence to their source and their on-disk representation (a mojopkg). Note that more of these closed source packages will become open source soon (stay tuned for news on this later this month!) at which point they will be part of the stdlib.mojopkg on disk.

[owenhilyard]

Thanks for the clarification! I wasn’t expecting _mlir and runtime to be allowed.