Very interesting discussion, and certainly helps us to shape the roadmap
This should not be as necessary anymore with pixi-build and the pixi-build-mojo backend. You can just depend on other Mojo packages by path or git source, if they carry the package definitions (then they will be built on the fly locally). Alternatively, you can just make your own channel on prefix.dev or elsewhere, of course. One could also change the model of the modular community channel to allow self-publishing through trusted publishing from user repositories.
In a way you can think of build backends as āpluginsā to Pixi, and we also have definite plans to add support for language specific linting / formatting and other ātasksā that the build backend can advertise automatically.
Hi @owenhilyard - I think this claim is a bit harsh. I am not even sure what you mean with ābinary repo to lean onā? On conda-forge we do not allow picking up āaccidental dependenciesā and I havenāt heard this claim from anyone in a long, long time. In the build tools, we do static analysis to warn or error if anything is linked from the āoutsideā that is not explicitly allowed. We do have sandboxing features in rattler-build that we can definitely improve and even default to, if that would help the Mojo usecase.
The problem Iām alluding to is the lack of a reliable way to build the world from source. rattler-build is a massive improvement here, but there are a lot of conda packages which donāt use it or similar backends, and thus building a conda environment only from itās precursor git repos is a fairly annoying endeavor. This is something Iāve had to do on occasion due to jumping to a new CUDA release early for one reason or another.
conda-forge does have good quality control over accidental dependencies, however, I think that the fact that conda-forge needs to use static analysis tooling at all for this is an issue. Itās very easy to accidentally create a conda package that relies on a too-new glibc, or which pulls in a vendor MPI implementation (both situations Iāve experienced).
pixi is a massive improvement to conda, but until everyone adopts both it and rattler-build it I think a lot of these core issues will remain.
Yeah - I would be down to ārebootā conda-forge as a mono-repo with pure rattler-build recipes that could be easily used to rebuild the world from scratch! Just takes a bunch of effort to get off the ground.
I would love to hear your ideas on how to fix these problems without static analysis? The glibc situation has improved quite a lot with conda-forge as we require it at build time now through the stdlib specifier.
The rattler-build adoption is going pretty strong these days: Are we Recipe v1 yet? - but of course we need to keep on fighting.
I had another thought for the problems with conda as a binary package manager. One library Iām quite familiar with is DPDK, which is a library which enables you to run device drivers fully in userspace for additional performance, starting with networking and later branching out. You only use DPDK when io_uring and friends fail you from a performance standpoint, which means that it cares deeply about performance. It uses meson, which is a pretty normal build system, but due to its performance constraints it has a lot of what one might call āweirdā configuration.
The instruction set, which is specified separately from the compiler flags so that DPDK can use .bytes and similar to make use of instructions which arenāt yet enabled in compilers (this comes up a lot in RHEL)
Which drivers to build, which can massively reduce the size of the output. This option can easily be several hundred characters long if you just want to eliminate things that donāt work on a particular platform (ex: ARM SoC drivers on x86)
Whether ibverbs is linked dynamically, statically, or used via dlopen (important for either performance or portability)
The maximum number of network ports supported (dramatic effect on memory usage)
The maximum number of CPU cores supported (dramatic effect on memory usage)
The maximum number of NUMA nodes supported (dramatic effect on memory usage)
Whether to allow for physical addressing (takes valuable space in a datastructure but is mandatory for working on AWS)
Whether the central datastructure for packets is atomically refcounted or not (big performance implications)
How much scratch space to leave in packet buffers (for adding vlan/vxlan/etc tunneling)
Whether to use C11 atomics from the compiler or ones hand-tuned by the CPU vendors (Intel, ARM, IBM, etc)
Whether to use the HPET timer (massively slower than the TSC, but the TSC is broken on some older systems)
If you try to shove this into conda packages, you get a very, very, very large feature matrix, much worse than pytorch, since the wrong value in many of these config options can either ruin portability or wreck performance. There are plenty of similar projects with very large numbers of tunables, like MPI, but some of them, like DPDK, change the ABI based on those options, which means you can just shove them behind an interface.
I would love to hear your ideas on how to fix these problems without static analysis? The glibc situation has improved quite a lot with conda-forge as we require it at build time now through the stdlib specifier.
In my opinion, Bazel and Nix do this correctly, with Nix being far less painful to make talk to other build systems. You have fully hermetic builds for everything and only the provided context is in the sandbox. If the build can only see specified dependencies, then itās unlikely to be able to accidentally pick up an unspecified one, especially if the build sandbox disables networking and other common means of adding new dependencies. They are also both very good at caching based on input config and smoothly falling back to source compiles when the binaries are missing.
The rattler-build adoption is going pretty strong these days: Are we Recipe v1 yet? - but of course we need to keep on fighting.
That looks great! The day when I can make 90% of my python version conflicts go away by compiling from source will be good.
I understand the downsides of binary packages - but we have already added source packages to pixi and itās pretty convenient with build backends (which can also be custom-developed, and there is already a specific one for mojo!).
We would be happy to extend the sandboxing in Pixi and our build tooling but so far it has not been a big problem. If you have some pathological cases I would always appreciate if you open an issue and we can work on a fix asap. If you want to help with our sandbox, you can opt-in with rattler-build and the --sandbox flag Sandbox - Rattler-Build
Iām a pixi fan and Iād love the community to adopt pixi as the āblessedā modular package/env manager. pixi-build works quite well and Iāve had no issues with it building from source with git and itās quite fast building multi-packages with multi-languages invovled. Unfortunately (the community built) pixi-build-mojo doesnāt as it uses mojo package which is unstable and tied to a very specific compiler version (weāll be renamed soon that better shows this Proposal: rename `mojo package` and `.mojopkg`) so if thereās a dependency that uses different mojo version itād fail to work as .mojopkg for 1.0.0 is different from 1.0.1.
My wish list for now:
The ergonomic should be better and I wish I didnāt need to write recipe.yaml (and maintaining its own version) if Iām not customizing my build and have normal dependency specification in pixi.toml similar to cargo.toml. I understand that packages that have multiple languages involved may need such lang-agnostic recipe but if my dependency knows how to deal with them then downstream packages should just work without repeating them in recipe.yaml.
Also would be great to optionally support similar build.rs-like script thatās language specific and in mojo case can be build.mojo as an idea!
Besides, nested workspace-like packages are harder to work with than cargo which I think the UX can be improved without manually specifying the manifest path for example.
Better way to enforce system level requirements as well as gpu vendor and arch support which is important for mojo
That is very nice! However, Iām running into issues trying to customize that file as something that depends on it (like toggling a feature flag in cargo). For instance to enable cuda for gpudev. The docs seem to be pointing me to needing to create " Variants", which would lead me to 2^{160} = 1461501637330902918203684832716283019655932542976 variants just to express the drivers, much less the tuning knobs. That number would come down somewhat from things like eliminating combinations that are impossible, like having drivers for multiple SoCs, but I suspect Iād still end up with a staggering number of packages if I tried to publish this feature matrix to conda forge. Am I completely missing the intended way to do this? DPDK very much prefers to be statically linked with LTO for performance reasons.
The sandbox flags look promising once reading the whole filesystem is turned off by default.
The definite goal of pixi-build is that you do NOT need to write a recipe.yaml but get all the knobs you need for the particular language from the build backend.
Build backends are very āfree to do what they wantā, and they usually template a build script dynamically. If you had a build.mojo script, or a build.py or whatever, the build backend could dynamically find it in the project, and inject the execution of this script into the build process. That would be easy and natural to do.
Another goal for Pixi & pixi-build is great mono-repo support. We already have users that have relatively large mono-repositories but I am sure there are areas where we can improve. Dependencies are currently specified as other-pkg = { path = ā../other-pkgā } which I believe is pretty similar to Cargo. However, Cargo has a concept of āworkspaceā dependencies, meaning you can also do other-pkg = { workspace = true } which will then pull the path argument from the top-level workspace TOML file. Something we can definitely also implement in the future.
For system level requirements - yes, let me just say we would love to collaborate and we can definitely come up with a plugin system, as well as additional āConda Enhancement Proposalsā to improve the default situation. We just helped NVIDIA engineers to push the __cuda_arch package over the finish line: https://github.com/conda/ceps/pull/157 (the vote is currently happening, and our Rust implementation is already ready to go).
@owenhilyard yeah, I have a prototype using PKL from Apple (https://pkl-lang.org/) for recipe.pkl / pixi.pkl. PKL would make it easy to override / extend existing recipes by downstream consumers. Today, the easiest would be to fork / copy the recipe and set the arguments that you want. At the same time, a given package could always produce some variants automatically (not 2^160), and users/the SAT solver could select from them.
We just helped NVIDIA engineers to push the __cuda_arch package over the finish line: https://github.com/conda/ceps/pull/157 (the vote is currently happening, and our Rust implementation is already ready to go).
Is there an equivalent for AMDGPU and Intel GPUs (in particular Alchemist to Celestial will have some pretty dramatic capability gaps)? Also, what about RISC-V ISA extensions, or even just the parts of AVX512? I donāt think that having a vote for every capability like this is going to scale, especially as Mojo picks up support for more hardware.
yeah, I have a prototype using PKL from Apple (https://pkl-lang.org/) for recipe.pkl / pixi.pkl
Pkl is a nice templating language, but it still seems dramatically less ergonomic than setting a few feature flags and config values. I can see how itās useful for the most advanced few percent of use-cases, but I see working via overrides as an ergonomics issue for even cases like DPDK, which are effectively a few sets of enum values, some numeric values and a few strings. Can you provide an example of what, say, taking the default config and re-enabling mlx5 would look like with your prototype for Pkl? Ideally, Iād like to see the enabled and disabled drivers as a set of enum values for correctness.
