Proposal: deprecating `mojo test`

We’ve posted a new proposal to deprecate mojo test. The proposal itself has a detailed background on the issues we’ve encountered with mojo test, as well as how to migrate your projects to the new Mojo TestSuite framework that the team is building.

In short, mojo test has had some problems throughout its history. Initially, it used the Mojo REPL to run tests, which led to slow execution and subtly broken behavior as code was not being run in the same way as compiled Mojo. The most recent form of mojo test is tightly coupled to the parser and located in publicly inaccessible C++ portions of the codebase. It has a number of bugs, and its emulation of a main function makes it incompatible with the test orchestration we want to do.

To provide a better path forward for testing in Mojo, the standard library team is building a new testing framework in Mojo. You may have noticed the addition of some reflection functionality in Mojo for discovering functions within a module. That has enabled a Mojo-based way to perform test discovery. The team is in the process of converting the standard library’s tests to use this new framework, and people have really liked the results so far.

We are therefore proposing to deprecate the current mojo test and migrate Mojo testing to this new test framework, along with user-defined test orchestration.

Steps to migrate off of mojo test

The first step in migration to this new framework is to add the following to your test_*.mojo files:

from testing import TestSuite

# ... The existing test functions starting with `test_`.

def main():
    TestSuite.discover_tests[__functions_in_module()]().run()

If you don’t already have a custom build system or process for test orchestration (Bazel, scripts, etc.), you can use a script like the following to find and run your test files:

#!/bin/bash
set -e

# Get test directory from first argument
if [ -z "$1" ]; then
    echo "Error: Test directory not provided"
    echo "Usage: $0 <test_directory>"
    exit 1
fi
test_dir="$1"

any_failed=false
echo "### ------------------------------------------------------------- ###"
while IFS= read -r test_file; do
    if ! mojo run "$test_file"; then
        any_failed=true
    fi
    echo "### ------------------------------------------------------------- ###"
done < <(find "$test_dir" -name "test_*.mojo" -type f | sort)

if [ "$any_failed" = true ]; then
    exit 1
fi

If you add that script to your project, you can then set up a Pixi or uv command for running tests with the aid of that script.

Feedback

We feel that building a robust testing framework for Mojo in Mojo is the right path forward, but we welcome your comments about this plan. Are there common use cases that we’ve missed? Will this cause trouble that we’re not aware of? Suggestions for improvement?

Feel free to let us know your thoughts on this proposal in the comments here.

5 Likes

I must admit that mojo test is so full of bugs that in practice I’ve already moved away from it for all of my projects. I’d be happy to get a true test runner in the future as it improves the DX massively. In the meantime bash is good enough!

1 Like

Same as @gabrieldemarmiesse , I’ve moved away from mojo test a couple months back (due to failed test discovery + mystery segfaults) and use bash + pixi tasks in the meantime. Excited for a proper test suite!

I appreciated the bash script template also, I’ll try it out

1 Like

I appreciate the information on why test is buggy and understand that removing it for now seems to be the cleanest solution. I would still like to have a clear standard way of doing testing rather than everyone piecing together their own test scripts.

I would not mind adding a couple lines to my code (using the test framework), but I would like some simple form of mojo test to persist, even if it is basically just executing the proposed test script. This will make it easier to introduce a nicer test orchestration tooling later, as everyone is starting from the same place.

Btw. the proposal mentions doc tests (“there is a solution”) but does not say anything more. Maybe this would be a good idea, though. (Of course, there is also modo…)

Awesome decision, less stuff is always great. Single file executables are the best test framework and that without any compiler magic.