Proptest: property-based testing for Mojo

Community Showcase
1

proptest is a property testing framework for Mojo (i.e., the QuickCheck
family) inspired by the Hypothesis framework
for Python, with ideas from Rust’s proptest and quickcheck, Haskell’s
QuickCheck, and C++'s RapidCheck:

With property-based testing, you state a property your code should have and
which inputs it should hold for. proptest generates many inputs, looking for
one that breaks the property. When it finds one, it shrinks it to a
minimal failing example before reporting it.

from std.testing import assert_equal
from proptest import given, text

def test_decode_inverts_encode() raises:
    def prop(s: String) raises:
        assert_equal(decode(encode(s)), s)
    given(text(), prop)

Falsifying example (examples/quickstart.mojo:41:14):
    given('0')
At examples/quickstart.mojo:38:21: AssertionError: `left == right` comparison failed:
   left:
  right: 0

With check, the inputs come from the property’s parameter types, and a
constraint can be part of the type:
from proptest import check, NonZero

def abs_is_positive(x: NonZero[Int8]) → Bool:
return abs(x.value) > 0

check(abs_is_positive)

Falsifying example (abs.mojo:7:10):
    given(-128)
check: the property returned False

Influences

Hypothesis supplies the engine. proptest is built on Hypothesis’s
Conjecture engine, not just its API. Strategies parse values from a
recorded sequence of choices, and the shrinker simplifies that sequence.
Shrinking therefore works for any strategy, including the ones you write
yourself, and you write no shrinking code. From Hypothesis there are also:

  • Strategies:
    • integers, floats, text and the other basic strategies;
    • lists, tuples, one_of, map, filter, flatmap, composite,
      recursive, shared and functions;
    • from_regex, with Unicode categories;
    • from_type, for building structs.
  • Engine: an example database that replays past failures, target()
    hill climbing, health checks, deadlines, reproduce_failure-style
    replay, the explain phase, and run statistics.
  • Stateful testing: rule-based state machines with preconditions,
    invariants and bundles.
  • NuMojo: array strategies, the counterpart of
    hypothesis.extra.numpy, as an optional extra.

The other libraries contributed most of the typed API:
Rust’s quickcheck: check(prop), which infers strategies from the
property’s parameter types.

  • Haskell’s QuickCheck:
    • constraint types such as Positive[T], NonEmpty[T] and Sorted[T],
      which put an input’s constraint in its type;
    • cover, tabulate and check_coverage, which check the
      distribution of inputs statistically;
    • Outcome results, expect_failure, and the PolyA stand-in types
      for testing generic code.
  • Rust’s proptest:
    • typed state machines, whose steps are values of a transition type;
    • filter_map, subsequence, Index, and weighted one_of.
  • RapidCheck (and through it, Erlang QuickCheck):
    • model-based testing, with one Command struct per operation;
    • build[T](field["x"](s)) for structs with some fields given, and
      apply.

Where proptest differs from Hypothesis, it’s mostly because of Mojo
itself. For example, map is a free function, map(s, f) rather than
s.map(f), and integers are 128-bit rather than arbitrary-precision. Each
difference and the reason for it is listed in
docs/differences-from-hypothesis.md,
and what came from the other libraries is in
docs/ideas.md.

It has already earned its keep. Testing MojoDEC (discrete exterior calculus
in Mojo) with it found a precision bug in simplex volumes, and proptest’s
own NuMojo tests found two indexing bugs in NuMojo.

Installing

proptest needs Mojo 1.3, so for now it runs on the nightly channel.
That’s also why it isn’t in the modular-community channel yet: that
channel builds only against stable Mojo releases, and the latest stable
release is 1.2. A recipe is ready, and I’ll submit it once 1.3.0 is out.

Until then, clone the repo and put it on your include path:
git clone GitHub - bpr/proptest: Property based testing library for Mojo · GitHub
mojo run -I path/to/proptest your_tests.mojo

Or build a package with pixi run package, which writes
build/proptest.mojoc.

Feedback, bug reports and pull requests are welcome. The project is MIT
licensed.

1 Like