[Proposal] Parameterized Deinitialization for Linear Types

Mojo
1

Linear Types: Parameterized Deinitialization

1. The Problem: The API Duplication Burden

Mojo has introduced elegant solutions to unify common language dualities—for example, raises Never unifies raising and non-raising functions, and the ref convention unifies immutable and mutable references.

However, supporting linear types in generic collections (List, Dict, Set) currently requires library authors to duplicate significant portions of API and memory management logic. This results in maintaining standard Deinitable methods (e.g., clear, __deinit__) alongside identical linear type variants (e.g., clear_with, deinit_with).

Semantically (ignoring performance), both versions perform the exact same underlying operations. For example:

struct List[T: AnyType]:
    # 1. Standard Deinitable version
    def clear(mut self) where conforms_to(Self.T, Deinitable):
        unsafe_destroy_n(self.data, count=len(self))
        ...

    # 2. Linear Type version (Duplicated API)
    def clear_with(mut self, deinit_func: Some[def(var T)]):
        for i in len(self):
            Pointer(to=self.data[i]).unsafe_deinit_with(deinit_func)
        ...

    # Proof of equivalence: We could implement `clear` using `clear_with`
    def clear(mut self) where conforms_to(Self.T, Deinitable):
        self.clear_with(Self.T.__deinit__)

(Note: We are not arguing against specialized APIs designed specifically to make linear types more ergonomic. We are simply addressing the duplication of API methods that share identical underlying semantics.)

Library code handles memory management uniformly across both Deinitable and linear types—the container’s internal cleanup logic remains identical in both cases. The requirement for flexibility really exists in user code, where linear types require an explicit choice of deinitialization function per call site. Currently, bridging this gap requires library authors to duplicate API methods.

1.1 Challenges with Existing Workarounds

While we could theoretically avoid duplication using existing language features, doing so in practice introduces a fair amount of friction:

Workaround A: Implement standard methods via _with variants

If we try to implement __deinit__ by simply calling deinit_with(Self.T.__deinit__), we face three main challenges:

  1. Loss of Trivial Deinit Specialization (Crucial for Performance!): Trivial type optimization (e.g., bulk destroying elements using unsafe_destroy_n) is executed by the standard library’s user code, not the compiler. If standard deinitialization routes through a generic function pointer path (deinit_with) to accommodate linear types, the stdlib can no longer specialize and optimize for trivial types.

  2. Loss of ASAP Deinitialization: The compiler cannot automatically invoke a custom deinit_func when an error is raised. Developers must manually wrap raising calls in try/except blocks, explicitly deinitialize the value, and re-raise. This adds significant boilerplate to control flow:

    # Standard Version: Compiler handles cleanup automatically on raise
def process_item(mut self) raises where conforms_to(Self.T, Deinitable):
    var item = self.extract_item()
    self.might_raise(item) # Compiler calls T.__deinit__(item) if this raises
    self.finish(item)

# Linear Version: Manual cleanup required on all exit paths
def process_item_with(mut self, deinit_func: Some[def(var T)]) raises:
    var item = self.extract_item()
    try:
        self.might_raise(item)
    except e:
        deinit_func(item) # Forced to manually catch, deinit, and re-raise
        raise e
    self.finish(item)
    deinit_func(item)

  3. Boilerplate and Delegation: To avoid duplicating core logic, standard methods must delegate to their _with variants. Repeating this delegation pattern across the entire API adds considerable boilerplate for library authors.

Workaround B: Wrap linear types in Deinitable structs

Developers could wrap a linear type to conform to Deinitable in order to use standard collection methods. However, this locks the type into a fixed deinit function, whereas real-world code often requires dynamic deinitialization strategies.

For example, when inserting a linear FileHandle into a Dict[String, FileHandle], we currently use the specialized insert method to apply different behaviors flexibly:

# Location A: we expect no elements to be replaced, so we abort on conflict
var displaced = my_dict.insert(key, file_handle^)
if displaced:
    abort("unreachable")

# Location B: we might want to move the old value to another place
var displaced2 = my_dict.insert(key, file_handle^)
if displaced2:
    move_to_other_place(displaced2.unwrap().value^)

# Location C: we might just destroy the old element
var displaced3 = my_dict.insert(key, file_handle^)
if displaced3:
    displaced3.unwrap()^.close()

If we instead wrapped FileHandle in a Deinitable struct to use standard collection methods (like __setitem__), we would be locked into a single __deinit__ implementation. We couldn’t dynamically choose between abort, move_to_other_place, or FileHandle.close across different call sites.

1.2 The Scaling Cost: Combinatorial Explosion

Relying on explicit _with methods can also lead to a combinatorial explosion for collections with multiple generic types that store their elements independently.

Consider a generic collection MultiStore[A, B] that stores A and B independently, where each has its own special handling for trivially-deinitable types. If a user wants to provide a custom deinit for just B but not A, the library author faces a difficult choice:

  1. To retain performance for the trivially-deinitable types while supporting custom linear types, they must write an exponential number of method overloads (in this case, 4 distinct overloads to cover all combinations of default and custom deinits for A and B).
  2. Alternatively, they could provide a single unified method (e.g., clear_with(..., a_deinit, b_deinit)). However, using the unified method forces the trivially-deinitable type (A) down an unoptimized function pointer path just to accommodate B, compounding the performance issues described above.

2. A Potential Semantic Direction

To bridge this gap without duplicating APIs, it might be beneficial to explore built-in language support.

Rather than implementing __deinit__ using deinit_with, we could invert the paradigm: algorithms are written once using the ergonomics of ASAP deinit, and the deinitialization function is parameterized per call site.

Under this model, all types can be Deinitable if they provide a deinit function:

  • Implicitly deinitable types automatically provide T.__deinit__ as the default deinit function.
  • Linear types would need to provide their custom deinit function per call site whenever an algorithm requires them to be Deinitable.

This means library authors can simply write algorithms for Deinitable types without worrying about what specific deinit function is used. They can write the algorithm with the ergonomics of ASAP deinit, and allow users to use it with linear types by providing an explicit deinit version per call site.

Specializing at the call site is quite important: the parametric function is specialized per call site according to the user-provided T.deinit, whether that is the default T.__deinit__ (retaining all trivial type optimizations) or a user-provided explicit function.

This approach preserves the strict linear nature of linear types, as they still require explicit lifecycle management from the user. The goal is simply to provide feature parity between implicit and explicit deinitialization in library implementations where the underlying semantics are identical. By doing so, we could gracefully solve both the combinatorial explosion and the loss of optimization.

3. Design and Syntax Ideas

Important Note: The exact syntax and design presented below represent a minimal direction, not a finalized specification. The primary goal of this proposal is to establish agreement on the problem (Section 1) and the proposed semantics (Section 2). If the Modular team agrees that this semantic direction is valuable, we would love to collaboratively iterate on the design and syntax challenges.

To make these semantics concrete, here is one potential approach using traits:

trait Deinitable:
    """
    Allows for normal ASAP deinitialization. 
    Supports Linear types when an explicit deinit function is provided at the callsite.
    """
    comptime deinit: Some[def(var Self)]

    comptime __deinit_is_trivial__: Bool

trait ImplicitlyDeinitable(Deinitable):
    """
    The deinit function implicitly used by the compiler for ASAP deinit.
    Can be overridden at callsites with alternative user-provided functions.
    """
    comptime deinit = Self.__deinit__
    def __deinit__(deinit self, /): ...

By default, all types would be ImplicitlyDeinitable. Linear types would opt out (e.g., struct Linear(ImplicitlyDeinitable where False)).

Functions parameterized on Deinitable could gain a special optional runtime keyword argument T.deinit = .... This allows developers to override the default deinit function (T.__deinit__) with a user-provided one at the call site. The type of the function would be inferred from the runtime value passed to this keyword argument. Using the type name prefix (T.deinit) helps avoid parameter name conflicts with standard function arguments.

3.1 API Consolidation: Eliminating Duplicated Methods

The primary benefit of this design is that we no longer need to write duplicated APIs (like lst.clear_with(custom_deinit) vs lst.clear()). By implementing a single regular method dependent on the Deinitable trait, the API automatically supports custom deinitialization.

This approach cleanly eliminates duplication across generic functions and collections:

Standalone Functions:

def process_item[T: Deinitable](arg: T): ...

# Call site overrides the deinit behavior
process_item(arg, T.deinit=custom_deinit)

Collection Methods:

# Uses default T.__deinit__
my_set.clear() 

# Replaces all internal T.__deinit__ calls with my_deinit
my_set.clear(T.deinit=my_deinit) 

# Avoids parameter shadowing via explicit path
my_set.clear(my_set.T.deinit=my_deinit)

Direct Struct Deinitialization:
Structs can also use deinit(T.deinit=...) directly as a replacement for the deinit_with API:

# Destroys the list, explicitly providing a custom deinitializer for its elements
lst^.deinit(T.deinit=custom_deinit)

3.2 Optional Overrides for Non-Linear Types

This mechanism is not exclusively for linear types. Any function parameterized on Deinitable could have its deinitialization behavior overridden. The core difference is that while a linear type must provide an explicit deinit function at the call site to use these APIs, an ImplicitlyDeinitable type has the option.

For non-linear types, developers can rely on the default implicit deinitialization, or choose to provide an explicit override. This can be useful if you want to change the lifecycle of standard elements—for example, moving an element somewhere else instead of destroying it.

3.3 Restoring Standard APIs for Linear Types

Because explicit deinitializers could be attached at the call site, linear types would regain access to standard APIs. For example, Dict.__setitem__ currently cannot be used with linear types, which necessitated the creation of a specialized Dict.insert. While Dict.insert is a great method and we are not suggesting removing it, if a collection lacked such specialized methods we would no longer be blocked from using linear types until an equivalent was implemented.

We could rewrite the complex FileHandle insertion examples shown earlier directly using __setitem__, maintaining exact parity while removing the boilerplate:

# Location A: abort on conflict
my_dict[key, V.deinit=lambda v: abort("unreachable")] = file_handle^

# Location B: move old value
my_dict[key, V.deinit=lambda v: move_to_other_place(v^)] = file_handle^

# Location C: destroy old element
my_dict[key, V.deinit=FileHandle.close] = file_handle^

This would automatically extend linear support to other methods without duplicated code:

d.pop(key, K.deinit=my_custom_key_deinit)

3.4 Solving the Combinatorial Explosion

It would also solve the combinatorics explosion by allowing selective overrides, gracefully retaining trivial optimizations for unspecified types:

# Modifies only the value's deinit, retaining trivial optimization for the key
my_dict.clear(V.deinit=custom_value_deinit)

3.5 Design Challenges

I acknowledge there is a conflation of using the deinit function as a compile-time parameter, since it can also be provided as a runtime kwarg. This might require exploring parametric traits or expressing it via some other syntax.

(Note: This design idea can be naturally extended to other traits beyond deinitialization. For instance, when calling a method parameterized on the Copyable trait, a user could provide T.copy=my_deep_copy at the specific call site to seamlessly swap the default copy behavior with a custom deep copy implementation.)

I would greatly appreciate your feedback on the problem described here, the proposed semantics, the design, syntax, and the overall direction of this idea.

2

This is a great writeup with some awesome ideas :smiley:
Just a few small thoughts from me - these aren’t major (dis)agreements with the post - just a brain dump of a few things to also consider.

  • If we had two traits, these would ideally be Deinitable and ExplicitlyDeinitable. There is not an exact parity here for (Implicitly)Copyable - the common case is Deinitable - aka “has a __deinit__ method”, and then the uncommon case of - does not have __deinit__ but instead has some other named deinitializer.
    • As a small nit for this, every type must have some sort of deinitizlier otherwise you can’t write a mojo program using that type (unless you do some MLIR hacks/purposeful resource leaking). This arguably means we could move the ExplicitlyDeinitable to just be a part of AnyType.
  • Some[def(var Self)] wouldn’t work for an ExplicitlyDeinitable trait. One of the purposes of linear types is to allow these named deinitializers to either raise error or have return values. You might be able to make it generic via Some[def(var Self) raises E -> R]) but then you step into territory of how to apply this generically across collection - which I have not thought too much about.
  • Re: trivial deinit specialization, there are likely solutions to this such as using function reflection to inspect the deinitializer and check if it is the __deinit__ method and then using the trivial flag. The other hope here is that we can rely on the Mojo compiler/LLVM to see that if the explicit deinitializer is a no-op it would optimize the loop away entirely.
  • Something to note too is not every collection needs to support linear types. Yes - ideally the stdlib’s types would and most people would model their generic collections after the stdlib, however, this doesn’t mean every collection needs to. Sometimes “picking the right tool for the job” means that some generic collection can’t easily/shouldn’t support linear types, and that’s okay too.
3

We can use a closure:

var failed = List[LinearRaising]()

def my_deinit(var val: LinearRaising) {mut failed}:
    try:
        var result  = val^.raising_deinit(123)
        do_something(result)
    except e:
        # move the value out instead of deinit
        failed.append(val^)

my_collection.deinit(T.deinit=my_deinit)

if len(failed) > 0:
    raise MyError(failed)

Maybe Deinitable and DefaultDeinitable, since we want to have ASAP (implicit) deinitialization for both. The difference is whether the type provides a default deinitializer or not.

4

Alternative Approach: Ad-Hoc Trait Conformance via Anonymous Sub-typing

Here is an alternative design direction to solve the same problem.

Instead of defining two distinct traits (e.g., ImplicitlyDeinitable and ExplicitlyDeinitable), we can maintain the current semantics of a single Deinitable trait. By default, linear types do not conform to it, preserving the status quo.

Using the proposed ad-hoc syntax, we effectively create a new, anonymous sub-type inline that conforms to Deinitable using a custom deinitialization function. The collection or algorithm’s parametric function is then specialized for this new type, and the original type is cast to it at the boundary.

Structural Prerequisites

Fully realizing this approach requires a progression of language features conceptually similar to struct inheritance or named extensions:

  1. Global Struct Inheritance / Named Extensions: The ability to declare a new named type that inherits from an existing struct, adding or overriding conformances globally.
  2. Local Struct Inheritance: Scoping these named extensions or inherited types locally to a specific block or context.
  3. Anonymous Struct Extensions: Finally, the ability to define these inherited types anonymously and inline at the callsite (the core mechanism for this alternative).

Design Space & Orthogonal Behaviors

When evaluating trait conformances, there are three orthogonal behaviors to consider:

  1. Type Creation: Does it create a new sub-type, or apply to the existing type?
  2. Behavior Modification: Does it extend the type with a new conformance, or override an existing one?
  3. Scope: Is the conformance global, or local to a specific context?

Here is a comparison of approaches within this design space:

Scope Modification Type Identity Description & Viability
Global Extend Existing Type Applies new conformances globally (Mojo’s existing Extension feature).
Global / Local Override Existing Type Highly dangerous. Doesn’t create a new type. Existing code relying on the original behavior could silently break or produce incorrect results.
Local Extend Existing Type Similar to the global Extension but local to a context. Affects existing code called indirectly from the context. Might also be dangerous.
Global Extend / Override New Sub-type Structurally equivalent to struct inheritance or composition, delegation, IoC, and manual orchestration.
Local Override New Sub-type Requires explicit casting between the new and old types. Only usable in parametric code. An interesting future extension, but not strictly necessary for linear types.
Local Extend New Sub-type The Focus of This Approach. The most conservative approach. It temporarily conforms linear types within a single function, entirely solving their inherent lack of Deinitable conformance.

Method Implementation: Function Pointers vs. Closures

Should ad-hoc conforming methods be strictly thin function pointers, or do they need to be closures?

  • Function Pointers Only: Sufficient for non-raising linear types.
  • Closures: Raising linear types cannot natively conform to the Deinitable trait (which requires a non-raising __deinit__ function). Closures allow us to wrap a raising deinitialization process into a non-raising __deinit__ function by handling the exception internally (see example above).

Semantic Model of Closures as Methods

Ignoring performance and memory layout optimization for a moment, here is how closures fit semantically into this model:

  • Struct: Semantically equivalent to a list of function pointers plus struct data (modeled as an explicit self argument).
  • Closure: Semantically equivalent to a struct with a single function pointer plus user data (modeled as the closure’s capture set).
  • Struct with Closure Methods (Ad-Hoc Conformance): Semantically equivalent to struct function pointers + closure function pointers + struct state + closure state. This closely mirrors struct composition or inheritance, where function pointers and state are added or replaced.

Examples in Practice

1. Conforming a Linear Type with Trivial Deinitialization
For a collection of linear types that can be trivially deinitialized, we can provide the conformance inline using initializer-list syntax:

# Destroys the list with trivially deinitable for optimization.
lst^.__deinit__(T={__deinit__=nop, __is_trivially_deinitable=True})

# Equivalent to:
struct Anonymous(Linear, Deinitable):

    comptime __is_trivially_deinitable = True

    def __deinit__(var self):
        forget_deinit(self^)

2. Overriding an Existing Conformance on the Same Collection
Ad-hoc conformance is particularly powerful when you need to dynamically swap out an existing trait conformance on the same container instance.
Here is an example where we override String’s default Comparable conformance in two different ways. This allows us to sort a list differently without having to allocate new mapped collections or wrapper structs:

var files = List[String]("script.sh", "Data.csv", "archive.tar.gz", "config.json")

# Override 1: Sort the list case-insensitively
files.sort(
    T.Comparable={
        __lt__=case_insensitive_lt, 
        __eq__=case_insensitive_eq
    }
)

# Override 2: Call the exact same method, but override the trait to sort by file extension
files.sort(
    T.Comparable={
        __lt__=extension_lt, 
        __eq__=extension_eq
    }
)

This design might be cleaner than the common workaround of passing an external comparator function. It allows the sort function to simply rely on the type’s native < and == operators.

I’d be curious to hear what the core team thinks about this alternative direction compared to the Parameterized Deinitialization approach.

5

I like the ideas here, especially avoiding the set of _with APIs for linear types. I wonder if we can solve this more directly without parameterized/ad-hoc trait conformances.

What if APIs that destroy values simply take a deinitializer, defaulted to T.__deinit__? (this requires a bit of compiler support/fixes but is likely doable).

fn clear(mut self, deinit: Some[def(var T)] = T.__deinit__):
    ...

For normal Deinitable types nothing changes:

strings.clear()

while linear types can explicitly provide how they should be consumed:

files.clear(deinit=File.close)
# or
files.clear(deinint=lambda (var f: File): ...)`

I like this distinction because Deinitable continues to mean “this type has a canonical/default way to destroy itself”, while linear types can have multiple valid ways to be consumed (close, commit, rollback, forget, etc.).

For trivial destruction, I think the compiler could determine this from the selected deinitializer itself:

is_trivial_deinitializer[deinit]()

E.g., something that only does forget_deinit(self^) could be recognized as trivial, while File.close would not. I like this for “trivial” deinitizlier for linear types - as the explicit deinitializer may be required to satisfy the type system but has zero runtime cost.

I’m a little hesitant about the ad-hoc conformance approach because it seems like a much larger feature than we need here. Once we allow something like:

foo(T.Deinitable={__deinit__=close})

we need to define how that conformance propagates into nested generic calls, interacts with existing/conditional conformances, affects type identity, etc.

Those may be useful problems to solve independently, but I’m not sure we need to solve them for this use case.

6

Thanks for the feedback! You raise some great points. Here are my thoughts:

The main downside here is that we lose the ergonomics of Mojo’s ASAP destruction inside the algorithms that consume the values. If an algorithm takes a custom deinitializer function, the author of that algorithm has to manually insert deinit(val) calls on every code path where a value is dropped, rather than just letting it fall out of scope.

While this might be fine for simple operations like list.clear(), it becomes tedious for more complex algorithms (essentially any method that requires Deinitable). Passing a deinitializer manually requires algorithm authors to opt out of the ASAP deinitialization ergonomics and efficiency.

I completely agree that it’s a much larger feature, but I think it offers cleaner semantics overall. By creating a subtype that conforms to Deinitable and casting it, we solve the problem in a way that naturally extends the type system and has other valuable use cases beyond just linear deinitialization. Given that it touches on subtyping, it might be an approach worth looking at alongside the design of classes, which are already on the roadmap.

I touched on this briefly, but I believe it naturally resolves itself through monomorphization if we restrict it to parametric functions. If we specialize the function for the ad-hoc subtype, the compiler knows exactly which type is being passed. Any nested generic calls that use the parameter would simply be specialized for that same ad-hoc subtype, propagating the conformance seamlessly.

That said, I agree it introduces complexity and would require careful design exploration if we decide to pursue it.

Absolutely. I suspect many of these challenges—type identity, existing/conditional conformances, etc.—are very similar to the problems that will need to be solved for implementing classes and subtyping anyway.