`deinit` proposal

Mojo Language Design
82

I also find the name confusing. In another programming language, deinit could as well mean to invoke the destructor, but in mojo it is exactly the opposite.

83
84

+1 to decorator

If we would stick to implicitly injecting code to disable destructor, as it is done currently, decorator is IMO much more reasonable as it emphasizes modifying behavior of decorated object while separated argument convention for this looks very magical.

85

I agree, and apologies for not explicitly saying it: deconstruct should only be usable from a method of that class. For now, outside users should not be able to deconstruct anything they’d like.

An interesting potential I want to mention: I like the ComptimeArcPointer case that @owenhilyard mentioned. The case from my talk whereLiveEntityHandle can be destroyed by LiveEntityList is similar to that. I think @clattner‘s solution of having LiveEntityList call a private LiveEntityHandle::_destroy(self) method is reasonable, especially if long-term we find a way to make it so only LiveEntityList can call it, perhaps via friend or a simpler Java-style package access control. I think getting this aspect right, combined with good memory safety design, could make it so Mojo’s linear types enable memory-safe back-references, which would be a massive win for us and also unlock a lot of OO patterns. If anyone wants me to expound on that, let me know and I can start a different thread about it.

Maybe I’m misunderstanding, but is that true of deinit? I’m not seeing anything in the proposal that says that we can put fields back / reassemble an object when inside a deiniting method. I would assume it can, but not sure.

If it helps, the imperative deconstructstatement composes cleanly with the reassembling mechanic:

struct Spaceship:
    var engine: Engine
    var fuel: Int
    ...
    fn __del__(var self):
        deconstruct self # we now own the fields
        self.engine^.explode()
        self.engine = Engine("Warp Engine")
        # At this point, `self` is whole again
        ... # presumably the user would deconstruct self again or throw self on a free-list or something

It’s unusual, and I can’t think of any cases off the top of my head for it. But it’s a good sign from the universe that this composes with / is decoupled from the putting-fields-back mechanism.

Actually, I take that back, I might know some cases. I vaguely recall times in the Google Chat android app where a Fragment/View’s “destructor” extracts some fields, replaces them with “empty-ish” values, and then threw this “ready for reusing” self onto a free-list (ask me how I know this, that’s a fun story). I wouldn’t be surprised if some virtual-dom-heavy web frameworks did this occasionally as well. The fact that deconstruct doesn’t interfere with reassembling could be a good thing.

Composability, huzzah!

That would be great, that would save me some time (to implement more features!) and give us something concrete to compare to the deinit proposal. No rush, I know your job takes a lot of your time as well. Thanks Nick!

86

We can already move fields away without this keyword, we just need to put them back so that x is in a properly initialized state at the end of the function. We can explain this to users this way:

Maybe I’m misunderstanding, but is that true of deinit? I’m not seeing anything in the proposal that says that we can put fields back / reassemble an object when inside a deiniting method. I would assume it can, but not sure.

What I’m trying to say is that in a normal function, you can move a field away on a mut self if you put something back before the function ends. Thus, the ability to take ownership of fields isn’t granted by deinit or deconstruct. Now, consider the following code:

@explicit_destroy
@fieldwise_init
struct WarpCore(Movable, Copyable):
    var id: UInt

@fieldwise_init
struct Engine(Movable):
    var id: UInt
    var warp_cores: List[WarpCore]

    fn __del__(deinit self):
        try:
            print("Engine {} destroyed".format(self.id))
            print("{} remaining warp cores".format(len(self.warp_cores)))
        except:
            pass

    fn print_id(self):
        print("Engine ID:", self.id)

@fieldwise_init
struct Spaceship(Movable):
    var engine: Engine
    var fuel: Int
    var used_warp_core_containment: List[WarpCore]


    fn engine_upgrade(mut self, var engine: Engine):
        var old_engine = self.engine^

        # disassemble old engine by removing warp cores
        for _ in range(len(old_engine.warp_cores)):
            self.used_warp_core_containment.append(old_engine.warp_cores.pop())
        
        # old_engine is now inert
        _ = List[WarpCore].__moveinit__(old_engine.warp_cores^)

       # old_engine.print_id()

        self.engine = engine^

When compiled, the compiler complains that field 'old_engine.warp_cores' destroyed out of the middle of a value, preventing the overall value from being destroyed. deinit/deconstruct gives us the ability to make that error go away, which means they’re telling the compiler to not call old_engine.__del__(), and not trying to call a function with a partially uninitialized value stops the error. If we don’t want __del__ to be special, it should conform to the same rules as other functions in that regard, and should give the same use of uninitialized value 'old_engine.warp_cores' that uncommenting old_engine.print_id() does.

I think that for a deconstruct statement, the following works well as implementation pseudocode:

fn deconstruct(var x):
    @parameter
    for field in x.fields: # owning iterator of magic static reflection values
        @parameter
        if field.is_init():
            field^.__del__() # destroy the field
    __disable_del x

This also means that, depending on how far we push static reflection and macros in the language, we might eventually be able to lift deconstruct out of the compiler.

If we want to go the “deconstruct x gives you ownership of the fields of x” route with a deconstruct statement, where deconstruct x is done before you start moving away fields, I think we need to determine semantics for how ownership transfer without a move work first, since we can’t move the fields of x or else you can’t have a linear type with an atomic variable in it.

I think that I’m moving away from liking the declarative version as I think more about how this will be implemented.

87

I don’t see why it is “better” for the position to matter. You can already move the fields away. There isn’t anything gained by such approach IMO.

I agree that the declarative design cuts off stuff like this, but I consider that to be a feature, not a bug. It provides a simple funnel point that is a dual to initializers.

Yes, I can see this concern, this is also pushed by wanting to make this terse and feel “argument convention like” even though it really isn’t (as many folks have pointed out).

New proposed approach

Summarizing the feedback I’m hearing:

  • there are some people that like the expressive capability of a statement - I remain very unmotivated by this. I would rather have a simple feature that is intentionally limited that is easy to explain, and I would like to tie it into “this method is a destructor” in a declarative way.
  • No one likes confusing this into an argument convention, but there is support for a decorator.
  • I want this to be simple and explainable, and low boilerplate.

How about this as a new direction:

  1. We change this to be a decorator with a long name, e.g. @destructor which has the same semantics as currently implemented. These methods are required to take a var self.

  2. You can therefore implement “named destructors” as well as implicit destructors using this decorator:

struct YourType:

    @destructor
    fn explode(var self, how_fast: Int): ...

    @destructor
    fn __del__(var self): ...

  1. I don’t like the boilerplate of adding this to every __del__ - so let’s just make that implicit. There is precedent for this in Python, where __new__ is implicitly a static method without having to declare it as such.

  2. We extend the del handling to support transferring the entire object away even if marked as @destructor . All together this would allow things like this to “just work”:

struct YourType:

    @destructor
    fn explode(var self, how_fast: Int): ...

    fn __del__(var self):
      self^.explode(42)

We would document this as a way to define “named destructors” and explain the behavior. This solves the argument confusion problem, and keeps it concise and limited - a dual to initializers.

Thoughts?

-Chris

1 Like
88

This would have been a foot gun anyway. Is there a reason to recursively call the destructor. Destructors are already special, because they are automatically inserted.

I also favor the go approach with simple and orthogonal features. Even though I don’t which feature go had chosen if it did not have garbage collector. However I remain convinced this resembles the spirit of go.

I am more satisfied with the name destructor. Naming is hard, if you negate the name, it becomes clearer but is this also more concise? Nevertheless destructor is a proper noun, thus there is less room for interpretation and there are only 2 outcomes.

-Dominic

1 Like
89

I like this design a lot better than deinit! That said, I think the deconstruct statement might have some significant advantages, so I will write it up as a formal proposal so that we can compare it against @destructor.

While we wait for that, here is my feedback your new design.

Firstly, “destructor” isn’t the right term, is it? Is __moveinit__ a destructor? Maybe @deconstruct would be a more accurate name. The feature we’re talking about is the phenomenon of “taking apart” an object piece-by-piece, emptying out its memory. __moveinit__ doesn’t really destroy anything, it just dismantles an object, salvaging the pieces to build another object.

In fact, I recall that Modular is considering replacing __moveinit__ and __copyinit__ with overloads of __init__. In that case, __moveinit__ is actually a constructor! It would certainly be strange to annotate a constructor with the word @destructor. :thinking:

I disagree. How often does a programmer write a custom destructor implementation? This will be very rare outside of the standard library, so I don’t see the benefit to leaving out the decorator for __del__. I’d much rather have a design that is uniform.

I don’t think we should be creating the unnecessary burden of requiring every Mojo tutorial to explain how __del__ is magical/different in this respect.

Agreed. I’m surprised to hear that this doesn’t already work.

Again—how would we explain __moveinit__?

90

Yes :+1: C.L and here is an simple idea to solve moveinit:

@destructor applies to every var arg of type Self of an method.

struct MyType:
    fn __moveinit__(init self, var other: Self):
        self.ptr = other^.scavage_pointer()
    @destructor
    fn scavage_pointer(var self) -> UnsafePointer[Int]):
        return self.ptr

struct SpaceShip:
    @destructor
    fn scavage_booster(var self, out booster: Booster):
        # This method safely scavage the booster
        # it is an complicated thing, so this method handle that.
        self.hull^.jettison() # jettison hull
        booster = self.booster^ # extract booster
        self.field1^.__del__() # move this one here

 

How do we explain __moveinit__

(Here is an attempt Nick)

What is __moveinit__ ?
It is just an constructor.
This constructors takes the arg other as var (owned value).

Why?
Instead of doing an copy of the data like __copyinit__,
having the ownership makes it possible to move the ownership of the data.

What is ownership ?
An owner is responsible for calling the destructor on the data,
this is why an change of ownership involve an custom destructor.

The argument other can go trough an scavaging destructor instead of the default __del__.
An custom destructor method can be created with @destructor decorator.

For example, the default destructor of an type (__del__) would free an pointer.
And an custom scavaging destructor would return that pointer but not free it.

This is how we moved the ownership of the data, instead of moving the data.

After the move, the previous owner did not free the data,
and the new owner have still an default destructor that will free the data.

This move tie in with the origin system:
Any existing pointer to the previous owner are not valid to dereference anymore,
since the new owner might have destructed it already.

1 Like
91

I believe we have retread all this ground many times over at this point. I am not convinced at all, but feel free to write up your thoughts, it could be a useful way to frame the argument.

Yes, moveinit can be a named destructor, even though it is also a constructor. When used with the @destructor decorator, move init destroys the value it runs on, and constructs a new value. It is both a destructor of the old value and a constructor of the new value.

That said, moveinit doesn’t /have/ to destruct the value, being “var” is required, but @destructor is optional. When you don’t use the @destructor decorator, Mojo will run the implicit destructor for you.

As we discussed upthread, while many people don’t think about it this way, moveinit acts as a method on an existing value - the design is consistent.

-Chris

92

While I disagree with your conclusions, there is something I like about your proposals, Chris. They have a certain ergonomic simplicity. They work for as high as 90-99% of cases without complications. You can explain how they should use it, and they can just use it and continue on with their lives. You have really high standards for this aspect, which is good because we don’t want users to swim in complexity that they shouldn’t have to care about. @destructor is really nice on these dimensions.

I’ll explain what I value about deconstruct:

  • deconstruct is decoupled from unrelated features. deinit raises questions about how it overlaps with var, how it interferes with the reassembling mechanic, and about how it affects the callsite (we know it doesn’t, but the question is raised). @destructor raises some of those questions, plus one about other var arguments.
  • deconstruct is composable. The user can deconstruct an object, move a field away, reinitialize the field, and toss the new object onto a free-list. It’s unusual, but hey, we don’t judge.
  • deconstruct builds on what the user already knows. They know about moving, and that’s all this is.
  • …which also makes it easy to explain. “It takes ownership of self, and gives you ownership of its fields.”
  • deconstruct gives the user has more freedom. Not as an absolute, but when the user can use your building blocks in new creative ways, that’s a nice thing. Especially because we’re a systems programming language.
  • It’s precise and flexible. We could use it to destruct non-self arguments in theory. @destructor appears to cut that option off. Maybe we want that, maybe we don’t, but that @destructor forces our hand is a hint from the universe that we should hesitate.
  • It reveals a truth to the user, that a “destructor” is just a composition of more fundamental concepts:
    • A function that’s automatically called when something goes out of scope (__del__)
    • The choice to use deconstructon self.
    • (This is also the truth that completely solves the __moveinit__ question)

That’s simplicity.

It’s a “fundamental simplicity” as opposed to @destructor’s “ergonomic simplicity”, in my terms.

If we had to choose between the two, I personally would choose the former every time. Fundamentally solid big rocks first, after all.

That said, maybe we don’t have to choose. I’d love to make deconstruct more ergonomic. Some promising angles:

  • Error messages will help, and be easy to code.
  • 90%+ (guessing) of destructors are auto-generated, which is nice.
  • An annotation for the common case perhaps.

That last item raises an interesting question: I wonder if we could have deconstruct, and then also offer @destructor as syntactic sugar to automatically deconstruct self at the right time. That way, most users can just use @destructor, and they can drop down to deconstruct when they want more freedom and power. It wouldn’t cut off any options, it would give users more power, and it would be ergonomically simple for the 90-99% of cases that don’t care about the power.

2 Likes
93

@clattner I think I might be able to replace the deconstruct statement with a solution that is entirely library-defined, so that destruction doesn’t require any new syntax.

The vague idea is for all structs to have a private method named self^.contents_ptr() that surrenders ownership of the object in exchange for an exclusive pointer to the contents of the object. Once you have that, you can move the data out field-by-field. This pointer type would be defined on top of a variant of lit.ref that does fieldwise destruction of the remaining fields when it is dropped. In other words, we would offer the equivalent of deinit through an MLIR type, rather than making it an argument convention.

Example:

struct Foo:
    var x: String
    var y: String
    fn __moveinit__(out self, var other):
        var c = other^.contents_ptr()
        self.x = c[].x^
        self.y = c[].y^

Notes:

  • If we wanted to avoid the [] boilerplate, we could either replace the pointer with a second-class ref, or we could add “auto-deref” support for pointers, like in Rust. This would allow us to write self.x = c.x^.
  • This design would need to tie into Mojo’s origin system—we need to prevent the use of other while the pointer c is live. (We can probably use “indirect origins” to model this constraint.)
  • The compiler needs to be able to keep track of the initialization state of the pointee’s fields. I address this concern at the end of my post.
  • contents_ptr would be defined on AnyType, and abuse of this method can be prevented by making it private. (While we wait for Mojo to add privacy, we can prefix it with __unsafe_.)

I need to spend more time on this design, but I think it could be the simplest design yet. It means that the concept of destruction would be a library feature built on top of lit.ref, rather than being baked into the language. All else equal, a “smaller Mojo” is better!

Furthermore, this design would allow the Mojo community to invent new destruction approaches, all built on top of a modified lit.ref. This is better than forcing Mojo users into one specific design pattern for value destruction, with its associated limitations.

Prerequisites

Note: If we define other^.contents() to return a second class -> ref, rather than a first-class pointer, the following feature is not technically required.

Required feature (?):
This design requires the ability to support transferring (^) fields out of a pointer’s target. (Both for Pointer, and this new “contents pointer”.)

This is a feature that Mojo needs anyway IMO, and I believe I know a means of achieving this via tweaking the definition of Pointer. In short: the internal lit.ref should be stored in a field named __item__, and the p[] syntax should desugar to something like __get_address_as_lvalue(p.__item__). This lets the compiler associate the pointee with a specific object, and therefore track the initialization state of the pointee’s fields. The compiler would need to forbid the mutation of __item__ (i.e. the address) within the window where one or more of the fields are uninitialized.

94

I just had a chance to read Mojo’s new vision doc and roadmap, and I must say, I agree wholeheartedly with the perspective on language design. This is the perspective I’ve been holding while trying to find a “clean” design for destructors.

Vision doc:

  • Emphasize composability and simplicity: Every Mojo feature must work reliably in all situations and combine seamlessly with other features (compose orthogonally). We’re not satisfied with features that work 80% of the time but fail in edge cases.
  • Defer syntactic sugar: Language sugar is often tempting, but we prioritize core “big rocks” first. Only once the fundamentals are solid do we revisit syntactic enhancements.

Roadmap:

we must resist the urge to chase short-term wins at the expense of long-term clarity, consistency, and quality.

:construction: Attribute macros: Replacing ad-hoc constructs like @register_passable("trivial"), @nonmaterializable, @value, etc., using traits and other existing language features (shrinking the language).

This last quote is especially relevant. I don’t feel comfortable with the idea of adding a new decorator to manipulate the behaviour of functions that take var self. It’s too magical IMO. Value destruction isn’t conceptually complicated, so we should be able to model it using a small, clean, composable primitive.

This is why I’m thinking we should introduce a variant of lit.ref (MLIR type) that does exactly what we want: treat its target as a raw bundle of fields that must be transferred away. We can then build ergonomic abstractions on top of this, using language features that already exist in Mojo! This allows us to avoid introducing a new syntax specifically for destructors.

This approach is consistent with the vision doc (“syntactic sugar is tempting, but we prioritize the core language first”) and the roadmap (“replacing ad-hoc constructs”; “shrinking the language”).

I believe we have an opportunity to find a design that’s small, simple, and beautiful.

1 Like
95
  • deconstruct builds on what the user already knows. They know about moving, and that’s all this is.
  • …which also makes it easy to explain. “It takes ownership of self, and gives you ownership of its fields.”

To me, this implies that deconstruct moves the fields. I’m strongly against that aspect if that’s what you intended because it would force using escape hatches to destroy anything with an atomic in it.

It’s precise and flexible. We could use it to destruct non-self arguments in theory. @destructor appears to cut that option off. Maybe we want that, maybe we don’t, but that @destructor forces our hand is a hint from the universe that we should hesitate.

I agree, I can envision a syntax for non-self destruction with the decorator, but I think that it might be more confusing than deconstruct. ex:

@destructor("other")
fn deconstruct_both_args(var self, var other: Self):
    ...

To the uninformed reader, this seems like it deconstructs only other, but it implicitly deconstructs self. if it doesn’t, then you have to write @deconstruct("self", "other"). To me, this feels like no matter what it’s special casing some behavior for destructors.

What I could see as a middle ground approach, if we make use of a terminal deconstruct x where it destroys the remaining fields in x, is to make @destructor a shorthand for "stick deconstruct self at the last line before every return statement in the function and before the function terminates. That keeps the “normal data type” path nice and short, and then if you want any other behavior then an LSP action to “expand” the decorator can add all of the deconstruct self instances and then you make your additions from there. It also keeps the implementation of the decorator simple.

This means that the default destructor would be:

@destructor
fn __del__(var self):
    pass

Which expands to:

fn __del__(var self):
    deconstruct self

Which, once we have some sort of static reflection capabilities in the language, can expand to (made-up syntax):

fn __del__(var self):
    for field in reflect.fields(self):
        if field.is_init(): # only there for more complex cases
           reflect.get_owned(self, field)^.__del__()
    __disable_del self #or unsafe replacement

Which removes any “compiler magic” from the process and provides an easy pseudocode explanation for what deconstruct does. This whole chain also means that, once users can write decorators, they can write a different decorator if they want different behavior instead of having deconstruct be a special macro that changes the compiler’s rules.

I think this lines up with what you’re asking for in your last paragraph, but it does involve slightly changing what deconstruct does.

edit: After talking with Nick more, the transfer sigil should work for this, I’ve run into a lot of things around immovable types that made me thing it wouldn’t.

1 Like
96

I like the change in direction towards a decorator instead of “muddying the water” with no-op argument convention changes.

I like the ideas of trying to make deinitialization be able to be done field-wise (and possibly reinitialized before the end of the scope), I never quite understood why deinit is limited to Self arguments.

If we can add more traits like AnyType that are automatically inherited with reflection and different destruction mechanics (or create decorators that insert the code), that is also much better IMO than hard-coding behavior in the compiler.

Just a quick example (with invented syntax) to give an idea of why being able to “deconstruct” other arguments might be valuable:

fn some_process(
  replace_val: List[Int], data: Dict[String, List[Int]], out removed: List[Int]
) raises:
  for read key, destruct mut value in data.items():
    if key == "some specific key":
      removed = value^
      # value is deinitialized, has to be reinitialized before exiting the scope
      value = replace_val
      return
  raise Error("key not found")
97

My only question with using a decorator is this: how does it fit into a future where the constructors are consolidated into just an __init__?

98

The behaviour shown in your example should work for ordinary refs that point to mutable data. It’s not a feature that would/should be associated with destructors.

@clattner It seems Mojo doesn’t support moving out of local refs yet. I assume this is planned at some point? For example:

words: List[String] = ["one", "two", "three"]
for ref word in words:
    foo(word^)
    word = "empty"

To make this safe, we would need to reason that word and words alias, and therefore the latter object should not be readable during the window where the former object is deinitialized.

I think the idea is for the move constructor to be declared as follows:

fn Thing:
    var x: Int
    @destructor
    fn __init__(out self, *, move: Self):
        self.x = move.x^

Here, the @destructor decorator is acting on the second argument, because it’s the first “non-out argument”.

I find this very unintuitive, but I believe that’s what the proposal is.

99

No, I don’t expect ownership transferring ref’s to be a thing, at least in the foreseeable future. It is possible we can add them some point in the distant future, but I’d want roughly-everything-else to be nailed down first, so it wouldn’t be for quite some time.

The way we can support that is with a destroying iterator, e.g.:

for var elt in words^.consume_elements():

Such an iterator would yield (by consuming move) the elements that are produced, and would destroy any unconsumed elements (eg due to a break out of the loop). The result is that the entire collection is consumed by such an iterator.

-Chris

100

In the text you quoted I didn’t mean ownership transferring refs just FYI. I meant being able to temporarily move out of a local ref, as long as you put the value back again. That doesn’t seem to work in today’s mojo, despite it working for args.

101

Oh yes, sorry for the confusion - I could see supporting that, so long as you move something back into it.