Struct Extensions Proposal

[Verdagon (Evan Ovadia)]

Welp, that was a long “week or two”

So, updates since last time:

• Around Oct 4, we landed the most conservative form of the feature (described in Intermediate struct extensions implementation ).
• Since then, most of my time has been on other things (traffic cop, vacation, LLVM conference, thanksgiving), but I did manage to get some bugfixes in. On Nov 12 (well, Nov 25 after a revert and re-merge), I successfully moved simd.mojo’s __extension SIMD: to other files (python/conversions.mojo and python/python_object.mojo), huzzah.

So, the most conservative form of struct extensions is in right now, and I have a moment to breath and look forward the final design a little bit.

Decision 1 (“Where can traitful/conforming extensions be?”):

1. Option A: Either in the struct’s package or the trait’s package.
2. Option B: Either in the struct’s file or the trait’s file.
3. Option C: Anyone can extend any struct for any trait.

I’m tentatively vetoing Option B since doesn’t support User Journey 1. Remaining choices are A and C.

Tentative conservative decision is A. We can relax this later, so I’m going to de-scope this for now.

We could relax this and allow traitful/conforming extensions in other packages, if we do something like Carbon does (thanks for the lead Nick!).

If anyone has strong opinions, please open a feature request!

Decision 2: Should we allow traitless/non-conforming extensions?

Resounding yes. Also I misspoke when I said Connor leans against this, he actually just doesn’t want traitless/non-conforming extensions outside the package of their target struct.

Also, our hand was kind of forced by Max here, they really wanted it, so it exists now. We can de-scope this for now.

Decision 3: Where can a traitless/non-conforming extensions be?

1. Option A: In the struct’s package.
2. Option B: In the struct’s file.
3. Option C: Anyone can add an extension anywhere for any struct.

I’m tentatively vetoing Option B since doesn’t support User Journey 1. Remaining choices are A and C.

Tentative conservative decisions is A. We can relax this later, so I’m going to de-scope this for now.

We could relax this and allow traitless/nonconforming extensions in other packages, if we have a nice way to import them, and maybe a nice way to disambiguate callsites who are accidentally calling methods from multiple overloads (“spooky action at a distance”).

If anyone has strong opinions, please open a feature request!

Decision 4: What decorators can be in an extension?

Tentative conservative decision is methods, aliases, requires clauses. No annotations. We can relax this later, so de-scoping for now.

(As Owen says, we might want some sort of @export decorator at some point)

Decision 5: Handling conflicts

The current behavior is that normal overload resolution rules apply to it, and if there is a tie, the user is kind of out of luck.

As Owen and Nate say, we’ll want a disambiguation syntax for when there’s a conflict. Perhaps L.Spaceship.fly_to(ship, “Corneria”).

Interestingly, this conflict isn’t possible today with top-level functions; we can’t import a foo and also define a function foo, we get an error on the latter’s definition because of the name conflict.

Yinon brought up an interesting possibility of renaming an extension on import. That might be really useful for this.

This is still in-scope, I want to figure this out before users gets into this situation.

Decision 6: Are extensions automatically imported?

• Option A: Explicitly import extensions
• Option B: Struct imports automatically import their extensions

People like A because it’s more explicit, and B because it’s more ergonomic. Tale as old as time!

Respectively:

• I conservatively implemented option A, and it turns out to be very un-ergonomic. For example, if a user wants to say var x = Float64(my_python_object) they need to put from python.conversions import SIMD to import the extension SIMD: in python/conversions.mojo. This is a bit of a deal-breaker; people using simple Mojo for Python stuff shouldn’t have to know what SIMD is. So, let’s consider option A was vetoed.
  • However, I’ll also mention an option A2: When main.mojo does from library_a.file_x import Spaceship, it imports any extensions in file_x (not just Spaceship extensions) and also any extensions in any files that file_ximports itself (as long as they’re in library_a).
• B is stated a bit imprecisely. It should be “If you can see a struct, you can see all extensions in the struct’s package.”

So it’s down to A2 vs B. A2 says that (direct+indirect) imports make an extension visible. B means that an extension’s presence in a package makes it visible.

This decision is still in scope, we’ll decide on it this week if all goes well.

Decision 7 (“Need import extension?”):

This was a pretty resounding no, but I gotta admit, it’s kind of growing on me. It could be useful if we want to have explicit control over which third-party traitless/non-conforming extensions we want to import.

However, it’s moot for first-party extensions, and we don’t support third-party traitless/non-conforming extensions yet. Let’s de-scope this. If anyone has strong opinions, please open a feature request!

Decision 8 (“What if we import only an extension but not its target struct?”):

As Yinon said, we accidentally got this for free. In this foo.mojo:

from A import Spaceship

extension Spaceship:
    fn barrel_roll(mut self):
        self.roll(360)
        maniacal_laughter()

when a user says from .foo import Spaceship, they actually will import the Spaceship because from A import Spaceship re-exported it.

Also, this is moot if we go with decision 6 option B.

Still, if we go with decision 6 option A2, we’ll need to decide this. Let’s keep it in scope.

Decision 9 (“Support importing multiple extensions?”):

Pretty resounding “yes”. Let’s consider this one closed.

Decision 10 (“Proposed Syntax”):

I’ll summarize this later.

Decision 11 (“What decorators can be on extensions’ methods”)

A lot of folks think we should have an allowlist. I actually think we shouldn’t, because I don’t really see why a method’s location should affect the decorators. I need to learn more about decorators. Let’s consider this still in-scope.

Other points

Josiah, I think you’re spot on. I encountered that exact same thing with Variant in my latest article about mojo metaprogramming. And yeah, i see how extensions could help with that, so really good point, thank you!

I am also discovering that there are some risks because we have both overloading + extensions. Spooky performance regressions at a distance.

rd4com has an interesting idea, a sort of way to define an extension up front, but then allowlist which types it applies to. This is kind of similar to defining a trait and a bunch of impls, then an extension on things that conform to that trait, I think. Let’s table this for now.

Nick, you had an interesting idea with using extensions with with. But I think we only need var x: Set[String with HashA] if Mojo made a misstep somewhere. In Java, they require a .hash()/.equals() method on the object itself, but in C++ they decouple the hashing logic from the object by having a separate hasher and equator. Maybe there’s another example we can use?

Summary

• Decision 1 (“Where can traitful/conforming extensions be?”): Either in the struct’s package or the trait’s package. We can relax this later, so de-scoping.
• Decision 2 (“Should we allow non-conforming extensions?”): Yes, considering closed.
• Decision 3 (“Where can non-conforming extensions be?”): In the struct’s package. We can relax this later, so de-scoping.
• Decision 4 (“What decorators can be in an extension?”): Methods, aliases, requires clauses. No annotations. We can relax this later, so de-scoping for now.
• Decision 5 (“Handling conflicts”): Still in-scope, I want to figure this out.
• Decision 6 (“Are extensions automatically imported?”): We’ll figure that out this week hopefully.
• Decision 7 (“Need import extension?”): Moot, so de-scoping for now.
• Decision 8 (“What if we import only an extension but not its target struct?”): Moot if we go with decision 6’s option B which we’ll figure out this week. We’ll see.
• Decision 9 (“Support importing multiple extensions?”): Yes. Closed.
• Decision 10 (“Proposed Syntax”): TBD.
• Decision 11 (new, “What decorators can be on extensions’ methods”): Still in-scope, I want to figure this out.

Summarizing the summary, remaining open decisions are:

• Decision 5 (“Handling conflicts”)
• Decision 6 (“Are extensions automatically imported?”)
• maybe Decision 8 (“What if we import only an extension but not its target struct?”)
• Decision 10 (“Proposed Syntax”)
• Decision 11 (new, “What decorators can be on extensions’ methods”)

Summarizing the summary of the summary: This week we’ll figure out decision 6, which will affect decision 8. Then we’ll think about the syntax and other details.

Summarizing the summary of the summary of the summary: struct extensions still goin’!