# \[stdlib\] \[proposal\] Codepoint as UTF-8 readonly view

**URL:** <https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859>\
**Category:** Standard Library\
**Tags:** discussion, mojo-proposal\
**Created:** [March 18, 2026, 9:14pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859 "2026-03-18T21:14:57Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![martinvuyk](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/martinvuyk/32/48_2.png) [@martinvuyk](https://forum.modular.com/u/martinvuyk)\
**Post date:** [March 18, 2026, 9:14pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/1 "2026-03-18T21:14:57Z")

</div>

[proposal PR link](https://github.com/modular/modular/pull/6200)

---

<div class="post-metadata">

**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\
**Post date:** [March 19, 2026, 8:34pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/2 "2026-03-19T20:34:33Z")

</div>

I think we should take a very hard look at what Windows and WASM ↔ JS support will require before we toss out the parameterized version. I don’t think we have any option aside from that if we want support that isn’t horrible. Doing that also resolves all of the complaints around the `"aaaa"[byte=1]` syntax as well.

If we don’t do that, then I think the linear typed reference for mutable views which requires a fixup makes the most sense.

> Since we won’t be dealing with origins

How do you plan to not have origins in the iterator?

---

<div class="post-metadata">

**Author:** ![martinvuyk](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/martinvuyk/32/48_2.png) [@martinvuyk](https://forum.modular.com/u/martinvuyk)\
**Post date:** [March 19, 2026, 8:58pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/3 "2026-03-19T20:58:56Z")

</div>

> [@owenhilyard](#):
>
> I think we should take a very hard look at what Windows and WASM ↔ JS support will require before we toss out the parameterized version. I don’t think we have any option aside from that if we want support that isn’t horrible. Doing that also resolves all of the complaints around the `"aaaa"[byte=1]` syntax as well.

Yeah I would also prefer parametrization, but doing it UTF-8 native for now doesn’t really make anything harder. The inner storage would still be 4 bytes regardless of the encoding (in fact a parametrized version would just bitcast the internal bytes)

> [@owenhilyard](#):
>
> How do you plan to not have origins in the iterator?

By copying 1-4 bytes for each unicode codepoint into the inline `SIMD[DType.uint8, 4]` storage. The string iterator reads the first byte of each string anyway to know how many bytes to jump forwards

---

<div class="post-metadata">

**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\
**Post date:** [March 20, 2026, 12:42am UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/4 "2026-03-20T00:42:00Z")

</div>

We need the origin to make sure the string still exists while you’re iterating over it.

---

<div class="post-metadata">

**Author:** ![martinvuyk](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/martinvuyk/32/48_2.png) [@martinvuyk](https://forum.modular.com/u/martinvuyk)\
**Post date:** [March 20, 2026, 12:06pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/5 "2026-03-20T12:06:44Z")

</div>

The iterator struct will probably still carry around the origin as a parameter, but we need the iterator values themselves to not be bound to an origin for things like `replace`, `lower` and `upper` to return completely new values (that don’t originate from the source data)

---

<div class="post-metadata">

**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\
**Post date:** [March 20, 2026, 2:26pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/6 "2026-03-20T14:26:26Z")

</div>

Oh, I see what you mean.

I disagree.

We should be trying to do zero copy as much as possible. If someone wants to do copies, they can move it to a codepoint iterator instead of a string span iterator.

---

<div class="post-metadata">

**Author:** ![martinvuyk](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/martinvuyk/32/48_2.png) [@martinvuyk](https://forum.modular.com/u/martinvuyk)\
**Post date:** [March 20, 2026, 2:59pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/7 "2026-03-20T14:59:44Z")

</div>

I’m still sketching some of it out mentally, but to give you concrete examples.

This is more or less what the current slice iterator does

```mojo
struct CodepointSliceIter[str_origin]:
  var slice: StringSlice[Self.str_origin]

  fn __next__ (mut self) raises StopIteration -> StringSlice[Self.str_origin]:
    var length = _utf8_first_byte_sequence_length(self.slice.as_bytes()[0])
    var ptr = self.slice.ptr
    self.slice = {ptr=ptr + length, length=self.slice.byte_length() - length}
    return {ptr=ptr, length=length}

```

I want to inline that data (1-4 bytes per codepoint) since we are dereferencing it anyway (we need to know the sequence length from the first byte).

What making the data inline and not related to an origin means is that we can do this (which is necessary):

```mojo
struct LowerIter[str_origin]:
  var slice: StringSlice[Self.str_origin]

  fn __next__ (mut self) raises StopIteration -> Codepoint:
    var length = _utf8_first_byte_sequence_length(self.slice.as_bytes()[0])
    var ptr = self.slice.ptr
    self.slice = {ptr=ptr + length, length=self.slice.byte_length() - length}
    # invented method, might return the same value or a new one based on the
    # unicode standard. But the value itself can't be bound to an origin it
    # doesn't belong to str_origin
    return Codepoint(ptr=ptr, length=length).lower()

```

What I am doubting however is how can we safely chain several `Iterator[Codepoint]` together. We will probably have to keep the struct pointing to the origin of the other iterator, something like

```mojo
struct LowerIter[iter_origin]:
  comptime iteratorType = Iterator[Codepoint, Self.iter_origin]
  var slice: Self.iteratorType

  fn __init__ (out self, ref[Self.iter_origin] iterator: Self.iteratorType): ...

  fn __next__ (mut self) raises StopIteration -> Codepoint:
    var length = _utf8_first_byte_sequence_length(self.slice.as_bytes()[0])
    var ptr = self.slice.ptr
    self.slice = {ptr=ptr + length, length=self.slice.byte_length() - length}
    return Codepoint(ptr=ptr, length=length).lower()

```

I do think that we’ll need for example the `GraphemeIter` to return `StringSlice` due to the data being potentially big. But for many other string methods I think it makes sense to just inline the data and thus allow new value creation within the iterators and allow a sort of “fusion” of them when chained

---

<div class="post-metadata">

**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\
**Post date:** [March 20, 2026, 3:03pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/8 "2026-03-20T15:03:29Z")

</div>

I think we’d need to have a type which is either an owning view of a single codepoint or a mutable borrowed view. My assumption is that most codepoints remain unchanged, so we’d want to hand out big spans of codepoints in many cases.

---

<div class="post-metadata">

**Author:** ![martinvuyk](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/martinvuyk/32/48_2.png) [@martinvuyk](https://forum.modular.com/u/martinvuyk)\
**Post date:** [March 20, 2026, 4:31pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/9 "2026-03-20T16:31:08Z")

</div>

> [@owenhilyard](#):
>
> I think we’d need to have a type which is either an owning view of a single codepoint or a mutable borrowed view.

I don’t think we should allow mutable string iteration at all. I can’t think of a single use-case where that is a good idea (other than ASCII vectorized ops).

> [@owenhilyard](#):
>
> My assumption is that most codepoints remain unchanged, so we’d want to hand out big spans of codepoints in many cases.

I agree, many iterators like `split` and `split_lines` should (like they currently do) return `StringSlice` when possible. But there are several cases where if they receive an `Iterator[Codepoint]` they won’t be able to construct a `StringSlice` since it doesn’t necessarily have a “pure” memory origin (values can get changed within). An example:

```mojo
var data = "Something;Something_else"

# s is a `StringSlice`
for s in data.split(";"):
  if s in ("something", "Something"): ...

# s is a `Codepoint` since it can switch e.g. s[0] from "S" to "s"
for s in s.lower(): ...

# s is an `Iterator[Codepoint]` since it can switch e.g. s[0] from "S" to "s"
for s in s.lower().split(";"):
  # `Iterator[Codepoint]` should be comparable to a string
  if s == "something": ...

# s is a `StringSlice`
for s in s.split(";"):
  # `Iterator[Codepoint]` should be comparable to a string
  if s.lower() == "something": ...

```

---

<div class="post-metadata">

**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\
**Post date:** [March 20, 2026, 4:34pm UTC](https://forum.modular.com/t/stdlib-proposal-codepoint-as-utf-8-readonly-view/2859/10 "2026-03-20T16:34:03Z")

</div>

I wonder if we need something like [bytes - Rust](https://docs.rs/bytes/latest/bytes/)