One of the challenges in processing data, or any workflow, is that individual steps fail all the time. You need to minimize the amount of lost work and one option leans on retries and state management.
A different option is to leverage a durable execution engine like https://restate.dev (just one of the players in the ecosystem). The following package enables integration with restate .
The API ended up surprisingly ergonomic, in my very biased opinion . It leverages comptime for building trampolines and context managers on top of the threads tin. If you want to see code examples check the following blog post.
Feedback appreciated.