Stateful jobs

Almost every job scheduler we rely on today was built on the assumption that work is stateless. In fact, in the last 15 years we mostly designed stateless schedulers, or stateful executors for niche topics. In the stateless world, you have a pool of tasks, a pool of workers, some CPU and memory constraints, maybe a priority queue and dependencies, and your goal is to find an efficient placement. If a node dies, you reschedule the task elsewhere. If a task fails, you retry it. Scheduling is fundamentally an allocation and packing problem.

Once jobs carry state, almost every core assumption breaks down.

Stateful job scheduling isn’t just “scheduling with some volumes attached.” It turns scheduler design inside out. The hard part isn’t only about finding where there are free CPU cores. The hard part is also managing identity, data lifecycle, protocol and schema drift across restarts, application-level layering and often coordination, and dealing with corrupt state.

Placement

In a stateless world, worker identity is ephemeral. Task 42 running on node A is interchangeable with Task 42 running on node B.

In a stateful system, identity is tied to durability. If a job maintains an embedded store, a local checkpoint, an in-memory index, or a partition of a distributed stream, moving that job to another machine incurs a massive transfer penalty. If the scheduler moves it too casually, you risk thundering herds of I/O saturating your network and storage subsystems.

Placement decisions in this world are different. They become longer-lived commitments. You must balance:

• Keeping work pinned to existing local state and warm caches vs. rescheduling to balance hot spots. When a job is evicted, you lose warmed-up resources, turning low-latency reads into cold storage starts.
• Tolerating temporary node unreachability without prematurely triggering expensive state rebuilds elsewhere.
• Dealing with zombie workers so two instances of the same job never mutate the same underlying state concurrently.

Upgrades

Code changes are constant and state persists. In stateless cases, rolling upgrades are straightforward: spin up version N+1 instances, route traffic or queue items to it, and drain version N.

Stateful jobs are often long-running entities by design. They don’t always finish in minutes; they run for weeks, months. When an urgent security patch or critical CVE lands in an underlying library, you don’t always want to wait for the job to naturally complete. You are forced to upgrade the binary in-place without losing the job’s accumulated state or suffering extended downtime.

In this context, every upgrade is an operational migration:

• State compatibility: Does the new binary understand the checkpoint, snapshot, or storage layout written by the previous version? What happens if you need to roll back to version N after version N+1 has already made partial mutations?
• Urgent patching: When a CVE demands an immediate restart across the fleet, can the job checkpoint cleanly on short notice? How do you patch a critical vulnerability without corrupting days’ worth of in-flight progress or forcing a days-long recompute from scratch?
• Heterogeneous clusters: Because long-running jobs take time to safely transition, multiple versions run side by side for extended periods. If they share persistent storage, participate in the same consensus group, or stream to one another with subtle serialization differences, data corruption can happen silently long before the rollout finishes.

Layering with Applications

Schedulers typically try to treat workloads as black boxes. They monitor CPU, memory, exit codes, and health checks.

For stateful systems, treating the job as a black box is not always enough. The scheduler needs deep semantic knowledge of what the application is doing:

• What kind of state persistence do I need? Snapshots or volume mounts?
• Can the scheduler preempt this job right now, or is it midway through a multi-step commit that will require expensive compaction if killed?
• How to reestablish connections during a resumption case?
• When am I idle and can be suspended automatically?

Corruption

In a stateless world, the universal hammer for almost every operational failure is simple: kill the process and restart it. If a worker leaks memory, gets into a deadlock, or faces a growing risk of an OOM event, systems routinely bounce the container or task. In stateless architectures, periodic restarts are treated as standard operational cleanup to shed accumulated memory pressure.

In a stateful system, an OOM kill or an unclean crash is catastrophic.

Stateful processes can sit dangerously close to their memory limits by design. They manage in-memory caches, write buffers, memtables, and connection states. Long running nature of these jobs make things even worse.

Idleness

In stateless cases, idleness is more straightforward: if there are no incoming requests or queued tasks, you scale to zero. When traffic arrives, you cold-start a container in hundreds of milliseconds.

For stateful jobs, deciding what it means to be “idle” is subtle and expensive:

• Cost of suspension: Suspending a job isn’t free. You must flush in-flight buffers, write a consistent snapshot to remote storage, and tear down local execution context. If a new request arrives thirty seconds later, rehydrating the working set and warming up caches completely destroys tail latency.
• Background work vs. client work: A job might not be serving active external traffic, but it might be running background compaction, garbage-collecting older state, or maintaining heartbeat connections to consensus peers. The scheduler cannot assume zero traffic means zero work.
• Connection and state lifecycle: Waking a stateful job back up requires cleanly reestablishing socket connections, rebuilding leases, and verifying that the persistent state wasn’t modified or invalidated while asleep.

A stateful scheduler has to decide when it is actually worth paying the penalty to put a job to sleep, and give applications clean coordination primitives to signal whether they are genuinely idle or merely waiting.

Why Build Something Different?

People are wondering why we are building something new. Scheduling is often assumed to be a solved problem, and the industry is full of mature orchestrators.

The reality is that there is a huge, underserved gap in between general-purpose infrastructure tools and specialized runtimes: a scheduler that understands jobs as long-running, state-carrying entities with distinct lifecycles, recovery profiles, and migration paths, while remaining flexible enough for arbitrary workloads.

Building this is notoriously hard because you cannot separate scheduling policy from storage semantics. You aren’t just solving a bin-packing problem; you are building a distributed systems substrate that must reason about everything. But until we bridge that gap, teams will continue reinventing brittle, ad-hoc state managers on top of schedulers that fundamentally wish the world were stateless.