Engine · Job routing

Job management

Follow one durable job across worker attempts, pauses and interruptions without mistaking a restart for completion.

Workflow

  1. Create or reuse jobCreate a durable job after admission. An owner-scoped idempotency key reuses the same request; a changed payload with the same key is refused. Queuing alone does not claim a worker.
  2. REFUSED · no new attemptA changed payload cannot reuse the idempotency key. A live worker lease or respected operator hold prevents claiming work; do not pretend that a new worker started.
  3. Claim a worker leaseClaim the next attempt. A live lease protects its worker; a stale worker cannot finalize a newer attempt. Respect the operator hold when using retry for controllable work.
  4. RUNNING · process unitsWorkers process bounded units, renew leases and persist checkpoints and unit receipts. Progress events describe what happened; they are not a substitute for a terminal result.
  5. PAUSED · retain progressAn operator hold prevents automatic re-claim when respected by the worker. Recovery parks an expired running attempt as paused, never complete. It refuses to interrupt a live lease. Pausing is cooperative at a worker boundary.
  6. Resume saved workRelease an operator hold and claim a new fenced attempt, then continue using recorded work. JobStore.resume alone is restricted to never-leased jobs; leased work uses the recovery/claim path.
  7. CANCELLED · record stopA cancellation request is not instant rollback. The worker records its safe boundary and cancellation outcome; completed work remains evidence. Operations must advertise which controls they support.
  8. Persist the terminal resultA worker publishes a terminal result only through the current lease. Complete, partial, failed and unavailable remain separate outcomes; a lease-free cancellation of queued or paused work is a separate control path. An expired worker must not publish a successful result.
  9. Return state and eventsReturn a receipt or events after a supplied sequence. The client reads the explicit state and result. A bounded event window may require a snapshot before replaying newer events.

Failure boundaries

  • Same key, changed payloadCreate refused: No duplicate job
  • Lease still liveRecovery refused: Worker keeps ownership
  • Expired lease claimed directlyNew generation interrupts old attempt: Job may remain running
  • Old worker finalizesLease fencing refuses: Newer attempt protected
  • Pause command unavailableOperation-specific refusal: No assumed pause
  • Event cursor too oldSnapshot/replay boundary: Partial history is explicit

engine.job.lifecycle