Long-running AI agents are brittle. One network blip, one timeout, or one ambiguous external state can turn a multi-hour task into a corrupted mess. Pyash, a tool for orchestrating long-running AI work, has been systematically hardening its recovery mechanisms since late 2025. By implementing structured run records, content-addressed artifacts, and idempotent session checkpoints, Pyash allows engineers to reconstruct completed work and resume interrupted tasks from saved evidence rather than starting from scratch.
The Foundation: Structured Run Records and Content-Addressed Artifacts
The recovery journey began on December 29, 2025, when Pyash introduced a structured run 'newspaper'—an ordered record capturing run boundaries, requested actions, outcomes, and tool activity. This was quickly followed by an --again path for rerunning and a replay command. On January 1, 2026, saved outputs gained content-addressed artifact identities. By using hash-based labels tied to the actual bytes of a file, Pyash can distinguish between a changed file and its earlier version. This allows the replay check to compare recorded artifact hashes against current bytes, ensuring that the saved output still matches what the run originally recorded. If a hash mismatches, it signals that the saved bytes changed after capture, alerting the operator to a potential integrity issue.
Identity Management: Idempotent Turns and Canonical Hashing
In early 2026, Pyash extended this record-keeping to agent sessions. Each conversation turn was given a unique identity to answer the critical question: 'Which request does this saved response belong to?' Pyash enforces idempotency, ensuring that one logical request keeps one identity across retries. The system uses a canonicalization algorithm to generate a SHA-256 fingerprint for each request. This process removes timestamp fields, sorts object keys, and serializes the normalized request before hashing. This ensures that equivalent maps produce the same fingerprint regardless of field order, while routine time changes do not alter the request identity. The turn ID then links the user input, agent reply, and checkpoint, allowing a verifier to accept the exact turn it checked.
Resumable Work: Checkpoints and Timeout Recovery
By August 2026, the focus shifted to making long-running backlog work safe to resume after crashes. A durable supervisor cycle and a resumable work backlog runner were introduced. A key addition was the ability to resume headquarters approval—Pyash’s human decision step—from a task checkpoint. The system also addressed timeout issues by changing from a fixed 900-second limit to a 900-second inactivity timeout, reset by progress through the Codex App Server, with a 1,800-second hard maximum. Checkpoints now retain partial output, turn identity, Git commit revisions, changed files, and diffs. If a remote turn’s outcome is ambiguous, the checkpoint marks it for reconciliation, comparing the recorded state with the remote service’s actual state before continuing.
Practical Challenges: Parsing Conflicts and Memory Limits
The recovery system isn't without its bugs. In November 2026, a saved coding session failed to replay due to a conflict between an assistant response and its checkpoint. The issue was traced to quoted shell text where a literal dollar-and-backtick sequence was interpreted as JavaScript replacement syntax during parsing. The repair involved treating quoted text literally and adding a regression test for this specific class of text. Additionally, the maintenance path revealed a critical limit: artifact-retention validation could exhaust the Node.js heap. On November 11, a shared test runner was introduced to divide quiz files into bounded batches and select concurrency based on available CPU and memory, ensuring that validation workloads stay within the host’s memory budget.
Key Takeaways
- Content-Addressed Artifacts are Critical: Using hash-based labels for outputs allows systems to verify integrity and detect post-capture changes.
- Idempotency Requires Canonicalization: Normalizing requests by sorting keys and removing timestamps ensures consistent identity across retries.
- Inactivity Timeouts Beat Fixed Limits: Resetting timeouts based on progress prevents false failures in long-running, slow-moving tasks.
- Reconciliation is Essential for Ambiguity: Checkpoints must support a reconciliation step to resolve conflicts between recorded state and external reality.
- Memory-Bounded Testing is Necessary: Validation workloads must be batched to prevent heap exhaustion during recovery checks.
The Bottom Line
Pyash’s evolution from simple run logs to a complex, idempotent recovery system highlights a hard truth for AI infrastructure: you cannot build reliable long-running agents without treating state management as a first-class engineering problem. The shift from fixed timeouts to inactivity-based limits and the introduction of content-addressed artifacts are not just features; they are survival mechanisms for any system that expects to run for hours, not seconds.