How do you let an agent recover after it crashes halfway through a task?
Some agent frameworks can save and reload agent state, but you still have to write the checkpoints, the recovery logic and the protection against running the same action twice yourself. As the business logic grows and the agent serves many customers, this gets painful. How do people handle it today?
Treat the agent run like any other long-running job and borrow the patterns that already exist for those. Split the work into steps, save the result of each finished step, and on restart skip what is already done instead of starting from scratch. Durable workflow engines do exactly this, and an agent loop fits inside them well.
The hard part is side effects, not state. Saving the conversation is easy; making sure an email is not sent twice or an order is not placed twice after a crash is the real work. Give every action an id, record it before or with the action, and check that record before repeating it.
When a step fails part way through a chain of changes, plan the undo as well as the retry. For actions that span several systems, a compensating action for each step, such as cancel, refund or revert, is often simpler than trying to make the whole chain atomic.
Retries need limits: back off between attempts, set timeouts, and stop after a few tries and report, so a broken step does not turn into an endless and expensive loop.
Listings mentioned
- Workflow orchestration patterns · skill by wshobsonCovers durable workflows with Temporal: separating workflow from activities, state handling and the determinism rules that make resuming safe.
- Saga orchestration · skill by wshobsonDesigns compensating actions for multi-step work that spans several services, so a failed step can be undone cleanly.
- Python Resilience Patterns · skill by wshobsonRetries with exponential backoff, timeouts and fault-tolerant wrappers for steps that fail for temporary reasons.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.