Engineering tutorial
Agentic workflows: what happens after the timeout?
An agent prepares a record, calls a tool and waits. The tool times out. Did nothing happen, or did the action succeed before the reply was lost? If the system cannot answer that question, asking the model to try again can turn one task into two.
Treat an unknown result as its own state
Here's a hypothetical service workflow. An agent creates a ticket, but the connection drops before the ticket ID comes back. The ticket is already in the destination system. If the agent starts the conversation again and repeats the call, you may now have two tickets for the same request.
Record an action as attempted before dispatch, and distinguish confirmed success, confirmed failure and an unknown result. Where the destination supports it, attach a stable request identifier and query the result using that identifier. A timeout tells you what the caller observed; it does not prove what the remote system did.
Keep the same identity when retrying the same action
An idempotency key lets a receiving service recognize repeated requests for one intended operation. AWS describes this approach in its guidance on making retries safe. The caller must preserve the key across retries. Generating a fresh key on every attempt defeats the purpose.
Bind the key to the approved action and its parameters. If the customer or requested change is different, it is a different operation. When a destination offers no deduplication mechanism or reliable lookup, send uncertain results to reconciliation. There is no prompt that can make an unsupported external API promise exactly-once execution.
Save the work that must survive a restart
Save the task ID, input version, approved plan, completed tool results and outstanding action somewhere the workflow can read after a restart. A chat transcript alone leaves too much to reconstruct. Keep this saved state behind the same access controls as the original records, and avoid storing data the recovery process doesn't need.
On restart, pick up the first unresolved step. Check whether the records used to prepare it have changed. If they have, set aside the affected plan and send it for review again. Otherwise an old approval can quietly authorize a change to a record that now says something else.
| Saved state | Next step after restart |
|---|---|
| Draft prepared; approval absent | Show the draft and request approval. |
| Approval recorded; action not dispatched | Check that the input version still matches, then dispatch. |
| Action dispatched; result unknown | Reconcile with the destination before attempting another write. |
| Result confirmed | Continue using the recorded result; do not repeat the action. |
Approval belongs to a particular proposal
'Approved earlier in the chat' is too loose for a workflow that can change records. Approval should identify the destination, operation and proposed values. If the model revises those values, the old approval no longer describes what it is about to do.
Routine reads can run under permissions agreed in advance. For an action that needs review, show the destination and proposed change in ordinary language. The operator needs to know what approving it will do; they shouldn't have to interpret the model's reasoning.
Try the failure you are designing for
Before release, deliberately interrupt the synthetic workflow after the remote action succeeds but before its result is saved locally. Resume it twice. Verify that there is one business action, that the recorded state eventually agrees with the destination and that unresolved ambiguity reaches an operator.
Also test a changed input, a refused permission and an expired approval. A system that continues confidently through those conditions needs a tighter boundary, even if its normal demo looks excellent.
Rubix Labs includes workflow architecture, approvals and evaluation in its agentic-systems offering. We'd put this failure drill in the pilot's acceptance checks. Before handing the system over, an operator should be able to stop it, inspect the last action and resume without guessing whether that action already happened.
