When an AI/LLM step retries, guard the side-effect — a field note on idempotency keys
I kept hitting a subtle failure in automations that call an LLM step and then do something with the result (send an email, create a row, post to an API). When the LLM step is slow or times out and the run retries, the model runs again — fine, it’s usually deterministic enough — but the side-effect after it fires twice. Two emails, two rows, two Slack pings. The model wasn’t the problem; the retry crossing a side-effect boundary was.
The fix that made this reliable for me is an idempotency key derived from the input, not the run:
Before the AI step, compute a stable key from the trigger payload — e.g. a short hash of the fields that define “this unit of work” (order_id, or sha256(email + subject)). Same input → same key, every retry.
Keep a tiny store of keys you’ve already completed (a datastore node, a KV, even a spreadsheet row keyed by the hash).
At the top of the side-effect step: if the key is already marked done, short-circuit and return the stored result instead of doing the action again.
Only mark the key done after the side-effect succeeds — so a genuine mid-run crash still retries, but a completed unit never repeats.
Two things I learned the hard way:
Don’t key on the run/execution id. The whole point is that a retry is a new run of the same work — the run id changes, so keying on it defeats the guard. Key on the input.
The LLM output is not a safe key. Temperature > 0 (or just model drift) means the same input can yield slightly different text, so hashing the output gives you a different key each retry and the guard never trips. Hash the input.
This is the same principle payment APIs use (Stripe’s Idempotency-Key), just applied one layer up in an automation. It turns “retry storm = duplicate havoc” into “retry storm = at-most-once side effect,” which is what you actually wanted from the retry.
Curious how others here handle at-most-once for the non-idempotent nodes — do you gate inside the flow like this, or lean on an external dedup service?
Following my own note here — there’s a second cost to that retry that bit me later, and it’s literal cost.
The idempotency key stops the side-effect from firing twice. But the AI step still runs twice, and you pay for both. If your prompt carries an accumulating context — prior messages, tool outputs, a growing scratchpad — each retry re-sends that whole payload to the model. On a flow that retried a few times a day, the duplicated token spend was quietly bigger than the “obvious” cost of the primary run.
Two things that helped once I could actually see it:
Measure the token cost per step, not per run. I started dumping the exact input token count of each LLM call so I could tell which retries were expensive vs. which were cheap. The expensive ones were almost always the steps dragging a big accumulated context. (I use a small CLI, tokenscope, for this — but even a crude len(prompt)/4 estimate logged per step surfaces the offenders.)
So my mental model now is two guards on the same retry: an idempotency key so the effect happens at-most-once, and a token-cost budget so the spend of the re-runs stays visible instead of drifting. Curious whether anyone here budgets token cost per node, or only watches the monthly bill after the fact.
Good write-up. One detail that makes this hold up: derive the idempotency key from the run’s input (for example a hash of the trigger payload plus the step name) rather than generating a random value inside the run, so a retried execution ends up with the same key.
For the cost side in your follow-up, the same hash can double as a cache key for the LLM output. If the first call finished but a later step failed, the retry can reuse the stored response instead of calling the model again, which also keeps the result identical across attempts.
To guard side-effects during retries, manage two-phase state transitions around the mutation step. Before triggering the external call, write a ‘pending’ record keyed by the deterministic hash with a modest lease/timeout. If a network retry triggers while the original call is still in flight, the retry detects the pending lease and backs off rather than firing a duplicate request. Once the external call succeeds, update the record to ‘completed’ alongside the downstream resource identifier so any future rerun immediately short-circuits. Where downstream APIs support native idempotency headers, passing that same deterministic hash gives defense in depth if the orchestrator fails before recording completion.