When an AI step retries, guard the side-effect — a note on idempotency keys

Tips & Tutorials
1

When an AI/LLM step retries, guard the side-effect — a field note on idempotency keys

I kept hitting a subtle failure in automations that call an LLM step and then do something with the result (send an email, create a row, post to an API). When the LLM step is slow or times out and the run retries, the model runs again — fine, it’s usually deterministic enough — but the side-effect after it fires twice. Two emails, two rows, two Slack pings. The model wasn’t the problem; the retry crossing a side-effect boundary was.

The fix that made this reliable for me is an idempotency key derived from the input, not the run:

  1. Before the AI step, compute a stable key from the trigger payload — e.g. a short hash of the fields that define “this unit of work” (order_id, or sha256(email + subject)). Same input → same key, every retry.
  2. Keep a tiny store of keys you’ve already completed (a datastore node, a KV, even a spreadsheet row keyed by the hash).
  3. At the top of the side-effect step: if the key is already marked done, short-circuit and return the stored result instead of doing the action again.
  4. Only mark the key done after the side-effect succeeds — so a genuine mid-run crash still retries, but a completed unit never repeats.

Two things I learned the hard way:

  • Don’t key on the run/execution id. The whole point is that a retry is a new run of the same work — the run id changes, so keying on it defeats the guard. Key on the input.
  • The LLM output is not a safe key. Temperature > 0 (or just model drift) means the same input can yield slightly different text, so hashing the output gives you a different key each retry and the guard never trips. Hash the input.

This is the same principle payment APIs use (Stripe’s Idempotency-Key), just applied one layer up in an automation. It turns “retry storm = duplicate havoc” into “retry storm = at-most-once side effect,” which is what you actually wanted from the retry.

Curious how others here handle at-most-once for the non-idempotent nodes — do you gate inside the flow like this, or lean on an external dedup service?