Failure Recovery & Determinism — Agentic Architectures at Scale

Every useful system eventually fails, because every useful system depends on things it does not fully control: networks, clocks, people, software versions, permissions, data, and outside services. The mature question is not whether failure can be eliminated. It cannot. The mature question is what the system will remember, what it will repeat, what it will skip, and how confidently an operator can explain what happened afterward.

Start with the shape of a failure

A failure is any gap between intended behavior and observed behavior. That gap may be obvious, such as a crashed process, or subtle, such as a report generated with yesterday’s exchange rate. Recovery is the work of moving from the bad state back to an acceptable state without making the damage worse.

To reason clearly, separate three ideas. Detection is how you notice something is wrong. Diagnosis is how you decide what kind of wrong it is. Recovery is the action you take next. Systems become dangerous when they jump straight from detection to action, because the same symptom can have different causes. A timeout might mean the remote service is busy, the network is broken, the request succeeded but the response was lost, or the request is still running.

From first principles, recovery needs memory. The system must know what it was trying to do, what it already did, and what evidence it has. Without that memory, it can only guess, and guesses turn small failures into duplicated orders, missing messages, and inconsistent records.

Separate the steps: a failure is noticed, its cause is named, and only then is recovery attempted.

Determinism means the same cause gives the same effect

A deterministic process gives the same output when it receives the same input under the same conditions. A calculator adding two numbers is deterministic. A distributed workflow calling remote services, reading the current time, and asking a large language model (LLM) to draft a response is usually not fully deterministic.

Determinism matters because recovery often means replay. If a system crashes after step three of a ten-step process, one recovery strategy is to replay the process from the beginning. That is safe only if replaying the earlier steps produces the same results or if repeated steps are harmless. Otherwise, replay can create a second payment, send a second email, or overwrite a human correction.

Perfect determinism is not always possible or worth buying. The practical goal is controlled determinism: know which parts must be repeatable, which parts may vary, and which variations must be recorded. For example, if an application uses Microsoft Foundry to run an LLM prompt, the prompt, model name, model version where available, parameters, tool choices, and returned output may all matter for later explanation and replay.

Deterministic steps give the same result and are safe to replay; non-deterministic ones may vary or duplicate.

State is the thing recovery protects

State is the remembered condition of the system: an order is pending, a file has been scanned, a message has been sent, a user has approved a request. Stateless work is easy to retry because nothing important has changed. Stateful work needs care because the world may already have been altered.

A common design mistake is to store final business data but not store the recovery trail. The system knows that an invoice exists, but not whether the notification was sent, whether the tax lookup succeeded, or whether an approval task was waiting when a deployment restarted the worker. Durable state should include progress, not just outcomes.

This is why long-running workflows often use checkpoints. A checkpoint is a durable record that says, in effect, the system reached this point with these inputs and these results. After a crash, the system resumes from a known point instead of guessing from memory that disappeared with the process.

📌

Progress is data. If a step matters during recovery, its completion and evidence should be stored as part of the system’s state.

Retries are useful, but only under rules

A retry is the simplest recovery action: try the same operation again. Retries work well for transient faults, which are temporary problems such as a dropped connection or a busy service. Retries are harmful for permanent faults, such as invalid input, missing permission, or a business rule violation.

Good retry design has four parts. First, classify the error so the system does not retry things that cannot succeed. Second, limit the number of attempts so one bad request does not consume the whole system. Third, use backoff, which means waiting longer between attempts, often with a small random variation to avoid many clients retrying at the same instant. Fourth, make the operation idempotent where possible, meaning that doing it more than once has the same effect as doing it once.

Idempotency is a practical bridge between recovery and determinism. If a payment request includes a unique idempotency key, the payment service can recognize a duplicate retry and return the original result rather than charging again. The repeated call may not be fully deterministic in timing or response path, but the business effect is controlled.

⚠️

Blind retries. A retry without classification, limits, and idempotency can turn one failure into many correct-looking wrong actions.

Classify the fault first: retry transient faults with backoff, but stop on permanent ones.

When you cannot undo, compensate

Some actions cannot be rolled back. You cannot unsend an email, unshow a notification, or pretend a third-party shipment was never requested. In these cases recovery uses compensation. A compensating action does not erase the original action; it creates a new action that repairs the business situation as far as possible.

Compensation is not just a technical pattern. It is a policy decision. Refunding a customer, canceling a shipment, issuing a correction, or escalating to a human reviewer all express a judgment about what acceptable repair means. The system should make that judgment explicit rather than hiding it in exception-handling code.

The most reliable designs define compensation at the same time as they define the happy path. For each step, ask what evidence proves it happened, what duplicate would mean, what timeout would mean, and what action is allowed if later steps fail.

For reversible steps, prefer rollback or replacement with a known good value.
For irreversible steps, design a compensating action and store the evidence needed to perform it.
For ambiguous steps, first reconcile with the system of record before doing anything new.
For high-impact steps, require human review when confidence is low.

Non-determinism has sources you can name

A process becomes non-deterministic when the same visible input can lead to different results. The cause is often not mysterious. Time changes. Random numbers change. Data stores are updated by other users. Remote application programming interfaces (APIs) return different results. Model outputs vary. Permissions change. Parallel tasks finish in different orders.

Once you name these sources, you can control many of them. Treat the current time as an input and record it. Store random seeds when a seed is meaningful. Pin dependency versions. Record tool inputs and outputs. Use stable ordering before merging parallel results. Capture the identity and authorization context used for an action; in many Microsoft environments, that context is governed through Microsoft Entra ID.

For AI-assisted systems, tools deserve special attention. If an LLM can call Foundry Tools or other external tools, the final answer may depend as much on those tool results as on the model. Logging only the final answer is not enough. Record the tool name, request, response, status, and relevant version information so that a future investigation can distinguish model variation from tool variation.

Non-determinism has nameable sources: once named, most can be recorded and controlled.

The leadership decision is how much certainty to buy

Failure recovery and determinism are not free. They require durable storage, workflow design, test environments, observability, operational runbooks, and sometimes slower release processes. The right investment depends on the cost of being wrong, the speed at which recovery must happen, and the level of explanation the organization owes customers, regulators, or itself.

A low-risk content suggestion can tolerate variation and manual correction. A payroll system cannot. A customer support copilot may need excellent audit trails even if its language varies. A medical, financial, or safety-related workflow may need strict approval gates, deterministic business rules, and controlled use of generative components.

The practical leadership move is to classify workflows by consequence. Do not demand maximum determinism everywhere; that wastes effort and slows learning. Do not accept improvisation in high-consequence paths; that creates hidden liability. Decide which paths need automatic recovery, which need human approval, which need replay, and which need only clear failure reporting.

What is the business cost of a duplicate, omission, or wrong result?
How quickly must the system recover before people notice or harm occurs?
Who is allowed to approve compensation when the system is unsure?
What evidence must be kept to explain the decision later?
Which parts must be deterministic, and which parts may vary within limits?

✅

Aim for explicitness. A good recovery design makes uncertainty visible early enough for software or people to act safely.

Key takeaways

Recovery starts with durable knowledge of what happened, not with guessing what to retry.
Determinism is valuable because it makes replay, diagnosis, and explanation more trustworthy.
Retries need classification, limits, backoff, and idempotency to avoid amplifying failure.
Irreversible actions require compensating actions, not pretend rollbacks.
Non-determinism often comes from time, randomness, changing data, external APIs, model behavior, permissions, and parallelism.
Leaders should buy the level of determinism and recovery that matches the consequence of being wrong.

Glossary

Determinism: Determinism is the property of producing the same result from the same inputs under the same conditions.
Recovery: Recovery is the process of returning a system from a failed or uncertain state to an acceptable operating state.
State: State is the information a system remembers about its current condition, past progress, and business facts.
Checkpoint: A checkpoint is a durable record that a process reached a specific point with specific inputs and results.
Retry: A retry is a repeated attempt to perform an operation after an earlier attempt failed or returned an uncertain result.
Idempotency: Idempotency is the property that doing an operation more than once has the same intended effect as doing it once.
Backoff: Backoff is a retry strategy that waits between attempts, often increasing the wait time to reduce pressure on a failing system.
Compensation: Compensation is a follow-up action that repairs the business effect of an earlier action that cannot be directly undone.
Replay: Replay is the act of running a process again from stored inputs or checkpoints to reproduce, continue, or investigate its behavior.