How much autonomy should an AI system have

The wrong question and the right one

"How autonomous should this system be?" invites an answer on a scale from cautious to ambitious, and it is usually settled by temperament — how comfortable the room feels — rather than by anything measurable.

There is a better question, and it has an answer that can be derived:

How easily can this system's actions be undone?

Autonomy should be a function of reversibility, not of confidence in the model. A highly accurate system taking irreversible actions is more dangerous than a mediocre one taking reversible ones, because the mediocre system's mistakes get corrected and the accurate system's mistakes get shipped.

Reversibility is a spectrum, not a flag

Most discussions treat actions as reversible or not. In practice there are several levels, and the distinctions carry real weight:

The bottom row is the one that matters most, and it is larger than teams expect. Anything that leaves your systems is irreversible: a message a person has read cannot be unread; a webhook another company acted on cannot be recalled.

Four factors that modify the baseline

Reversibility sets the starting point. Four things move it.

Blast radius

One record or a hundred thousand. The same error rate produces a different outcome at different scale, so an action that is safe individually can require a gate in bulk. A useful pattern is a volume threshold on an otherwise autonomous action: automatic below N, approval above it.

Detectability

If the action goes wrong, how quickly would anyone notice?

An error that surfaces immediately is far safer than one that sits undetected for a quarter. Actions with poor detectability deserve tighter autonomy even when they are technically reversible, because "reversible in principle" is worthless if nobody knows to reverse it.

Confidence, used correctly

Confidence should modulate autonomy within a tier, never move an action between tiers.

High confidence on an irreversible action does not earn autonomy. Low confidence on a reversible one is a reasonable trigger to escalate. This asymmetry matters: confidence is an input to the decision, not a licence.

And it only means something if it is calibrated. An uncalibrated confidence score used to grant autonomy is a random number with authority.

Novelty

Has the system seen this situation before?

A case that matches a known pattern is a reasonable candidate for autonomy. One that does not is exactly where a model will be confidently wrong. Systems that can recognise "this is unlike anything in my mapped set" and escalate on that basis are meaningfully safer than systems that cannot.

Deriving the envelope, step by step

1. List every action the system can take. Actual actions, not categories.
2. Assign a reversibility level from the table above. When uncertain, choose the more severe level.
3. Set the baseline autonomy from that level.
4. Apply the modifiers. Large blast radius or poor detectability moves it down. Nothing moves an irreversible action up.
5. Add a volume threshold where an action is individually safe and collectively risky.
6. Add a confidence floor below which the action escalates regardless of tier.
7. Add a novelty check that escalates unmapped situations.
8. Write down what happens at each boundary — hold, escalate, drop — because an undefined boundary resolves itself in whatever way is most convenient at runtime.

Autonomy should be earned, then widened

The most common structural error is setting the envelope once, at design time, and never revisiting it.

A better pattern is to start deliberately tighter than necessary and widen on evidence:

1. Shadow. The system proposes; a human decides everything. You are measuring agreement, not saving time.
2. Assisted. The system proposes with reasoning; the human approves. Time savings begin here.
3. Bounded autonomy. Reversible, low-radius actions execute automatically. Everything else is gated.
4. Widened autonomy. Thresholds move outward as override rates and outcome data justify it.

Each transition should be a decision supported by data from the previous stage, not a schedule agreed in advance. And the movement should work in both directions: if override rates rise or drift is detected, narrowing the envelope is a legitimate, reversible mitigation.

Where the boundary is a product decision

PSX Invest is the clearest example we operate. Placing an order is irreversible, so it sits at the bottom row of the table, and the product has no execution capability at all. That is not a roadmap gap — the system has no route to the market by design, and the trader places every order themselves.

That constraint made everything else easier to build, because the hardest autonomy question was answered before anything else was designed.

Barion Agents is being specified with the full tiered model, because operational work genuinely requires execution rather than proposals. It is not available yet, which is stated plainly on that page.

The summary

Autonomy is not a measure of how good your system is. It is a measure of how much it is allowed to break before someone can intervene.

Set it from reversibility, modify it for radius, detectability, calibrated confidence and novelty, then widen it on evidence rather than on optimism.

The neighbouring decisions are covered in approval gate design and in the readiness assessment in when not to use an AI agent.