Making AI decisions explainable, not just narrated

Explanation is a system property, not a feature

"Can it explain its reasoning?" is usually answered by pointing at text the model produced describing what it did. That text is not an explanation. It is another generation, produced after the fact, and it is under no obligation to correspond to what actually happened.

A real explanation has a harder requirement:

An explanation is only useful if acting on it would have changed the decision.

If a reader could disagree with the stated reasoning and that disagreement would not have altered the output, the explanation is decorative. Which gives a test: change the reason, and see whether the answer moves. If it does not, the reason was not the reason.

Three things that get called explanation

Most products ship the first and describe it as the third.

The distinction matters most when something goes wrong. A narrative explanation of a bad decision tells you what a model would say about it now. A structural explanation tells you what actually happened, which is the only one useful in a review.

Build the explanation from the decision, not about it

The reliable approach is to make the decision path itself the explanation, which means designing the system so its reasoning is legible by construction rather than reconstructed afterwards.

Concretely, that means recording:

1. The options considered, including the rejected ones. A ranking with only the winner shown is not a comparison.
2. The criteria applied, and their relative weight.
3. Each option's score against each criterion. This is the substance.
4. Which constraints bound, and on which options.
5. What would change the order — the sensitivity. Which single factor, if different, would promote the runner-up?

Point 5 is the one that separates a genuinely useful explanation from a defensible one. "Option A ranked first because it scored highest on cost and availability" is fine. "Option A ranked first, and B would overtake it if the lead time were two days shorter" is actionable, and it tells the reader exactly where to push back.

Rejected options are the substance

The strongest single improvement most systems could make is to show what they did not choose.

A decision presented alone invites acceptance. A decision presented alongside three alternatives and the reason each lost invites scrutiny — and scrutiny is what the explanation exists for. It also makes the system's criteria visible, which is how people discover that the encoded objective is not quite the one they meant.

When a system starts behaving oddly, "what else was on the table, and why did it lose?" is almost always the more diagnostic question than "why did it pick that?"

Match the explanation to the reader

One explanation format does not serve everyone, and the differences are substantial:

• The operator, deciding now, needs the short form: the top options, the deciding factor, what would change it. Anything longer will not be read under time pressure.
• The reviewer, weeks later, needs the full structural record: inputs as they were, criteria, scores, constraints, approver.
• The affected person — the customer, the patient — needs plain language, with no jargon and no false precision about certainty.
• The engineer, debugging, needs the attribution view plus versions.

Build the structural record once and render it four ways. Building four independent explanations guarantees they will diverge.

Explaining uncertainty honestly

Explanations tend to be more confident than the decisions behind them, because prose does not naturally carry error bars.

Two habits help:

• State the confidence and what it means. Not "high confidence" but the calibrated figure and what it has historically corresponded to.
• Say what was missing. "This ranking did not account for X, which was unavailable" is far more useful than silence, and it is the sentence that most often prevents a bad decision.

A system that never mentions what it did not know is implicitly claiming to have known everything.

Where it becomes non-negotiable

The requirement scales with consequence and with how long the decision stays reviewable.

In Clinical Insight Engine, which is in development and not available for clinical use, the design position is that a ranking that cannot be explained is a defect rather than an output. If the reasoning cannot be shown, the option should not be presented at all.

In PSX Invest, the indicator rationale travels with every published call. A trader can see which of the seven checks supported it and what each read at the time, which means they can disagree with the reasoning rather than only with the conclusion. That is the whole point of shipping it.

Building it, in order

1. Record the option set at decision time, not just the winner.
2. Record the criteria and weights as data, not as prose.
3. Record each score, so the comparison can be rebuilt exactly.
4. Record which constraints bound, from the validation and policy layers.
5. Compute sensitivity — what would flip the order.
6. Render per audience from that single record.
7. Use generated prose only as a summary of the structural record, never as the record itself.

Step 7 is the discipline that holds the whole thing together. Narrative is a presentation layer over structure. Once it becomes the source, the explanation has stopped being evidence.

The wider control model this sits inside is on the governance and observability page.