What a calibrated confidence score actually means

Most confidence scores mean nothing

Almost every system that produces recommendations attaches a number to them. Very few of those numbers have been checked against reality, which means they communicate the system's enthusiasm rather than anything about the world.

A confidence score is only meaningful if it is calibrated, and calibration has a precise definition:

A score is calibrated if, across many predictions, the things it rates at 70% happen about 70% of the time.

Note what that does not say. It says nothing about any individual prediction. A calibrated 70% call that fails is not a broken system — it is one of the 30%. Calibration is a property of a distribution, and it can only be judged in aggregate, over enough cases for the arithmetic to mean anything.

Confidence is not accuracy

These get conflated constantly, and they are close to independent.

• Accuracy asks: how often is the system right?
• Calibration asks: when the system says it is 70% sure, is it right about 70% of the time?

A system can be accurate and badly calibrated. If it is right 85% of the time but stamps everything "95% confident", every downstream decision that relies on the number is being made on a false premise.

A system can also be calibrated and not very accurate. If it says 55% and is right 55% of the time, it is being honest about a hard problem. That is far more useful than false precision, because it lets a human size their exposure correctly.

For decision-making, calibration usually matters more than accuracy, because the number is what people act on.

How you actually check it

The procedure is not complicated, and the fact that it is rarely done is mostly a matter of discipline rather than difficulty.

1. Record the score with the prediction, at the moment it is made. Not recomputed later.
2. Wait for outcomes. They must be defined in advance and resolvable without judgement. "Did price reach the target before the stop?" resolves. "Was it a good call?" does not.
3. Bucket the predictions by score. Everything scored 60–70% goes together.
4. Compute the hit rate in each bucket.
5. Compare predicted against observed. Plot predicted on one axis, observed on the other. Perfect calibration is the diagonal.
6. Check the buckets are large enough. Ten predictions in a bucket tells you very little. This is the step most often skipped, and it produces confident conclusions from noise.

The resulting curve tells you the shape of the problem:

That last row is more common than the industry admits. A flat reliability curve means high-confidence and low-confidence predictions perform identically, and the number should not be shown to anyone.

Why overconfidence is the default

Systems drift toward overconfidence for structural reasons, not by accident:

• Selection pressure. The cases used to build and tune the system are the ones that were resolvable and well-recorded. Messy cases are absent, and they are the hard ones.
• Optimisation targets. Most training objectives reward being right, not being honest about uncertainty. Nothing penalises a confident wrong answer more than an uncertain one.
• Presentation pressure. A system that says "55%" feels weaker than one saying "90%", and there is commercial gravity toward the second.

Resisting that last pull is a product decision more than a technical one.

What calibration cannot do

Three limits worth stating plainly, because they are routinely elided:

It is backward-looking. Calibration describes how scores behaved historically under historical conditions. If conditions change, the calibration is stale before the numbers show it. This is why calibration needs drift monitoring alongside it.

It says nothing about an individual case. A calibrated 80% call that fails is not evidence of a broken system, and a calibrated 55% call that succeeds is not evidence of a good one. Judging either on a single outcome is a category error, and it is how people lose confidence in systems that are working correctly.

It is not a prediction of returns. A calibrated confidence score on a trading signal says how often that class of setup has historically resolved as expected. It does not forecast a market, and it cannot be compounded into an expected return.

How this works in PSX Invest

PSX Invest publishes a calibrated confidence score with every signal, and the calibration is against real outcomes rather than asserted.

The structure that makes this measurable:

• A defined resolution rule. Every call ships with an entry range, a target and a stop. Whether it reached target before stop is resolvable without judgement, which is what makes outcome recording possible at all.
• A consistent universe. The scan runs across 500+ symbols after the session closes, so every symbol is evaluated on the same complete picture rather than on partial data.
• A published floor. Any call that goes out carries a minimum 2:1 reward-to-risk. Below that it is not published.
• A hard boundary. The system places no orders. Execution stays with the trader, so the score informs a human decision rather than triggering an automatic one.

If you are building one

1. Define the outcome so it resolves mechanically. If two people could disagree about whether a prediction came true, the definition is not finished.
2. Record the score at prediction time, with the model and policy version attached.
3. Wait for enough volume per bucket before drawing conclusions.
4. Plot reliability, do not compute a single summary number. The shape carries the information.
5. Publish the range you trust and suppress the rest. If the extremes are unreliable, do not show them.
6. Re-check on a schedule, because calibration decays.

And if the curve comes back flat, remove the number. A meaningless confidence score is worse than none, because people act on it.