How to rank options in a decision system

Ranking is where the objective becomes real

A decision system's output is an ordered list. Everything upstream — ingestion, context, option generation — exists to produce it, and everything downstream depends on it being defensible.

Ranking is also where vague organisational intent gets forced into arithmetic. "We want the best outcome for the customer" cannot rank anything. Turning it into something that can is the hard, political, unavoidable part of the work, and it is usually mistaken for a technical problem.

Start with the option set, not the score

Before anything can be ranked, the candidates have to exist, and most systems generate them badly.

Three failures are common:

• Too few. The system proposes one option and a justification. That is a recommendation dressed as a comparison, and it cannot be interrogated.
• Too many. Forty candidates is a search result, not a decision. Beyond roughly seven the human stops comparing and starts skimming the top.
• The null option is missing. "Do nothing" is a legitimate candidate and frequently the correct one. A system that cannot recommend inaction will always recommend action.

A good option set is small, genuinely distinct, and includes doing nothing. Distinctness matters: three variations of the same approach and one alternative is effectively a two-option comparison with padding.

Scoring, in four parts

1. Criteria

Name the dimensions that matter. Keep the list short — three to six. Every additional criterion dilutes the others and makes the result harder to explain.

Each criterion must be measurable from available data. A criterion nobody can score is a value, and values belong in the constraint set or in the human's head, not in the ranking function.

2. Normalisation

Criteria arrive in different units: currency, days, probabilities, counts. They cannot be combined until they are on a common scale.

The choice of normalisation is not neutral and is routinely made by accident:

Min-max is the common default and has a trap: scores become relative to whatever happened to be in today's set, so the same option can score differently on different days. If a score needs to mean the same thing across runs, use fixed bounds.

3. Weights

Weights encode what the organisation actually values, which is why they are contentious.

Two rules keep this honest:

• Weights are a business decision, not a modelling one. They belong to the decision owner, in writing, and should be reviewable.
• Test their sensitivity. If a 5% change in one weight reorders the top three, the ranking is unstable and the result is being driven by an arbitrary number. Say so rather than shipping it.

4. Aggregation

How the scores combine. The choice has strong behavioural consequences:

• Weighted sum. Simple, explainable, allows compensation — a very strong score on one criterion can offset a poor one elsewhere. Usually right.
• Weighted product. Penalises weak scores much harder. Use when a serious deficiency should not be compensable.
• Lexicographic. Sort by the first criterion, break ties with the second. Use when one dimension genuinely dominates.

Most systems use a weighted sum without considering whether compensation is appropriate. Ask directly: should an option that is excellent on cost and terrible on risk be able to win? If not, a weighted sum is the wrong aggregator.

Constraints are not criteria

The most consequential distinction in this whole area.

• A criterion is something you want more of. It trades off.
• A constraint is a line that cannot be crossed. It does not trade off.

Constraints must be applied as filters before ranking, never as heavily weighted criteria. An option that breaches a hard limit should not appear at position eleven with a poor score — it should not appear at all.

Encoding a constraint as a large weight means that, given a sufficiently attractive option elsewhere, the system will eventually recommend crossing the line. That is precisely the failure the constraint existed to prevent, and it will happen on the unusual case rather than the routine one.

The ranking must carry its own uncertainty

Two options separated by 0.3 points are not meaningfully ordered, and presenting them as first and second implies a precision the data does not support.

Three practices:

1. Show the margin. A clear leader and a near-tie should look different.
2. Group where separation is within noise. "These three are equivalent on current data" is more honest and more useful than a false ordering.
3. Report what would change the order — the sensitivity. This is the single most actionable line in any ranked output.

A worked structure

1. Generate candidates, including the null option.
2. Apply hard constraints as filters. Log what was excluded and why.
3. Score each survivor against each criterion from available data.
4. Normalise, with a method chosen deliberately.
5. Apply weights owned by the decision owner.
6. Aggregate with an appropriate function.
7. Compute margins and sensitivity.
8. Group where separation is within noise.
9. Attach the explanation: scores, binding constraints, and what would flip the order.
10. Hand it to a named human.

In production

PSX Invest implements this shape against the Pakistan Stock Exchange. The universe of 500+ symbols is filtered, scored across seven indicator checks, and ordered so attention goes to a shortlist rather than to everything that moved.

The constraint-before-ranking rule is visible in the output: a minimum 2:1 reward-to-risk applies to anything published. A setup that fails it is not ranked poorly, it is not published at all. And the ranking ends at a human — the system places no orders.

The general pattern, and where it applies beyond markets, is on the decision intelligence page, which includes an explorer that composes the structure of a system around the objective and constraints you choose.