Determine eligibility before ranking candidates
The cheapest model is not necessarily suitable for a scanned document, a constrained transformation or a task with restricted data handling. Routing should first eliminate candidates that do not meet the contract’s capability and policy requirements. Only eligible candidates belong in a quality, latency and cost comparison.
A useful routing input includes modality, document length, output structure, configured attempt budget and permitted providers. Data location or retention requirements must be supported by an actual provider arrangement and configuration. A selector cannot turn an unsupported deployment or contractual requirement into a supported one.
Measure performance on the operation that matters
A general benchmark score does not establish whether a model reliably preserves invoice identifiers or catalog units. Evaluate candidates against a frozen, representative task set and the same verifier definitions. Inspect both aggregate results and failure categories: missing line items, unsupported values, invalid references and cases that should have requested review.
Selection should account for the whole attempt path. A low-cost first attempt can become expensive if it repeatedly needs a larger model, manual review or reconciliation. Report first-pass check results, escalation frequency and total observed run cost separately; avoid presenting a successful recovery as an error-free initial attempt.
| Decision input | What it should constrain |
|---|---|
| Input modality and size | Whether a candidate can process the source |
| Provider and data policy | Whether the candidate may receive the data |
| Task-specific evaluation | Which eligible candidates are credible choices |
| Attempt and cost policy | When to stop or request intervention |
An escalation is a new attempt, not a waived check
A routing policy can permit bounded re-attempts when a candidate fails a check that another attempt could reasonably resolve. The next attempt still faces the same contract and verification boundary. A different model does not receive permission to loosen required fields, change the destination or write around a failed supplier match.
Some failures should not trigger another model call. Missing credentials, a deleted reference record or an ambiguous downstream write require operational handling. Routing policies should preserve that distinction and expose why an attempt was selected, escalated or stopped. Confirm model eligibility, attempt limits and the treatment of retries when scoping a workflow.
Missing a detail or found a problem?
Send a documentation question →