Blog

How ReasonRank works, in plain terms

ReasonRank Engineering · 2026-07-28

If you run AI agents in production, you are paying a model bill every month for a decision nobody re-checks: which model should each job use? The flagship you picked in March might be 5× the price of a model that now does the job just as well — but swapping feels risky, because "just as well" is a vibe, not a number.

ReasonRank turns that vibe into a number you can defend. That's the whole product. Here's the plain-language version of how.

The one-paragraph version

You show ReasonRank examples of a job your agent does. It runs those examples across the model you use today and cheaper candidates — on your own API keys — and scores every answer. If a cheaper model holds quality, ReasonRank tells you, shows you the per-example evidence, and projects what the switch saves at your real traffic volume. After you switch, it watches production and reports the actual dollars — and keeps watching so quality never quietly rots.

That's it. Everything else — judges, statistics, replay, CI gates — exists to make that one answer trustworthy.

The four steps, in order

1. Describe the job with examples. Ten to twenty real examples of what your agent handles ("classify this support ticket", "summarize this document") with what a good answer looks like. Start from a template, paste your own, or let ReasonRank propose examples mined from your real traffic (with identifiers masked). This is your test suite — think of it as unit tests for the job.

2. Run the comparison. Pick your current model and one or two cheaper candidates. ReasonRank runs every example on every model — using your provider keys, with the cost estimated up front (typically well under a dollar) and hard spend caps so an experiment can never surprise you.

3. Read the recommendation. You get a ranked table: quality vs cost per call, with one model marked recommended — the cheapest one that statistically holds quality. Click any number and you see the evidence behind it, example by example. If there isn't enough data to be sure, ReasonRank says "unproven" and tells you what to add. It never bluffs.

4. Switch and verify. Apply the switch (in your stack or ours), and ReasonRank compares production cost before vs after — reporting verified savings only once the new model is actually observed handling your live traffic. If quality regresses, you'll know, with rollback one click away.

"But how does it know the answers are good?"

The question everyone asks. Three honest answers, in increasing order of rigor:

• Deterministic checks — exact matches, required structure, numbers within tolerance, sources actually cited. No AI judging AI; just checks.
• An AI judge, on a leash — for fuzzy quality, a judge model scores answers against your rubric. But judges aren't trusted by default: your team labels a handful of answers first, and the judge only counts once it demonstrably agrees with your people — and it's re-checked on a schedule.
• Statistics that refuse to overclaim — a switch is only recommended when the math clears a strict bar. Small suite? You get "add more examples," not a confident-sounding guess.

Your first 15 minutes

The fastest path from zero to a defensible answer:

1. Sign up and add one provider API key (2 min). Keys are encrypted; ReasonRank never resells or marks up your tokens.
2. Create an agent from a template (2 min) — templates ship with working example suites, so you can skip authoring on day one.
3. Run your first comparison (5–10 min) — current model vs one cheaper candidate. Watch the cost estimate before you click; expect cents.
4. Read the recommendation and the evidence page — this is the moment the product clicks: a real quality-vs-cost answer about your job, with the receipts attached.

From there, value compounds with two optional connections:

• Connect your traffic (5 min with a framework adapter or any OpenTelemetry exporter, metadata-only by default) — now projections use your real volume, savings get verified in production, and ReasonRank can propose test examples from real usage.
• Add the CI gate — one GitHub Action, and every prompt or model change gets a quality report on the PR before it merges.

Where the value shows up

• Week one: your first evidence-backed "this cheaper model holds quality" (or the equally valuable "it doesn't — don't switch").
• Month one: a verified $/month savings line on a real agent, plus CI catching a regression before customers do.
• Quarter one: every agent right-sized, drift alerts watching live quality between decisions, and a paper trail your staff engineers — and your CFO — can audit.

Ready to try it? Start here — or if you'd rather see it on your own agents first, book a demo and bring one job you suspect is overpaying.