A week of candidate evaluation, done in under five minutes

Hiring teams were spending a week evaluating candidates using the weakest signal they had, and getting inconsistent answers.

• Resumes stopped meaning much. With AI writing polished resumes for everyone, the document itself says less about what a person can actually do.
• Review was manual. Recruiters read each resume by hand, which doesn't scale past a few dozen applicants per role.
• Scoring was inconsistent. Different reviewers judged the same candidate differently, with no shared scale.
• Interview evidence was scattered across docs, emails and spreadsheets, so decisions rarely drew on all of it.

The obvious fix, asking an LLM to score candidates, has its own flaw: LLM scores bunch together in a narrow band, so strong and weak candidates end up looking alike. Solving that became the core technical problem.

Compelo runs seven specialized agents: an orchestrator plans each evaluation and hands the work to six specialists covering discovery, parsing, scoring, retrieval, verification and reporting.

Splitting the work this way keeps each agent's job narrow and testable, and the verifier means no single agent's output is trusted unchecked. Together they cut a week-long evaluation cycle to under five minutes.

I designed Dynamic Reference Bucketing (DRB), a self-calibrating scoring framework that replaces an LLM's unreliable absolute scores with a small number of pairwise comparisons against an evolving reference set. It recovers 99% of the true score spread (raw LLM scores recover 42%) at about 2 comparisons per candidate, and agreed with recruiters on 22 of 23 real pilot decisions.

Why raw LLM scores fail

Ask an LLM to score candidates from 0 to 100 and the scores cluster around 70 with a spread of only about 8 points, whatever the real spread of quality is. I modeled this as compression toward the mean plus noise:

In plain terms, each point of real quality difference moves the LLM's score by only about 0.3 points, and noise sits on top. In simulation, raw scores kept only 42% of the true spread and ranked 19.6% of candidate pairs in the wrong order.

The key observation: LLMs are poor at absolute numbers but much better at saying which of two candidates is stronger. DRB reduces scoring to those comparisons, while avoiding the cost of comparing every candidate with every other.

How it works

1. Buckets. The 0 to 100 range is split into $k$ buckets ($k = 10$, so each is 10 points wide).
2. Monotonic anchors. The LLM generates an anchor profile at each bucket's target quality. A comparison-based merge sort then orders the anchors, so the scale is guaranteed to increase monotonically. This is a one-time $O(k \log k)$ cost.
3. An evolving reference set. Each bucket keeps up to $m$ real candidates as references ($m = 3$). A new candidate is admitted only if it is at least a minimum distance from the existing references; once a bucket is full, it replaces the reference crowded closest to another. References stay spread across the bucket, and the scale calibrates itself as more candidates arrive.
4. Find the bucket by binary search. Start in the bucket the raw LLM score suggests, compare the candidate against that bucket's references, and move up if they're stronger, down if weaker, stop if equal. This takes at most $\lceil \log_2 k \rceil$ steps, and 2.12 comparisons on average.
5. Place them inside the bucket with Bradley-Terry. Compare the candidate against each of the bucket's $m$ references (wins count 1, ties count ½), map each reference's score to a Bradley-Terry strength, and solve for the candidate's strength by maximum likelihood.

The Bradley-Terry step

Reference scores in bucket $b$, spanning $l_b$ to $u_b$, map to strengths on $[1, e)$:

With $W$ wins out of $m$ comparisons, the candidate's log-likelihood is:

I proved this is strictly concave, so there is exactly one best estimate, and solve it with Hunter's MM iteration, which converges monotonically from any starting point:

The calibrated score inverts the strength mapping:

Guarantees

• Cost: $O(\log k + m)$ comparisons per candidate, and $O(k \log k + n(\log k + m))$ in total, versus $\Theta(n^2)$ for full pairwise ranking such as Elo-style LLM-as-judge.
• Placement: with comparator error rate $\varepsilon$, a candidate lands in the right bucket with probability at least $(1 - \varepsilon) + \varepsilon(1 - \varepsilon)^{n_c - 1}$, where $n_c = \lceil \log_2 k \rceil$.
• End-to-end error: I derived an upper bound splitting error into within-bucket quantization and mis-bucketing. At $\varepsilon = 0.10$ the bound is 19.7 points; the measured error is 2.04 points, so the bound is about 9.7× conservative.

Results

In Monte Carlo simulation (N = 5,000, 10% comparator error), DRB placed 91.9% of candidates in the correct bucket and 99.9% within one bucket. Accuracy stayed between 89% and 93% as comparator error rose from 1% to 30%.

• 8× lower error than raw LLM scores, with near-perfect rank preservation.
• Fast warm-up: the system gets most of its benefit once about 60% of buckets have references, which takes roughly 15 candidates.
• Real pilot: on 23 real candidates for real roles, DRB agreed with recruiters on 22 (95.7%), with zero false positives. The one disagreement was a candidate with non-traditional qualifications whom recruiters valued for transferable skills.

Known limits. LLM comparators can occasionally break transitivity, which I mitigate by comparing against several references per bucket; the error model assumes a fixed rate; and 23 decisions is too small a pilot for statistical significance, so larger-scale evaluation is the next step.

Candidates get a verified picture of their skills instead of a resume that says whatever they typed into it.

• Verified skill profile. Experience is mapped onto a standard skills taxonomy, using O*NET and ESCO as the spine, with proficiency levels based on the Dreyfus and SFIA frameworks.
• Fit and gap feedback. For a given role, candidates see where they match and which skills are missing.
• A portfolio of work artifacts. Real work samples serve as proof, rather than claims on a page.

On the recruiter side, I built an ATS-style CRM that lets a team manage 1,000+ candidates across a five-stage hiring lifecycle.

• Kanban pipeline for moving candidates between stages
• SLA alerting when a candidate sits in a stage too long
• Role-based access control so each team member sees and changes only what their role allows
• Bulk import for bringing in existing candidate lists

Compelo exposes its hiring data to other platforms through a hardened public API, and every push is gated by a large automated test suite.

API

• 26 REST endpoints, shipped as a Dockerized microservice on Express, MongoDB and AWS
• HMAC-signed webhooks, so receivers can verify every event came from Compelo
• Scoped API keys, giving each integration access only to what it needs
• Tiered rate limiting to protect the service from heavy or abusive clients

Testing

• 2,000+ unit tests across 105 test files, plus end-to-end tests in Playwright
• 90%+ coverage on the core pipeline modules
• CI gating with GitHub Actions on every push, blocking regressions before they reach a deploy

Compelo turned a week of manual, inconsistent evaluation into a calibrated result in under five minutes.

• A week to under 5 minutes per evaluation cycle
• 95.7% agreement with recruiter decisions across 23 real pilot evaluations, with zero false positives
• 99% of the true score spread recovered by DRB, versus 42% from raw LLM scores, at about 2 comparisons per candidate
• 1,000+ candidates managed through the CRM across five hiring stages
• 2,000+ tests with 90%+ coverage on core modules and CI gating on every push