Hiring results in under an hour, not 3 to 5 weeks

CMS was hiring at the scale of a company but with the tools of a single school. Every year 5,000 to 7,000 candidates applied across two hiring cycles, and every step after the application was manual.

• Video screening was a guess. Candidates sent a resume and an interview video. Staff watched what they could, but at this volume most videos were never seen, so the school never knew who it had missed.
• Exams were a logistics project. The 1,000 to 2,000 candidates shortlisted each cycle were called in to sit written exams. That meant printing papers for 150+ different exams, securing rooms, handing answer sheets out to teachers for marking, collecting them back and typing every grade into the HR system by hand.
• Practicals doubled the work. Many roles also needed a practical test in Word, Excel, PowerPoint, Scratch or programming, which teachers had to set up and check one by one.

A single round took 3 to 5 weeks, and the result was slow, inconsistent between markers, and incomplete.

Looma plugs into the school's existing HR system instead of replacing it. Candidates arrive from Savvy HRMS through Looma's REST API, and results go back through signed webhooks.

Candidates and HR staff start and finish in Savvy; everything in between runs inside Looma, and results flow back with no manual data entry.

Written answers are graded by several models that check each other, and a human only sees the rare answer they can't agree on. In internal testing, the grading matched or beat the accuracy of human teachers.

Input. Candidates still write on paper in the exam hall. The HR team scans the answer sheets and uploads them as PDFs, so the pipeline handles typed, handwritten and diagram answers across 5 languages.

How a score is decided

1. A grader model scores the answer against the question's rubric.
2. An independent verifier model checks that score.
3. If they agree, the score stands.
4. If they disagree, a third model regrades the answer once, and the verifier checks it again.
5. Only if they still disagree does the answer go to a human for manual review.

Why this design. A single model call is cheap but can be confidently wrong, and a human on every answer recreates the old bottleneck. Escalating only on disagreement keeps the cost close to one grading pass for most answers, while putting the uncertain ones in front of a person.

Every interview video now gets watched, scored and checked for cheating, with 98% accuracy and under 0.5% false positives on a hand-labeled set of 1,000 videos. Integrity flags go to staff as a recommendation to review, never as an automatic rejection.

Processing. A background Python worker takes each video and:

1. Normalizes it (rotation, frame rate).
2. Transcribes the speech with word-level timestamps.
3. Tracks face and eye landmarks frame by frame.

It returns two things to the platform: an interview score and an integrity report.

Scoring

• Fluency is rule-based, so it is consistent and easy to explain. From the word timings I measure speaking rate, how often and how long the candidate pauses, and long silences. Filler words are scored as their own dimension.
• An LLM grades grammar and relevance against a fixed rubric.
• Model changes are tested first. Before switching models, I re-score past transcripts to make sure scores stay stable.

Detecting screen reading and off-screen gaze

Everything is measured against the candidate's own baseline, taken while they pause naturally, so camera angle and face shape don't skew the result.

• Reading shows up as a rhythm: repeated slow sideways eye sweeps at a steady pace, or eyes held low while speech stays fluent.
• Peek and recite is subtler: short glances to the same spot mid-sentence, followed by speech with no hesitation.
• Off-screen gaze looks at how often, how long and in which direction the eyes leave the camera. Glancing around in mixed directions is normal thinking, so it isn't flagged.
• Other checks flag a second person in the frame and edited videos (cuts, splices, editing-app watermarks).
• Gaze checks are skipped when the face is too small or tracking is unstable, rather than guessing.

Rebuilding for precision

The first version flagged far too many people. I rebuilt it to favor precision, because a false flag hurts a real candidate.

• I hand-labeled videos as reading, not reading or edited, measured each version against those labels, and reviewed the flags myself.
• I tried an automated judge for labeling, but it missed the subtle peek-and-recite cases, so the labels are human.
• Each signal now needs a minimum duration and rate, and a video is only flagged when several signals agree strongly.

That is what took the detector to 98% accuracy with under 0.5% false positives.

Practicals are marked mechanically wherever a rule can decide, and an LLM is used only for criteria that genuinely need judgement. The mechanical results go to the LLM as verified facts it cannot overrule.

The exam environment

Candidates take practicals in the browser on a locked-down exam page, in a proctored hall. Each exam type opens a tool that looks and works like the real one:

• Word, Excel and PowerPoint editors
• A code editor with a Run button for Java, Python and JavaScript
• The Scratch editor
• A Windows-style file explorer
• A Gmail-style compose window
• A virtual multimeter

Work saves automatically. Copy and paste are blocked everywhere except the code editor. Candidates never see whether an answer is right, though they can run their programs on their own input. If they never press Save, the last stored copy is still marked.

Isolation

• Candidate code never runs on our app servers. Each run starts a fresh, short-lived sandbox VM from a prebuilt image, with time and output limits, and the VM is thrown away afterwards.
• Office files are never executed. Docx, xlsx and pptx are zipped XML, so the marker parses them as data and reads facts straight from the file.

What gets checked mechanically

Where the LLM steps in

Only for criteria that need judgement: the wording of a letter or email, whether a picture suits the task, and layout and placement.

• Input: the rubric, a text rendering of the file, page images of the saved file as the candidate saw it, and the pictures inside the file.
• Fixed facts: the mechanical results are passed in as verified facts the model can't overrule.
• Output: marks in quarter steps with evidence, then a second pass checks the grade.
• Model changes: before switching models, I re-grade real past answers with both and compare.

Exam day is spiky: a whole hall starts within a minute or two, saves constantly and runs toward the same deadline, on school Wi-Fi and lab machines that can fail at any point. Most of the engineering went into making each of those failures boring, and no live exam has had an incident so far, including a peak of 1,200 to 1,300 candidates at once.

Autosave under load

• Save only on change. Saves are debounced (quicker after a click, a little later while typing), with a cap so nothing stays unsaved for long.
• No contention. Each candidate writes only their own rows, so candidates never compete with each other.
• Fixing read amplification. Watching a real exam day showed autosave was most of the database reads, because every save loaded the whole exam definition (20 to 60 KB) just to validate one answer. I moved validation to a compact spec about 15 times smaller, and reordered the checks so a rejected save reads nothing at all.
• Decoupling heartbeats. Candidate heartbeats lived on the main attempt record, so every heartbeat re-ran the staff dashboards. I moved them to their own record.

Flaky Wi-Fi and crashes

• Offline-tolerant. Answers are kept in the browser as well as on the server. If the network drops, the candidate keeps working, sees a small notice, and pending saves go through when the connection returns.
• Crash recovery. If the browser or machine dies, the candidate signs back in, the invigilator confirms with a session PIN, and they pick up from the server copy. Invigilators can also reopen a sitting or add time.

Timers and submission

• The server owns the clock. The end time is stored when the exam starts and the screen only displays it, so a refresh, a wrong system clock or a crash can't change it.
• Idempotent, two-step submit. A submit cut off halfway is finished by a recovery job.
• Offline sweep. A background job auto-submits anyone whose time ran out while they were offline.

Grading queue

• Submissions go into durable job queues, one per kind of grading (code runs, typed answers, rubric grading), each with its own concurrency cap and retries.
• A malformed answer fails on its own, so it can't take down the rest of the batch.

This is what lets a full hall's results reach the HR system in under an hour.

CMS now runs its hiring cycles on Looma, with 5,000+ candidates evaluated and 3,500+ teacher-hours saved per round.