Strata Phase 2: Scaling to 32B for evidence-grounded hiring decisions

We trained Qwen2.5-3B and Qwen2.5-32B adapters on the same frozen corpus with the same recipe, then scored both on a locked 200-item evaluation. The 32B adapter met every product target. The 3B adapter met one of three.

The problem

Strata turns interview evidence into a structured signal read for hiring managers: competency signals with evidence references, coverage gaps, concerns with severity, and a recommendation of hire, lean_hire, lean_no_hire or no_hire. The hiring manager makes the call.

Two requirements come before accuracy:

• Schema compliance. Every downstream feature reads these fields.
• Commitment. When the evidence triggers a hard-no rule, the model has to say no_hire.

Our earlier 3B adapters failed both. Two causes were plausible: noisy training data, or too little model capacity. We built one experiment to separate them.

Our approach

One variable

Qwen2.5-3B-Instruct and Qwen2.5-32B-Instruct, from the same model family, each got a fresh QLoRA adapter. Corpus, prompts, schema, seed, hyperparameters, evaluation and hardware were identical. A preflight script blocks training if the two configs differ in anything beyond the model name.

Clean, gated data

• A flagship teacher model wrote targets for 654 real hiring records and 175 synthetic profiles built to fill the borderline classes.
• A validator checked every target for schema, for grounding of each claim in the input, and against a written hard-no policy.
• We kept 600 balanced examples, 538 for training and 62 for development, and froze them with content hashes.

A locked evaluation

200 held-out items, 50 per label, with zero overlap with the training data. Before the freeze, three rounds of boundary audits resolved every disagreement between teacher and reference labels through human adjudication. The set is fingerprinted, and scoring refuses runs whose fingerprints differ.

Verdicts written before training

• 3B passes: clean data was the fix, and we keep the small model.
• 3B ambiguous or failing, 32B passing with a significant paired test: capacity was the constraint.
• Gates were tiered, schema compliance first and no_hire recall second. Four-class accuracy was reported and never gated.

Recipe

4-bit NF4 base with double quantization, bf16 compute, LoRA rank 16 and alpha 32 on the query and value projections, 3 epochs at learning rate 2e-4, best checkpoint by dev loss. Both arms trained and evaluated back to back on one 80 GB A100.

Results

The 32B adapter met all three product targets: schema compliance of at least 0.90, no_hire recall of at least 0.90, and at least ten committed predictions at each extreme label. The 3B adapter met only the last one.

Locked eval, tuned	3B	32B	Paired test, p
Schema compliance	0.525	0.905	3.2e-16
no_hire recall	0.68	0.94	0.00024
Binary direction accuracy	0.86	0.92	0.0042
Compliant and grounded	0.52	0.815	—
Four-class label accuracy	0.555	0.66	—
hire recall / precision	0.48 / 0.83	0.42 / 0.91	—
no_hire precision	0.55	0.68	—
Calibration error, ECE	0.21	0.178	—

What the numbers measure

• Schema compliance is the share of outputs that parse, contain every required field and pass all validator rules.
• no_hire recall is the share of the 50 true no_hire candidates the model called no_hire. Precision is the share of no_hire calls that were correct.
• Binary direction accuracy checks only the side: hire or lean_hire against lean_no_hire or no_hire.
• Compliant and grounded requires a compliant output where every reasoning claim resolves to a field in the evidence.
• Calibration error is the gap between the model's stated confidence and its actual accuracy. Lower is better.
• The paired test is McNemar's, run on the same 200 items. It counts only items where the two models disagree, so a small p means the gap is unlikely to be noise.

Intervals matter at this sample size. 181 compliant outputs out of 200 gives a range of roughly 0.86 to 0.94. 47 of 50 no_hire items gives roughly 0.84 to 0.98. The 32B adapter sits on the target line with little margin, so a fresh holdout is the next real test.

Both models learned. Only one reached the bar.

Compliance rose from 0.0 to 0.525 for 3B and from 0.26 to 0.905 for 32B. Dev loss fell steadily in both runs, to 0.538 and 0.373. The corpus taught both models, and capacity set how far each one got.

The student tracks its teacher

The 32B student is 0.06 behind the teacher on compliance and within 0.01 on four-class accuracy. It is slightly ahead on no_hire recall, binary direction and adjacent accuracy. Both score 0.42 on hire recall. These figures describe this task under our schema and nothing broader.

Where the errors go

Over-calling no_hire

The 32B adapter made 69 no_hire calls for 50 true cases, a precision of 0.68. Of the 22 false calls, 19 landed on lean_no_hire, the adjacent label, 3 on lean_hire and none on hire. Across the 100 hire-side candidates, 32B said no_hire 3 times and 3B said it 8 times. For a screening aid, this is the direction of error we accept, and the product will say so.

Under-calling hire

hire recall is 0.42: 28 of 50 hire candidates were called lean_hire. The teacher scores the same 0.42 on this set, so the student copied its teacher's caution. Positive-side rules and targets are the fix. More capacity is unlikely to help.

Confidence

A calibration error of 0.178 means raw model confidence stays out of the interface.

Scope

The result holds for this recipe. A 3B adapter with a different rank, target modules or schedule is untested and will get its own pre-registered run.

Lessons for teams training adapters

Keep adapters unmerged on quantized bases

Merging a LoRA into a 4-bit base re-rounds the weights to NF4. One earlier adapter kept 22.9% of its per-token loss improvement after merging. Another kept 5.3%. Under 0.3% of the packed bytes changed. Our old eval path merged, so part of the failure we blamed on earlier adapters was an evaluation artifact. Applying adapters at runtime costs about 9% in decode speed.

Budget for evaluation

Training took 16 minutes for 3B and 98 minutes for 32B. Evaluation took 29 of the roughly 31.5 pod hours, with the GPU 27 to 50% utilized. Items ran one at a time, and validator failures triggered a repair retry. We will switch to batched evaluation and fix that procedure before the next pre-registration, so every arm is scored the same way.

Freeze every reference

Two teacher runs with identical settings swapped hire and lean_hire on about ten items. Without a frozen eval set, a frozen corpus and a fixed teacher reference, that variance reads as progress.

What's next

• Serving parity. The 32B model in 4-bit with the adapter unmerged fits on one 80 GB GPU, or a 32 GB card for inference. Any serving quantization has to match these results on the locked set before it ships.
• The positive boundary. We will write explicit hire rules the way we wrote hard-no rules, then regenerate the affected targets.
• Smaller models. Our goal is a cost-effective model teams can run in their own environment. Intermediate sizes and a revised 3B recipe will each run as pre-registered experiments on the same locked set.
• A bias audit. Counterfactual testing comes before any external use. This post makes no fairness claims.
• Field-level evaluation. Coverage gaps and follow-up questions will get their own labeled evaluation slice.

For hiring teams

• Strata is decision support. The hiring manager decides.
• Every claim cites a field from the interview evidence.
• A validator checks each output before anyone sees it.
• One written hard-no policy applies identically to every candidate.
• The model is built to run in your own environment.

If your team runs structured interviews and wants the evidence behind each recommendation, reach us at debrev.com. Developers who want to compare notes on evaluation design or adapter training can reach us there too.