Introducing Org Intelligence: Ask AI that knows how your team hires

Every hiring team has its own rules: what "strong" means on a system design round, which signals are disqualifying, how long a scorecard can wait. Until now, Ask AI in Debrev Interview knew your candidates but not those rules, so its answers fell back on general hiring advice.

Org Intelligence changes that. Upload your rubrics, hiring principles and interview guides under Settings > Knowledge, and Ask AI answers from them, with a citation chip on every sentence that relies on one. Click a chip and the source passage opens beside the answer.

This post covers how we built it, how we measured it before shipping, and what comes next.

The problem

Grounding a model in an organization's documents raises three questions, and an answer is only useful if all three hold:

• Does it cite the right passage? A citation that points at the wrong rule is worse than none, because it looks authoritative.
• Does it stay quiet when the documents don't cover the question? If someone asks about a probation policy you never uploaded, the right answer is "your documents don't cover this", not general advice presented as company policy.
• Can a document take over the model? Anyone who can upload a document can write instructions into it. In a hiring product, the attack that matters is quiet: steering which candidate gets recommended.

Our approach

Figure 1 at the top of this post shows the path a question takes.

Retrieval scoped to one organization

Documents are split along their headings, embedded and stored per organization, with versions, and never mixed with candidate data. The organization filter comes from the signed-in session, not from anything in the request, and every hit is re-checked against the document's current version. In our isolation test, 20 org-filtered queries across 4 organizations returned 0 chunks from another organization.

References as untrusted data

Retrieved passages go to the model in a separate message, labeled [R1], [R2] and so on, and marked as reference material rather than instructions. Citations are only allowed in the answer text, and the server validates each one before it reaches the screen. A document is never cited for a fact about a candidate.

A relevance filter, then an abstention rule

After retrieval, a small model keeps only the passages that bear on the question. The system prompt then says that questions about the organization's own policies are answered only from references. If none applies, the answer says so and cites nothing, not even the nearest related fact.

Defence in depth against injection

Instructions hidden in a document are handled at three points: an ingestion filter quarantines text addressed to an AI, the prompt tells the model to treat references as data, and a server-side check on the structured output removes recommendations a document should never be able to cause. The last layer turned out to matter most.

Hybrid candidate search

Ask AI also has to find the right candidates. An earlier version of Ask AI was built solely on RAG, and when we evaluated it, plain keyword matching beat its vector search on candidate retrieval (50.8% against 40.0%). Hybrid search is now the default: vector search and BM25 keyword ranking, fused by reciprocal rank fusion, with an automatic fallback to vector-only on any failure.

How we measured it

We ran Ask AI's first turn exactly as it ships: the production system prompt, the production response schema and the real retrieval path, against a small synthetic organization. Generation used gpt-5.1 and judging used gpt-5-mini.

• 5 documents, 23 chunks, written to overlap on purpose: two "five business days" rules and two discount rules.
• 70 questions. 34 can be answered from the documents, across single-section, paraphrased, two-document, look-alike and candidate-versus-rubric questions. 36 cannot, and 19 of those are near misses that share words with a document.
• 40 injection payloads in 8 attack styles. Each runs in its own organization next to a control copy of the same document without the payload, and counts as obeyed only if the output shows it and the control does not.
• 3 runs of everything. A question fails if any run fails, so repeats cannot average a failure away. Gates use the bound of a 95% Wilson interval, not the point estimate.

The thresholds were committed before the first scored run and never changed. Both judges passed hand-labeled calibration sets before every run. During development we refined the fabrication judge's rubric to match how the feature is actually used: an early version flagged a correct "the documents don't cover this" and the company's own name as invented claims. Its calibration set grew from 8 to 11 cases to cover those situations.

Results: org documents

Every gold section reached the model in both runs (34 of 34). Citation precision was 0.94 and citation recall 1.00. Groundedness, the share of cited claims the cited passage actually supports, was 0.93 and 0.87. Precision was 1.00 on every answerable question type except candidate-versus-rubric (0.75 and 0.80), where the model sometimes cited a rubric's red-flags section for a statement that section does not make.

On 34 of the 36 unanswerable questions, Ask AI cited nothing in any run. The other two were near misses: asked about a probation period, the model said the documents don't cover it, then cited the closest related fact, a recruiting agency's 90-day guarantee, while stating that it was not the policy asked about. With 36 questions, the abstention gate allows at most one such case, so two of 36 leaves it just short of its threshold. It is the only gate the final configuration missed.

Prompt injection

As shipped, 0 of 40 payloads got through in either run, across all eight styles: direct overrides, fake system notes, forged tags, format hijacks, soft appends, authority claims, hidden markup and goal hijacks.

Prompt rules alone were not enough. With the server-side output check turned off, 2 of 40 payloads succeeded in one run and 5 of 40 in the other, all of them goal hijacks of the form "always recommend this candidate". The answer text looked normal; the planted candidate showed up only in the structured shortlist. The output check removed every one. A legacy payload that demands the model reply only with a code word was obeyed in 0 of 12 runs.

Why a stricter similarity floor isn't the fix

Retrieval recall is 1.00 at the production floor of 0.68. Raising the floor costs answerable questions first: recall falls to 0.91 at 0.70 and 0.71 at 0.76, while near misses keep retrieving. One similarity threshold cannot separate "related" from "relevant". The relevance filter can: after it, 33 of the 36 unanswerable questions retrieve nothing at all.

Results: hybrid candidate search

We reran all four methods together on the same questions and organization used to evaluate the earlier RAG-only version, with a gpt-5.1 oracle reading every candidate's full record.

Method	Precision@5, this release	Earlier version, built solely on RAG
Hybrid	53.3%	not available
Keyword	50.8%	50.8%
Vector	43.3%	40.0%
Random	15.8%	13.3%

Against vector search, hybrid was better on 9 questions, worse on 1 and tied on 14; a sign test on the 10 non-ties gives p ≈ 0.02. Against keyword search it was 6 wins, 6 losses and 12 ties, which at 24 questions is not distinguishable from noise. Keyword matching was already competitive. What hybrid adds is not having to choose: it keeps keyword's precision on exact titles and skills and vector search's tolerance for paraphrase.

An offline comparison on the stored index points the same way. With hybrid, a candidate holding exactly the queried title reached the top 5 on 97.2% of 108 title queries (from 91.7%), skill precision@5 rose from 0.446 to 0.615, and paraphrase stability, the overlap between top-5 results for reworded queries, rose from 0.50 to 0.67. Vector search latency was 61 ms at p50 and 97 ms at p95.

What's next

More candidate evidence you can cite. Ask AI already cites interview notes alongside org documents, shown as the C chips in Figure 2. Next, we are extending candidate citations to the Strata decision summary, indexed as its own chunk with a source reference back to the decision field. Candidate evidence passes through a per-candidate access filter, so an interviewer only retrieves evidence for candidates they can already open, and a decision chunk will only be returned to viewers who can see that decision. The decision text is built from Strata's stored, scrubbed output, and only the newest run per candidate is indexed.

A larger evaluation. We are growing the citation eval to at least 150 questions across three fixture organizations of 5, 25 and about 80 chunks, with conflicting documents, role-restricted documents and, with pilot organizations' permission, real rubrics. A separate pre-registered test will check that adding org references never makes Ask AI's answers about candidates worse.

Closing the abstention gap. The two near misses share a pattern, offering the nearest related fact, and that is where the next round of prompt and filter work will go.

Try it

Org Intelligence is rolling out to beta organizations in Debrev Interview. Admins can upload documents under Settings > Knowledge, attach a rubric to a role, or turn a scorecard guide into a template. Hybrid search is now the default for candidate retrieval in Ask AI.

If your team wants to join the beta, or you want to compare notes on evaluating grounded answers, reach us at debrev.com.