How to Audit Whether Your Screen Filters or Just Defers

A screening funnel that rejects 96% of resumes and one that’s silently broken can produce the exact same topline number. Both would show a narrow pass rate. Both would look, from the outside, like an aggressive but working filter. The only way to tell them apart is to actually look inside the funnel — not at the ratio, but at the decisions the ratio is made of.

We published a real July 2026 usage dataset from one HireQwik deployment recently: 24,327 resumes, 1,074 interviews, a roughly 4% conversion rate, and one role accounting for most of the volume. Every one of those numbers is the kind of thing you’d want to audit before trusting, ours included. This post is the audit framework itself — the questions we’d actually run against a dataset like that one, and the specific checks that separate a funnel doing real filtering from one that’s just deferring the decision downstream.

Why the pass rate alone tells you nothing

A 4% interview rate could mean the resume scoring is doing careful, relevance-aware filtering — correctly identifying that 96% of applicants weren’t a fit for that specific role. It could also mean the scoring threshold is set so aggressively that it’s rejecting qualified candidates along with unqualified ones, and nobody would know from the ratio alone. Both scenarios produce the same number. The audit has to go past the ratio into the actual rejected cases.

This is the trap every “results” post in AI screening falls into if you let it: a strong-sounding percentage, presented without the sampling work behind it, tells you the vendor ran a filter — not that the filter was correct. Treat any funnel number, including the ones in our own usage data, as a starting point for an audit rather than a conclusion.

The audit, step by step

Here’s the actual sequence we’d run against a month of funnel data:

1. Pull a random sample of rejected resumes, not the ones that failed obviously. Skip the resumes with zero relevant experience — those are easy to agree on. Sample from the borderline band instead, where a resume score sits close to the reject threshold, because that’s where a scoring error would actually show up.
2. Have a human reviewer independently score the same sample against the job description, without seeing the AI’s verdict first. This is the only step that catches disagreement — comparing a human’s cold read to the AI’s verdict after the fact, rather than checking whether the human agrees once they already know what the AI decided.
3. Check whether disagreements cluster. One or two disagreements out of fifty is noise. A pattern — every disagreement involves candidates from a particular background, education path, or resume format — is a signal the rubric is systematically weighting something it shouldn’t.
4. Look at the interview stage separately from the resume stage. A candidate can clear resume scoring and still exit early on a knockout question — check that knockout exits are tagged with a specific, reviewable reason, not just a bare Reject with no trace of why.
5. Check the Maybe/borderline band specifically, not just the clear Reject and clear Go bands. This is where a funnel that “defers” instead of filters usually hides — a wide Maybe band that HR still has to fully re-review by hand means the AI didn’t actually reduce review load for that segment, even if the topline pass rate looks selective.
6. Compare volume-weighted results, not just role-blended ones. If one role dominates volume the way it did in July 2026’s data, a blended pass rate across 9 roles can hide a very different, and possibly worse, rate for the smaller roles buried underneath it.
7. Re-run the audit after any rubric or threshold change. A funnel that passed this audit in July isn’t guaranteed to still be healthy in September if the job description, the rubric, or the auto-decide thresholds changed in between.

Filtering vs. deferring, side by side

The two failure modes look different once you know what to check for, even though the topline numbers can look the same:

None of these signals show up in a single aggregate pass-rate number. That’s the whole reason the audit has to go past the ratio — the ratio genuinely cannot distinguish the left column from the right one.

Why this matters beyond just funnel quality

There’s a compliance dimension to this too, not just an efficiency one. A screening process that can’t produce a specific, reviewable reason for a reject decision is a harder position to defend if a rejected candidate — or a regulator — asks why. A funnel that filters, with tagged reasons attached to every reject, gives HR something concrete to point to. One that defers leaves HR with nothing to show beyond “the score was low.” Building the audit habit is partly about being able to answer for those decisions after the fact, which matters more the larger the volume gets.

In some jurisdictions this has already moved from good practice to legal obligation. New York City’s rules on automated employment decision tools require an independent bias audit before such a tool can be used in hiring, with the results published. India has no equivalent statute today, but a team that already samples its own rejects and keeps traceable reasons attached to them is doing most of the underlying work regardless of what any particular rulebook demands.

What “deferring” actually looks like in practice

A deferring funnel usually shows up in one of two shapes. The first is a wide Maybe band: instead of confidently rejecting or advancing most candidates, the system pushes a large share into an ambiguous middle tier that still requires a full human review — meaning the screening layer added a step without removing the review burden it was supposed to reduce. The second shape is the opposite: an aggressively narrow pass rate with no reviewable reasoning behind the rejects, which looks efficient on a dashboard but means nobody — human or AI — can actually defend individual reject decisions if challenged.

HireQwik’s auto-decide thresholds are built specifically against the first failure mode: JDs can be configured to auto-reject below one match-score threshold and auto-fanout above another, so HR only reviews the genuine middle band — not because the middle band is being hidden, but because it’s the only segment where a human call actually adds information the scoring didn’t already resolve.

A worked example against the July 2026 dataset

Take the 22,761 resumes screened for the DL/ML Research Intern role in July 2026. Running this audit against that specific role would mean pulling a sample from the borderline resume-score band for that role, having a human independently re-score it against the DL/ML-specific rubric, and checking whether the 621 candidates who did reach an interview cluster around a particular resume-score range or are spread widely — a tight cluster near the pass threshold would suggest a hard cutoff is doing most of the work, while a wide spread would suggest richer, more nuanced scoring.

We haven’t published that role-by-role rejected-sample breakdown, and that’s a fair thing to flag: this post gives you the audit method, not a completed audit of our own July numbers at that level of granularity. If you’re evaluating any screening vendor’s usage data, ours included, ask for exactly this kind of role-level sample, not just the aggregate ratio.

How big a sample actually catches a problem

A common reason audits don’t happen is that “sample the rejected resumes” sounds like it requires reviewing hundreds of them, which feels like exactly the review-time cost the screening layer was supposed to remove. It doesn’t need to be that large. A sample of 30–50 resumes, pulled specifically from the borderline band rather than randomly across the whole reject pool, is usually enough to surface a real pattern if one exists — because the borderline band is where a scoring error would actually change an outcome. A resume scored 5 points below the reject threshold and a resume scored 40 points below it are not equally informative to re-check; the first tells you something about where the threshold sits, the second almost never does.

Weight the effort toward whichever roles are pulling the most volume rather than spreading it evenly — a sampling error on a role with 22,761 applicants costs far more than the same error on one with 200.

Who should run the audit, and how often

The person running this audit doesn’t need to be the same person who set up the rubric or configured the auto-decide thresholds — in fact, an independent reviewer is better, precisely because they haven’t already committed to a view of what the “right” answer looks like for a given resume. A recruiter who reviews the Needs Review queue regularly is well positioned to run this: they already have context on what a strong match for the role looks like, without necessarily having tuned the scoring themselves.

On cadence: run it once after any month with meaningful volume, and again any time something changes that could plausibly shift the funnel’s behavior — a new job description, an adjusted per-JD rubric, a changed auto-decide threshold, or a new hiring season with a different applicant mix than the last one. A single audit that passed in July doesn’t carry forward automatically into September, especially if anything about the role or the rubric changed in between — the audit answers “is this healthy right now,” not “will this stay healthy.”

When a low pass rate is genuinely fine

Not every narrow funnel needs to be treated with suspicion. If the sampled rejects consistently show resumes with no relevant experience for the role, no matching skills, and no plausible transferable background, a 4% pass rate is simply an accurate reflection of a mismatched applicant pool relative to that specific opening — not evidence of over-filtering. Narrow is also the industry default rather than an outlier: benchmark data across hiring generally puts about 3% of applicants through to an interview, which means a manual process is already discarding a comparable share, just without the sampled evidence to back it up. The audit isn’t meant to prove every low pass rate is wrong. It’s meant to replace “trust the number” with “check the number,” which produces a defensible answer either way.

What to do with the results

If the audit turns up a real problem — a cluster of disagreements, a wide unreviewed Maybe band, un-tagged rejects — the fix isn’t to abandon AI screening, it’s to adjust the specific thing the audit found: the rubric, the threshold, or the tagging on knockout exits. If the audit comes back clean, you now have something better than a topline pass rate to point to: an actual sampled comparison against human judgment, which is the evidence that should back any claim about a screening funnel’s accuracy — including the ones a vendor makes about their own numbers.

Run this audit against your own funnel, whatever tool produced it, before deciding a narrow pass rate means the screen is working. Pull a sample from your own review queue and check it against a human read the same way — that comparison is the only thing that actually answers the question a topline ratio can’t.