How to evaluate an AI screening vendor in 30 minutes: the founder demo checklist

Most AI screening vendor demos run 45 minutes and leave you with an impressive recording, a follow-up calendar link, and no framework to compare what you just saw. After sitting through enough of these, a pattern emerges: the feature that gets the most screen time during the demo is almost never the feature that determines whether the platform works for your actual hiring reality.

The voice quality sounds impressive. The candidate dashboard looks clean. The UI demos beautifully on a 27-inch monitor. And then you go back to your team and realize you forgot to ask whether the platform can handle 3,000 applicants in a single evening, or what happens when a candidate files a complaint about an AI-driven rejection decision six weeks from now.

This is a checklist for the 30 minutes before or during a demo that cuts through that pattern.

Why most vendor evaluations fail at the comparison stage

The failure mode in AI screening evaluation is not lack of diligence — it’s that TA directors evaluate vendors on the vendor’s preferred framing. The vendor controls the demo flow. They show you the features that perform best. They do not volunteer the constraints.

SHRM’s 2025 AI-in-HR survey found 88% of HR leaders see AI screening as a compliance risk. The EU AI Act classifies hiring AI systems as high-risk under Article 6, which means vendors operating at scale need audit trail documentation that is ready today — not planned for a future sprint. Most demos do not surface either requirement because the buyer has to ask first.

The 30 minutes you spend asking the right questions before the demo are worth more than the 45 minutes watching the demo.

Five questions that differentiate platforms

Question one: Is the scoring rubric JD-specific or shared across roles?

Platforms that use a single scoring model across all JDs are cheaper to build and faster to deploy. They also produce shortlists that conflate a BDE candidate with a CSM candidate. Ask the vendor to show you two rubrics for two different roles — specifically what varies between them and what does not. If the answer is “the system adapts automatically,” ask what specifically adapts and how that adaptation is verified against the individual JD. Vague answers indicate a template-with-minor-tweaks approach, not genuine per-JD scoring.

Question two: What happens in the first 60 seconds when a candidate should be disqualified?

Phase-0 knockout questions — questions that fire first and end the call within one to two minutes for clear no-gos — protect recruiter time and give candidates a fast, respectful exit. Ask whether the vendor supports this pattern. A platform that runs every candidate through a full 15–20 minute structured screen regardless of a first-question fail is wasting candidate time and your per-interview budget simultaneously.

Question three: Who sets the accept/reject threshold, and can it be configured per JD?

Auto-decide bands — where the platform automatically rejects candidates below a threshold or advances candidates above one — are real features in mature screening platforms. Ask how the threshold is set, who can change it, and whether it’s configured per JD or set globally. A global threshold set by the vendor means a tier-3 college candidate applying for an ops role and a tier-1 candidate applying for a sales role are judged against the same bar. That is a quality problem and a compliance problem simultaneously.

Question four: What does the audit trail look like for a rejected candidate?

If your legal team receives a bias complaint six weeks from now, what documentation does the platform produce? Ask to see a sample audit export for a rejected candidate — the scoring breakdown, the question sequence, the rubric weights at the time of evaluation. If the vendor says they can build that report on request, it means it does not exist today in an exportable form. SHRM’s data shows 88% of HR leaders already view AI screening as a compliance exposure. A platform without a ready audit export is transferring that liability to you.

Question five: What is the candidate completion rate at your hiring scale?

A platform might schedule 1,000 interviews and complete 700. The 30% dropout is candidate experience data the vendor has and you should ask for. Request actual completion rate data from campaigns at your headcount scale — 500 or more candidates in a single drive. Completion rates below 70% indicate a UX problem, a scheduling friction issue, or a candidate communication gap. All three are addressable. All three should be disclosed rather than discovered after you’ve signed a contract.

What the pilot experience says about which questions matter

HireQwik ran 1,099 structured screening interviews across pilot campaigns, including a single campaign that screened 3,000 candidates through a 15–20 minute voice conversation in one evening. The 89% reduction in HR time per candidate came from Phase-0 knockout efficiency and per-JD rubric fidelity — not from the voice persona, the branded landing page, or the real-time dashboard aesthetics.

When those pilot campaigns were being designed, the questions that consumed the most evaluation time were about rubric specificity, rejection transparency, and scheduling completion rates — exactly the five areas above. The features that looked most impressive in vendor demos were consistently the features that mattered least at actual operating scale.

The thing most vendors will not volunteer

Ask any vendor whether they have completed an independent bias audit, and ask who conducted it. Then ask whether their system produces documentation that satisfies EU AI Act Article 6 high-risk hiring AI requirements — even if you are operating entirely in India today, enterprise clients with EU operations will increasingly ask for this in their own vendor due diligence processes.

The vendor that answers both questions clearly and specifically in the first conversation is not the norm. Make it your baseline expectation rather than a pleasant surprise.

What to leave the demo with

Not a proposal deck. Not a case study PDF. A reference — a TA director who ran a campaign at your scale, in India, in the last 12 months. A name and a phone number, not a company logo on a slide. If the vendor hesitates, ask what the largest Indian campus drive completed end-to-end on their platform was in the last calendar year. The answer tells you whether they are genuinely operating in this market or entering it with a polished outbound motion.

A detailed look at the ATS integration layer — which is where vendor demo promises most often diverge from production reality — is covered in the AI screening ATS integration guide.