Suggested guest: Hugh Seaton, alongside a contractor estimator.
Why it matters: The denominator, consequences and test conditions matter more than a headline percentage. The percentage in this question is hypothetical, not a verified supplier result.
Good answer should cover: Expected versus returned fields, missing pages, holdout data, severity classes, false assurances and correction time. Ask how the evaluation changes when the output affects a binding quotation rather than an internal draft.
Follow-up: “What evidence would make you reduce the scope of the pilot?”
Evidence to request: A sample evaluation design and a clear account of uncertainty. Use NIST's voluntary generative AI profile as a discussion reference, not a pass certificate.
Expert quote: Awaiting interview and publication approval.
Sources cited on this page
- NIST's voluntary generative AI profilenvlpubs.nist.gov