AI Detector Accuracy: A Reproducible 500-Sample Test Protocol

Claims about the “most accurate AI detector” are only as credible as the test behind them. A useful benchmark must define what counts as human or AI, prevent the detector threshold from changing after the results are seen, include difficult edited cases, and report errors—not just a headline accuracy number.

This article is a preregistration-style protocol for a 500-sample English benchmark. It is not a completed results report. The tables contain the planned sample counts and empty result fields so no number can be mistaken for an observation. Publishing the method first creates a dated record and makes any later changes visible.

The question this benchmark can answer

The target question is narrow: how well do selected detector versions distinguish the defined sample groups, in five English-language domains, at the vendors’ documented thresholds on the test date?

It cannot answer whether a detector works equally well in every language, whether future versions will behave the same way, or who authored an individual document. It also cannot turn a probabilistic classification into evidence of misconduct.

The NIST GenAI Text Challenge likewise frames detection as a measurement problem requiring rigorous, science-based evaluation. Our smaller protocol borrows that discipline while remaining practical enough for an independently reproducible website study.

Build the 500-sample dataset

The dataset should be stratified by provenance and domain. A random collection of convenient web paragraphs would hide important failure modes and make the ground-truth labels difficult to defend.

The mixed-provenance groups should not be silently folded into the main human-versus-AI accuracy score. Report them as separate challenge sets. If a product exposes only a binary label, record that label while preserving the more precise dataset category.

Balance five writing domains

Use 100 samples from each domain: student-style essays, news or explanatory writing, marketing or blog content, business correspondence, and creative writing. Allocate provenance groups across all five domains as evenly as the totals permit.

Within each domain, use predefined length buckets such as 100–199, 200–399, and 400–800 words. Do not truncate a sample after seeing a detector response. Strip names and sensitive details before testing, but keep a record of every normalization step.

Record generation conditions

Raw AI samples should come from multiple model families and at least two decoding settings. For every sample, store:

• A random sample ID, domain, word count, and provenance group.
• Model and version, access date, prompt, system instruction, and generation settings where available.
• The untouched output plus each later revision as a separate file.
• Human editor instructions and a concise change log for edited samples.
• Source license or participant consent and the anonymization status.

Do not publish private student submissions or scraped personal material. A reproducible study can release hashes, metadata, permitted samples, and anonymized detector outputs without exposing confidential text.

Prevent leakage and cherry-picking

Freeze the dataset before submitting any sample to a commercial detector. Generate sample IDs, calculate hashes, and save a dated manifest. The person assembling the material should not remove difficult items after seeing scores.

Check human reference text for likely contamination from model-training or public benchmark corpora where feasible. No check can prove that a model never encountered public text, so record the source period and selection rule instead of claiming perfect independence.

Deduplicate within and across categories. Near-duplicate passages let repeated wording dominate the result and make confidence intervals look stronger than they are. Keep prompt templates separated across the development and final test sets if any threshold or analysis code is tuned.

Freeze each detector configuration

Create a detector manifest before the first run. It should contain the product name, URL or API endpoint, account tier, displayed version, test timestamp, supported language, minimum length, and documented decision threshold.

For each submission, capture the raw score, displayed label, warnings, processing error, and a screenshot or structured API response when the terms permit it. Keep the input formatting consistent. If retries produce different scores, retain all attempts and apply a retry rule written in advance.

Do not choose a custom threshold separately for each detector using the final 500 samples. Either use the vendor’s documented threshold or tune one on a separate development set. Otherwise, the evaluation rewards knowledge of the answers.

Report metrics that expose mistakes

Accuracy alone is insufficient. A detector that labels nearly everything as human can look good when the test set contains mostly human text. Report the confusion matrix and class-specific metrics at the frozen threshold.

Report a 95% confidence interval for each primary metric. A stratified bootstrap over sample IDs is a practical approach, provided repeated variants from the same source document stay in the same resampled cluster. State the number of bootstrap repetitions and random seed.

The practical examination of AI-text detectors emphasizes performance at specific false-positive rates. This is especially relevant in high-stakes use: a broad AUROC can look respectable even when recall is weak at a threshold strict enough to protect human writers.

Predefine the stress tests

Report raw AI and transformed AI separately. Combining them into one average can conceal a detector that performs well on untouched output and poorly after light revision.

The challenge conditions should include ordinary copyediting, sentence restructuring, paraphrasing, and translation round-trips. Apply a consistent operation count or editing-time budget. Record semantic similarity or have blinded reviewers confirm that the core meaning survived.

This is not an artificial concern. The ACL 2024 paper Stumbling Blocks evaluated editing, paraphrasing, co-generation, and prompting attacks and reported an average performance decline across attacks, with no tested detector robust to every condition. A current benchmark should therefore measure transformed text directly rather than extrapolate from raw generations.

Publish results without manufacturing certainty

Use a table like the one below when the runs are complete. Every dash is intentionally blank.

The final report should also include confusion matrices, domain-by-domain results, length-bucket results, confidence intervals, and a count of unsupported or failed submissions. Publish the analysis script and a machine-readable results file when licensing and detector terms allow it.

Rankings should be conditional: “best on this frozen English dataset at this threshold and date” is defensible. “The world’s most accurate detector” is not.

A safe decision policy

Even a strong aggregate result does not prove the origin of one document. Institutions should decide in advance what a flag triggers. A sensible policy is a manual review of the text, drafts, citations, document history, and the writer’s explanation—not an automatic penalty.

Try the GPTHuman AI Detector for an initial signal, then apply the same caution to our score as you would to any competitor’s. Our editorial policy describes how articles are reviewed and updated. The technical companion, How AI Detectors Use Perplexity and Burstiness, explains why statistical patterns are clues rather than proof.

Methodology and disclosure

This protocol was published before collecting the 500 commercial-detector results. No detector has been assigned a performance value in this article. The header is an original conceptual illustration, not a screenshot of a completed experiment.

When the study is run, the report should add a frozen manifest, detector screenshots or permitted responses, anonymized raw scores, analysis code, exclusions, deviations from this protocol, and a dated change log. If a test cannot satisfy those requirements, it should be described as an exploratory comparison rather than an accuracy benchmark.