Spotting AI writing: How reliable are the detectors?
Paris, FRANCE—Claims that Canadian-Haitian writer Thelyson Orelien used artificial intelligence (AI) tools to write his best-selling debut novel are rocking the French literary world and posing questions about how AI can be detected.
An X account with a handful of followers sparked the controversy by claiming that the AI detecting program Pangram had concluded the tale of a Haitian migrant traveling to Canada had been largely written by a text-generator, such as Claude or ChatGPT.
Orelien, 38, denies the allegations and has said he will prove that he wrote the book, which has been sold to publishers in more than 20 countries.
“I didn’t write this novel with my head, but with my guts and my heart,” he told Agence France-Presse (AFP) recently.
AFP ran excerpts of the book through various AI detection tools and came up with a variety of results—from it being “very likely” to have been written by a human to “100 percent” AI generated.
How do AI detectors work?
Dozens of tools are now available online to detect AI generated text—many of them designed for use in education or publishing.
They generally have large troves of AI-generated text alongside content written by humans—and compare the two.
Their performance has improved significantly in recent years.
In a paper published in October 2025 called “Artificial Writing and Automated Detection,” researchers at the University of Chicago found Pangram “achieving near-zero error rates” on medium-to-long extracts.
Thierry Poibeau, a researcher at France’s national CNRS research institute and specialist in natural language processing, said the programs look for “linguistic tics.”
“The classic example is triadic phrasing (using three clauses), or em dashes,” Poibeau explained.
These telltale signs used to be common in generated texts because the tools had been trained on American scientific writing or American novels, he said.
Pangram claims it is able to “detect AI text by analyzing and understanding subtle cues in the writing.”
Some words or phrases show up regularly, while monotonous sentence structures are also a warning sign.
Are the tools reliable?
Many tools publish percentage figures about their accuracy, but the figures are impossible to verify and do not take into account the risk of a false positive.
Pangram CEO and cofounder Max Spero stressed that the tool should not be the ultimate arbiter, but rather a starting point for a more in-depth investigation.
According to Poibeau, these detection systems also run up against a “moving target problem” because AI programs are “constantly evolving.”
The expert also points to a potential problem when it comes to detecting the use of AI in novels.
“The tool may pick up something particular in the writing, something repetitive, but that might be exactly what constitutes a writer’s style,” he said.
