A Hobbyist's Detector Caught AI Text at About 85%. The Mechanism Is Older Than the Hype.

Over a long weekend, a developer called lyc8503 built a text classifier that can sort chapters of Chinese web fiction into "human" and "large language model" with about 85% sentence-level accuracy on the author's own test set. The code is on GitHub, the model runs entirely in the browser via a pure-JavaScript demo, and the writeup is unusually candid about what that number does and does not mean. The interesting part is not the 85%. It is the stack underneath: not a transformer, not a perplexity score, but scikit-learn linear support vector machines trained on bag-of-words features. The same statistical machinery that filtered spam in the early 2010s turns out to be a stubborn detector of LLM prose, at least on this corpus.

lyc8503 scraped roughly 8,500 chapter-length samples from a Chinese fiction platform, half written by humans before ChatGPT's public release, half generated by seven LLMs the author prompted to imitate the platform's style: gemini-3-pro, qwen-coder-plus, glm-5, glm-4.7, kimi-k2.5, doubao-seed-code, and deepseek-v3.2. Each sample was turned into a TF-IDF vector over a 500,000-word vocabulary — meaning the model tracks which words and short phrases appear, and how distinctively, rather than reading for meaning. lyc8503 then trained a separate binary linear SVM per LLM family and combined the seven with a simple majority vote: if at least two of the seven flag a passage, the page lights up. Per-sentence accuracy lands in the 0.83 to 0.89 range for the final in-browser model.

That is a long way from a general AI-text detector, and the author says so. The training data is one genre on one platform in one language, so claims about "mainstream LLM-generated text" do not transfer. But the mechanism is the news: classical n-gram features can pick up surface regularities in LLM output that more elaborate approaches miss. A high-engagement Hacker News thread on the same post pushed the opposite view — that text provenance is information-sparse enough that any detector is closer to tarot card reading than signal recovery. The project, taken at face value, is counter-evidence to that claim on its own narrow turf.

The bypass tests are the most useful part of the writeup for a general reader, because they show where the detector breaks and where it does not. The author's baseline is an 89.9% score on strongly machine-flavored text. Running that text through a translation roundtrip — into another language and back — drops the score to the high 70s or mid 80s but does not erase the signal. Prompting the LLM with an "AI-flavor remover" instruction ("make this sound more human," in effect) pulls the score down from 89.9% to roughly 79–83%. Neither trick produces clean human-mimicry on this corpus. That is consistent with the mechanism story: the patterns the SVM is matching live in word choice and phrase distribution, not in surface polish, and they survive the cheap edits people actually apply.

The other number worth pausing on is not in the eval table. lyc8503 ran the detector over the top-20 trending tags on Lofter, a Chinese fanfiction platform, and found 32.22% of posts scoring above 50% AI-likelihood, with none of the flagged authors declaring AI authorship. The scan is a single weekend probe on a single platform, not a survey, but it is the kind of data point that makes the project feel less like a benchmark and more like a prototype for a reading-time problem: machine-written text is already arriving unannounced in reader-facing spaces, and the tools to flag it are uneven.

The honest read of the work is the one the author offers. Detection on prose is tractable, sometimes surprisingly so, but the social problem — the volume of low-quality machine-written text — is larger than any detector one developer can build. A 500,000-feature linear SVM running in 38 megabytes of gzipped JavaScript is a useful instrument, and a long way from a solved one.