Building a Better AI Comment Classifier: Lessons from a $30 Dataset That Cost Way More

Detecting whether a code comment was written by a human or an LLM is a niche but increasingly relevant problem. One developer recently rebuilt their AI comment classifier from scratch, this time on public data and a more solid foundation, and shared the gory details of what it took.

Performance Numbers

The headline metric is a balanced accuracy of 77% on cross-validation. That means the classifier gets the human-vs-robot verdict right 77% of the time when human and robot comments are equally likely. The classifier also outputs a calibrated probability, so you can read the confidence as a true likelihood of correctness.

On a small manual test set of real-world comments, accuracy jumped to 88%, suggesting that real-world cases are actually easier to distinguish than the synthetic training data. The confusion matrix shows a true positive rate of 80% for robot comments and a true negative rate of 73% for human comments on the cross-validation set.

But the author cautions against over-indexing on aggregate error rates. Because the classifier is calibrated, you can trust the confidence score per prediction. When confidence is 80% or higher, the false positive rate drops to 5%. When it's around 50%, you're basically guessing.

Data Collection: The Devil's in the Details

The dataset was built by pulling permissively licensed repositories, checking out a 2021 commit, extracting human comments, then having LLMs generate new comments for the same files. The goal was to balance token counts across classes to avoid subject matter leakage.

The author lists several mistakes that turned a theoretical $30 dataset into a much more expensive endeavor:

• Accidentally using different source files for each LLM, causing subject matter leakage.
• Generating LLM comments for files with very few human comments, again leaking subject matter.
• Failing to strip docstrings, which let LLMs mimic repository style.
• Missing comment syntaxes in some languages, leaving contamination.
• Not filtering out very short human comments like "main task structure" or "chIcon" — teaching the classifier that humans write poorly.
• Using a fixed prompt for LLM generation, narrowing dataset variation.

Some of these were fixable with preprocessing, but others required full regeneration. The author notes that classifier power scales with the log of money spent, so diminishing returns kick in fast.

Feature Evaluation

Rather than exhaustively training on all feature subsets (30,000+ classifiers for 15 features), the author used a guided approach: train classifiers on individual features for different discrimination tasks (human vs. robot, Claude vs. Grok, GPT vs. Gemini) and rank features by their individual power.

The resulting graph shows feature rankings with arrows indicating agreement across comparisons. Features like bigwords (fraction of words longer than 5 letters) and wordlen (distribution of word lengths) are classic stylistic markers but weak discriminators on their own.

The author also built an interactive UI where you can click any part of a classified comment to see which features activate and how they contribute to the verdict — a nice debugging tool that makes the classifier's reasoning transparent.

Scope and Limitations

The classifier is trained only on code comments. It can handle other text types, but accuracy is not guaranteed. The author makes no promises beyond that domain.