Why is it Slop?

How It Works

A transparent look at the methodology, data, models, and limitations behind the detector.

1. The Approach

This tool identifies AI-generated text through a two-layer system:

1. 46 hand-crafted features in two complementary groups: 31 AI red flags that check for linguistic patterns statistically associated with LLM output, and 15 human green flags that detect markers of authentic human writing (disfluencies, informal language, personal voice, citations). Strong green flags push the probability down.
2. A Gradient Boosting classifier trained on all 46 feature scores, which learns the optimal combination weights from labeled data.

The ML model runs client-side via ONNX Runtime Web (WebAssembly). When the model loads successfully, the tool uses its probability as the primary signal. If it fails to load (e.g. very old browser), it falls back to a weighted heuristic sum.

Crucially, the tool doesn't just give a score — it shows you which patterns triggered and where, so you can form your own judgment about whether the flags are meaningful in context.

2. Why Heuristics + ML (Not a Black Box)

Pure neural AI detectors (like GPTZero, Originality.ai, or OpenAI's classifier) are opaque — they give a probability but can't explain why. This is a problem for educators, because:

• You can't have a conversation with a student about a probability score.
• False positives are devastating without evidence to back up the accusation.
• Neural detectors are famously unreliable on non-native English speakers, second-language learners, and highly structured academic writing.

Our approach is the opposite: every detection is grounded in observable textual evidence. If the tool flags a passage, you can hover over the trigger to see exactly which phrases, word choices, or structures caused it — and decide for yourself whether the flag is meaningful.

3. The 46 Features (31 AI Red Flags + 15 Human Green Flags)

The AI red flags fall into three groups, and are joined by 15 human green flags that score evidence of authentic human writing. Each has a weight reflecting how strongly it discriminates between AI and human text (calibrated on the training data). Negative weights belong to green flags — a high score there reduces the AI probability:

The heuristics are derived from Wikipedia's "Signs of AI Writing" article and refined through empirical testing on labeled data. They use a mix of regex pattern matching, compromise.js NLP tagging, and statistical measures.

4. Training Dataset

Training used the balanced training split (2,119,856 samples, 1,059,928 per class). Performance was estimated using 10-fold stratified cross-validation on the training set, then the final model was retrained on the full training split. The 374,092-sample held-out test set was never used during training or hyperparameter selection — it exists solely to measure real-world performance, and it's the same pool that the "Try an example" buttons and Quiz Mode draw from.

5. ML Model Details

5a. Cross-Validation Protocol

5b. Model Comparison (10-Fold CV, mean ± std)

Four classifiers were compared, all using the same 46 feature scores as input (31 AI red flags + 15 human green flags):

5c. Per-Fold Results (Gradient Boosting)

5d. Selected Model & Hyperparameters

Gradient Boosting was selected as the production model based on highest mean F1 across all 10 folds. Full hyperparameters for reproducibility:

5e. Heuristic-Only Baseline vs ML

The ML model outperforms the heuristic-only weighted-sum baseline by a significant margin (measured via 10-fold CV on the 2.1M training set):

Note: heuristic baseline has higher recall but much lower precision (many false positives). The ML model trades a small amount of recall for dramatically better precision.

5e′. Held-Out Test Set Performance

Final performance on the 374,092-sample held-out test set the model was never trained on:

5f. Top Feature Importances

paragraph_asymmetry alone accounts for 23.5% of the model's splitting decisions — uneven paragraph lengths are a strong signal of natural human writing (AI tends to produce uniform blocks). The top four features collectively account for 72.7% of model importance. Two of the top-11 features are human green flags (paragraph_asymmetry, academic_citations), directly confirming their discriminative value.

5g. Reproducibility

All parameters needed to reproduce this exact model are saved in ml_metadata.json alongside the model file. The training script is train_ml_model.py. To retrain from scratch:

python train_ml_model.py --fresh

The --fresh flag clears any cached intermediate results. Without it, the script will resume from cached feature extractions and CV results if they exist. The model is saved as ml_model.pkl and loaded automatically at server startup.

6. Verdict Tiers

7. How Trigger Ranking Works

In the detector interface, triggered patterns are ranked by importance, calculated as:

Importance = Raw Score × |Heuristic Weight|

This bubbles the most discriminative and strongest-firing patterns to the top. AI red flags and human green flags are ranked separately within their own sidebar tiers, each with a percentage showing that pattern's share of its group's total signal.

8. Caveats & Limitations

Known limitations:

• Formal writing looks like AI. Highly polished academic prose, professional copywriting, and non-native English speakers who learned formal English may trigger structural and vocabulary heuristics.
• Edited AI text evades detection. If a student rewrites AI output in their own voice, the distinctive patterns disappear. The tool detects unedited or lightly edited AI output best.
• Dataset era matters. The training data reflects AI writing patterns circa 2023–2024. As LLMs evolve and lose some of these tics, detection accuracy may degrade.
• Short text is unreliable. Fewer than ~100 words provides too little statistical signal for confident detection. Doc-level features like sentence uniformity need enough sentences to measure.
• English only. All heuristics, vocabulary lists, and training data are English. The tool should not be used on other languages.
• No individual heuristic is conclusive. "Furthermore" at the start of a sentence is perfectly valid English. It only becomes suspicious in aggregate — when combined with other AI-typical patterns. Always look at the full picture.
• Doc-level stats have no spans. Some triggers (sentence uniformity, avg word length, etc.) are statistical properties of the whole document with no specific passage to highlight. They still contribute signal but are harder to point to in conversation with a student.

9. Philosophy: Why Transparency Matters

Most AI detectors are black boxes. You paste text in, get a number out, and have no idea why. This makes them dangerous tools for high-stakes decisions like academic integrity.

This tool takes a different approach: it makes every part of its reasoning visible. You can see exactly which patterns it found, how much each one contributed, and where in the text they appear. This means:

• You can evaluate the evidence yourself rather than trusting a number.
• You can have a specific conversation with a student: "I noticed your essay uses 'Furthermore' to start three sentences and contains the phrase 'it is important to note' twice — can you walk me through your writing process?"
• You can identify false positives by seeing that the only triggers are low-signal ones like passive voice or sentence length — not the high-signal structural patterns that are truly distinctive of AI.

Built with compromise.js, scikit-learn, ONNX Runtime Web, and Flask. Detection runs entirely in your browser — no text is ever sent to a server. Source heuristics derived from Wikipedia's Signs of AI Writing.