from tf-idf to bert: lessons from clickbait classifier
· 3 min read · case study, machine learning, nlp, bert
case study of Clickbait Classifier · python · pytorch · pandas
I wanted to know how much a fine-tuned transformer actually gains over a classical model at spotting clickbait, measured on held-out data.
- Role: Solo personal project: data prep, baseline, fine-tuning, evaluation and error analysis.
- Stack: Python, PyTorch, Hugging Face Transformers (
bert-base-uncased), pandas. - Result: 0.89 F1 and 91.4% accuracy, up from a TF-IDF baseline's 0.75 F1 and 82.1%.
- When: 2026.
1. The problem #
Clickbait is about structure as much as vocabulary. "You won't believe what happened next" holds back the point; "The most mind-blowing discovery ever" oversells it. But the same words show up in real reporting, so keyword lists aren't enough.
2. Approach #
Baseline first. TF-IDF over unigrams and bigrams with regularized logistic regression. It trains in under 2 seconds, and its weights are easy to read: "SHOCKING", "UNBELIEVABLE" and "THIS REASON" get large positive weights. Its blind spot is word order:
"Why Scientists Are Shocked by New Battery Data" real news
"You Will Be Shocked by This Trick" clickbait
───────
TF-IDF sees: both contain "shocked by": similar features
BERT sees: who is shocked, and by what
Then BERT. I fine-tuned bert-base-uncased on 18,000+ labeled news headlines, with a stratified split so both sets keep the same class balance:
18,000+ labeled headlines
├── 80% train
└── 20% held-out test (never seen during training)
model = BertForSequenceClassification.from_pretrained(
'bert-base-uncased', num_labels=2, hidden_dropout_prob=0.3
)
optimizer = AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
Linear warmup over the first 10% of steps, then cosine decay, so early updates don't wreck the pretrained weights:
2e-5 ┤ ••••••••• │ • •••••••• │ • ••••• │ • •••••• │ • ••••••• 0 ┤• •••••••••• └────────────────────────────────────────────────── ↑ warmup ends at 10%, then cosine decayHidden dropout raised to 0.3 and weight decay of 0.01 for regularization.
Evaluated on a stratified, held-out 20% test split.
3. Results #
| Model | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| TF-IDF + logistic regression | 0.78 | 0.72 | 0.75 | 82.1% |
| BERT (fine-tuned) | 0.90 | 0.88 | 0.89 | 91.4% |
precision ███████████████████████████████ 0.78 TF-IDF
████████████████████████████████████ 0.90 BERT
recall █████████████████████████████ 0.72 TF-IDF
███████████████████████████████████ 0.88 BERT
F1 ██████████████████████████████ 0.75 TF-IDF
████████████████████████████████████ 0.89 BERT
accuracy █████████████████████████████████ 82.1% TF-IDF
█████████████████████████████████████ 91.4% BERT
1 block = 0.025 (accuracy: 2.5 points)
F1 rose by 0.14, and most of that came from recall (0.72 → 0.88). That suggests BERT catches clickbait that doesn't lean on the buzzwords the baseline keys on.
Reading through the errors, two patterns kept coming up:
recurring misclassifications
├── satire clickbait's structure, real events
└── real questions "Can Solar Geoengineering Reverse Warming?"
shaped like a bait question
4. Next steps #
- Fix the two failure modes: add labeled satire and real question-style headlines, and report accuracy on each group separately.
- Test on unseen publishers: split by outlet instead of at random, to check the model learned clickbait style and not which sites publish it.
- Set the threshold on purpose: choose it from the precision–recall curve for the use case (flag for review vs. filter out), not the default 0.5.
- Make it cheaper: distill to DistilBERT and compare F1 with inference speed.
What I learned #
Context beats keyword counts when the label depends on how something is said. Building the cheap baseline first was worth it: it set the bar, and it turned "BERT is better" into a measured +0.14 F1.