NLP

Hallucination Detection with Logistic Regression

A lightweight, from-scratch classifier flagging factual vs hallucinated summaries.

2025

OVERVIEW

The Project

A factuality classifier that labels generated text summaries as factual or hallucinated, using a custom logistic-regression model built from scratch with NumPy and Bag-of-Words features. It shows that simple, interpretable models can compete on factuality detection.

Objectives

Detect hallucinated summaries with binary classification.

Build logistic regression without external ML libraries.

Extract features with Bag of Words and analyze errors.

Tools & Technologies

PythonpandasNumPyscikit-learn (evaluation only)

METHODOLOGY

The Approach

Load the XSum Hallucination dataset (summary + is_factual).

Clean and tokenize text; build a BoW vocabulary and feature vectors.

Implement logistic regression (sigmoid + gradient descent, cross-entropy loss).

Evaluate with accuracy/precision/recall/F1 and k-fold cross-validation.

OUTCOME

Results & Learnings

Strong, consistent scores across folds despite model simplicity.

Error analysis surfaced hallucination cues: vague terms, wrong entities, abstractions.

Key Learnings

Simple models yield meaningful NLP insights; BoW is a strong baseline.
Building from scratch deepens understanding of optimization.

Related Case Studies

NLP