Transformer-Based Text Classification on Reuters Dataset

A custom Transformer encoder built from scratch in TensorFlow/Keras for 46-class news classification — with architectural depth analysis and RNN comparison.

---

Overview

This project implements a custom Transformer encoder architecture from scratch using TensorFlow and Keras for multi-class text classification on the Reuters newswire dataset — 46 categories, 11,228 samples.

Rather than fine-tuning a pre-trained model, every component of the Transformer is built and trained explicitly: token embeddings, positional embeddings, multi-head self-attention, feed-forward layers, residual connections, and layer normalization. This makes the architectural trade-offs visible and measurable.

The core research question:

How does Transformer encoder depth (3, 5, 7 layers) affect classification performance, and how does it compare to RNN-based sequence models on the same task?

---

Architecture

flowchart TD
    A[Reuters Dataset\n8,982 train · 2,246 test\n46 categories] --> B[Text Preprocessing\nVocabulary: 10k tokens\nMax length: 200]
    B --> C[Token Embedding\ndim = 32]
    C --> D[Positional Embedding\ndim = 32]
    D --> E

    subgraph Transformer["Transformer Encoder Block × N layers"]
        E[Multi-Head Self-Attention\n4 heads] --> F[Add and Norm\nResidual Connection]
        F --> G[Feed-Forward Network\nDense → ReLU → Dense]
        G --> H[Add and Norm\nResidual Connection]
    end

    H --> I[Global Average Pooling\nDocument-level representation]
    I --> J[Dense Classifier\n46-class Softmax]
    J --> K[Evaluation\nAccuracy · Weighted F1\nConfusion Matrix]

    style Transformer fill:#1a1a2e,color:#fff,stroke:#FF6F00

Loading

Depth Experiment

Three configurations trained and evaluated independently:

Transformer-3  →  3 encoder layers
Transformer-5  →  5 encoder layers
Transformer-7  →  7 encoder layers

---

Results

Transformer Depth vs. Performance

Encoder Layers	Accuracy	Weighted F1
3 (best)	0.7511	0.7461
5	0.7427	0.7377
7	0.7235	0.7156

Finding: The 3-layer Transformer achieves the best performance. Deeper configurations show decreasing accuracy and F1, indicating overfitting and optimization difficulty on a dataset of this scale — consistent with known depth-vs-data-size trade-offs in Transformer literature.

Transformer vs. Recurrent Models

Architecture	Weighted F1
Simple RNN	lower
LSTM	moderate
GRU	moderate
Bidirectional RNN	moderate
Transformer (3L)	0.7461

The 3-layer Transformer matches or outperforms all recurrent baselines, confirming the advantage of self-attention over sequential computation for multi-class news classification.

---

Model Configuration

Parameter	Value
Vocabulary size	10,000
Max sequence length	200
Embedding dimension	32
Attention heads	4
Output classes	46
Optimizer	Adam
Loss function	Sparse Categorical Crossentropy
Evaluation	Accuracy, Weighted F1, Confusion Matrix

---

Project Structure

Deep-learning/
├── KLAadithya_Deep_learning_Project.ipynb   # Full pipeline notebook
├── KLAadithya_Deep_learning_Project.pdf     # Project report
└── README.md

---

Getting Started

Prerequisites

pip install tensorflow numpy pandas matplotlib seaborn scikit-learn jupyter

Run

git clone https://github.com/Aadithya-kl/Deep-learning.git
cd Deep-learning
jupyter notebook KLAadithya_Deep_learning_Project.ipynb

The notebook is self-contained — the Reuters dataset loads automatically via tensorflow.keras.datasets.reuters.

---

Key Concepts Demonstrated

• Custom Transformer encoder implementation without pre-trained weights
• Positional embedding design for sequence-aware representations
• Multi-head self-attention mechanics and implementation
• Residual connections and layer normalization for training stability
• Architectural ablation study across encoder depths
• Controlled comparison against RNN, LSTM, GRU, and BiRNN baselines
• Multi-class evaluation with weighted F1-score to account for class imbalance

---

Visualizations

The notebook generates:

• Training and validation accuracy/loss curves per configuration
• Weighted F1-score comparison across all architectures (Transformer vs. RNN family)
• Per-class confusion matrices for each Transformer depth
• Bar charts comparing model performance across all experiments

---

Author

K L Aadithya B.Tech Computer Science and Data Science, Sai University

---