Dog Breed Classification using CNN-Transformer Hybrid (93% Accuracy)

This project focuses on developing an advanced deep learning-based dog breed classification system using a hybrid CNN-Transformer architecture. The model achieves 93% test accuracy on 120 different dog breeds using a combination of Xception (as a feature extractor) and Transformer blocks for enhanced pattern recognition.

Overview

The project is implemented as a Jupyter notebook (dogbreedHybrid.ipynb) that combines the power of convolutional neural networks with transformer attention mechanisms. The hybrid architecture leverages a pre-trained Xception model for feature extraction and stacks multiple Transformer blocks on top to capture complex inter-feature relationships.

Project Structure

Notebook Sections

1. Setup & Configuration

   • Mount Google Drive for data access
   • Import all necessary libraries (TensorFlow, PyTorch, timm, einops, etc.)
   • Configure constants (IMG_SIZE=224, BATCH_SIZE=32, NUM_EPOCHS=20, DEVICE='cuda')
   • Set base path for dataset access

2. Data Pipeline

   • Custom PyTorch Dataset class (DogDataset) for efficient data loading
   • Automatic label assignment from folder structure (120 dog breeds)
   • Custom 10-fold cross-validation with manual data splits
   • Three different train/val/test configurations for robust evaluation
   • Data loaders with batch processing, shuffle, and GPU optimization

3. Model Architecture - CNN-Transformer Hybrid

   • Feature Extractor: Pre-trained Xception model (frozen weights)
   • Transformer Blocks: Multi-head self-attention mechanism
     • 3 stacked transformer blocks
     • 8 attention heads
     • Feed-forward dimension: 4096
     • Dropout: 0.1 for regularization
   • Classifier: Linear layer for 120 dog breeds
   • Architecture flow: CNN → Reshape → Transformer → Global Pooling → Classification

4. Advanced Training Techniques

   • Label Smoothing Cross Entropy: Prevents overconfidence and improves generalization
   • Early Stopping: Patience-based mechanism with model checkpointing
   • AdamW Optimizer: Learning rate: 1e-4
   • Training History Visualization: Loss and accuracy plots over epochs
   • Custom Cross-Validation: Multiple data splits for robust evaluation

5. Model Evaluation

   • Comprehensive evaluation on test dataset
   • Performance metrics (accuracy, loss)
   • Multiple training runs with different data splits
   • Test accuracy reporting with CrossEntropyLoss

6. Data Analysis & Visualization

   • Class Distribution Visualization: Bar charts showing sample counts per breed
   • Min/Max Sample Analysis: Identifies breeds with lowest/highest representation
   • Random Prediction Visualization: Top-5 predictions with probability bars
   • Side-by-side image and probability distribution display

Model Architecture Details

CNN-Transformer Hybrid Components

1. Feature Extraction Layer

• Pre-trained Xception model from timm library
• Frozen weights (no fine-tuning of CNN backbone)
• Classification head removed and replaced with Identity layer
• Outputs 2048-dimensional feature maps

2. Transformer Processing

• Multiple stacked transformer blocks (default: 3)
• Each block contains:
  • Layer Normalization
  • Multi-head Self-Attention (8 heads)
  • Residual connections
  • Feed-forward network (2048 → 4096 → 2048)
  • Dropout regularization (0.1)

3. Classification Head

• Global average pooling over spatial dimensions
• Linear layer: 2048 → 120 classes

Training Configuration

Component	Value
Image Size	224 × 224
Batch Size	32
Epochs	20 (with early stopping)
Optimizer	AdamW (lr=1e-4)
Loss Function	Label Smoothing Cross Entropy (smoothing=0.1)
Early Stopping Patience	10 epochs
Data Split	Custom 10-fold cross-validation

Key Features

• Hybrid Architecture: Combines CNN feature extraction with Transformer attention mechanisms
• Transfer Learning: Leverages pre-trained Xception weights as a frozen feature extractor
• Label Smoothing: Improves generalization and prevents model overconfidence
• Custom Cross-Validation: 10-fold data split with multiple training configurations
• Advanced Training: Early stopping, model checkpointing, and AdamW optimization
• GPU Acceleration: Automatic CUDA detection and utilization for faster training
• Comprehensive Visualization: Class distribution analysis and top-5 prediction displays
• Model Persistence: Automatic saving of best models during training

Results

Performance Metrics

The CNN-Transformer Hybrid architecture achieves 93% test accuracy on the dog breed classification task across 120 different breeds. This impressive performance demonstrates the effectiveness of combining CNN feature extraction with Transformer attention mechanisms.

Key Achievements

The hybrid approach enables the model to:

• 93% Test Accuracy: Accurately classify dog breeds across 120 classes
• Capture fine-grained breed-specific visual features through CNN layers
• Model complex inter-feature relationships using multi-head self-attention
• Generalize robustly through custom 10-fold cross-validation
• Maintain architectural efficiency with frozen CNN backbone

The model undergoes multiple training rounds with different data splits to ensure robust evaluation and generalization capabilities, with the best configuration achieving 93% accuracy on the test set.

Usage

1. Setup Environment

   • Open dogbreedHybrid.ipynb in Google Colab or Jupyter
   • Mount your Google Drive
   • Ensure dataset is at drive/MyDrive/data/cropped
   • Place pre-trained Xception weights at drive/MyDrive/xception.pt

2. Data Preparation

   • Run data loading and splitting cells
   • Adjust val_index and test_indices for different cross-validation splits
   • Verify class distribution with visualization cells

3. Model Training

   • Run model architecture definition cells
   • Execute training loop with early stopping
   • Monitor training/validation loss and accuracy plots
   • Best models are automatically saved to Google Drive

4. Evaluation & Visualization

   • Load saved model checkpoint
   • Run evaluation on test set
   • Visualize random predictions with top-5 probabilities
   • Analyze class distribution across splits

Requirements

Core Libraries

• PyTorch >= 1.x: Main deep learning framework
• TensorFlow >= 2.x: For GPU detection
• timm: PyTorch Image Models (for Xception)
• einops: Tensor operations and reshaping

Data Processing

• OpenCV (cv2): Image loading and preprocessing
• NumPy: Numerical computations
• scikit-learn: Train/test splitting

Visualization & Monitoring

• Matplotlib: Plotting and visualization
• tqdm: Progress bars for training loops

Optional

• wandb: Weights & Biases for experiment tracking (imported but optional)

Installation

pip install torch torchvision timm einops opencv-python numpy scikit-learn matplotlib tqdm tensorflow

Cross-Validation Strategy

The notebook implements a custom 10-fold cross-validation approach:

• Dataset is divided into 10 equal parts
• Different configurations are tested by varying validation and test indices
• Example splits demonstrated:
  • Split 1: Parts 1-7 (train), Part 8 (val), Parts 9-10 (test)
  • Split 2: Parts 3-9 (train), Part 10 (val), Parts 1-2 (test)
  • Split 3: Parts 1,5-9 (train), Part 2 (val), Parts 3-4 (test)

Dog Breeds Covered

The model classifies 120 different dog breeds from the Stanford Dogs Dataset, including:

• Toy breeds (Chihuahua, Pomeranian, Maltese, etc.)
• Working dogs (German Shepherd, Rottweiler, Boxer, etc.)
• Sporting dogs (Golden Retriever, Labrador Retriever, English Setter, etc.)
• Hounds (Beagle, Bloodhound, Afghan Hound, etc.)
• Terriers (Yorkshire Terrier, Scottish Terrier, Airedale, etc.)
• And many more!

All breed labels are automatically extracted from the dataset folder structure following the ImageNet naming convention (e.g., n02085620-Chihuahua).

Model Checkpoints

The notebook saves model checkpoints at:

• drive/MyDrive/cnn_transformer_hybrid_best_model.pt
• drive/MyDrive/cnn_transformer_hybrid_optimized_best_model.pt

These checkpoints can be loaded for inference without retraining the model.