🏁 CodeWizards 2.0 · Duality AI — Offroad Semantic Scene Segmentation

Team Vortex · Ajay Kumar Garg Engineering College
SRMIST Delhi-NCR Campus · 17–18 April 2026

---

📁 Google Drive — Project Assets

All training checkpoints, backup runs, and the full dataset are hosted on Google Drive.

→ Access the Full Project Drive Folder

The Drive contains:

• Offroad_Segmentation_Training_Dataset/ — official train / val / test splits
• offroad_project_backup/ — all Colab training runs and checkpoint history
• CodeWizards_Offroad_Submission/ — this submission package

---

🧠 What This Project Does

We tackle Duality AI's Offroad Semantic Scene Segmentation challenge — classifying every pixel of synthetic desert RGB images into 10 terrain classes using the Falcon digital-twin pipeline.

Our solution fine-tunes SegFormer-B2 (nvidia/mit-b2, ImageNet-22k pretrained) with:

• Median-frequency class weighting to handle severe pixel imbalance
• Strong colour + geometric augmentation for domain robustness
• Mixed-precision (AMP) training for GPU efficiency
• Multi-scale + horizontal flip TTA at inference — boosting mIoU without touching test labels

Setting	Value
Backbone	nvidia/mit-b2
Classes	10 terrain classes
Image size	512 × 512
Optimizer	AdamW (lr=6e-5, wd=0.01)
Epochs	60
Val mIoU (flip TTA)	~0.658
Val mIoU (multi-scale + flip TTA)	~0.662

---

📋 Compliance

Rule	Practice
Train only on train/ (+ val/ for metrics)	✅ Yes
Never train or tune using testImages/ labels	✅ Test folder has RGB only — no masks used
Outputs use official class IDs in PNGs	✅ Saved as uint16 with values 100, 200, … 10000

---

🗂️ Repository Layout

Duality-AI-main/
├── README.md
├── requirements.txt
├── SUBMISSION_CHECKLIST.md
├── run_evaluate.bat          ← Windows shortcut
├── run_test.bat              ← Windows shortcut
│
├── configs/
│   ├── config.yaml           ← Default: Google Colab / Drive paths
│   └── config.windows.yaml  ← Local Windows (P:\ drive) paths
│
├── src/
│   ├── dataset.py            ← Custom Dataset + Albumentations augmentation
│   ├── model.py              ← SegFormerWrapper + checkpoint normalization
│   ├── train.py              ← Training loop (AMP, checkpointing, resume)
│   ├── test.py               ← Inference + multi-scale TTA → uint16 PNGs
│   ├── evaluate.py           ← Quick per-epoch mIoU
│   ├── evaluate_full.py      ← Full report: mIoU, confusion matrix, per-class IoU
│   ├── eda.py                ← Class distribution analysis
│   ├── bench_infer.py        ← Latency benchmark (median ms)
│   └── visualize.py          ← Side-by-side prediction panels
│
├── runs/
│   ├── checkpoints/          ← Place best_model.pth here (copy from Drive backup)
│   ├── training_curves.png
│   ├── confusion_matrix.png
│   ├── per_class_iou.png
│   └── class_distribution.png
│
├── predictions/              ← Output: <stem>_pred.png (uint16, official IDs)
│
└── report/
    ├── CODEWIZARDS_REPORT.md
    └── CODEWIZARDS_REPORT.docx

---

⚙️ Setup

Prerequisites

• Python 3.10 (Miniconda / Anaconda recommended)
• NVIDIA GPU with CUDA (for training); CPU inference is possible but slow

1 — Create environment

conda create -n EDU python=3.10 -y
conda activate EDU
cd CodeWizards_Offroad_Submission
pip install -r requirements.txt

2 — Install CUDA-enabled PyTorch (if needed)

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

3 — Verify GPU

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU only')"

---

🔧 Configuration

Two config files are provided under configs/. Copy the right one to configs/config.yaml before running.

Google Colab / Drive (default)

configs/config.yaml is pre-configured for Colab after mounting Drive:

data:
  train_rgb:   "/content/drive/MyDrive/Offroad_Project/Offroad_Segmentation_Training_Dataset/train/Color_Images"
  train_masks: "/content/drive/MyDrive/Offroad_Project/Offroad_Segmentation_Training_Dataset/train/Segmentation"
  val_rgb:     "/content/drive/MyDrive/Offroad_Project/Offroad_Segmentation_Training_Dataset/val/Color_Images"
  val_masks:   "/content/drive/MyDrive/Offroad_Project/Offroad_Segmentation_Training_Dataset/val/Segmentation"
  test_rgb:    "/content/drive/MyDrive/Offroad_Project/Offroad_Segmentation_Training_Dataset/testImages"

Shared Drive note: If the dataset is "Shared with me", right-click Offroad_Project → Add shortcut to Drive so the MyDrive/... path resolves in Colab.

Local Windows

# Copy windows config over the default
copy configs\config.windows.yaml configs\config.yaml

data:
  train_rgb:   "P:/SRM Hackathon/Offroad_Segmentation_Training_Dataset/train/Color_Images"
  # ... adjust P:\ paths to your local extract

---

☁️ Google Colab Quick Start

from google.colab import drive
drive.mount('/content/drive')

%cd /content/drive/MyDrive/Offroad_Project/CodeWizards_Offroad_Submission
!pip install -r requirements.txt
!python src/evaluate_full.py

Always run commands from the CodeWizards_Offroad_Submission root so runs/checkpoints resolves correctly.

---

🏋️ Placing Trained Weights

Training saves checkpoints to offroad_project_backup/ on Drive. Copy your best checkpoint into this repo:

runs/checkpoints/best_model.pth

Supported checkpoint formats (all handled automatically by normalize_training_state_dict() in src/model.py):

Source	Key prefix	Handled?
This repo's train.py	model.segformer.*	✅
Colab / Hugging Face raw	segformer.*, decode_head.*	✅ auto-remapped
DDP multi-GPU	module.* prefix	✅ stripped

---

🚀 Commands

Run all commands from the project root.

Task	Command
Class distribution analysis	python src/eda.py
Training (if re-running)	python src/train.py
Full validation report (mIoU + plots)	python src/evaluate_full.py
Inference speed benchmark	python src/bench_infer.py
Generate test predictions	python src/test.py
Visualize predictions (8 samples)	python src/visualize.py --limit 8

Outputs:

• Predictions → predictions/<stem>_pred.png (uint16, official Falcon class IDs)
• Plots → runs/training_curves.png, runs/confusion_matrix.png, runs/per_class_iou.png

---

🔍 Inference Settings

Configured in configs/config.yaml under inference::

inference:
  use_multiscale_tta: true         # Multi-scale + flip averaging (recommended)
  tta_scales: [448, 512, 576]      # Reduce if GPU OOM (remove largest scale)

TTA recommendation: Keep use_multiscale_tta: true for submission-quality predictions. It provides a ~0.4% mIoU gain over single-scale without touching any test labels.

---

📊 Class Taxonomy

Raw Falcon ID	Class	Notes
100	Trees	Sparse vegetation
200	Lush Bushes	Often confused with Dry Bushes
300	Dry Grass	Large region coverage
500	Dry Bushes	Frequent desert vegetation
550	Ground Clutter	Rarest class; hardest to segment
600	Flowers	Small, rare pixels
700	Logs	Thin structures
800	Rocks	High texture diversity
7100	Landscape	Dominant ground cover
10000	Sky	Dominant background

Internal training uses 0–9 indices with ignore_index=255 for void pixels. Output PNGs encode the original Falcon IDs as uint16.

---

📈 Results

Evaluation Mode	mIoU
Single forward (evaluate_full.py)	0.6556
Checkpoint metadata (best epoch)	0.6576
Flip TTA (Colab eval)	~0.658
Multi-scale + flip TTA	~0.662 ✅

The mIoU improvement from ~0.658 → ~0.662 comes entirely from inference-time ensembling on the same weights — no additional training data or test labels used.

Strongest classes: Sky, Trees, Landscape
Weakest classes: Ground Clutter, Dry Bushes, Rocks (confusion with adjacent texture classes)

---

✅ Pre-Submission Checklist

• configs/config.yaml — all five data paths exist on your machine
• runs/checkpoints/best_model.pth — final weights copied from Drive backup
• python src/evaluate_full.py — completes cleanly; mIoU noted for report
• python src/test.py — predictions/ has one *_pred.png per test image
• python src/bench_infer.py — median ms recorded for report
• report/CODEWIZARDS_REPORT.md — team name filled, PNGs embedded, exported to PDF
• Zip folder + PDF → Vortex_CodeWizards_Submission.zip

---

👥 Team

Name	Role
Shivanshu Tiwari	Model architecture & training pipeline
Anirudh Agarwal	Data augmentation & evaluation
Mohammad Raees	Inference, TTA & benchmarking
Aditya Gaur	EDA, visualisation & report

Institution: Ajay Kumar Garg Engineering College
Team Name: Vortex
Event: CodeWizards 2.0 · SRMIST Delhi-NCR · April 2026
Challenge: Powered by Duality AI — Falcon digital-twin platform

---

_{Built for CodeWizards 2.0 · Duality AI Offroad Segmentation Challenge · April 2026}