ITWS 4100 - Team 8 Final Project (Client - Twinner)

A full-stack web application for uploading, classifying, and visualizing 3D point cloud data using AI-powered object recognition and interactive 3D rendering.

---

Table of Contents

• Overview
• Features
• Tech Stack
• Third-Party Libraries
• Architecture
• Getting Started
• Usage
• Project Structure
• Development
• Production Deployment
• API Documentation
• Contributing

---

Overview

This platform combines machine learning and 3D visualization to provide a comprehensive solution for point cloud data analysis:

• PointNet Classification: Upload .txt point clouds and get instant AI-powered object classification (40 classes including chairs, desks, lamps, airplanes, etc.)
• Potree Visualization: Upload .laz/.las files and explore massive 3D datasets interactively in your browser
• Real-time Processing: Asynchronous job queue handles long-running conversions without blocking
• Containerized Deployment: Docker Compose ensures consistent environments across dev and production

---

Features

AI-Powered Classification

• PointNet Deep Learning Model trained on ShapeNet/ModelNet40 (40 object classes)
• Upload .txt point cloud files for instant classification
• View confidence scores and top-5 predictions
• Interactive 3D visualization of uploaded point clouds using Plotly

PointNet++ Semantic Segmentation

• PointNet++ semantic segmentation model for per-point scene understanding
• Upload a .txt point cloud (with optional RGB) and receive per-point semantic labels
• Sliding-window block decomposition with overlapping windows and majority-vote ensemble (3 passes)
• Colored .obj output written to the visual directory, browsable via the frontend viewer
• Supports custom class sets via a JSON class-names file — not limited to S3DIS/indoor categories
• Two upload modes: individual file upload (file + weights + class_names_file) or a single bundled .zip

Interactive 3D Visualization

• Potree WebGL renderer for massive point cloud datasets (millions/billions of points)
• Upload .laz/.las files with drag-and-drop interface
• Progressive loading with Level of Detail (LOD)
• Pan, rotate, zoom, and measure tools
• Manage multiple point clouds with status tracking (processing, ready, failed)

Developer-Friendly

• Hot reload in development (Vite HMR + Flask debug mode)
• Docker Compose for one-command setup
• RESTful API design
• Clean separation between frontend and backend
• Comprehensive error handling and logging

---

Tech Stack

Frontend

• React 19 - UI framework
• Vite 7 - Build tool & dev server
• Tailwind CSS 4 - Styling
• Plotly.js - 3D visualization
• React Router 7 - Client-side routing

Backend

• Flask 3 - Web framework
• TensorFlow 2.17 - Deep learning (PointNet classification)
• PyTorch - Deep learning (PointNet++ segmentation)
• PointNet - 3D object classification model
• PointNet++ (via third_party/Pointnet_Pointnet2_pytorch) - Semantic segmentation model
• scikit-learn 1.5.2 - Used by the segmentation pipeline utilities
• Potree - Point cloud visualization library
• PotreeConverter - LAS/LAZ to octree conversion

Infrastructure

• Docker & Docker Compose - Containerization
• Python 3.11 - Backend runtime
• Node.js - Frontend build environment

---

Third-Party Libraries

backend/third_party/Pointnet_Pointnet2_pytorch

This directory contains a vendored copy of the Pointnet_Pointnet2_pytorch repository by Xu Yan et al., used under the MIT License.

Only the files needed at runtime are committed to this repo:

• models/pointnet2_sem_seg.py — the PointNet++ semantic segmentation architecture
• data_utils/indoor3d_util.py — the standard S3DIS label-to-color mapping (g_label2color) used when coloring OBJ outputs
• official_single_scene_infer.py — the canonical single-scene inference script that ml_segmenter.py wraps and extends

Large training data (data/) and generated visual outputs (log/.../visual/) are excluded via .gitignore. Place your trained segmentation weights (.pth checkpoint) anywhere accessible to the container and pass the path via the /segment API.

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CLIENT BROWSER                           │
│  React Frontend (localhost:5173 dev / localhost:80 prod)    │
└────────────────┬────────────────────────────────────────────┘
                 │ REST API (HTTP/JSON)
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  Flask Backend (localhost:8080)                             │
│  ├─ /predict                    → PointNet classification   │
│  ├─ /segment                    → PointNet++ segmentation   │
│  ├─ /official_segment_files     → List segmentation results │
│  ├─ /official_segment_result/<> → Serve segmentation result │
│  ├─ /api/pointclouds            → List & manage point clouds │
│  ├─ /viewer/<name>              → Potree 3D viewer           │
│  └─ Worker Thread               → Async LAS/LAZ conversion  │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  Data Storage                                               │
│  ├─ backend/log/model.ckpt              → PointNet weights  │
│  ├─ backend/data/raw/                   → Uploaded files    │
│  ├─ backend/data/pointclouds/           → Converted octrees │
│  └─ third_party/.../visual/            → Segmentation OBJs  │
└─────────────────────────────────────────────────────────────┘

---

Getting Started

See QUICKSTART.md for full setup instructions including:

• Updating NVIDIA drivers for GPU acceleration
• Installing Docker Desktop
• Cloning the PointNet++ third-party repo
• All Docker commands for development and production

---

Usage

1. PointNet Classification

1. Navigate to "Try Classifier" from the home page
2. Upload a .txt point cloud file (format: each line contains X Y Z coordinates)
3. View the 3D visualization in Plotly
4. See the predicted object class and confidence score
5. Example classes: chair, desk, lamp, airplane, car, etc.

2. PointNet++ Semantic Segmentation

1. Navigate to the Segmentation page from the home page
2. Choose an upload mode:
   • Individual files: Upload a .txt point cloud, a .pth weights file, and a class names .json
   • Bundle zip: Pack all three into a single .zip and upload that
3. The backend runs the segmentation pipeline and returns per-point labels
4. Use "View Official Results" to browse previously computed .obj segmentation outputs

Point cloud format (.txt): one point per line, whitespace-separated:

X Y Z           # minimum — RGB defaults to zero
X Y Z R G B    # optional RGB in [0, 255] or [0, 1]

Class names JSON — any of the following formats are accepted:

["ceiling", "floor", "wall", "beam", "column", "window", "door",
 "table", "chair", "sofa", "bookcase", "board", "clutter"]

{"classes": ["ceiling", "floor", ...]}

{"0": "ceiling", "1": "floor", ...}

3. Potree 3D Visualization

1. Navigate to "Open Viewer" from the home page
2. Click "Upload Point Cloud"
3. Drag & drop a .laz or .las file
4. Add a name and description
5. Wait for processing to complete (status shows in sidebar)
6. Click on the point cloud name to launch the interactive 3D viewer

---

Project Structure

ITWS-Capstone-Project-Team8/
├── frontend/                  # React application
│   ├── src/
│   │   ├── pages/            # PointNetPage, PotreePage
│   │   ├── components/       # Reusable UI components
│   │   └── main.jsx          # App entry point
│   ├── package.json          # Node dependencies
│   ├── vite.config.js        # Vite configuration
│   └── Dockerfile            # Frontend container
│
├── backend/                   # Flask application
│   ├── server.py             # Main API server
│   ├── worker.py             # Background job processor
│   ├── ml_segmenter.py       # PointNet++ segmentation pipeline wrapper
│   ├── segmenter.py          # Legacy cluster-based segmenter (kept for reference)
│   ├── models/               # PointNet (TF) architecture
│   │   └── pointnet_cls.py   # Classification model
│   ├── log/                  # Pre-trained model checkpoints
│   │   └── model.ckpt.*      # PointNet weights
│   ├── potree/               # Potree library & converter
│   ├── data/                 # Storage for point clouds
│   │   ├── raw/              # Uploaded LAS/LAZ files
│   │   └── pointclouds/      # Converted Potree octrees
│   ├── third_party/          # Vendored third-party repos (git-ignored)
│   │   └── Pointnet_Pointnet2_pytorch/  # PointNet++ (PyTorch) — clone separately
│   │       ├── models/       # pointnet2_sem_seg.py architecture
│   │       ├── data_utils/   # indoor3d_util.py (label→color map)
│   │       ├── log/sem_seg/pointnet2_sem_seg/visual/  # Segmentation OBJ outputs
│   │       └── official_single_scene_infer.py  # Canonical inference script
│   ├── requirements.txt      # Python dependencies
│   └── Dockerfile            # Backend container
│
├── docker-compose.yml         # Development orchestration
├── docker-compose.prod.yml    # Production orchestration
└── README.md                  # This file

---

Development & Deployment

All Docker commands (development, production, logs, shells, rebuilds) are documented in QUICKSTART.md.

Hot-Reload Behaviour

• Frontend: Changes in frontend/src/ auto-reload instantly (Vite HMR)
• Backend: Changes in backend/*.py auto-reload Flask server
• No restart needed for most code changes

Custom Domain (Production)

Create frontend/.env with:

VITE_API_URL=https://your-domain.com:8080

Then rebuild: docker compose -f docker-compose.prod.yml up --build -d

---

API Documentation

Classification Endpoint

POST /predict

Upload a point cloud for classification.

Request:

• Method: POST
• Content-Type: multipart/form-data
• Body: file (.txt format, each line: X Y Z)

Response:

{
  "predicted_label": "chair",
  "predicted_index": 8,
  "confidence": 94.23
}

---

Segmentation Endpoint

POST /segment

Run the PointNet++ semantic segmentation pipeline on a single scene.

Mode 1 — Individual files:

• Method: POST
• Content-Type: multipart/form-data
• Body:
  • file: .txt point cloud (XYZ or XYZRGB)
  • weights: .pth model checkpoint
  • class_names_file: .json class-name definition

Mode 2 — Bundle zip:

• Method: POST
• Content-Type: multipart/form-data
• Body: bundle_zip: .zip containing scene.txt, weights.pth, and class_names.json

Response:

{
  "num_points": 50000,
  "num_instances": 5,
  "instances": [
    {
      "id": 0,
      "semantic_id": 0,
      "semantic_label": "ceiling",
      "count": 12000,
      "bbox_min": [-1.0, -1.0, 0.9],
      "bbox_max": [1.0, 1.0, 1.0]
    }
  ],
  "points": [[x, y, z, label_id, label_id, 1.0], ...]
}

---

List Official Segmentation Results

GET /official_segment_files

Lists all *_pred.obj files produced by previous segmentation runs.

Response:

{
  "files": ["scene_001_pred.obj", "office_B_pred.obj"]
}

---

Load a Segmentation Result

GET /official_segment_result/<filename>

Loads a named OBJ from the visual output directory and returns it in the frontend viewer JSON format. Downsamples to at most 200,000 points before returning.

Response:

{
  "file_name": "scene_001_pred.obj",
  "num_points": 75000,
  "num_points_returned": 75000,
  "num_instances": 13,
  "instances": [...],
  "points": [[x, y, z, label_id, label_id, 1.0], ...]
}

---

List Point Clouds

GET /api/pointclouds

Returns all point clouds grouped by status.

Response:

{
  "ready": [
    ["building_scan", null, "Office building"],
    ["landscape", null, "Outdoor terrain"]
  ],
  "processing": [
    ["new_scan", null, "Processing..."]
  ],
  "failed": [
    ["bad_file", "Conversion error", "Invalid format"]
  ]
}

---

Upload Point Cloud

POST /api/pointclouds/upload

Queue a LAS/LAZ file for conversion.

Request:

• Method: POST
• Content-Type: multipart/form-data
• Body:
  • file: .laz or .las file
  • name: Unique identifier
  • description: Optional description

Response:

{
  "status": "queued",
  "name": "building_scan"
}

---

Delete Point Cloud

POST /api/pointclouds/delete

Remove a point cloud and its metadata.

Request:

{
  "name": "building_scan"
}

Response:

{
  "status": "deleted"
}

---

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

Notes

• PointNet model is pre-trained on ShapeNet/ModelNet40 (40 object classes)
• PointNet++ segmentation requires a separate .pth checkpoint — not included in the repo
• third_party/Pointnet_Pointnet2_pytorch source files are committed; only large data/output directories are gitignored
• Potree conversion can take several minutes for large files
• Development mode uses volume mounts for hot-reload
• Production mode bakes code into images (rebuild required for updates)
• GPU acceleration: the segmentation pipeline auto-detects CUDA; CPU fallback is used if unavailable

---

License

This project is part of the ITWS Capstone Project - Team 8.

The vendored third_party/Pointnet_Pointnet2_pytorch library is © Xu Yan and contributors, distributed under the MIT License.

---

Acknowledgments

• PointNet: Qi et al., "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" (CVPR 2017)
• PointNet++: Qi et al., "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space" (NeurIPS 2017)
• Pointnet_Pointnet2_pytorch: Xu Yan — PyTorch implementation of PointNet/PointNet++ used as the segmentation backbone (GitHub)
• Potree: High-performance WebGL point cloud renderer
• ShapeNet/ModelNet40: 3D object datasets for training
• S3DIS: Indoor 3D dataset; color mapping convention used for segmentation outputs

---