Clean up data pipeline + docs, add batch local YOLO import + FP review

.gitignore

@@ -82,6 +82,9 @@ celerybeat-schedule
 # Environments
 *.env
 .venv
+
+# Per-environment credential files (do not commit)
+creds.json
 env/
 venv/
 ENV/

CLAUDE.md

@@ -0,0 +1,144 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Pyro-Annotator is a suite for annotating wildfire detection sequences. It combines a FastAPI backend, a React frontend, and data transfer scripts to collect, manage, and annotate fire detection data from multiple sources.
+
+## Repository Structure
+
+```
+pyro-annotator/
+├── annotation_api/      # FastAPI backend + data transfer scripts
+├── frontend/            # React/TypeScript annotation UI
+├── sam_based_bbox_propagation/  # SAM-based semi-automatic bbox tool (Dash, port 8050)
+├── docker-compose.yml   # Full stack orchestration
+└── Makefile             # Docker build/push targets
+```
+
+Each submodule has its own `CLAUDE.md` with detailed context — read those when working within a specific module.
+
+## Quick Start
+
+```bash
+# Start all services (PostgreSQL, LocalStack S3, API, Frontend)
+docker compose up -d
+
+# Services:
+# Frontend:  http://localhost:3000
+# API:       http://localhost:5050  (docs at /docs)
+# Database:  localhost:5432
+# S3:        http://localhost:4566
+```
+
+## Backend (`annotation_api/`)
+
+**Stack**: FastAPI, Python 3.11+, uv, PostgreSQL (SQLModel/SQLAlchemy), S3-compatible storage, JWT auth
+
+```bash
+cd annotation_api
+
+# Dev environment via Docker
+make start          # Start dev containers
+make stop           # Stop (preserves data)
+make clean          # Remove containers and volumes
+
+# Local dev (requires uv: curl -LsSf https://astral.sh/uv/install.sh | sh)
+uv sync --group dev
+uv run uvicorn app.main:app --reload --app-dir src
+
+# Quality
+make lint           # Format check + ruff + mypy
+make fix            # Auto-fix formatting/lint issues
+
+# Tests
+make test                                          # Full suite in Docker
+make test-specific TEST=tests/test_foo.py::test_bar  # Single test
+```
+
+## Frontend (`frontend/`)
+
+**Stack**: React 18, TypeScript, Vite, Tailwind CSS, Zustand, TanStack Query v5, Axios
+
+```bash
+cd frontend
+npm install
+npm run dev          # Dev server at http://localhost:5173
+
+npm run build        # TypeScript compile + Vite build
+npm run lint         # ESLint (strict)
+npm run type-check   # TypeScript check
+npm run quality      # All checks
+npm run quality:fix  # Fix all issues
+npm run test         # Vitest
+```
+
+## Data Transfer Scripts
+
+Scripts live in `annotation_api/scripts/data_transfer/ingestion/platform/`. Run from `annotation_api/` with `make` targets.
+
+### TP Pipeline (true positives — fire sequences)
+
+Pull annotated sequences, enrich bboxes with YOLO, visual check, push back.
+
+```bash
+cd annotation_api
+
+# 1. Pull seq_annotation_done sequences (remote → local files, marks remote in_review)
+make pull-seq-annotations MAX_SEQUENCES=20
+
+# 2. Auto-fill missing bboxes with YOLO model
+make auto-annotate
+
+# 3. Visual review in FiftyOne — tag "issue" on bad frames
+make visual-check
+
+# 4. Push results: clean → annotated, issue → needs_manual
+make apply-review
+```
+
+### FP Pipeline (false positives — no fire sequences)
+
+Pull sequences, visually confirm no fire was missed, push back with empty labels.
+
+```bash
+cd annotation_api
+
+# 1. Pull seq_annotation_done sequences (separate output dir)
+make pull-fp MAX_SEQUENCES=20
+
+# 2. Visual check in FiftyOne — tag "issue" if fire was missed
+make visual-check-fp
+
+# 3. Push results: clean → annotated (no labels), issue → needs_manual
+make apply-review-fp
+```
+
+### Platform Import
+
+Credentials live in `annotation_api/.env` (copy from `.env.example` once). Each data-transfer script loads it at startup via `python-dotenv`; Make does not parse `.env`.
+
+```bash
+# Import sequences from platform API (.env must define PLATFORM_LOGIN, PLATFORM_PASSWORD,
+# PLATFORM_ADMIN_LOGIN, PLATFORM_ADMIN_PASSWORD, MAIN_ANNOTATION_LOGIN, MAIN_ANNOTATION_PASSWORD)
+make import-platform DATE_FROM=2024-01-01 DATE_END=2024-01-02
+```
+
+## Key Architecture Concepts
+
+**Backend data flow**: Platform API → ingestion scripts → annotation_api DB → frontend UI → human annotations
+
+**Processing stages** (sequence): `IMPORTED` → `READY_TO_ANNOTATE` → `UNDER_ANNOTATION` → `SEQ_ANNOTATION_DONE` → `IN_REVIEW` → `ANNOTATED`
+
+**Backend patterns**: CRUD modules per entity, Pydantic schemas separate from SQLModel, dependency injection, fastapi-pagination, IoU-based annotation generation service.
+
+**Frontend patterns**: Zustand for client state, TanStack Query for server state, canvas-based bbox drawing utilities, 13+ focused utility modules in `src/utils/`.
+
+## Pre-commit Hooks
+
+```bash
+# Hooks run: ruff (format + lint), mypy, prevents commits to main
+pre-commit install   # Install hooks
+pre-commit run --all-files  # Run manually
+```

README.md

@@ -48,11 +48,12 @@ npm run dev  # Vite dev server on port 5173
 
 All workflows below assume the services are running locally (`docker compose up -d`) and that you have credentials to the remote annotation API. Run the `make` targets from the `annotation_api/` directory.
 
-Before anything that talks to the remote API, export your credentials once per shell (or put them in `.envrc`):
+Before anything that talks to the remote API, configure your credentials in `annotation_api/.env` (loaded by the data-transfer scripts at startup via python-dotenv):
 
 ```bash
-export MAIN_ANNOTATION_LOGIN=<remote_user>
-export MAIN_ANNOTATION_PASSWORD=<remote_pass>
+cd annotation_api
+cp .env.example .env
+# then edit .env and set MAIN_ANNOTATION_LOGIN / MAIN_ANNOTATION_PASSWORD
 ```
 
 All make targets accept variable overrides inline, e.g. `make pull-sequences MAX_SEQUENCES=50 CLONE_STAGE=under_annotation`. Common variables: `REMOTE_API`, `LOCAL_API`, `MAX_SEQUENCES`, `CLONE_STAGE`, `DATA_ROOT`, `SMOKE_TYPE`, `DATASET_NAME`, `LOGLEVEL`. See `make help` for the full list.
@@ -116,6 +117,30 @@ make apply-review
 - To preview changes without writing to the API, call the underlying script with `--dry-run`.
 - Override `DATASET_NAME` / `DATA_ROOT` if you used non-default values.
 
+#### C. False-Positive (FP) Review
+
+For sequences with no fire (`smoke_types` is empty), confirm they are true false positives and push them as annotated with empty labels.
+
+**Step 1 — `pull-fp`**: pull `seq_annotation_done` FP sequences locally (moves remote stage to `in_review`):
+
+```bash
+make pull-fp MAX_SEQUENCES=20
+```
+
+**Step 2 — `visual-check-fp`**: review in FiftyOne — tag frames with `"issue"` if fire was actually missed:
+
+```bash
+make visual-check-fp
+```
+
+**Step 3 — `apply-review-fp`**: push results back to the remote API:
+
+```bash
+make apply-review-fp
+```
+- Clean sequences (no `"issue"` tags) → moved to `annotated` with empty labels (confirmed FP).
+- Issue sequences → moved to `needs_manual` for reannotation.
+
 ##### Other commands
 
 **Reset stages on the remote API** (e.g., move `in_review` back to `seq_annotation_done` to retry a workflow):
@@ -133,8 +158,9 @@ make update-stage-local FROM_STAGE=seq_annotation_done TO_STAGE=needs_manual MAX
 **Export images + YOLO labels from the remote API** (use smaller pages and a longer timeout for large datasets):
 
 ```bash
-make export-dataset USERNAME=<remote_user> PASSWORD=<remote_pass> OUTPUT_DIR=outputs/datasets LIMIT=1000 TIMEOUT=120
+make export-dataset OUTPUT_DIR=outputs/datasets LIMIT=1000 TIMEOUT=120
 ```
+- Filter by category: `make export-dataset CATEGORY=fp` (also `wildfire`, `other_smoke`). Omit to export all.
 
 **Import a single sequence from an exported YOLO folder** (images + labels) into an API:
 
@@ -156,14 +182,20 @@ make import-yolo-sequence \
 
 If you manage the main dataset and have platform credentials, import directly from the platform into the target annotation API. This is the only entry point that brings new data into the system.
 
-```bash
-export PLATFORM_LOGIN=<platform_user>
-export PLATFORM_PASSWORD=<platform_pass>
-export PLATFORM_ADMIN_LOGIN=<platform_admin_user>
-export PLATFORM_ADMIN_PASSWORD=<platform_admin_pass>
-export MAIN_ANNOTATION_LOGIN=<target_user>
-export MAIN_ANNOTATION_PASSWORD=<target_pass>
+Set the platform + target credentials in `annotation_api/.env` (see `.env.example`):
+
+```
+PLATFORM_LOGIN=...
+PLATFORM_PASSWORD=...
+PLATFORM_ADMIN_LOGIN=...
+PLATFORM_ADMIN_PASSWORD=...
+MAIN_ANNOTATION_LOGIN=...
+MAIN_ANNOTATION_PASSWORD=...
+```
+
+Then run:
 
+```bash
 cd annotation_api
 make import-platform DATE_FROM=2025-03-04 DATE_END=2025-03-04 MAX_SEQUENCES=10
 ```
@@ -184,19 +216,24 @@ docker compose up -d
 curl http://localhost:5050/docs
 ```
 
-**Required Environment Variables:**
-```bash
+**Required Environment Variables (in `annotation_api/.env`):**
+
+Copy `annotation_api/.env.example` to `annotation_api/.env` and fill in the values you need:
+
+```
 # Remote annotation API credentials (required for all workflows)
-export MAIN_ANNOTATION_LOGIN="remote_user"
-export MAIN_ANNOTATION_PASSWORD="remote_pass"
+MAIN_ANNOTATION_LOGIN=remote_user
+MAIN_ANNOTATION_PASSWORD=remote_pass
 
 # Platform API credentials (admin ingestion only)
-export PLATFORM_LOGIN="your_platform_username"
-export PLATFORM_PASSWORD="your_platform_password"
-export PLATFORM_ADMIN_LOGIN="your_admin_username"
-export PLATFORM_ADMIN_PASSWORD="your_admin_password"
+PLATFORM_LOGIN=your_platform_username
+PLATFORM_PASSWORD=your_platform_password
+PLATFORM_ADMIN_LOGIN=your_admin_username
+PLATFORM_ADMIN_PASSWORD=your_admin_password
 ```
 
+Each data-transfer script loads `annotation_api/.env` via `python-dotenv` at startup — no shell `export` or manual `source` needed. (Make does **not** parse `.env`, because Make's variable expansion would mangle values containing `$`, spaces, or quotes.) Shell-level env vars still take priority, so you can override per-invocation with `MAIN_ANNOTATION_LOGIN=foo make ...`.
+
 ### Deployment Environments
 
 **Local Development (default):**

annotation_api/.env.example

@@ -0,0 +1,35 @@
+# Copy to .env and fill in real values.
+#
+# Each data-transfer script loads this file via python-dotenv at startup,
+# which respects dotenv quoting (single quotes for literal $, etc.).
+# Make does NOT parse .env (it would mangle values with $ / quotes / spaces).
+# To override a value per-invocation, pass it in the shell environment:
+#   MAIN_ANNOTATION_LOGIN=other_user make export-dataset
+
+# --- Database ---
+POSTGRES_USER=dbadmin
+POSTGRES_PASSWORD=changeme
+POSTGRES_DB=pyroannotation
+
+# --- S3 / object storage ---
+S3_ENDPOINT_URL=https://s3.gra.io.cloud.ovh.net/
+S3_ACCESS_KEY=your-access-key
+S3_SECRET_KEY=your-secret-key
+S3_REGION=gra
+
+# --- JWT signing secret ---
+JWT_SECRET=change-me-to-a-secure-random-string
+
+# --- Annotation API auth (local target, used by scripts hitting localhost) ---
+ANNOTATOR_LOGIN=admin
+ANNOTATOR_PASSWORD=admin12345
+
+# --- Remote annotation API auth (export-dataset, push-annotations, etc.) ---
+MAIN_ANNOTATION_LOGIN=admin
+MAIN_ANNOTATION_PASSWORD=changeme
+
+# --- Platform API credentials (only needed for `make import-platform`) ---
+PLATFORM_LOGIN=your_platform_username
+PLATFORM_PASSWORD=your_platform_password
+PLATFORM_ADMIN_LOGIN=your_admin_username
+PLATFORM_ADMIN_PASSWORD=your_admin_password

annotation_api/.gitignore

@@ -4,6 +4,7 @@ acme.json
 # Environment files (may contain secrets)
 .env
 .env.*
+!.env.example
 
 # Python
 .venv/