fix(platform-import): adapt to new pyro-api sequence/detection schema#121
Merged
MateoLostanlen merged 1 commit intomainfrom May 1, 2026
Merged
fix(platform-import): adapt to new pyro-api sequence/detection schema#121MateoLostanlen merged 1 commit intomainfrom
MateoLostanlen merged 1 commit intomainfrom
Conversation
The platform API restructured its sequence/detection fields, breaking the import pipeline (silent KeyErrors masked by a per-sequence try/except, ending with "0 records from N sequences"). Updates the script to the current schema and tightens correctness around bbox parsing and azimuth handling. Schema changes addressed: - Sequence: `camera_azimuth` (= pose.azimuth at sequence creation), `sequence_azimuth` (smoke cone direction, ignored), `pose_id`, `cone_angle`. The old single `azimuth` is gone. - Detection: `bbox` (string `"[(x,y,x,y,c), ...]"`) plus `others_bboxes` (siblings of the same image). The old `bboxes` and `azimuth` are gone. Changes: - utils.py: rewrite `to_record` for the new fields. The internal record key is renamed `sequence_azimuth` → `camera_azimuth` to reflect what it actually is (camera/pose direction, not smoke direction). Bbox parsing is hardened (`ast.literal_eval`, defensive against flat singular tuples and malformed values), and merges `bbox + others_bboxes` so each image carries all detected boxes. - sequence_fetching.py: stop swallowing per-sequence exceptions. Dedupe detections by `bucket_key` (the platform emits one row per bbox even for shared images, which would otherwise duplicate predictions). Fetch a small buffer above `--detections-limit` so the dedupe doesn't under-fill, and preserve the "0 = no limit" convention. - client.py: `list_cameras` now passes `include_non_trustable=true` so the camera index covers every camera that can show up on a sequence, not just the trustable subset. - shared.py: round float `camera_azimuth` to int and wrap with `% 360` (annotation API stores `Optional[int]`; rounding `359.6` would land on an out-of-range 360). - import.py / batch_import_local_yolo.py: rename to `camera_azimuth` in the clone path; CSV consumer falls back to legacy keys. - .gitignore: ignore local `.claude/` worktree state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The platform API restructured its sequence/detection fields, which silently broke
make import-platform— every sequence ended up withKeyErroron the new field names, swallowed by a per-sequencetry/except, leaving0 records from N sequenceswith no diagnostic. This PR adapts the import script to the current schema and tightens a few related correctness issues that codex review surfaced.Schema changes addressed
camera_azimuth(=pose.azimuthat creation),sequence_azimuth(smoke cone direction, ignored),pose_id,cone_angle. Old singleazimuthgone.bbox(string"[(x,y,x,y,c), ...]") +others_bboxes(siblings of the same image). Oldbboxesandazimuthgone.What's in the patch
utils.py:to_record— rewritten for the new schema. Internal record keysequence_azimuth→camera_azimuth(the value is the camera/pose direction; the platform'ssequence_azimuthis the smoke direction we don't want). Bbox parsing usesast.literal_evalwith defensive handling for flat singular tuples and malformed values; mergesbbox + others_bboxesso each image carries all detected boxes.sequence_fetching.py:process_single_sequence_detections— dropped the baretry/exceptthat returned[]and hid the schema drift. Detections are deduped bybucket_key(the platform emits one row per bbox, so without dedupe a multi-box image would be imported N times with N-box predictions each). Fetchesmin(limit + 10, 100)so dedupe doesn't under-fill, and preserves0 = no limit.client.py:list_cameras— passesinclude_non_trustable=true. Without this, sequences from non-trustable cameras of the same org couldn't be matched against the camera index (your sdis-40 has 5 cameras total, not 4).shared.py:transform_sequence_data— rounds floatcamera_azimuthtointand wraps with% 360(annotation API storesOptional[int];round(359.6) = 360is out of range).import.py/batch_import_local_yolo.py— clone path usescamera_azimuth; CSV consumer falls back to legacy keys for older exports..gitignore— ignore local.claude/worktree state.Codex review
Multiple rounds. Net result: codex went from blocker findings → P2 → P2 false-positive. Findings addressed in commits: defensive bbox parser, bucket-key dedupe with fetch buffer, azimuth modulo, missed clone-path key rename,
detections_limit <= 0short-circuit. Final P2 (claim that platform doesn't returncamera_azimuth) is incorrect — verified directly against the API response.Test plan
make import-platform DATE_FROM=2026-04-15 DATE_END=2026-04-15 --max-sequences 2(smoke) — 1 sequence + 30 detections + 1 annotation, end-to-end cleanmake import-platform DATE_FROM=2026-04-15 DATE_END=2026-04-17 MAX_SEQUENCES=0(3-day on clean DB) — 48/48 sequences, 978/978 detections, 48/48 annotationssequences == annotations in ready_to_annotate(no orphans)make pull-sequences) end-to-end — pending local stack bring-up (postgres volume mismatch unrelated to this PR)🤖 Generated with Claude Code