From ee9f80d7f24b510f168742ef625d0439e73187b7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 04:55:14 +0000 Subject: [PATCH 1/5] Add Cellpose-SAM retrain worker Adds a cellposesam_train worker that fine-tunes Cellpose-SAM models on user-corrected annotations, mirroring the cellpose_train paradigm (training tag + optional training regions, saved to Girder for reuse by the inference worker) while adopting the Cellpose-SAM interface and training API. Key differences from the older cellpose_train worker: - Channel-slot input (Slot 1 required, Slots 2/3 optional) matching the cellposesam inference worker, instead of the cyto/nucleus Primary/Secondary channels + Nuclear Model checkbox. Cellpose-SAM ingests up to 3 channels natively, so there is no separate cytoplasm/nucleus concept. - Training via cellpose 4.x train.train_seg: no `channels` arg (uses channel_axis=-1 on channels-last stacks), AdamW (SGD deprecated), learning_rate default 1e-5, weight_decay default 0.1, bsize=256, native resolution (no diameter). - Model instantiated with CellposeModel(pretrained_model=...) from the cpsam checkpoints (models_config.py, shared with the inference worker) or a custom Girder model path; saved to .cellposesam/models. Registers the worker in build_machine_learning_workers.sh and REGISTRY.md, and includes models_config unit tests. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0199mMkZxJV7p7p8zTs7wYFe --- REGISTRY.md | 1 + build_machine_learning_workers.sh | 3 + .../cellposesam_train/CELLPOSESAM_TRAIN.md | 103 +++++ .../annotations/cellposesam_train/Dockerfile | 75 ++++ .../cellposesam_train/download_models.py | 16 + .../cellposesam_train/entrypoint.py | 379 ++++++++++++++++++ .../cellposesam_train/environment.yml | 10 + .../cellposesam_train/girder_utils.py | 66 +++ .../cellposesam_train/models_config.py | 42 ++ .../cellposesam_train/tests/__init__.py | 0 .../tests/test_models_config.py | 76 ++++ .../annotations/cellposesam_train/utils.py | 50 +++ 12 files changed, 821 insertions(+) create mode 100644 workers/annotations/cellposesam_train/CELLPOSESAM_TRAIN.md create mode 100644 workers/annotations/cellposesam_train/Dockerfile create mode 100644 workers/annotations/cellposesam_train/download_models.py create mode 100644 workers/annotations/cellposesam_train/entrypoint.py create mode 100644 workers/annotations/cellposesam_train/environment.yml create mode 100644 workers/annotations/cellposesam_train/girder_utils.py create mode 100644 workers/annotations/cellposesam_train/models_config.py create mode 100644 workers/annotations/cellposesam_train/tests/__init__.py create mode 100644 workers/annotations/cellposesam_train/tests/test_models_config.py create mode 100644 workers/annotations/cellposesam_train/utils.py diff --git a/REGISTRY.md b/REGISTRY.md index 57de4e9..0cccb4e 100644 --- a/REGISTRY.md +++ b/REGISTRY.md @@ -26,6 +26,7 @@ Create new annotations by segmenting images or connecting existing annotations. | Cellpose | This tool runs the Cellpose model to segment the image into cells. Learn more | Yes | | [docs](workers/annotations/cellpose/CELLPOSE.md) | | Cellpose retrain | This tool trains a Cellpose model using user-corrected annotations. Learn more | Yes | | [docs](workers/annotations/cellpose_train/CELLPOSE_TRAIN.md) | | Cellpose-SAM | This tool runs the Cellpose-SAM model to segment the image into cells. Learn more | Yes | | [docs](workers/annotations/cellposesam/CELLPOSESAM.md) | +| Cellpose-SAM retrain | This tool fine-tunes a Cellpose-SAM model using user-corrected annotations. Learn more | Yes | Yes | [docs](workers/annotations/cellposesam_train/CELLPOSESAM_TRAIN.md) | | CondensateNet segmentation | Segments biomolecular condensates using CondensateNet deep learning model | Yes | | [docs](workers/annotations/condensatenet/CONDENSATENET.md) | | Connect Sequential | This tool connects objects sequentially across time or z-slices.It is useful for connec... | | Yes | [docs](workers/annotations/connect_sequential/CONNECT_SEQUENTIAL.md) | | Connect Time Lapse | This tool connects objects across time slices. It allows you to connect objects even if... | | Yes | [docs](workers/annotations/connect_timelapse/CONNECT_TIMELAPSE.md) | diff --git a/build_machine_learning_workers.sh b/build_machine_learning_workers.sh index 935ebdc..64c63d3 100755 --- a/build_machine_learning_workers.sh +++ b/build_machine_learning_workers.sh @@ -31,6 +31,9 @@ docker build . -f ./workers/annotations/cellpose_train/Dockerfile -t annotations echo "Building Cellpose-SAM worker" docker build . -f ./workers/annotations/cellposesam/Dockerfile -t annotations/cellposesam_worker:latest $NO_CACHE +echo "Building Cellpose-SAM retrain worker" +docker build . -f ./workers/annotations/cellposesam_train/Dockerfile -t annotations/cellposesam_train_worker:latest $NO_CACHE + echo "Building Stardist worker" docker build . -f ./workers/annotations/stardist/Dockerfile -t annotations/stardist_worker:latest $NO_CACHE diff --git a/workers/annotations/cellposesam_train/CELLPOSESAM_TRAIN.md b/workers/annotations/cellposesam_train/CELLPOSESAM_TRAIN.md new file mode 100644 index 0000000..e02c6fa --- /dev/null +++ b/workers/annotations/cellposesam_train/CELLPOSESAM_TRAIN.md @@ -0,0 +1,103 @@ +# Cellpose-SAM Retrain Worker + +This worker fine-tunes a Cellpose-SAM model on user-corrected annotations, producing a custom model that can be used by the Cellpose-SAM worker. + +It is the Cellpose-SAM counterpart to the older [Cellpose Train worker](../cellpose_train/CELLPOSE_TRAIN.md). It follows the same annotation-gathering paradigm (training tag + optional training regions) but adopts the Cellpose-SAM interface and training API, which differ substantially from earlier Cellpose versions. + +## How It Works + +1. **Annotation Loading**: Retrieves all polygon and rectangle annotations from the dataset +2. **Tag Filtering**: Selects training annotations by the specified training tag, and optionally crops to training region annotations +3. **Image Assembly**: For each location (XY/Z/Time), loads the selected input-slot channels and stacks them channels-last, then renders training annotations into label masks +4. **Region Cropping**: If training regions are specified, crops each image/mask pair to the bounding box of each region polygon +5. **Fine-tuning**: Fine-tunes the selected base Cellpose-SAM model using `cellpose.train.train_seg()` (AdamW optimizer) +6. **Upload**: Saves the trained model to the user's Girder `.cellposesam/models` folder + +## Interface Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| **Cellpose-SAM retrain** | notes | -- | Informational text with documentation link | +| **Base Model** | select | cellpose-sam | Base model to fine-tune. `cellpose-sam` starts from the `cpsam_v2` checkpoint; `cellpose-sam (legacy cpsam)` starts from the original April 2025 `cpsam` checkpoint. User-trained models from Girder are also listed | +| **Output Model Name** | text | -- | **Required.** Name for the saved model. Will appear in the Model dropdown of the Cellpose-SAM worker | +| **Channel for Slot 1** | channelCheckboxes | -- | **Required.** Source channel(s) for the model's first input slot. If multiple selected, only the first is used | +| **Channel for Slot 2** | channelCheckboxes | -- | Optional second input slot channel | +| **Channel for Slot 3** | channelCheckboxes | -- | Optional third input slot channel | +| **Training Tag** | tags | -- | **Required.** Tag identifying the corrected annotations to use as training ground truth | +| **Training Region** | tags | -- | Optional tag identifying region annotations that define training crops. If empty, uses the full image | +| **Learning Rate** | number | 0.00001 | AdamW learning rate (range: 0.000001-0.1). Cellpose-SAM fine-tunes best with a small learning rate (1e-5) | +| **Epochs** | number | 100 | Number of training epochs (range: 10-2000) | +| **Weight Decay** | number | 0.1 | AdamW weight decay regularization (range: 0-1) | + +## Implementation Details + +### Channel Slots vs. Primary/Secondary Channels + +Unlike the older `cellpose_train` worker (which used Primary/Secondary channel selectors and a "Nuclear Model?" checkbox to build the classic `channels=[cyto, nucleus]` pair), Cellpose-SAM works directly on the channels you provide. It ingests up to three channels natively and reorders/normalizes them internally, so there is no separate cytoplasm/nucleus concept. + +This worker therefore uses the same three `channelCheckboxes` slots as the Cellpose-SAM inference worker: + +- **Slot 1** is required; an error is raised if no channel is selected +- **Slots 2 and 3** are optional +- If multiple channels are checked in a single slot, a warning is issued and only the first is used +- Selected channels are stacked channels-last and passed to training via `channel_axis=-1` + +Matching the inference worker's channel layout matters: a model trained on a given channel arrangement should be run with the same arrangement. + +### Training Data Preparation + +- Annotations are grouped by location (Time, Z, XY) to batch image loading +- Both polygon and rectangle annotations are retrieved and combined +- Each training annotation is rasterized into a label mask using `skimage.draw.polygon2mask()`, with each annotation assigned a unique integer label +- Images are stacked channels-last as `(H, W, num_selected_channels)`; Cellpose-SAM standardizes to three channels internally (padding with zeros or truncating as needed) + +### Training Regions + +- Training regions are polygon/rectangle annotations with the specified region tag +- When regions are specified, images and label masks are cropped to each region's bounding box, producing one training sample per region per location +- When no regions are specified, the full image is used as a single training sample (a warning is displayed) +- Using regions is recommended to focus training on relevant areas and reduce memory usage + +### Cellpose-SAM Training API Differences + +Cellpose-SAM (cellpose ≥ 4) changed the training API relative to the versions used by `cellpose_train`: + +- **No `channels` argument**: the network ingests channels directly, so training passes `channel_axis=-1` instead of a `channels=[...]` pair +- **AdamW optimizer**: the old `SGD` flag is deprecated (AdamW is always used), so it is not passed +- **Small learning rate**: the recommended default is `1e-5` (vs `0.01` for the older worker) +- **`weight_decay=0.1`** is the Cellpose-SAM default (vs `0.0001`) +- **`bsize=256`**: cpsam requires a block size of 256 +- **Native resolution**: training runs with `rescale=False`, so no diameter parameter is needed +- **Model instantiation**: `CellposeModel(gpu=..., pretrained_model=)` replaces the old `model_type=` argument + +### Model Storage + +- The trained model is saved locally to `/root/.cellposesam/models/` (via `save_path` + `model_name` passed to `train_seg`) +- After training completes, the model is uploaded to the user's Girder `Private/.cellposesam/models/` folder +- If a model with the same name already exists in Girder, it is replaced +- Uploaded models automatically appear in the Model dropdown of the Cellpose-SAM worker + +### Built-in Checkpoints + +The base-model labels map to cellpose built-in checkpoints in `models_config.py` — `cellpose-sam` → `cpsam_v2`, `cellpose-sam (legacy cpsam)` → `cpsam`. Both checkpoints are pre-downloaded at build time by `download_models.py` so neither downloads on first run. `models_config.py` is kept in sync with the inference worker and is the single source of truth for both the interface and the build-time download. + +### GPU Handling + +The worker checks for GPU availability via `cellpose.core.use_gpu()` and instantiates the model with that result; cellpose falls back to CPU if no GPU is available. + +## Testing + +`tests/test_models_config.py` unit-tests the base-model mapping (`models_config.py`) in isolation — it has no heavy imports, so it runs in the lightweight local venv: + +```bash +.cache/testvenv/bin/pytest workers/annotations/cellposesam_train/tests -q +``` + +## Notes + +- Does not use `WorkerClient` for batch processing; instead directly manages annotation retrieval and image loading (same approach as `cellpose_train`) +- Requires a training tag and an output model name; the worker errors if either is missing +- Training regions are optional but recommended for efficiency and to avoid training on irrelevant parts of the image +- The base-model list is dynamically populated from both built-in checkpoints and user models in Girder +- Progress updates are sent at key stages: loading annotations (10%), processing (20%), loading images (30%), training (40%), and saving (95%) +- Related workers: [Cellpose-SAM](../cellposesam/CELLPOSESAM.md) (inference), [Cellpose Train](../cellpose_train/CELLPOSE_TRAIN.md) (older retrain) diff --git a/workers/annotations/cellposesam_train/Dockerfile b/workers/annotations/cellposesam_train/Dockerfile new file mode 100644 index 0000000..8fd873e --- /dev/null +++ b/workers/annotations/cellposesam_train/Dockerfile @@ -0,0 +1,75 @@ +FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 as base +LABEL isUPennContrastWorker=True +LABEL com.nvidia.volumes.needed="nvidia_driver" + +ENV NVIDIA_VISIBLE_DEVICES all +ENV NVIDIA_DRIVER_CAPABILITIES compute,utility + +RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -qy tzdata && \ + apt-get install -qy software-properties-common python3-software-properties && \ + apt-get update && apt-get install -qy \ + build-essential \ + wget \ + python3 \ + r-base \ + libffi-dev \ + libssl-dev \ + libjpeg-dev \ + zlib1g-dev \ + r-base \ + git \ + libpython3-dev && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + +ENV PATH="/root/miniforge3/bin:$PATH" +ARG PATH="/root/miniforge3/bin:$PATH" + +RUN wget \ + https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh \ + && mkdir /root/.conda \ + && bash Miniforge3-Linux-x86_64.sh -b \ + && rm -f Miniforge3-Linux-x86_64.sh + +FROM base as build + +COPY ./workers/annotations/cellposesam_train/environment.yml / +RUN conda env create --file /environment.yml +SHELL ["conda", "run", "-n", "worker", "/bin/bash", "-c"] + +RUN pip install rtree shapely + +RUN git clone https://github.com/Kitware/UPennContrast/ + +RUN pip install -r /UPennContrast/devops/girder/annotation_client/requirements.txt +RUN pip install -e /UPennContrast/devops/girder/annotation_client/ + +# models_config.py is the single source of truth for the built-in checkpoints, +# so it must be present before download_models.py runs. +COPY ./workers/annotations/cellposesam_train/models_config.py / +COPY ./workers/annotations/cellposesam_train/download_models.py / +RUN python /download_models.py + +COPY ./workers/annotations/cellposesam_train/utils.py / +COPY ./workers/annotations/cellposesam_train/girder_utils.py / +COPY ./workers/annotations/cellposesam_train/entrypoint.py / + +COPY ./annotation_utilities /annotation_utilities +RUN pip install /annotation_utilities + +COPY ./worker_client /worker_client +RUN pip install /worker_client + +LABEL isUPennContrastWorker="" \ + isAnnotationWorker="" \ + interfaceName="Cellpose-SAM retrain" \ + interfaceCategory="Cellpose" \ + description="Fine-tunes Cellpose-SAM models based on user-selected images" \ + advancedOptionsPanel="False" \ + annotationConfigurationPanel="False" \ + defaultToolName="Cellpose-SAM retrain" \ + annotationShape="polygon" + +# Fast env activation in place of `conda run` (see todo/worker-startup-latency.md) +COPY ./workers/base_docker_images/run_worker.sh /usr/local/bin/run_worker.sh +RUN chmod +x /usr/local/bin/run_worker.sh +ENTRYPOINT ["/usr/local/bin/run_worker.sh", "/entrypoint.py"] diff --git a/workers/annotations/cellposesam_train/download_models.py b/workers/annotations/cellposesam_train/download_models.py new file mode 100644 index 0000000..6c0bfd6 --- /dev/null +++ b/workers/annotations/cellposesam_train/download_models.py @@ -0,0 +1,16 @@ +"""Pre-download the built-in Cellpose-SAM checkpoints at build time. + +Fine-tuning starts from one of these base checkpoints, so baking them into the +image avoids a multi-GB download on the first run. Instantiating +``CellposeModel(pretrained_model=)`` fetches the weights to +``models.MODELS_DIR`` on first use. ``gpu=True`` matches runtime; on a GPU-less +build host cellpose falls back to CPU but still downloads. +""" + +from cellpose import models + +from models_config import BASE_MODEL_CHECKPOINTS + +for checkpoint in set(BASE_MODEL_CHECKPOINTS.values()): + print(f"Downloading Cellpose-SAM checkpoint: {checkpoint}") + models.CellposeModel(gpu=True, pretrained_model=checkpoint) diff --git a/workers/annotations/cellposesam_train/entrypoint.py b/workers/annotations/cellposesam_train/entrypoint.py new file mode 100644 index 0000000..3c60795 --- /dev/null +++ b/workers/annotations/cellposesam_train/entrypoint.py @@ -0,0 +1,379 @@ +import argparse +from collections import defaultdict +import json +import sys + +from skimage import draw +import numpy as np + +from shapely.geometry import Polygon + +import annotation_client.workers as workers +import annotation_client.tiles as tiles +import annotation_client.annotations as annotations +from annotation_client.utils import sendProgress, sendError, sendWarning +import annotation_utilities.annotation_tools as annotation_tools + +import girder_utils +from girder_utils import CELLPOSE_DIR, MODELS_DIR + +from models_config import ( + BASE_MODELS, BASE_MODEL_CHECKPOINTS, DEFAULT_MODEL, build_model_items) + + +def interface(image, apiUrl, token): + client = workers.UPennContrastWorkerPreviewClient( + apiUrl=apiUrl, token=token) + + girder_models = [model['name'] + for model in girder_utils.list_girder_models(client.client)[0]] + # Reserved base labels win over any custom model of the same name (see + # build_model_items); this keeps the dropdown consistent with compute()'s routing. + models = build_model_items(girder_models) + + # Available types: number, text, tags, layer + interface = { + 'Cellpose-SAM retrain': { + 'type': 'notes', + 'value': 'This tool fine-tunes a Cellpose-SAM model using user-corrected annotations. ' + 'Cellpose-SAM works directly on the channels you provide (up to three), so there ' + 'is no separate cytoplasm/nucleus channel concept — just pick the channels you ' + 'want the model to see. ' + 'Learn more', + 'displayOrder': 0, + }, + 'Base Model': { + 'type': 'select', + 'items': models, + 'default': DEFAULT_MODEL, + 'tooltip': 'The model used as a starting point for fine-tuning.\n' + '"cellpose-sam" starts from the current default checkpoint (cpsam_v2).\n' + '"cellpose-sam (legacy cpsam)" starts from the original April 2025 model.\n' + 'Custom models you have previously trained are also listed here.', + 'noCache': True, + 'displayOrder': 1, + }, + 'Output Model Name': { + 'type': 'text', + 'tooltip': 'The name of the retrained model (saved to your .cellposesam/models folder).\n' + 'It will appear in the Model dropdown of the Cellpose-SAM worker.', + 'displayOrder': 2, + }, + 'Channel for Slot 1': { + 'type': 'channelCheckboxes', + 'tooltip': "Select source channel(s) for the model's first input slot. If multiple are selected, only the first will be used. This slot is required.", + 'displayOrder': 3, + }, + 'Channel for Slot 2': { + 'type': 'channelCheckboxes', + 'tooltip': "Select source channel(s) for the model's second input slot. If multiple are selected, only the first will be used. (Optional)", + 'displayOrder': 4, + }, + 'Channel for Slot 3': { + 'type': 'channelCheckboxes', + 'tooltip': "Select source channel(s) for the model's third input slot. If multiple are selected, only the first will be used. (Optional)", + 'displayOrder': 5, + }, + 'Training Tag': { + 'type': 'tags', + 'tooltip': 'Train the model on objects that have this tag.', + 'displayOrder': 6, + }, + 'Training Region': { + 'type': 'tags', + 'tooltip': 'These objects define the regions that the training will be performed on.\n' + 'If you do not select any objects, the training will be performed on all objects in the image.\n' + 'You can and probably should select multiple regions.', + 'displayOrder': 7, + }, + 'Learning Rate': { + 'type': 'number', + 'min': 0.000001, + 'max': 0.1, + 'default': 0.00001, + 'tooltip': 'The learning rate for fine-tuning. Cellpose-SAM fine-tunes best with a small ' + 'learning rate; 1e-5 (0.00001) is the recommended default.', + 'displayOrder': 8, + }, + 'Epochs': { + 'type': 'number', + 'min': 10, + 'max': 2000, + 'default': 100, + 'tooltip': 'The number of epochs to train the model for. 100 is a good starting point for Cellpose-SAM.', + 'displayOrder': 9, + }, + 'Weight Decay': { + 'type': 'number', + 'min': 0, + 'max': 1, + 'default': 0.1, + 'tooltip': 'The weight decay (AdamW regularization) for training. 0.1 is the Cellpose-SAM default.', + 'displayOrder': 10, + }, + } + # Send the interface object to the server + client.setWorkerImageInterface(image, interface) + + +def get_slot_channels(workerInterface): + """Resolve the selected input-slot channels into an ordered channel list. + + Mirrors the cellposesam inference worker: Slot 1 is required, Slots 2 and 3 + are optional, and if multiple channels are checked in a slot only the first + is used (with a warning). + """ + slot1 = [k for k, v in workerInterface.get('Channel for Slot 1', {}).items() if v] + slot2 = [k for k, v in workerInterface.get('Channel for Slot 2', {}).items() if v] + slot3 = [k for k, v in workerInterface.get('Channel for Slot 3', {}).items() if v] + + stack_channels = [] + + if not slot1: + sendError("No channel selected for Slot 1. This is a required field.") + raise ValueError("No channel selected for Slot 1.") + if len(slot1) > 1: + sendWarning( + f"Multiple channels selected for Slot 1 ({slot1}). Using the first: {slot1[0]}.") + stack_channels.append(int(slot1[0])) + + if slot2: + if len(slot2) > 1: + sendWarning( + f"Multiple channels selected for Slot 2 ({slot2}). Using the first: {slot2[0]}.") + stack_channels.append(int(slot2[0])) + + if slot3: + if len(slot3) > 1: + sendWarning( + f"Multiple channels selected for Slot 3 ({slot3}). Using the first: {slot3[0]}.") + stack_channels.append(int(slot3[0])) + + return stack_channels + + +def compute(datasetId, apiUrl, token, params): + """ + params (could change): + configurationId, + datasetId, + description: tool description, + type: tool type, + id: tool id, + name: tool name, + image: docker image, + channel: annotation channel, + assignment: annotation assignment ({XY, Z, Time}), + tags: annotation tags (list of strings), + tile: tile position (TODO: roi) ({XY, Z, Time}), + connectTo: how new annotations should be connected + """ + + # Lazy import: keeps cellpose off the interface/startup path (~seconds). See todo/worker-startup-latency.md + from cellpose import models, train, core + + workerInterface = params['workerInterface'] + + # Get the model and training parameters from interface values + base_model = workerInterface['Base Model'] + output_model_name = workerInterface['Output Model Name'] + training_tag = workerInterface.get('Training Tag', None) + training_regions = workerInterface.get('Training Region', None) + learning_rate = float(workerInterface['Learning Rate']) + epochs = int(workerInterface['Epochs']) + weight_decay = float(workerInterface['Weight Decay']) + + print(f"Training tag: {training_tag}") + print(f"Training regions: {training_regions}") + + if not output_model_name: + sendError("No output model name provided.", + info="Please provide a name for the retrained model.") + raise ValueError("No output model name provided.") + + if training_tag is None or len(training_tag) == 0: + sendError("No training tag selected.", + info="Choose a tag for training annotations.") + raise ValueError("No training tag selected.") + if training_regions is None or len(training_regions) == 0: + sendWarning("No training regions selected.", + info="Training will be performed on entire image that the annotations are in.") + print("No training regions selected. Training will be performed on entire image that the annotations are in.") + + # Resolve the input-slot channels (Slot 1 required, Slots 2 & 3 optional). + stack_channels = get_slot_channels(workerInterface) + print(f"Using channels for Cellpose-SAM input (slots 1, 2, 3): {stack_channels}") + + client = workers.UPennContrastWorkerPreviewClient( + apiUrl=apiUrl, token=token) + annotationClient = annotations.UPennContrastAnnotationClient( + apiUrl=apiUrl, token=token) + tileClient = tiles.UPennContrastDataset( + apiUrl=apiUrl, token=token, datasetId=datasetId) + + # Resolve the base checkpoint / model path to fine-tune from. + if base_model in BASE_MODELS: + # Pass the checkpoint name explicitly so the starting point is pinned to + # the selected model rather than cellpose's internal default, which can + # change between versions. + pretrained_model = BASE_MODEL_CHECKPOINTS[base_model] + else: + girder_utils.download_girder_model(client.client, base_model) + pretrained_model = str(MODELS_DIR / base_model) + + # Print the contents of the models directory + print(f"Models directory contents: {list(MODELS_DIR.glob('*'))}") + + # Initial loading phase + sendProgress(0.1, "Loading annotations", + "Retrieving annotations from server") + blobAnnotationList = annotationClient.getAnnotationsByDatasetId( + datasetId, limit=1000000, shape='polygon') + rectangleAnnotationList = annotationClient.getAnnotationsByDatasetId( + datasetId, limit=1000000, shape='rectangle') + # Add the rectangle annotations to the blob annotations + blobAnnotationList.extend(rectangleAnnotationList) + + trainingAnnotationList = annotation_tools.get_annotations_with_tags( + blobAnnotationList, training_tag, exclusive=False) + if training_regions is None or len(training_regions) == 0: + regionAnnotationList = [] + else: + regionAnnotationList = annotation_tools.get_annotations_with_tags( + blobAnnotationList, training_regions, exclusive=False) + print(f"Training on {len(regionAnnotationList)} region annotations.") + + if len(trainingAnnotationList) == 0: + sendError("No training annotations found.", + info="No annotations with the training tag were found.") + raise ValueError("No training annotations found.") + if len(regionAnnotationList) == 0 and training_regions and len(training_regions) > 0: + sendWarning("No region annotations found.", + info="No annotations with the training region tag were found.") + print("No region annotations found. Training will be performed on entire image that the annotations are in.") + + sendProgress(0.2, "Processing annotations", + "Grouping annotations by location") + # Group the training annotations by location so that we can batch the image loading. + grouped_training_annotations = defaultdict(list) + for current_annotation in trainingAnnotationList: + location_key = (current_annotation['location']['Time'], + current_annotation['location']['Z'], current_annotation['location']['XY']) + grouped_training_annotations[location_key].append(current_annotation) + + # Group the region annotations by location so that we can batch the image loading. + grouped_region_annotations = defaultdict(list) + for current_annotation in regionAnnotationList: + location_key = (current_annotation['location']['Time'], + current_annotation['location']['Z'], current_annotation['location']['XY']) + grouped_region_annotations[location_key].append(current_annotation) + + training_images = [] + label_images = [] + + sendProgress(0.3, "Loading training data", "Loading training images") + # Loop through each location and load the image for the training. + for location_key, training_annotations in grouped_training_annotations.items(): + time, z, xy = location_key + + # Load and stack the selected channels (channels-last) for this location. + channel_images = [] + for channel in stack_channels: + frame = tileClient.coordinatesToFrameIndex(xy, z, time, channel) + channel_image = tileClient.getRegion(datasetId, frame=frame).squeeze() + channel_images.append(channel_image) + # Shape: (H, W, num_selected_channels). Cellpose-SAM standardizes to + # three channels internally (padding with zeros / truncating as needed). + stacked_image = np.stack(channel_images, axis=-1) + + label_image = np.zeros(stacked_image.shape[:2], dtype=np.uint16) + for i, current_annotation in enumerate(training_annotations): + polygon = np.array([list(coordinate.values())[1::-1] + for coordinate in current_annotation['coordinates']]) + mask = draw.polygon2mask(label_image.shape, polygon) + label_image[mask] = i + 1 + + if training_regions is None or len(training_regions) == 0: + training_images.append(stacked_image) + label_images.append(label_image) + else: + region_annotations = grouped_region_annotations[location_key] + for region_annotation in region_annotations: + region_polygon = Polygon([(coordinate['x'], coordinate['y']) + for coordinate in region_annotation['coordinates']]) + + # Use shapely to get the bounding box + min_x, min_y, max_x, max_y = region_polygon.bounds + + # Crop both the stacked image and the label mask to the bounding box. + stacked_image_crop = stacked_image[int( + min_y):int(max_y), int(min_x):int(max_x)] + label_image_crop = label_image[int(min_y):int( + max_y), int(min_x):int(max_x)] + + training_images.append(stacked_image_crop) + label_images.append(label_image_crop) + + using_gpu = core.use_gpu() + print(f"Using GPU: {using_gpu}") + + # Cellpose-SAM (cellpose >= 4) drops model_type in favor of pretrained_model; + # passing 'cpsam'/'cpsam_v2' or a custom model path starts fine-tuning from + # those weights. + model = models.CellposeModel(gpu=using_gpu, pretrained_model=pretrained_model) + + print(f"Training with {len(training_images)} images.") + sendProgress(0.4, "Training model", + f"Training with {len(training_images)} images, be patient...") + + # Cellpose-SAM training differs from earlier Cellpose versions: + # - no `channels` argument (the network ingests up to 3 channels directly); + # channel_axis=-1 tells train_seg our stacks are channels-last. + # - AdamW is always used (the old SGD flag is deprecated), so we do not pass SGD. + # - bsize must be 256 for cpsam. + # - training uses native resolution (rescale=False), so no diameter is needed. + model_path, train_losses, test_losses = train.train_seg( + model.net, + train_data=training_images, + train_labels=label_images, + channel_axis=-1, + normalize=True, + weight_decay=weight_decay, + learning_rate=learning_rate, + n_epochs=epochs, + bsize=256, + save_path=str(CELLPOSE_DIR), + model_name=output_model_name) + + print(f"Trained model saved to: {model_path}") + + # Upload the trained model to Girder + sendProgress(0.95, "Saving model", f"Uploading model {output_model_name}") + girder_utils.upload_girder_model(client.client, output_model_name) + + +if __name__ == '__main__': + # Define the command-line interface for the entry point + parser = argparse.ArgumentParser( + description='Fine-tune a Cellpose-SAM model on user-corrected annotations') + + parser.add_argument('--datasetId', type=str, + required=False, action='store') + parser.add_argument('--apiUrl', type=str, required=True, action='store') + parser.add_argument('--token', type=str, required=True, action='store') + parser.add_argument('--request', type=str, required=True, action='store') + parser.add_argument('--parameters', type=str, + required=True, action='store') + + args = parser.parse_args(sys.argv[1:]) + + params = json.loads(args.parameters) + datasetId = args.datasetId + apiUrl = args.apiUrl + token = args.token + + match args.request: + case 'compute': + compute(datasetId, apiUrl, token, params) + case 'interface': + interface(params['image'], apiUrl, token) diff --git a/workers/annotations/cellposesam_train/environment.yml b/workers/annotations/cellposesam_train/environment.yml new file mode 100644 index 0000000..e80333c --- /dev/null +++ b/workers/annotations/cellposesam_train/environment.yml @@ -0,0 +1,10 @@ +name: worker +dependencies: +- python=3.10 +- pip +- imageio +- rasterio +- shapely +- pip: + - cellpose==4.2.1.1 + - scikit-image diff --git a/workers/annotations/cellposesam_train/girder_utils.py b/workers/annotations/cellposesam_train/girder_utils.py new file mode 100644 index 0000000..6ce4f54 --- /dev/null +++ b/workers/annotations/cellposesam_train/girder_utils.py @@ -0,0 +1,66 @@ +from pathlib import Path + +# Assuming these are defined elsewhere in your cellpose worker +CELLPOSE_DIR = Path('/root/.cellposesam') +MODELS_DIR = CELLPOSE_DIR / 'models' + + +def mkdir(gc, parent_id, folder_name): + + folders = gc.get('folder', parameters={'parentId': parent_id, + 'parentType': 'folder', 'name': folder_name}) + + if folders: + _id = folders[0]['_id'] + else: + new_folder = gc.post('folder', parameters={ + 'parentId': parent_id, + 'parentType': 'folder', + 'name': folder_name + }) + _id = new_folder['_id'] + + return _id + + +def get_cellpose_dir(gc): + + user_id = gc.get('user/me')['_id'] + private_folder_id = gc.get('folder', parameters={ + 'parentId': user_id, 'parentType': 'user', 'name': 'Private'})[0]['_id'] + cellpose_folder_id = mkdir(gc, private_folder_id, '.cellposesam') + + return cellpose_folder_id + + +def list_girder_models(gc): + + cellpose_folder_id = get_cellpose_dir(gc) + models_folder_id = mkdir(gc, cellpose_folder_id, 'models') + girder_models = list(gc.listItem(models_folder_id)) + + return girder_models, models_folder_id + + +def list_local_models(): + """List models in local .cellposesam/models directory""" + MODELS_DIR.mkdir(parents=True, exist_ok=True) + return list(MODELS_DIR.glob('*')) + + +def download_girder_model(gc, model_name): + + girder_models, _ = list_girder_models(gc) + girder_model = [model for model in girder_models if model['name'] == model_name] + if girder_model: + gc.downloadItem(girder_model[0]['_id'], MODELS_DIR, girder_model[0]['name']) + + +def upload_girder_model(gc, model_name): + + girder_models, models_folder_id = list_girder_models(gc) + girder_model = [model for model in girder_models if model['name'] == model_name] + if girder_model: + gc.delete(f"{girder_model[0]['_modelType']}/{girder_model[0]['_id']}") + + gc.uploadFileToFolder(models_folder_id, MODELS_DIR / model_name) diff --git a/workers/annotations/cellposesam_train/models_config.py b/workers/annotations/cellposesam_train/models_config.py new file mode 100644 index 0000000..9c9dc0f --- /dev/null +++ b/workers/annotations/cellposesam_train/models_config.py @@ -0,0 +1,42 @@ +"""Built-in Cellpose-SAM model options for the cellposesam_train worker. + +Maps the human-friendly names shown in the Base Model dropdown to the cellpose +checkpoint identifiers passed to ``CellposeModel(pretrained_model=...)`` as the +starting point for fine-tuning. + +Kept in sync with the cellposesam (inference) worker's ``models_config.py`` so +the base-model choices match. Kept deliberately import-free (no +cellpose/annotation_client) so the mapping can be unit-tested in the lightweight +local venv without the full worker stack. +""" + +# Human-friendly dropdown label -> cellpose built-in checkpoint name. +# Insertion order lists the default (cpsam_v2) first. +BASE_MODEL_CHECKPOINTS = { + # New default: SAM-ViTL backbone, released June 2026; fixes spurious masks + # in low-contrast regions relative to the original cpsam. + 'cellpose-sam': 'cpsam_v2', + # Original Cellpose-SAM model, released April 2025. Kept selectable so prior + # results remain reproducible. + 'cellpose-sam (legacy cpsam)': 'cpsam', +} + +# The dropdown option selected by default. +DEFAULT_MODEL = 'cellpose-sam' + +# The list of built-in model labels offered in the interface. +BASE_MODELS = list(BASE_MODEL_CHECKPOINTS) + + +def build_model_items(girder_model_names): + """Build the sorted Base Model-dropdown options from the built-in labels plus + custom Girder model names. + + The built-in labels are reserved: a custom Girder model whose name collides + with one is dropped, because ``compute()`` routes any name in ``BASE_MODELS`` + to the built-in checkpoint — so a same-named custom model could never be + loaded and would only create a confusing duplicate entry. + """ + custom = [name for name in girder_model_names + if name not in BASE_MODEL_CHECKPOINTS] + return sorted(set(BASE_MODELS) | set(custom)) diff --git a/workers/annotations/cellposesam_train/tests/__init__.py b/workers/annotations/cellposesam_train/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/workers/annotations/cellposesam_train/tests/test_models_config.py b/workers/annotations/cellposesam_train/tests/test_models_config.py new file mode 100644 index 0000000..16dfae4 --- /dev/null +++ b/workers/annotations/cellposesam_train/tests/test_models_config.py @@ -0,0 +1,76 @@ +"""Unit tests for the built-in Cellpose-SAM base-model mapping. + +These exercise ``models_config`` in isolation — it must stay free of heavy +imports (cellpose/annotation_client) so it runs in the lightweight local venv +without the full worker stack. Run with: + + .cache/testvenv/bin/pytest workers/annotations/cellposesam_train/tests -q +""" + +import sys +from pathlib import Path + +# Put the worker directory (parent of tests/) on the path so we can import the +# standalone mapping module without installing the whole worker. +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +import models_config # noqa: E402 + + +def test_default_model_resolves_to_cpsam_v2(): + """The default dropdown selection must fine-tune from the cpsam_v2 checkpoint.""" + checkpoint = models_config.BASE_MODEL_CHECKPOINTS[models_config.DEFAULT_MODEL] + assert checkpoint == 'cpsam_v2' + + +def test_legacy_label_resolves_to_original_cpsam(): + """The legacy option must still map to the original April 2025 cpsam checkpoint.""" + assert models_config.BASE_MODEL_CHECKPOINTS['cellpose-sam (legacy cpsam)'] == 'cpsam' + + +def test_default_model_is_a_selectable_base_model(): + """The configured default must be one of the offered base models.""" + assert models_config.DEFAULT_MODEL in models_config.BASE_MODELS + + +def test_base_models_offers_both_builtins(): + """Both the v2 default and the legacy option must remain available.""" + assert set(models_config.BASE_MODELS) == { + 'cellpose-sam', + 'cellpose-sam (legacy cpsam)', + } + + +def test_build_model_items_includes_base_and_custom(): + """Custom Girder model names appear alongside the built-in labels.""" + items = models_config.build_model_items(['my custom model']) + assert 'cellpose-sam' in items + assert 'cellpose-sam (legacy cpsam)' in items + assert 'my custom model' in items + + +def test_build_model_items_excludes_reserved_name_collision(): + """A custom model named exactly like a base label is dropped, not duplicated. + + Otherwise it would silently route to the built-in checkpoint in compute() + and the custom weights would never be used as the fine-tuning base. + """ + items = models_config.build_model_items( + ['cellpose-sam', 'cellpose-sam (legacy cpsam)']) + assert items.count('cellpose-sam') == 1 + assert items.count('cellpose-sam (legacy cpsam)') == 1 + assert set(items) == set(models_config.BASE_MODELS) + + +def test_build_model_items_sorted_and_deduped(): + """Output is sorted and free of duplicates.""" + items = models_config.build_model_items(['zeta', 'alpha', 'alpha']) + assert items == sorted(set(items)) + assert items.count('alpha') == 1 + + +def test_build_model_items_empty_returns_base_models(): + """With no custom models, only the built-in labels are offered.""" + items = models_config.build_model_items([]) + assert set(items) == set(models_config.BASE_MODELS) + assert models_config.DEFAULT_MODEL in items diff --git a/workers/annotations/cellposesam_train/utils.py b/workers/annotations/cellposesam_train/utils.py new file mode 100644 index 0000000..31b5e43 --- /dev/null +++ b/workers/annotations/cellposesam_train/utils.py @@ -0,0 +1,50 @@ +from itertools import chain + + +def process_range_list(rl): + + g = parse_range_list(rl) + first, g = peek_generator(g) + if first is None: + g = None + + return g + + +def parse_range_list(rl): + ranges = sorted(set(map(_parse_range, rl.split(','))), key=lambda x: (x.start, x.stop)) + return chain.from_iterable(_collapse_range(ranges)) + + +def peek_generator(g): + + first = next(g, None) + g = chain([first], g) + + return first, g + + +def _parse_range(r): + parts = list(_split_range(r.strip())) + if len(parts) == 0: + return range(0, 0) + elif len(parts) > 2: + raise ValueError('Invalid range: {}'.format(r)) + return range(parts[0], parts[-1] + 1) + + +def _collapse_range(ranges): + end = None + for value in ranges: + yield range(max(end, value.start), max(value.stop, end)) if end else value + end = max(end, value.stop) if end else value.stop + + +def _split_range(value): + value = value.split('-') + for val, prev in zip(value, chain((None,), value)): + if val != '': + val = int(val) + if prev == '': + val *= -1 + yield val From 80e6083b05c142305abb078e958eb120a797d611 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:29:44 +0000 Subject: [PATCH 2/5] Ensure .cellposesam/models dir exists before training cellpose 4.x train_seg saves to save_path/models/model_name and only calls (save_path/"models").mkdir(exist_ok=True) without parents=True. On the default base-model fine-tune path nothing creates /root/.cellposesam beforehand, so the save would raise FileNotFoundError. Create the models directory up front. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0199mMkZxJV7p7p8zTs7wYFe --- .../__pycache__/entrypoint.cpython-311.pyc | Bin 0 -> 18709 bytes .../annotations/cellposesam_train/entrypoint.py | 9 +++++++++ 2 files changed, 9 insertions(+) create mode 100644 workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc diff --git a/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc b/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc new file mode 100644 index 0000000000000000000000000000000000000000..71030f03e00cedef399736312a0fd242d402df18 GIT binary patch literal 18709 zcmc(HTWlLymRON2iWDC*B}yXoF1;TXsh1^N58JJlWm~o+YizmQR=cC7R3(`iCq?N#9~N8jnlrdbbb__47no1_kLuM_xdy3>^{<~r{j&&Cfyc0-vj zTejeXls}j9L#ij2>VZ^mHub>3dezb7OAQpX*2wudKi9)`A08FfaE)TCG*}NnHgMf+ zb)Isp^%=DD}i_@#s-&Tkfd2Ya36jxbJ5WHYBbVm zQXB#oVdwbhD$famQjDodo{#d19iuk_d?XlI?KCQmiOK2dxhvNu7tc*yRctd?&P`5V z!+0mH*xw$9d^Jm{%qJEnE>2Edp1U$NJAX}ao|`;Betmjgg;PqG;=vHR7>=@BXfYV$ z!oqT{J3xFX{9*cQ76km~R4gav6m=&T7Y(BEiQzMtB43bc0`OGQ#2P=MVrGo1&z&4z z(R3##{Q5yd`1Q4DFcR~x zt-rF3g@QtCIT~I&lc<9{j$a8ZbN;bVbU6?b&OjtvTnnuT{Sa(W2b7`Wx}h-4RMj7t zsLGnHp^x7$%va-ILoJF&G8HqB#tHF%4=k9!9Tb>YG#X-7f)UOai$^$t2{02H#lCCf zGYsh<6Bl5zK*;gF!qi&^i;1FtOVlvwRzBa3O(NkH(ob zK6)?6a!f~j4M>1;p5r>Z83Cgl&oMw*B+3ZfT7VD4pzeAsx)uru;hso*ImE?JnV$6Tt_?Wp97h=p>aQW`h_U;EFF_Nox2$~c)&d-bm7-)IrbkprvY)u&H>0zVG zfZ`P_k{y@5raDiFjqhXb|agYMwsHTQtG`QuvDyojXAzHcNHG(&tXMSS6$=hZEV!mvBGHKe zbcIJ%0Nmvs#1apYdpPNt=nCmN&t;oZMX5soN>~(PkZYdtXBh`f=+Wq1iH2ZIfFDya z=6)~~VghSxFqTYEoftE?Rxm6G2^+m1$+n}LhN@=qzr-orX(HX=n<1`JTB=R02^@u` z=mSnjY+fVd4-+cRFX!f7htNre8mA+ntsTZR$iu7&M%LmnMhHb?er9Tg3B%eNTnhm! zft{)-acnotWaKndRx6RdEODq9SU*G531Yy>%X5DahxyO?U*am2NPcH>1P*2o`+d;U zab*psU?9}_5_Nn+{LVDQZU_1An*?0AL8Jn!Lc%|f>p+1v0tKn50mux*7$Vz11Zd~L zs{bW!1QLG@awvchswxI29_VaMAXjmvQLEzMWHpE?YcM{bd|-lyK}w?$5W*EQf!HW$ zSQv6uP683o(<<(6g7sUGzzhpz!Lw4Vs!Q8VQ0~it2&@h)D*2_r5{Qg&I~otM*|u`R z#dCym|4Uram7+*Iyu|UV|KflAI`zN*?IjDb#Cu4)+C`k=+*)+`HZH&F zaFMoDg#tbDGkv|iNI`&EjYffSN40gKdhow2#unbe7VdMw)!V3;mjmle$2c1Z-+(@> z#zO%>(7rUFm~QPnfO{=RHy3z6n}+i6df>k2dt_{#Qn03?!VuUDm4;xW`9sxF%a zefg?vLS68Q#<_-iq3<<${jbRzc;z5d1P6IOu=?s)E_Ibi%qu5Z3bFVqSO-ecBw^+& z#|^df=dkmgR>h(jWD0YAj*CPlqF{LigxC#Yrp@8n!QGz-1z{mm9IG(pIesx73x))x z3>S|@EsKnbpcE|=NWKI|*$IK-5jYLk6q*WCS>vtK0!s|$@M!JxWf+{p3eH0qHO$31 zvroW@z^D)J#d=fO08_%hgJo=k+H;o6&cV#FimmGhCPPWfR}=()hr!1AgQC1;xYy*D zo5r8F{RdyVX-;mMlYVejZn_3lz;=W>5JDvgH)g)EQY8&r$+XiaJAHt}Mpe`v80pGu zCWzM69~`qb+72iT%BhOFOhwI}r{%zC^4>HYSSfF9rnc_DZSw#K241S7=D=w1E*SP9 zzt~(4`52T?6&(jg%DZGlcnjSC`4~8~{2On`f9Gg^O~!LUMN)-GAOIxLsL?xXI4DAj z?M1avf`JPucGqOwl^IVx(shl{b*Lp(5W*3U;Un06q#)zy!y1E>et(lfo94))-{T$G^BGvvx3ocjFKrlE^S887^u4rU zT7~_qXb|cPcf_nkzlSwYG32oJuvUHPj|?$Xo7y89^rdX^U9=h~zP50e%-Tf5FDTXy zyF*8QcL+JoPtE%J2G~)jvPXE?M#yjMSU#!8qsO360X`#tLr1Yu5AgXQ*u!sgYEU{gq_zD0Z!RIEv65@*C zu41?+puPi^sLLD9p+qa}au<=b#VqBoJO20(K)*&4YlD&;@2KxjPdw^-6p-exLvr&6 zxoo0n`BMP<_a&X51|+{X$v=e3{1psP^YgLq3H)S->JO;rUMK_6G$H4h)ltCLb1JrCn@SNu_?qwc(n8J>~Ib2f5ik7z0<55 zQ>SXq+EdIZx_#;d^ac10&f36>NUAnN{@&<|2{z6hd<$KV&s9j1u&#$IyF3L3*cWY*MYG%_lb8^ibK%^b-$c}e5CNpGO{KLWTS9msN z_wB9&%4iBS$+`{({}dQ_L+Tq_L9x`AB{X5*)sg(%;uRrhbQCu;90BW`T*bS3(|YH zNiN!JI^jB^Q_fe@O3aNkMdCLy`^~ljg7~S%fQW;R%1zB3f8FZdn2Ut5Kj9W zAxjD^D%K&|enEYD$Tk8?u)ykKoud79Fk&$*(YR;R9=zFj;?na4FyZo-gV6WyM+VXM zhwd0IecB`1^bo8;y8zR-1a~LDrKM6m9%#vas3p4|qM#*DmKqpoxSM%RONHF2(Bpv~ z=UGpQhivTRF&$?s#bdAIY^)|v!5x%q@{oHjkDk9tSF9mh4tHX%Z2YybrLpF0Il$sZ zn>+|BQct}4%fw`vXpOaHVJPYz#h1i#*QgXFTJzmZx+8tD+(jwiMgxXCqU%nMo1#@u zJ8-Hm3rA5gysOb3(VBp}Ha%aHZXM^UVuM*|s5_94-^uZgt=41Er(VnV;XEF=b%Pr> zJx@S`l&0$PF!`8B;oaO5Mh~AX|5o_MBk&bsc?wU{k38a0DLb17y2E7SV!3Ezo1Qcu zqNDOVOF!-uJ&}{KH(|HG2D?W$GA+9K>^aPJw)N9P`x=mgLJd%Cm%ml()-4ezI@4wDOoLg#R8Xu z6mBSvJYtoe9s~98!gr*+M)drHQC{==Q4aE73x9RG(dlOg*b|3GNA#u&^l6?ghtHHA z8Q=GlEnjuAUaY%lt>)BL+7SBWKsZ6$tO>Y?8)+3{PGj=79C+}J#HnBrbPU_oEw&`Fyv-QNb zyps)UVNVsh6_#&FM}J$d&Yu|3}tvGWj(U1A42mVi29yVx}bzYnLsvnK8qYu_*;hcp|k zN_B65Y}fa}04`(?FN(UEkvuLQ2X6g!vO5;Y_73#fY2Z%x@4y|O=)03!&2NE*O!|{O z$=+n2*ru~#d-N3P%KG$WKQ`)0{rXZ}S+BnA$HtWc7q4W$*sg>A$Dm>0jicGGr~O+- zbKo%K!@qfCeAuhAesWe6dxkypvYjvZpph$W&!bg~z8Hfkgbb=W!;i=SNl3~bgf;7@&so*zn{OrFy9d`cX8 z9WQmO%_a6SJ1rIyxI;bAS=o4GM;@J(U99Ik_;p#}ly)ZYH*iow4k*z{C@e4?%jA|B zPnlM^&Jj|CM=sz{uo7I2!*wy-lTWeTxdQaufjc~17AGw8L2|S@qWVU$`J8-BbiC2Z zF`6$3vRZbqpdEA{(XugG0FAeVBMkT#f!`QI?hNyV+G*JcobP7c$+8D^d7ywG1Xm;I zoTP)xXEPoD^xd1?%tALa9}IKJ&U{t$MbK5k3~rK`Zr_j@b0iggF9967PDW$9o(feCzFFlww)a+=`8gITQ>BV{poQI{>bC#pvM(+9t%EhhZ<~9xR8zZ}1+M z$ontA97&X;2L{fTg4X>+?sD=`Py+rE@d(IWFAPdyHo2R@5zJLWp}LRWl3I;gsByHV zZ0X70ichP17%ls4XumJvSXI4!{_1R^a*mHKgVXk*VM#C-_-K6XP%g8y&Y-I-DkH_3 zi_xg;6vKjIeD~(+0D1oN*+fNF3UVAoF-mxjWD`v+z>u0(U}fQ6{$F@-UI!io1s*pa zSj3{{*WKq7K?!cyzg~hsyBq6H8Va4fMZ$9uRT4m;~!YXQM#b~t>7_)db4b@$NIaY zs~Yga-1QL#AWW-kam58*tVAF=?X!V`Rg28|gjLnyf?t~?|K)4}2wAfdl^RGlgZ_Rn z9iU;q-~W<=4t3fT8);w>#a(fz@laq1oUs*$ri~X-ww2-t_z4C=!36jdtDaM z!Wu}t?iSJOPO(N7RR=c3yd)rWF?zu+Lc_Ne8=+_sEuDmS9@jOvSyorptgGw@v${f_ z|9LjyxE3H9SnY@F`dUa+)oPW_Va1xgGAEV*zX-#_e}KU?2w;jrg&2Q>#KCbGe7x{7 zkDtfjItF-)N3PfSw=gyifnqPvtV*5EJ#0K@ZfU+@W;A&OrCbMfPPH9*G>>`wHiCZ} z@m8P-@%{}(R&J}+kEsqsu*iWABS(b59N_uDIuBc8N^xjQWPdmy+~siw zD@9AVjtTIQ#9jClK(T;_!6M4LGrQ!^px?7ntcs(+ttzGnOa%1FQQUDN20A8o4)ds2 z3#@=5)$l(o7HAzRwk5FH)X#8~Qq{cEKFUGC)FuHeCde?wMQFg3x&#XRN2CX;jPw5t zo4tj>Im|8?3Qa<6C?RT`Pa;U!5{zwrM5j?PdveCYp@Pb6>xc5oRbme1)~@0*2)ZI$ zt561%vSrmXzaUd_WS7OoyWF}`ovp88F8~yjV>z7a3J6tEODt-s@?3EsH%A0;Y9?!K zLNPvAS4y(#M5|)a(tH@%;?$KYP9kQ|6#}f-3mQ`l;E#z_XcQ==>@~!e9?4Tmv@A^+ zLG*%=#RuxoI{XA4kQDr^D`mRI7ni~PUU3yrpym{VJqKa|a?9i4cviU)MVD2iQ2{>h zQS7k5#uh^)A>h^?x1PV(LtJANi?%lNJAmsK@Fxtz)PQxVg);jjy8BJe#gFZrgFUr9ONbOS{Kp_t?gC#?!Xz>Duvh zNyo>Yv1!kQ?3s}0^1bq^t)A^3xq2jBJ}Q@wZp`dCU5^@4Ub(a@?L00!k8hmcb9=WM zw(I4pQ)&0G>>l2jenA&Yw$80F2;fcAT{7Jz(Or9(-6u8lU})=0)BQ5tFVX$`_Syr= zWF93>1^wXM2~MiTQ|P1$A-V-j5x0OT!wYmX7*3ar$R#6^b!6Y-mMk@&3aPPlL$BP> z`*eJ#VNhxqBzSxo&soGXPVkHqJmUn2@$ab5laxU1QBa^VRcF+@wD}xY`rI0?|oB5Rko*on65Y} zSDf6KCH{Xlf`G((!O!oH=gczTkd!p zvf)kBlQKOi(US;T`p7L+9?z5xf^UWSBoJag309E#Zq>w2)kMZxy5-p}2E(cP z9P+Pv0D!6oB)W3H-oIOaai{*G3StM41~TRig8%~KNY{V>n?1X9-40zR)eolWA(qnth{isA&@4sm@GYyTBwHDqN4Lx$hRK`&+F}=?a z12ciNK?pb?L=YU<=})gojf3gBA-Qe{QtF$w&dAmo$vX4GVw3ERX-kuAX-WllEbWq| zJuAUeC@`0az+5H*bD0Rt<--yTjxv(HA#Gt~3zHh(v9wC?c8x&z8$p`5mOeW#pPdJ9 znVqxmN@w3C;@F?5x{BE63HEt{oml6PL#U3Wt#R2Jm#py@v##I3onsX63fgP(eL0_Q?3Wv1#@A=+`>~~I z(%v*M{P$_EDK#crdv~oTcdRF+QM@5Ysky8=}a;ek);xtOM>WO_=X zr}q7qcKvfZ{<*aOs_egdK((8PzQQ1F?b>umm0g+2ra!W8(wl)tWgvsqwVNdwPb&~y z+(&}V3z_oz-SVNG@}Z~vv%Y8S^9$+nX}NrQb23xgwp-h~Q``Hr>UpbF+ncVvDA!)x zt)1JcolDnVm20nV&SV_byN<>kM`OyDcJ#`QUWvTGqpF(6)>Qq|CCNLS_72P5;Z5g@ zO7F+Fw~}&0|I_}b<4^nLhN0gq{bf+9yqK=MC|6$GwC}Z|b6C3doZNa2+@gxxNRW0k zZjMXG8uze1jCp?V);IBdM(#ryf;<#^B13B2Lp}8HA5YKo$DOWZ&CqSJxx+qxcq=x(?&m zHt_3#OnDVdg%dSU>qHIczpeNuBacS_--))p_O9*0KTZ5H@$2TN+`sDjH(gTu7#5rc z*tRp0t7*U4|E2JB@bkoz#BZ8aM5Bmkv<9f_t=X$>NSQym^!U=R1Qm}|+XwY~`vIh7 zKyooaLMgNBYTI$ON$taF*NE&Ikz6AgSM9E=dB@c(wVX)124&ZvMBcCJfDzyLDf2Oz zF5jBpJ}1$zWcJB)A4K+D8$#qI+;V(LkOM6u#EEc|HYL*zml;!9~Og&k*R3RRCqFFl^CgP&XjsG zC8c|16+a85%bMk~=8U`Lz(e`QzM|~Dw!gz5O?PfwdgR!;`%N`jg!)nohYSUE$ti#I zq0~Mjja)L%O(_7(1W+ z=9vlT4BYALW7ty!;Ga3m$) z1ShOMpZYoae8S@MkuTB{#XcXxfeD##_du{b-L~dy zl8>_Bx`7k?1({?M%>N8%=fMpkf>7v*p~L$TYTX-v?9H?1k0Id zDO8ZB_G#NLO`vSs?x(Tm7bMz~rl)0kTB4^jxx(UY(^K~ONm6)OrY}qM<$R%iyZIUY z{01qUk?9$Up2-)MY+uVXwr;QH8ZR)%fk;Q$uDy20Ub}Ve@w=%&YALn!_*UB9CEL3; z&h0xroAVDBw&=&l(#{Uq*|Bjx<0{9MCF3l8_#UJju7{U4CiiV654$tY%FUZUxw#4d zG7k6Vg-26c_fqlgm9%3}b_`0{H{ Date: Tue, 7 Jul 2026 09:33:20 +0000 Subject: [PATCH 3/5] Address Codex review: region fallback and min_train_masks - Fall back to full-image training when a Training Region tag is supplied but matches no annotations. Previously the region-cropping branch was chosen based on whether a tag was given, so an unmatched tag produced zero crops and handed train_seg an empty training set despite the "using entire image" warning. The branch now hinges on whether region annotations were actually found. - Pass min_train_masks=1 to train_seg. cellpose 4's default of 5 silently drops any crop/image with fewer than five labeled objects, which is common for user-corrected crops and could empty the training set. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0199mMkZxJV7p7p8zTs7wYFe --- .../__pycache__/entrypoint.cpython-311.pyc | Bin 18709 -> 18706 bytes .../cellposesam_train/entrypoint.py | 16 ++++++++++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc b/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc index 71030f03e00cedef399736312a0fd242d402df18..e81e4d69d6d72c371c39d9e13edce70f1716bc96 100644 GIT binary patch delta 1112 zcmaJ=O>7%g5Z^ck`@IC(87TOXF#Y`(i#+gdf*19I-9X{o@r2mbDqy|XTQ$(Jbb_Y( z!t~zYszX9wUU=;k3t)gAC=vu{V%Hv3Km?Mb(|EUeOYvsT?2*L)} zwt%owTo71@YacDD3P^NS)g$tpjpLD7809$*l=ZU#Hppuf6!{%5GZo!8RVqci(jX3C zKdS3|f{ooNT76jr-P8nM7YZ1mSj4bLQK6lvMpq&dDD+9hBgJYj>|x-pqIcF9qdSo) zQLDYui3kFQDHQdGbr#rnCfTVlbC(59(A%*D#Oa-=-_R=YP7dSie3T8YZD9%2&GCzT zgBm8rxnG&(v=3p3-c|gb*g<-*uN8kvI~ZSu6$3e`(q3Oa^bihla(-jUTIt`QU`}vs z$rd*G7#^z(cQSLFES6$vHq4ImnXhU=a%xWcSrh4DOr&SDBz>-lp#4Y2Y0tC~*Y%^e zHb%%0=w1ksUq9_NYOOc6C*~v?Yr+j14SPLQ*L_|CJ)cb-;3Z4n>M}q5i$2Y5C_WL& zRz^%_Y98E^Zf(XpYg8_@y~PP z_#-f=e&Q!^&>NZ8kBEzB&dt}BO584%s%NX#51FzJa#m`j8+?xCYI%OCym03Hx$3XM zg{AY$Z5F5PpdEG@e$qYL?$-~;+GufY5CFn=08qUpjU&t5KMpibWuQt&DLRWM-33ZE*2_fgns}Ecoen( delta 1048 zcmaJ%?-KNvoT+UWu`Jj=JLB%3iJyh;V&j$s) zHN-p_Q3SnYfxgs@dWfJ$dx*AW5rl*dl-F!EO&$G2@JO?5=K;0fmzeH4U?9hh=9W5F{mG{Vj_Ilhr{0u;8^Px zCw$xSl|K-kriFdL*JrC{A}k0&zJL#`C4cgdlVp3Z5NKIzM+`wI*H(>;@ujdZYh}Aw z8z1Pk+|5x>!+25=(XW{Bs$$Z$gZz(3N2MKe`{s*-ru8--E8FplqNls?hcax53nlimd+f4Jc6rnJzBa=b!G0j%`qO=!rgCl;blWzQD`n1TuTPm-Bj7& zEAl7M&Gqx~V*hX{IkJ=-DIOUuCr69PQLSh9^_oUOEDF-S@nW#2Bqx^S#0y%MQ$;zY zz3cl*X#+0q@hZy=mf3Tq25G55dT;iwT1hDJl?WmK_cc|jRWBWw-g@)w54&dVH&R+g ze|nJK!twM7ox;VmL<{&iJxph@Z=jDyFAqfMHGDGAN*~~-fhhf=Irg5WmP0z4p{r!W KaZ0<90s0GY*#)uy diff --git a/workers/annotations/cellposesam_train/entrypoint.py b/workers/annotations/cellposesam_train/entrypoint.py index 56099dd..3adc21d 100644 --- a/workers/annotations/cellposesam_train/entrypoint.py +++ b/workers/annotations/cellposesam_train/entrypoint.py @@ -247,7 +247,14 @@ def compute(datasetId, apiUrl, token, params): sendError("No training annotations found.", info="No annotations with the training tag were found.") raise ValueError("No training annotations found.") - if len(regionAnnotationList) == 0 and training_regions and len(training_regions) > 0: + + # Decide whether to crop to regions based on whether region annotations were + # actually found, not merely on whether a region tag was supplied. If a tag + # was given but matched nothing, fall back to full-image training (as the + # warning promises) instead of silently producing zero crops and handing + # train_seg an empty training set. + use_regions = len(regionAnnotationList) > 0 + if not use_regions and training_regions and len(training_regions) > 0: sendWarning("No region annotations found.", info="No annotations with the training region tag were found.") print("No region annotations found. Training will be performed on entire image that the annotations are in.") @@ -293,7 +300,7 @@ def compute(datasetId, apiUrl, token, params): mask = draw.polygon2mask(label_image.shape, polygon) label_image[mask] = i + 1 - if training_regions is None or len(training_regions) == 0: + if not use_regions: training_images.append(stacked_image) label_images.append(label_image) else: @@ -341,6 +348,10 @@ def compute(datasetId, apiUrl, token, params): # - AdamW is always used (the old SGD flag is deprecated), so we do not pass SGD. # - bsize must be 256 for cpsam. # - training uses native resolution (rescale=False), so no diameter is needed. + # - min_train_masks defaults to 5, which would silently drop any crop/image + # with fewer than five labeled objects; user-corrected crops commonly have + # only a handful of objects, so lower it to 1 to keep every non-empty + # sample (samples with zero labels are still excluded by cellpose). model_path, train_losses, test_losses = train.train_seg( model.net, train_data=training_images, @@ -351,6 +362,7 @@ def compute(datasetId, apiUrl, token, params): learning_rate=learning_rate, n_epochs=epochs, bsize=256, + min_train_masks=1, save_path=str(CELLPOSE_DIR), model_name=output_model_name) From ffcd7aa62609abcb243d2d6eb9565cd65cc2ed4d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 09:37:54 +0000 Subject: [PATCH 4/5] Apply same region-fallback and min_train_masks fixes to cellpose_train The cellposesam_train worker inherited two latent bugs from cellpose_train; fix them at the source too: - Fall back to full-image training when a Training Region tag matches no annotations, rather than producing zero crops and training on an empty set. (Also avoids a latent len(None) crash when no region tag is set.) - Pass min_train_masks=1 so crops with fewer than five labeled objects are not silently dropped. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0199mMkZxJV7p7p8zTs7wYFe --- .../__pycache__/entrypoint.cpython-311.pyc | Bin 0 -> 16547 bytes .../annotations/cellpose_train/entrypoint.py | 17 ++++++++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) create mode 100644 workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc diff --git a/workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc b/workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc new file mode 100644 index 0000000000000000000000000000000000000000..f0b04bfa4c5be5b4c24bb07e42642a87a1b0eef8 GIT binary patch literal 16547 zcmc(GTTC2TmS7|q2}wR6At5t)CBd5nGQm7;@S|+NHW;wm#`U1elbsL&B+dsN5!fat za)ustYuHh(#Eg2?^tMaN8Fkmt)2#E%J@piMDi0sR)mO!1U~G8A1#mp^lTPC^a%S058`W{nD{QTeTExW9Fcs&8GR`W-cz^j;PIGUQsb zo~Avt>4!x%dXhN7y%ylFd9771W?I(T^!N$Lb|~x6%N9By)tO6mLMo6;1t8U>r=FN- zuR5A+g^7UH>X;6ulL;{GCr5=g-6Lq92I~dLTBe<@$y1JMeQt)4gn#t`nL>56|3K|) zlsy+PDR!D!j%>tuI=aL|lBC(lWB4t(lZ$shKVjNZLx5 z(l**oJDwKPPBaT?7hUm`gxTe$tLSQg^3Yz$_t7!E8QUPsr{4Non!pKhcn4CAMp z=w=*ip~ds@30;gOmm)FlGDP&^ zdT4^{hhU31CQQ}$O@*Oc<3BbNp1R%PA3Z3{SL0tpErLZfDi)xN;ZlD;zZ&HzKADVB zm@1BnP~#e%6z&qW!NFXCkYR&MNtR`nc!s75`W2!sMyOSmS-#Y~%Jb{oXm2l_T;f8B zXnYas#v?0C2>OLGH{#G~%ne{nB(|9X%s~thq$7MJ_&Cb129t|x&;|#MB@z%ptP4gG zba0(bt}`qjWw>5MFBe=+vcY)dA;O{6Y0+66#GQeXW{QunD-3_BIlLH)Bpx^g0;TQ_&Qke>>F)=Ed zpca$-DzyY8av|H)GPMbaw7P1|9%_Aq$GyVo4enHB)Z;m<5UFtuFKTT(*P5QY9A#oO z7n*;K=2W1Vt-~@G<(W9AlxXT(DM91NN7t2-L~=Z`1S~*10mTZjrH6~jC)ZPh8krH6 z$+1-LXoOV-vd&=}RS-aer9` z3k>j69o$Y}>moyOkw+?vfmJLO(xs~x1931qF_4l#=r!7s1H#u^ngSxF1APL9~sya}Zs9avI0VGLhVJIADT>a!gb^+&O?%=b2;EJI^1zW#f7QZL`pD~^HLOaEi~rBeE>Iu&DJxHK_9bDipp^6E^1 zp^UC1V9L`}bQva8EXL3_CZ786xCk_k%FH;@_OsnuMw zd}Jl`8m&pG{|wB5QP$x=;-jD_co=Y&SwSjIt%@>qRh1(Zv>eR)I1rmmbQF_)^{SIBA_jFQnaq9dKoK|rAPwwISrziT#PKnKo(b%8!=jMD-JuY=z1(i zdFVAdo1l$Re34;Sg5&l7`@ap>URi-mc#f{D{C?HHx$$39Ke&>*h9m?=1@E?~oXRWg zbcpH;^_@fdE<&v&lh7#WYaR)^x-!1D15RCknB&oeVva|ioS4yBB!z%^Gh8`U~`B%*Votg4g!$wPcaEb}PJJRXllVZ~PL*RI~146A!(#jz4ai(`0$kH$Eq675UE zS{CLirxY)#)rz?!3!4RnL@?^1HUrl`G z-hDq4&o&HXYX+s7!EDu#R5g@!4ol8qz-uF_>jAmvp82T0;keXNXFDcPP)1ZY$klbS z*MD4O_PtkhY$bg4GJv>kUI0OXkEm{xz4fwp7UdhNLcJA~Swi{&7zGZZ+J8(I`QA4j z7ZWvpR9J4Uhe8xoL80uek$nxvMMb_<1k&J#a#R@wP!1Sxs2FPyMhF079Ri=j;El)j zVuTI_E=0S#Ms|B-FVy>3uJ<21OKPDM1(mqeBfGu0lvYdKs1yRVv zU>z}mRfLF}(qz0SZHb%H#c?uSVj=`{;W;X86|6Lw-{YrCA#Ev0+aO(BkhTlfOzwQ9 zH;sPj6547!d!!v_!j_(bb)E@pI|W;ICanDwY_jZ+jSU%cw}O0`JBZE>gTvA zxYv+Y#g9;Gr0>KS*6_VLBth`!^6|AC$LLxkH;k#bN`T}0#e##bd+RuZec>(KFC6E3 zV+%0Ob{1JAOe;lcPoi0H0(NH(yNAzF3Q`1Ci__S1bxDXdAuRk*zNcW_0LzAv*Pwf< zG6iFvhZYEA19{eAgim|l3Ez1N-Y0l7$VPDL5q!qp!+4G5f!2`G13!XSD5C?c)rF1nxn$`DYLIhAnqU+QC9a2`oGSR3-yz;4?9 z?4nVpr#pm>Oluw{A2U)1aL(^fW8H~%Uj#ex4^ZX&M^V)&bpBRUbtbC#TX~w-a%-H> zVU)FWrva`*uOoEi{UBiVZYuP(@Y6aJtdP0Y^4X;m91QUH^Ke45v0aebTX;;n(hUX* zy7fBr#pQy&q=Gl5o5ZW|46S3bOrTzVNZna6EyY+U9=zh1cZRla}j<&odF&8 zfq|l&-VizjBe%}v>WZ1Jd-mRmwt@n6#{55_ZDFNsIs-Cb?6V1ciJ$HWhMAfU2_cv> zrUz5$UOudI0kr;A;Aii*;AfxEx7MfE{{(b)`dqp{J&+z0It+W|ej`nWvO#0n4~qp>1pe7+7&${yhlF86NnTQijC_5oq@F)hQqLP} zze7?-PC|mD-r6bp^nVB?nSwpBE??l4gJ^+A?}A}pbWOPM)?KkBgB(Jq9-;KvxUv28 zr~%)EVTVE)VPT&&`IdcF;cnwPeS`joaNGnOuqG7xb%o;0z$s!OLl|8f)bj;OXc9d1 zlu!x(E}??<^SN0FkhT<92A@rT4&g81Uw!bJIF`PczGPT+Z_zXKtYE{OJ3)!zl#}b3 z;gn-D>M!smTEYJ}7x^zHg0LaG(|UxB#5t;S30qB(oqvT1jN&3x43_Za=nC@qfZcbB z?#UG(mlv4B3|S1f#75T<f*Ym4 zF$ca%3Y*{Zg_^%&w7}J;TjTRU5sm}HI&xeX;PTm2=kI^`Q4h7yL(NCyOgfOS3bu02 zPzhyFoyb&(jJ6!lf(4T+tPqW%Za@gOVJ?bzM0L*Jp13_qvB_xQ?3Oe~;(T(HT1`F% zx7p)@K5@FW8+-&BAVcFJ!hTY?6;KpSvNIrG;4wyKSGXkrjs&G#wVvh-nCu9EuoqBp z9tEQi%(3Wv#iI3;y@Z0xC_sC0_B{wvWjV_&^nH{{Rp!!~lK}E+<}|Z4vgZPrW|e2? zP^x`=H3{xyDuVM1y^e+b3n5o^G_jmaH58)K-4Ln+5TUz~YaM$RfggpA2nxm+96+j+ z4wZ8*j}uZ<|b6C4qMbYiUyLG_$2t(U~GBl z0*IVq>SDVQ`$vFI-TkIXdyc!zp2UL30{bA)-I361 z&=f>!5}M2V#X2?5TR@*w#-{UbJ#7CjXuKV+wJd?>7s-R_nm^fkG7;Mhy@rDVvGT8f zf}VeOCFPz;MsSsa0ZTRC1-EAu4E=zrAfH-hlW`!2eZ;WQ_pX%_O3Au*v!+O~#G>&i zuUNR%2zbp)5qm-3#hB|LW~I!Nr5Mokh)LyL6fjFuRhrkUu$55CUnnPr_LCb45Xe3# zFU&@_1B}itG#?&=yAwtW*QkdYs$d%`YNcqr-;GbJdlD`EF0}tt@~n;3f4GwJ;1LGR zY8;jnb%RZAte?oG7B?w$M+B)u#hQzeNZl%?1*Pc2k5&fo^PjGys&y&gbq5lfGAZw= zY`hk5Fr=1cSlbvBq0vfXxGqx+5{8T}UoS$SU6<)en+mT(38oB^{Zo%|lVjX=(Aa4I z!-8v0aOEZlORkyy01(aL$r~tQZ$d&T#@9BuUlg&`C}~?tCRvbk@Ox=jaa5O(4+>Lt z=yuHp|JF-2aKVw_W&sLcs-fo5rJ7tQw*4{?6WUHVmy9~j%>vsd{H(I4|zq=lwe&<3tI0Bc6dWN4bJ-hN^`X%?P+hW&D zA%Ggky-eBfV}ZSK=l&=Ky|iJy6-Gi+u+HHroU*E`2p7_1{%@}ofZ%#5j|S3%u3@Z$ zbc5~=g+lY-x3pp54I{}_9BMokS%kY+=;8>>Zxky9XMa2ri>Ba4kLtD!W4EnhzIS6n zaq2$lF!D(&)@$-MKq#XAmSY@h|y6&&XG~KGI z>cB7%qFte4@1S8py9f3=D7XuOVut%jFtMN(I7PvasaQaR!;d+}Bv|ClRBZ2~%Siaf zB8x65uvbxV4Hel7G?>z0@Z4U_-8IwPwidKm;*=@_)P%b0WXBQ8352qmPCibcJ{a7% z*!_wp8H;W-DC9~kxyYh9uhc*X@{MuVHZ`16v{DS$Yv6{Z;>wd5#<#U_rEor%3nN(w zgR13N*ptIdj;yakO_o9lZ^4wAPhLklXevQ7Pn1$<`Yv{JD<$0bHW&s*xG1r%*isCe zjrT4+%POna36rd!)=MMY-E##gFK-)w1ur74L2-md}Qe%i>9Me zyoi<*xU2-1b;2v_8;Vo+>(3%rzEY}69^B4R%n2Az_XZXN0}BI}S!DVt*rF7xXN9+k z1$z%QYeP54!dQ@$iX^@rpr*n|1{J5aCD9$)d$iaq0tsu_)#FVNf&d1y~hRltmUgg~UQvmTN)< zt$YiW6{qr)GA&CJV7Pw@s_TjRvk5AZMAL#r-auVVf}UQ@spKRF_Y4&~EW~^`h7%k*{-6`_@BZM`Tq)5Oa`rz0!cF++ z#^6d4tYEE#<(x?Nz56}#)Qb%v$@kte3S!cK8?B2S5==SY4?)jyf^I7)=$$epK<`r2g+5)>5A%Hhac1dKHNOm2f z>~mscFA8nvvShzR_KRfyQHfhDsr!t}T+BB1NsWEaukJSviH$>t)(WvQD0{EKm6p=0 z=t>KOaHR!8SkobVRpt}nB1_qjR5m19hvX8kSkn0O0kLBw+j3rNIsfA7e#<4X<fa1yvP+WOoYZ?a9TKQR>Skembp|wh~cI^#6yY!+fYn_y= zlcIIP~82^`dn@7IjW){0&4UMcM2Y9|oh+5rHn9S}*+ zkw0|czq#+fse;%6q;U#?G!6j>2nksS#9O=vWWzq$Ao_>0+ zW^r&L+czooP5#ET-!~=pO<}q3muv2#mgaFw^SC8^a)2D%13qirkgOY`b>kK3LSq$@ zD+bV5As$`%qmpm<`6J19IhWuh z-bjy7aO5$#xF&m1#xsH$28y@wAUQ`Vm{l{ zFExRR@XP*w)Y1%YZw8qEyR_GwxhPrt4y+^l))Ddibk=%HvfdK$d*tzfzI0b3-GzA& z6nk%G$tj7P63MBf(DXs*&VJ}lHgs1C-907-EyG`-AZzW~cEHfcp5}jH-zK*sJCz`h z6?F%${(V=!>?qqY?F{ZV0y8VRfSDCt+m<6=^MS8t-`BJE(F;cO^<;e$l5b+$Df`+E ze8GKRFzf4;e7)OFJOMk!;N_Ri`@sn@I3ZX052}XutA=G)<<8X|9@YmyT-^tVtNXSm zkNlkn{{DS`|MLqkCq;jM);}%zr?+p*{@{UsXdmW=e?;<+Y~S9#{mM^0jqlB7{iBk9 z6p|j_k5+fnQe*$~{^wVp_e+h#zh3-RRP@};dTvUdo7?uowvgC6o^6|u+9u%AU1>WG zvW}+ht73W6A*v5zl0S-`8-F<~okL;=d1%&Pu~Xw-d@T{e?HjM$Udi2>c>w3s7advm zHOYN#dqSp~GFSiHvVCip%sLuGN5fHh?J-ebI(*pB`1C`$!k;Psvhr^lUZk_xX1z|p+#XwF>DI{GC?zvzG=@f|oC_Z^KHd)Cn-IeJ7# z4@8cd#$L=wO;e)7FSmhqs&>NoLLj=D4x2imYx&dW*vxMpNMj#EgOCReLI@42@#fz{ zaR_(H;F@IUq7=NAb#?7nb~bjI%zcoI_JJ=4A?O4I ztw$}PFSzGJzeqhx{kla(G=>n3)j=ow>JIB0GnSuBKb`(LrxGI8p92E=`T?YMKy*=X z3$-KgvsJmeLkwK{HT$cyI2+Dh`L1;3yJA&DTwMd;ju>JKgir@U(bWRHsGts9?fb5F zv12sr8k1aOqH9cc)gQQ8_FZTflXVSAt|8Gi1d*@0f%C^9qN4i8jz4q!Y1xm;wx;mb zp=B5L4=wO!$ySMk^|tlMQF-8?_8rvTwCJF+juFW*vNegf9~V9w*!{z7U9VIJ+l>qA zn!j4UQRXdF} z45xnBMTb+9V@?}10z^L z;7i4vLnmbq;;)>r%jY^eY=R#_|7xU*fNgYf1b^<7vIc|dM{4t_l3);jrRxr6#0m3q=9*Ep~hnz>Cl z-Et4{sY5*tTi}emzRCVeM2ia>c)^go!X)EhCf-z!#WneJP*}a?9n!9mV;hCyJzb8C z5bXC6(uUKe`5cb@Lj>~{4EUxwHa%eT2sVi*ZTXt`DmplX;Jc=ROl)>w{}^y$GY7U} zs8)|Tl!wj^3Jf_qSBJw&*;RH0f1Y*+aScBG!rWsOM}(%sz)V~QOygO_hKo=p2gW>h z`77;~BZJIDjO9Krn= z81Z4=n@loMzJ>o}qD3^mGGX7se==d)!hbU16OFG-REqk0M6`;=_lW2gjjv2}h{jhY z21Mg46F0@PUzvDMG`>fKTh!mvy*MiC7xBAgKQ8{R$pm|&g7CWew%LR>!-gNam>?b7 zseN-z)?9PsY~B7)a<-X*TyZ4@cn2BT-GmB3@$=bUl zd)L;)k<+_9|LMXm`LsOi?3A3HTi0b*71H6dv*Oc_A?0v=I=wY{WGnl$M|OI)Kl+o8 zw&729xVLZYOzl3(Z0s#(9Yd01NYr21>HCw9w?CF0)!S1$%ex=^D4BV*w{ZgK_#8pj Risl+vIm&9bKgQbf{{iLR6lnke literal 0 HcmV?d00001 diff --git a/workers/annotations/cellpose_train/entrypoint.py b/workers/annotations/cellpose_train/entrypoint.py index eb2c861..37ad229 100755 --- a/workers/annotations/cellpose_train/entrypoint.py +++ b/workers/annotations/cellpose_train/entrypoint.py @@ -217,7 +217,14 @@ def compute(datasetId, apiUrl, token, params): sendError("No training annotations found.", info="No annotations with the training tag were found.") raise ValueError("No training annotations found.") - if len(regionAnnotationList) == 0 and len(training_regions) > 0: + + # Decide whether to crop to regions based on whether region annotations were + # actually found, not merely on whether a region tag was supplied. If a tag + # was given but matched nothing, fall back to full-image training (as the + # warning promises) instead of silently producing zero crops and handing + # train_seg an empty training set. + use_regions = len(regionAnnotationList) > 0 + if not use_regions and training_regions and len(training_regions) > 0: sendWarning("No region annotations found.", info="No annotations with the training region tag were found.") print("No region annotations found. Training will be performed on entire image that the annotations are in.") @@ -267,7 +274,7 @@ def compute(datasetId, apiUrl, token, params): mask = draw.polygon2mask(primary_image.shape, polygon) label_image[mask] = i + 1 - if training_regions is None or len(training_regions) == 0: + if not use_regions: training_image = np.stack( [primary_image, secondary_image, np.zeros_like(primary_image)], axis=-1) training_images.append(training_image) @@ -306,11 +313,15 @@ def compute(datasetId, apiUrl, token, params): sendProgress(0.4, "Training model", f"Training with {len(training_images)} images, be patient...") + # min_train_masks defaults to 5, which would silently drop any crop/image + # with fewer than five labeled objects; user-corrected crops commonly have + # only a handful of objects, so lower it to 1 to keep every non-empty sample. model_path, train_losses, test_losses = train.train_seg(model.net, train_data=training_images, train_labels=label_images, channels=channels, normalize=True, weight_decay=weight_decay, SGD=True, learning_rate=learning_rate, - n_epochs=epochs, model_name=MODELS_DIR / output_model_name) + n_epochs=epochs, min_train_masks=1, + model_name=MODELS_DIR / output_model_name) # Upload the trained model to Girder sendProgress(0.95, "Saving model", f"Uploading model {output_model_name}") From c76d54ea065684de095424eef97db197579887f5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 09:40:34 +0000 Subject: [PATCH 5/5] Second review pass: drop committed .pyc, hoist models dir creation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove two __pycache__/*.pyc files accidentally committed during local py_compile checks (nothing else in the repo tracks compiled artifacts). - Create .cellposesam/models once, up front in compute(), before it is used as the download destination for a custom base model and before the glob that lists it — not only just before train_seg. Avoids relying on the Girder client to create the destination directory on the custom-model path. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0199mMkZxJV7p7p8zTs7wYFe --- .../__pycache__/entrypoint.cpython-311.pyc | Bin 16547 -> 0 bytes .../__pycache__/entrypoint.cpython-311.pyc | Bin 18706 -> 0 bytes .../cellposesam_train/entrypoint.py | 18 +++++++++--------- 3 files changed, 9 insertions(+), 9 deletions(-) delete mode 100644 workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc delete mode 100644 workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc diff --git a/workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc b/workers/annotations/cellpose_train/__pycache__/entrypoint.cpython-311.pyc deleted file mode 100644 index f0b04bfa4c5be5b4c24bb07e42642a87a1b0eef8..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 16547 zcmc(GTTC2TmS7|q2}wR6At5t)CBd5nGQm7;@S|+NHW;wm#`U1elbsL&B+dsN5!fat za)ustYuHh(#Eg2?^tMaN8Fkmt)2#E%J@piMDi0sR)mO!1U~G8A1#mp^lTPC^a%S058`W{nD{QTeTExW9Fcs&8GR`W-cz^j;PIGUQsb zo~Avt>4!x%dXhN7y%ylFd9771W?I(T^!N$Lb|~x6%N9By)tO6mLMo6;1t8U>r=FN- zuR5A+g^7UH>X;6ulL;{GCr5=g-6Lq92I~dLTBe<@$y1JMeQt)4gn#t`nL>56|3K|) zlsy+PDR!D!j%>tuI=aL|lBC(lWB4t(lZ$shKVjNZLx5 z(l**oJDwKPPBaT?7hUm`gxTe$tLSQg^3Yz$_t7!E8QUPsr{4Non!pKhcn4CAMp z=w=*ip~ds@30;gOmm)FlGDP&^ zdT4^{hhU31CQQ}$O@*Oc<3BbNp1R%PA3Z3{SL0tpErLZfDi)xN;ZlD;zZ&HzKADVB zm@1BnP~#e%6z&qW!NFXCkYR&MNtR`nc!s75`W2!sMyOSmS-#Y~%Jb{oXm2l_T;f8B zXnYas#v?0C2>OLGH{#G~%ne{nB(|9X%s~thq$7MJ_&Cb129t|x&;|#MB@z%ptP4gG zba0(bt}`qjWw>5MFBe=+vcY)dA;O{6Y0+66#GQeXW{QunD-3_BIlLH)Bpx^g0;TQ_&Qke>>F)=Ed zpca$-DzyY8av|H)GPMbaw7P1|9%_Aq$GyVo4enHB)Z;m<5UFtuFKTT(*P5QY9A#oO z7n*;K=2W1Vt-~@G<(W9AlxXT(DM91NN7t2-L~=Z`1S~*10mTZjrH6~jC)ZPh8krH6 z$+1-LXoOV-vd&=}RS-aer9` z3k>j69o$Y}>moyOkw+?vfmJLO(xs~x1931qF_4l#=r!7s1H#u^ngSxF1APL9~sya}Zs9avI0VGLhVJIADT>a!gb^+&O?%=b2;EJI^1zW#f7QZL`pD~^HLOaEi~rBeE>Iu&DJxHK_9bDipp^6E^1 zp^UC1V9L`}bQva8EXL3_CZ786xCk_k%FH;@_OsnuMw zd}Jl`8m&pG{|wB5QP$x=;-jD_co=Y&SwSjIt%@>qRh1(Zv>eR)I1rmmbQF_)^{SIBA_jFQnaq9dKoK|rAPwwISrziT#PKnKo(b%8!=jMD-JuY=z1(i zdFVAdo1l$Re34;Sg5&l7`@ap>URi-mc#f{D{C?HHx$$39Ke&>*h9m?=1@E?~oXRWg zbcpH;^_@fdE<&v&lh7#WYaR)^x-!1D15RCknB&oeVva|ioS4yBB!z%^Gh8`U~`B%*Votg4g!$wPcaEb}PJJRXllVZ~PL*RI~146A!(#jz4ai(`0$kH$Eq675UE zS{CLirxY)#)rz?!3!4RnL@?^1HUrl`G z-hDq4&o&HXYX+s7!EDu#R5g@!4ol8qz-uF_>jAmvp82T0;keXNXFDcPP)1ZY$klbS z*MD4O_PtkhY$bg4GJv>kUI0OXkEm{xz4fwp7UdhNLcJA~Swi{&7zGZZ+J8(I`QA4j z7ZWvpR9J4Uhe8xoL80uek$nxvMMb_<1k&J#a#R@wP!1Sxs2FPyMhF079Ri=j;El)j zVuTI_E=0S#Ms|B-FVy>3uJ<21OKPDM1(mqeBfGu0lvYdKs1yRVv zU>z}mRfLF}(qz0SZHb%H#c?uSVj=`{;W;X86|6Lw-{YrCA#Ev0+aO(BkhTlfOzwQ9 zH;sPj6547!d!!v_!j_(bb)E@pI|W;ICanDwY_jZ+jSU%cw}O0`JBZE>gTvA zxYv+Y#g9;Gr0>KS*6_VLBth`!^6|AC$LLxkH;k#bN`T}0#e##bd+RuZec>(KFC6E3 zV+%0Ob{1JAOe;lcPoi0H0(NH(yNAzF3Q`1Ci__S1bxDXdAuRk*zNcW_0LzAv*Pwf< zG6iFvhZYEA19{eAgim|l3Ez1N-Y0l7$VPDL5q!qp!+4G5f!2`G13!XSD5C?c)rF1nxn$`DYLIhAnqU+QC9a2`oGSR3-yz;4?9 z?4nVpr#pm>Oluw{A2U)1aL(^fW8H~%Uj#ex4^ZX&M^V)&bpBRUbtbC#TX~w-a%-H> zVU)FWrva`*uOoEi{UBiVZYuP(@Y6aJtdP0Y^4X;m91QUH^Ke45v0aebTX;;n(hUX* zy7fBr#pQy&q=Gl5o5ZW|46S3bOrTzVNZna6EyY+U9=zh1cZRla}j<&odF&8 zfq|l&-VizjBe%}v>WZ1Jd-mRmwt@n6#{55_ZDFNsIs-Cb?6V1ciJ$HWhMAfU2_cv> zrUz5$UOudI0kr;A;Aii*;AfxEx7MfE{{(b)`dqp{J&+z0It+W|ej`nWvO#0n4~qp>1pe7+7&${yhlF86NnTQijC_5oq@F)hQqLP} zze7?-PC|mD-r6bp^nVB?nSwpBE??l4gJ^+A?}A}pbWOPM)?KkBgB(Jq9-;KvxUv28 zr~%)EVTVE)VPT&&`IdcF;cnwPeS`joaNGnOuqG7xb%o;0z$s!OLl|8f)bj;OXc9d1 zlu!x(E}??<^SN0FkhT<92A@rT4&g81Uw!bJIF`PczGPT+Z_zXKtYE{OJ3)!zl#}b3 z;gn-D>M!smTEYJ}7x^zHg0LaG(|UxB#5t;S30qB(oqvT1jN&3x43_Za=nC@qfZcbB z?#UG(mlv4B3|S1f#75T<f*Ym4 zF$ca%3Y*{Zg_^%&w7}J;TjTRU5sm}HI&xeX;PTm2=kI^`Q4h7yL(NCyOgfOS3bu02 zPzhyFoyb&(jJ6!lf(4T+tPqW%Za@gOVJ?bzM0L*Jp13_qvB_xQ?3Oe~;(T(HT1`F% zx7p)@K5@FW8+-&BAVcFJ!hTY?6;KpSvNIrG;4wyKSGXkrjs&G#wVvh-nCu9EuoqBp z9tEQi%(3Wv#iI3;y@Z0xC_sC0_B{wvWjV_&^nH{{Rp!!~lK}E+<}|Z4vgZPrW|e2? zP^x`=H3{xyDuVM1y^e+b3n5o^G_jmaH58)K-4Ln+5TUz~YaM$RfggpA2nxm+96+j+ z4wZ8*j}uZ<|b6C4qMbYiUyLG_$2t(U~GBl z0*IVq>SDVQ`$vFI-TkIXdyc!zp2UL30{bA)-I361 z&=f>!5}M2V#X2?5TR@*w#-{UbJ#7CjXuKV+wJd?>7s-R_nm^fkG7;Mhy@rDVvGT8f zf}VeOCFPz;MsSsa0ZTRC1-EAu4E=zrAfH-hlW`!2eZ;WQ_pX%_O3Au*v!+O~#G>&i zuUNR%2zbp)5qm-3#hB|LW~I!Nr5Mokh)LyL6fjFuRhrkUu$55CUnnPr_LCb45Xe3# zFU&@_1B}itG#?&=yAwtW*QkdYs$d%`YNcqr-;GbJdlD`EF0}tt@~n;3f4GwJ;1LGR zY8;jnb%RZAte?oG7B?w$M+B)u#hQzeNZl%?1*Pc2k5&fo^PjGys&y&gbq5lfGAZw= zY`hk5Fr=1cSlbvBq0vfXxGqx+5{8T}UoS$SU6<)en+mT(38oB^{Zo%|lVjX=(Aa4I z!-8v0aOEZlORkyy01(aL$r~tQZ$d&T#@9BuUlg&`C}~?tCRvbk@Ox=jaa5O(4+>Lt z=yuHp|JF-2aKVw_W&sLcs-fo5rJ7tQw*4{?6WUHVmy9~j%>vsd{H(I4|zq=lwe&<3tI0Bc6dWN4bJ-hN^`X%?P+hW&D zA%Ggky-eBfV}ZSK=l&=Ky|iJy6-Gi+u+HHroU*E`2p7_1{%@}ofZ%#5j|S3%u3@Z$ zbc5~=g+lY-x3pp54I{}_9BMokS%kY+=;8>>Zxky9XMa2ri>Ba4kLtD!W4EnhzIS6n zaq2$lF!D(&)@$-MKq#XAmSY@h|y6&&XG~KGI z>cB7%qFte4@1S8py9f3=D7XuOVut%jFtMN(I7PvasaQaR!;d+}Bv|ClRBZ2~%Siaf zB8x65uvbxV4Hel7G?>z0@Z4U_-8IwPwidKm;*=@_)P%b0WXBQ8352qmPCibcJ{a7% z*!_wp8H;W-DC9~kxyYh9uhc*X@{MuVHZ`16v{DS$Yv6{Z;>wd5#<#U_rEor%3nN(w zgR13N*ptIdj;yakO_o9lZ^4wAPhLklXevQ7Pn1$<`Yv{JD<$0bHW&s*xG1r%*isCe zjrT4+%POna36rd!)=MMY-E##gFK-)w1ur74L2-md}Qe%i>9Me zyoi<*xU2-1b;2v_8;Vo+>(3%rzEY}69^B4R%n2Az_XZXN0}BI}S!DVt*rF7xXN9+k z1$z%QYeP54!dQ@$iX^@rpr*n|1{J5aCD9$)d$iaq0tsu_)#FVNf&d1y~hRltmUgg~UQvmTN)< zt$YiW6{qr)GA&CJV7Pw@s_TjRvk5AZMAL#r-auVVf}UQ@spKRF_Y4&~EW~^`h7%k*{-6`_@BZM`Tq)5Oa`rz0!cF++ z#^6d4tYEE#<(x?Nz56}#)Qb%v$@kte3S!cK8?B2S5==SY4?)jyf^I7)=$$epK<`r2g+5)>5A%Hhac1dKHNOm2f z>~mscFA8nvvShzR_KRfyQHfhDsr!t}T+BB1NsWEaukJSviH$>t)(WvQD0{EKm6p=0 z=t>KOaHR!8SkobVRpt}nB1_qjR5m19hvX8kSkn0O0kLBw+j3rNIsfA7e#<4X<fa1yvP+WOoYZ?a9TKQR>Skembp|wh~cI^#6yY!+fYn_y= zlcIIP~82^`dn@7IjW){0&4UMcM2Y9|oh+5rHn9S}*+ zkw0|czq#+fse;%6q;U#?G!6j>2nksS#9O=vWWzq$Ao_>0+ zW^r&L+czooP5#ET-!~=pO<}q3muv2#mgaFw^SC8^a)2D%13qirkgOY`b>kK3LSq$@ zD+bV5As$`%qmpm<`6J19IhWuh z-bjy7aO5$#xF&m1#xsH$28y@wAUQ`Vm{l{ zFExRR@XP*w)Y1%YZw8qEyR_GwxhPrt4y+^l))Ddibk=%HvfdK$d*tzfzI0b3-GzA& z6nk%G$tj7P63MBf(DXs*&VJ}lHgs1C-907-EyG`-AZzW~cEHfcp5}jH-zK*sJCz`h z6?F%${(V=!>?qqY?F{ZV0y8VRfSDCt+m<6=^MS8t-`BJE(F;cO^<;e$l5b+$Df`+E ze8GKRFzf4;e7)OFJOMk!;N_Ri`@sn@I3ZX052}XutA=G)<<8X|9@YmyT-^tVtNXSm zkNlkn{{DS`|MLqkCq;jM);}%zr?+p*{@{UsXdmW=e?;<+Y~S9#{mM^0jqlB7{iBk9 z6p|j_k5+fnQe*$~{^wVp_e+h#zh3-RRP@};dTvUdo7?uowvgC6o^6|u+9u%AU1>WG zvW}+ht73W6A*v5zl0S-`8-F<~okL;=d1%&Pu~Xw-d@T{e?HjM$Udi2>c>w3s7advm zHOYN#dqSp~GFSiHvVCip%sLuGN5fHh?J-ebI(*pB`1C`$!k;Psvhr^lUZk_xX1z|p+#XwF>DI{GC?zvzG=@f|oC_Z^KHd)Cn-IeJ7# z4@8cd#$L=wO;e)7FSmhqs&>NoLLj=D4x2imYx&dW*vxMpNMj#EgOCReLI@42@#fz{ zaR_(H;F@IUq7=NAb#?7nb~bjI%zcoI_JJ=4A?O4I ztw$}PFSzGJzeqhx{kla(G=>n3)j=ow>JIB0GnSuBKb`(LrxGI8p92E=`T?YMKy*=X z3$-KgvsJmeLkwK{HT$cyI2+Dh`L1;3yJA&DTwMd;ju>JKgir@U(bWRHsGts9?fb5F zv12sr8k1aOqH9cc)gQQ8_FZTflXVSAt|8Gi1d*@0f%C^9qN4i8jz4q!Y1xm;wx;mb zp=B5L4=wO!$ySMk^|tlMQF-8?_8rvTwCJF+juFW*vNegf9~V9w*!{z7U9VIJ+l>qA zn!j4UQRXdF} z45xnBMTb+9V@?}10z^L z;7i4vLnmbq;;)>r%jY^eY=R#_|7xU*fNgYf1b^<7vIc|dM{4t_l3);jrRxr6#0m3q=9*Ep~hnz>Cl z-Et4{sY5*tTi}emzRCVeM2ia>c)^go!X)EhCf-z!#WneJP*}a?9n!9mV;hCyJzb8C z5bXC6(uUKe`5cb@Lj>~{4EUxwHa%eT2sVi*ZTXt`DmplX;Jc=ROl)>w{}^y$GY7U} zs8)|Tl!wj^3Jf_qSBJw&*;RH0f1Y*+aScBG!rWsOM}(%sz)V~QOygO_hKo=p2gW>h z`77;~BZJIDjO9Krn= z81Z4=n@loMzJ>o}qD3^mGGX7se==d)!hbU16OFG-REqk0M6`;=_lW2gjjv2}h{jhY z21Mg46F0@PUzvDMG`>fKTh!mvy*MiC7xBAgKQ8{R$pm|&g7CWew%LR>!-gNam>?b7 zseN-z)?9PsY~B7)a<-X*TyZ4@cn2BT-GmB3@$=bUl zd)L;)k<+_9|LMXm`LsOi?3A3HTi0b*71H6dv*Oc_A?0v=I=wY{WGnl$M|OI)Kl+o8 zw&729xVLZYOzl3(Z0s#(9Yd01NYr21>HCw9w?CF0)!S1$%ex=^D4BV*w{ZgK_#8pj Risl+vIm&9bKgQbf{{iLR6lnke diff --git a/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc b/workers/annotations/cellposesam_train/__pycache__/entrypoint.cpython-311.pyc deleted file mode 100644 index e81e4d69d6d72c371c39d9e13edce70f1716bc96..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 18706 zcmc(HTWlLymRON2iWDDuQ4%RpkJ9^Lk$PE@CChfJW!aW3$r@X3x7F^bDOE|P&4;#% zvMr|Aqn+&uv;!~VS@dSq88^TMW;*J$8|wgRzzKE(e+0?quc$3ROzjE~vM~aGBv7Nb zk-$I6Ik$?BqV1lZ+1_lqba@}=o_p@O=bn4+y{G)IoK71B&tLuLncIo;6!o{5$UH?_ z|w{ELyJ4ZC5t6myfj?4SVrRJPcM z?UqcZS;u4NX9kM;0{+xzv6X90wq;WJzibKD&X%%ekIO%!p$7b^kAZq)jS8sY`mJhI z{?;|Be(M_5-%+C@)2sYnj=tTQNwaR&@Ka+=CP@`=uM7BVy4{^A=DKbl%ft&Hd!Wph zDO>bG%AZa7A=R5r^+KvIle%wUJ?d!k#&d$86k}?N=cBx0$LRF{9|=a*x{Qira%yH~{_@qSrL)sl6x;0Ovr{uy zG2TTh_O~Y>U(HggbBU$N3saMq<}XjrEnHQcXQ$3hT$@=?;gs^_jbMmf3P)Kkv=oeS zVPPfP9U#6O{xJOw3j+QNDwY*5nYbMMr&6-&gTlC1pT10cqk%MKeY%yzN z?T?CB2kT@@*iwKgW6RkJE#_h?wOAEfO=51=!`5iATGmTqb!JXAovBfMv2t>wuC`6uvCtBKQ}(G11^xXNFdC8PJ!SU{e6m= z46}fP#4s-h??w4L953_+B9Uk;5DP{lLhlL}3av*4P6&jTVtgPN>E$9Zeset05`@_OmlS3*7r3>!qi&?Xa&FtJ-4vvLb+a3O)&jBYUN zeDrRR<(STmbsz!Cd5-JqVFZkFJjVcKktic@>j6FxgSwls=z1t1gnJ_!DT3@MCmAha0=hWH4Cj}62EzI(ygEnjr`HuP8k9zjP} z$oTsL5!MHtUFY~%kP~{5-GXm5%KO5BJBUN8(~^-B9}yBLX=Y*pevONrYF=6n1tNEv zPtS0mwO~YeZoD^e8gv{T+g#@q^W8vbgHs%APyiNfUglYjSBeBK1SAB&NPuzXvg3&3 zhJ!2s@)!;(5kv+E$_fGG`fbfwdO&cSRd9h><)dMhw;Lb@z)?*N!)S0TcT`lJE9)rh zcL%x(urQq=ZY{8~$q<09SCK~f;93xv%S^2E!4NahH!!5q?YB+B$U@WVbRzpYifN)@F}0VrWnh(WG-#-CvvFrinYcQqP2bvuftd}L3Lux;#$G5AS7(`UL@0wZW^kZ#s31QaF>a6gKvSjN@=+^wI*>C zmZSGMA-;W;j6Y1M4Spp%_d10xGSoO732hxPra>NNO)#>)5o3fB};Qih=brM4cc8oV+~udmAwSS^o=MrQ(V2O^(39>|uWh zdO9z!;}i^px?Z4;kBi@%hS;qj|3j023pa>VU`>eo7jPZO(?*~mH8lX4ffz$%8;Ahy z99Z+ez>PrsuR#t45JFYO;KT!+tqJ5Bt~6>@9Gr{>QDqIr2b2#?@GwYeGy+1nN+u8+ z1q};BuF6Ru0(x4--9xZ`ClZ)pp$vFNidA)Ky8+65B@lttfkh?199RaC5pG2{LTsk3 ztZ?xhq1^uhS9GN)vJqb9__aU!pI;^a_kVhF`hWb(Pw@G7XV$*H^|#w!{h$ASI{wdq z2}}spAgV61G8Aeq{YsR#uq|^S=BTDo`-QOsIz%U#e$F?AX5Yfc_Fa&vNW5zLL}y;lPrZ;dEOk0mXOjEbNXtq@4Q1V`Bkf#Mc84c9c93ey?mt;+&S4Ce4??ei5FoW%;x0*o5w z;;h*x;6z~5hxcH;=}dqr;orkDwnZH{D`n?Uy0mKN+L6gn()u+8!QWx9b?&GrXBi$e z`{m||XYK#dmujAuo9Cq;U6GrwLKUzbp$>#l3Bs+}Z>>~G<4z*w^vO;iAhA(Zbw@_J z`l<<{^$ka*)+XB#g+V1%RiCc%9=KbNj3&He42FS;tgsSR1GE$ypBf?wgM##s&q2*tHL;gFj=6lob^D2@WL;?XIfkusI~N-?M))hpAk zN>Un4*EK*XHjBY!<7Gp6(Gi8=*7VV`kuux&>E=DUIXSf3m7Tk^0hFAy1fSu zH?EK0Y(oe~Jcf^8_mPB*S0C0Gr1blnB-%8uJo-J}D|SX2!FVhJ5jg{t-dOUgz`V`qi*_i=VU~q!ngR8LKh$r~K(m)%Y z>D26_d)o1spOorU7ETIsT!LFE?YN})ChSWRhJ;Zx{+UHIz+s8)D#cd;00}-f@zoGl z40ja6T>&@&1@z!!<(B=xptV&cx9KLxkgT!aL$n}<`_dx ztzH`Z-w(|Ip!r_y4KkfzAeun(_$Fiut`myw9Xv@Ap(PzRzcv|){|Q>75WQt!zCvk z-TK7$$d{r!Wx7+MI}a@mF#F8D1AEE##vhJ77~3)Y?DY2O1BW|Z_QBIOsq6!MuPsXN z-ITAruO-VqfM%S_*bIcw41|Z)>V0edp0)ndm1Ntl;a8oXgdc@d)*ji~BUyX?e>G#T zk?0!LIwUp+NC7@*ClKopWPwJqC+z`9+Z!RB1vu?* zge=Los91++`z7^hfo%kqV4l^*Iz{{IV8mirqH)irJ$SS6xJ=I%z=X?R7DC^@pBO~j z@4I5S^l6W1(?hTZ?E*~S65O5qhL+0pc%UVFK}&W$L|#ko3^g#)a5wXsmh!n%rN;w3 z&as{n3vBE}sgASNV(IHR8}sHUxQ#n#@{oHjx1PU=vRGrL9PY%*GV#~Kmd9E$P z+JRI3890iH;a!dPh}JmVwdwhqaOpT#6C27vL*2n#{C1XiY^@%HKJ{9@kLK{etsC69 z>3ISgq&!)lgUQ823h(9~GkW+$<#)n2y#ik)RwnT@{gp?2Rm#rffNnFHxL7IL*yhJA z1$0z@Z|TRKqB}AYdlPp1Yp}a@Bh#vz&+bC5vu&Ri>}x;{@-;xUzDC}>%-1z=>-m~_ znXhZw^>|?an%_zRUC;{A^a+pX5r#xNKd6I#RaX#bo{43p8+1h{+w~fn@^wYGo+jvd zzQi44dsv@-r1tooNEa}gcteO&D zu}1Xfa+0{Ic;yib=9+i$-6)4SP!E3%y3rY62ifC=qa)TN^K@#C zErsVw4~_48$+pj%XcWCS!A?mqiKav|w9+D$LjT`Qw2BqU)*OahOl%gLL`H1YJBTFO z0KP2;m5agMe%s?Ay{=8>Y4RMTp2|eK*z$(aEtpF%x;J8%b9i#@HDU`p{CGs)cA`TE z+m)#&cH|swSPOe1-z_ILmw|%SB)hUa9xcG!8M~2#6C3q&0=RmK@VqNgtD_+6w!@CW z$y+UZGAG@ICxCj4iS8KgJG3X~;v;tEdLnig(0EMjV#ni9N9+`jjl=JQ8PG93V*ML< z-Jx$2)|`enKz0^FJ~TWS5_K}4=n;E>4}YET#R3_+K~tTAe*1ole)~oL?d)286Lepq zH_?~qPYj41It#T=Pm!){KwtJ#qpq}9U#ctX*O&d&xSHp{l^7H|b-AgW(Ccg=IFHR+ zZF0E8A<^}CLf>*?Scma#-P&2N?-!g^PQJzh7|pf7o@J-l--P2F-~cU}@BAj3lX*5_ z4xc!DdpMI4>%|6km_03)hz`-oo@d8(HfSURS7?2T#g8w120mS1Wc|8`4j)OJNQ~Tkw<4m7wdTsep?neg`Exj4IG4!14wiN3JXl< z3b{qbQ=~PnYm5}(kqS5rtOnOM;5r!Y!Kc}tYyo=az#W|~ixXD(AUQf6Q+=P?KEr*&TljBVwr=u98f?I zf@=|UKGMPEvYF0*`tFS$X0eA^2!^>tSFS4h8t5uv1~}os)h1p*3o#xpP=)*o!4P%> zLhuR_g2+v#bK&yY%Ww)84R*b`CC%SyAv(s~ir!-)+`YU$37Ok}a754mIhG6&Iobnn zmJ1*f;F*UP(h3d!WgP!5Wn_=kO7!$9^rZwLJ z$ZLR8%BXrfO)gwjZywD74Nfur^WX>o7ugw?w15hLN0WFCS>W&EM<+lyw~xka052gL z@1ddj?5U@d=whRd1%gOdz10B$@k=7Kq$b)Mina{ z6&y#9-kdan`v4Y4{BiC(+hKmFW=7Q0+w3tfc3kH245i z`Stm#7aRC8-sSkf!71*o0Jzo_qX#2sn-F&nhP{}(zY+q! zz`I;L=br#`BwmRg7C2k-TK5yV%gRSy3HU>7L_qHPU{Lb2$=wT%V73wp)jjlv)N0g0 zjaOUBl#cvHd|KT?wCs1G{h@?oP4(vaFK6P_^L%s#oU#jsCC;4ZqZ{i5xyU-)->}lbN^jtEh2u zimll5h$4N^u@Ub=2pjzn(b!l~=y44<+}$I z8kZB&>F)Qj)7dcX8v!5cp5Z4ekIy{4_)Pfo%To7DK7bl0U0eW4EMS+Q9#X6v$W3f1 zdM9pIHQ@RAYhw&RnAX-elrr#AB?8H5p9SQtT4c`0t*QuV?Z=$eI-QJjBo51v$jf}|8*wrxEdfTS?yTawe^suyVWXPqlz_iM@~!wehJ2l{{Vxl z5Wt*;3NijViG!ywICbIe9lwCVH4O0Dj@+m5Z((c#0>z%EYL$AOKiEXp{?eSm%xD7% zN~I3!tZGd1aE?pyXf*S0Bi<_X*~K2{+{ILXFB8~UJh}n$Jc80|q3AM?N<^uF`@dYX z!rPk~_Gwlr0uwuPYUEoa3Vczlu1DHh{DJ!?5as(;`Qt!W`iF zz$Op7WlC{qYvuro1CLW$DO$#bO@OZ=?tt&VVgZ+eC7b}xj14f0j?PN4DxU(krkEly zGtdb~acvMW&@r)Vj7LSAXBHHxJAp#6Kyy*CErSuKes`mktM;Du6%GofHY;FtL53-1 zga%BhYofq^M0%hKJ^u&T>`e^LVs_q8XcA&W2~p#G0zoR4VQh0FI)j?powXhc3aT@$ z7vxu}#1!P(k&hagO5GYRlAuX4DDnkDsaR3{^7AqkM`qDny2EWMwVC=V!aP7hDb3t`K0wp4XUS z0B=mJLZfsk6|W(-{FOYVM9b176XXzd)_wJ76Mh2sNeX^8l?q+sODo`fuaxCcpym{V zwFfc+Ld@f_ct+C^t(Q@=Q2{>gQS7k%#+E`PA)sFcZcKl-hj_*)7H#R~WkB{z_!CB8 zqQJ7%N|}8U-SejRu)0pF_siA2TbIETvZ5(jCzt!TE~Y&lQs+t8GcM8K>DavQYTt9U zCqH}=OS#5n*Z9^<+TFhI?%s2EOUK5avMKkZ?4Fe9%7e<9o!;GExppj7c~Y)Cxix#> zEPL3P^vLDiDd#cSd2H+4fy=YgxZ5DtoJhGwW!LD|%yYU}vUTl@LjZ4z?w09piS9nY z?0%`S7eiZriXM>Z0f`WaK*J&{KCDT(9J%ym<4_#9Av2^(mxK@})fDrQt7)j=l z!3fP8m`S7!Lcjqbg5bc;e0o)C8cNj<%k{&MQs0zyR<_Pc z*4gJ4n`CcFS(;@_b26}J>5wcP83~?7fw@Ek<`NN@OGIEU6-w~8C6udLs_srz9Fr@K z?FRNLdbcj6?bSFP?iwWTF*Wj)NjiQu)i)*gP5q@|uWwrFnj>&A1s49Ci!E+53PLLQrkaIA)499w(8jHOzR%Prun%%mHRW2@jv zO+v|9oAz|?j>(?kCwFDf>1;xfJtw78m!L0ieaRV%T;ZT3eGhW$VDc zb#%`MqE27xwGs_v+?TbywuNE8DYaNA13&Y0uG= z^rakqvZGHTFYu_w`^cJXc(N>cMpK?q*)zKBd|vJO_|{HBZX9?r@MPl2fZRCz7t4Ph zl&UYJsxQdZ7q;yOZRjVKYC9{podpl6;&u|G98KF3Qfbox)`v0AZS(pkpUuksC_|8k zQ-H{j8h4ZL10dW!|J>!4U9HLYxaVk0+Eb1m+0i38dLVMxbn@wp+%zpY z8q#f`I!c_wB#_FQ4w^dE{gxW0PtN@1`|`>6fK=G90jUsz46E^G;yec7!xGSf{&Cs& zcB-uVq50v)4wt+J!`43d)nK}^1|Kc>gPcA;X z_-jGMBh~do{k{PJX&scx7$BjX*)MC~D{GfJMpI>Da@m+vHkK}{+b?U`D{GNjkEhCp zwUOH9iStg9{X zrs!6ghLxffKHG4*_8rWggGt_yCcrh8Nja{_jw@SJWNjY(ba3a#DQ~arg|&E8eN*(9 zOpi(Q*nv7t+SBE8zivrJQ_aJ2^RQGtr@p8?=8^RfLgz88rF;W_cKKglPFtN1iov=_ zS2d-p-06yHj8wOz%iZac^0cSsprYzeLaB-txuPZQYCWo@BHe0>G)KNJ}1-X zq|B?HN?4a*?+Pa2QhfYe=7VP@pg*Bh_n>f;2S)cBK6Zw)A^7hHveyW4yN`SVOfDME z;Z6!pu;J*YjskGRtP-YS~1=!%-A(pkeUo*ol zZhezSOHwH#r<|6(<=%VM)Cd z%G<3w5x4|{X$J_R=~Iqb2V1Y89KqYwsb>xXd+O|& zgnc$b3NOj@C5gV2E41&nJf)vqCxx>zJuA_(xx$j&tLdh;-L-7vdB!*p>8RMZ*X`Ns zcCJ2pHyKDSCzl`HOxe3-d-vAaL#KOt;lbh#{irnM?3A6ITj$bcmAJU1o#hYSgOsD} z!NskqLtDv%p0u-i`^L|1Y{S2_!?k_>;q=bk