SAM3 v0 by JonasWurst · Pull Request #3 · lightly-ai/lightly-studio-plugins

JonasWurst · 2026-04-28T13:59:20Z

What has changed and why?

This PR add SAM3 support as plugin.

To test it, run

uv pip install "git+https://github.com/lightly-ai/lightly-studio-plugins.git@jonas-add-sam3-plugins#subdirectory=plugins/sam3_segmentation"

in the env that you are running lightlyStudio in. This will automatically register the plugin.

You may also require access to sam3. for this follow the steps in the README

How has it been tested?

MacBook Pro M4. Successfully able to segment with SAM

Did you update Readme.md and plugins.toml?

Yes
Not needed

Summary by CodeRabbit

New Features
- Added SAM3 Segmentation plugin for prompt-driven instance segmentation of image collections; configurable model, prompt, confidence threshold, and optional GPU acceleration
- Packaged for independent pip installation (includes HuggingFace model requirement)
Documentation
- Updated main README and added plugin README with install, authentication, parameters, and GPU setup
Chores
- Added license and packaging/installation metadata

_{⚠️ Note: Feature available only on Pro plans or higher.}

coderabbitai · 2026-04-28T13:59:31Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a new SAM3 Segmentation plugin: a pip-installable Lightly Studio operator that runs HuggingFace SAM3 prompt-driven instance segmentation, registers the plugin, includes packaging, docs, license, and implements image inference and annotation creation.

Changes

SAM3 Segmentation plugin

Layer / File(s)	Summary
Module imports, helper, and operator metadata `plugins/sam3_segmentation/src/lightly_plugins_sam3_segmentation/operator.py`	Adds module imports/constants, `_get_or_create_label` helper, and defines `SAM3SegmentationOperator` dataclass with parameters and supported scopes.
Model loading and caching `plugins/sam3_segmentation/src/lightly_plugins_sam3_segmentation/operator.py`	Implements `_load_model` with caching, device placement, and processor/model loading for the operator runtime.
Execution loop: parameter parsing, collection resolution, inference, post-processing, annotations `plugins/sam3_segmentation/src/lightly_plugins_sam3_segmentation/operator.py`, `plugins/sam3_segmentation/src/lightly_plugins_sam3_segmentation/utils.py`	`execute` parses/validates parameters, selects device, resolves collection and filters, iterates samples, runs SAM3 inference, post-processes boxes/masks/scores into RLE and `AnnotationCreate` objects, and bulk-creates annotations or returns a no-op result.
Registry & Docs `README.md`, `plugins.toml`	Adds plugin entry to README and registers `lightly_plugins_sam3_segmentation` in `plugins.toml`.
Package metadata & build `plugins/sam3_segmentation/pyproject.toml`, `plugins/sam3_segmentation/Makefile`, `plugins/sam3_segmentation/LICENSE`, `plugins/sam3_segmentation/README.md`	Adds plugin package `pyproject.toml` with entry point and dependencies, Makefile targets, Apache-2.0 license, and plugin README with setup/usage instructions.

Sequence Diagram(s)

sequenceDiagram
    participant Ctx as ExecutionContext
    participant Op as SAM3SegmentationOperator
    participant Sam as Sam3Model/Processor
    participant IS as Image Store
    participant AR as AnnotationResolver

    Ctx->>Op: execute(session, context, parameters)
    Op->>Op: resolve collection, params, device
    Op->>Sam: load model & processor on device
    Op->>IS: iterate samples
    loop per sample
        IS->>Op: provide image bytes
        Op->>Op: load RGB image, compute target size
        Op->>Sam: run processor & model -> boxes, masks, scores
        Sam->>Op: return detections
        Op->>Op: prepare_segmentation_entries (clamp boxes, masks->RLE)
        Op->>Op: accumulate AnnotationCreate objects
    end
    Op->>AR: create_many(annotations)
    AR->>Op: return created count
    Op->>Ctx: return OperatorResult(success, count)

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

IgorSusmelj · 2026-04-28T14:05:01Z

Does not work for me:

INFO:     Waiting for application startup.
2026-04-28 16:03:59,638 WARNING: Failed to load plugin 'sam3_segmentation' from lightly_plugins_sam3_segmentation.operator:SAM3SegmentationOperator
Traceback (most recent call last):
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/src/lightly_studio/plugins/operator_registry.py", line 110, in discover_plugins
    operator_class = ep.load()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/lightly_plugins_sam3_segmentation/operator.py", line 14, in <module>
    from transformers.models.sam3.modeling_sam3 import Sam3Model
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/transformers/__init__.py", line 27, in <module>
    from . import dependency_versions_check
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 57, in <module>
    require_version_core(deps[pkg])
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/transformers/utils/versions.py", line 117, in require_version_core
    return require_version(requirement, hint)
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/transformers/utils/versions.py", line 111, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
  File "/Users/igorsusmelj/GitProjects/lightly-studio/lightly_studio/.venv/lib/python3.9/site-packages/transformers/utils/versions.py", line 44, in _compare_versions
    raise ImportError(
ImportError: huggingface-hub>=0.34.0,<1.0 is required for a normal functioning of this module, but found huggingface-hub==1.7.1.
Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main

Will check for access and the hub

…d-sam3-plugins

MalteEbner

Preliminary review.

MalteEbner

Also asked codex for a review, 4 out of its 5 points are valid, posted them here.

MalteEbner

My previous comments have all been addressed, thanks :) Found some new points and mypy is failing.

MalteEbner

LGTM^2

v0

759500c

update deps

5168178

LeonardoRosaa reviewed May 8, 2026

View reviewed changes

Comment thread plugins/sam3_segmentation/src/lightly_plugins_sam3_segmentation/operator.py Outdated

JonasWurst added 5 commits May 15, 2026 10:13

Collection Name

c7fa744

fix tests

9670536

Merge commit '0dd6ec963b31cabc9a17c0f947c73cbaec6354bf' into jonas-ad…

e590975

…d-sam3-plugins

Merge commit 'a290db4f3d9d94a40d304e5f3d543fa7ae2e8fe6' into jonas-ad…

35789ad

…d-sam3-plugins

Merge commit 'f2bc6b8658aae8d3fd8e3f27288011a663502226' into jonas-ad…

eb1e5f3

…d-sam3-plugins

MalteEbner reviewed May 15, 2026

View reviewed changes

Comment thread plugins/sam3_segmentation/README.md Outdated

Comment thread plugins/sam3_segmentation/README.md Outdated

Comment thread plugins/sam3_segmentation/README.md Outdated

MalteEbner reviewed May 15, 2026

View reviewed changes

Review Changes

93ea21d

MalteEbner reviewed May 18, 2026

View reviewed changes

JonasWurst added 3 commits May 18, 2026 10:08

Fix review

9a7c838

mypy

28a88cc

Fixing issues

1f6c1a1

MalteEbner approved these changes May 18, 2026

View reviewed changes

JonasWurst merged commit ef7b0f7 into main May 18, 2026
5 checks passed

JonasWurst deleted the jonas-add-sam3-plugins branch May 18, 2026 12:08

Conversation

JonasWurst commented Apr 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What has changed and why?

How has it been tested?

Did you update Readme.md and plugins.toml?

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

IgorSusmelj commented Apr 28, 2026

Uh oh!

Uh oh!

MalteEbner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MalteEbner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MalteEbner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MalteEbner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JonasWurst commented Apr 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 28, 2026 •

edited

Loading