Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ service keys.
|---|---|---|---|---|
| [Self-hosted product search in 5 min](./ecommerce-product-search) | Showing the fastest local product-search path with extraction, embeddings, and reranking | `extract`, `encode`, `score` | Local SIE Docker image, Python or TypeScript app | Runnable |
| [Find the best retrieval strategy for your RAG](./retrieval-ablation) | Picking a production RAG retrieval pipeline by evals on real financial documents | `encode`, `score` | SIE endpoint, Turbopuffer key, optional SIE API key for auth-enabled clusters | Runnable benchmark |
| [Rank exact primary-source passages](./rerank) | Testing a reranker on verbatim SEC, CMS, NTSB, and Supreme Court excerpts | `score` | SIE endpoint with Qwen3 Reranker; standalone `uv` project | Runnable verified example |
| [Extract custom entities from primary sources](./named-entity-extraction) | Changing zero-shot labels across financial, healthcare, rail-safety, and legal text | `extract` | SIE endpoint with GLiNER; standalone `uv` project | Runnable verified example |
| [Search licensed images with text](./multimodal-search) | Recomputing a six-image hard-negative ranking from full SigLIP vectors | `encode` | SIE endpoint with SigLIP; standalone `uv` project | Runnable verified example |
| [Find SOTA embedding models by MTEB task](./sie-hugging-face-mteb-semantic-search) | Searching ~14K HF embedding models ranked by task-specific MTEB scores | `encode`, `score` | Backend seed script plus Vite frontend; falls back without a live SIE endpoint | Runnable |
| [Private fine-tuned compliance RAG](./regulatory-rag) | Hot-loading a domain LoRA encoder and a custom token-pruning adapter on SIE | `encode`, `score`, `extract` | Custom SIE Docker image, GPU recommended | Advanced runnable example |
| [Build a multimodal wine recommender with OCR](./wine-recommender) | Combining preference-based retrieval with OCR-driven label detection in one UI | `encode`, `score`, `extract` | Docker Compose app plus local SIE endpoint; API key optional for unauthenticated SIE | Runnable demo |
Expand Down
3 changes: 3 additions & 0 deletions examples/multimodal-search/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.venv/
__pycache__/
run-output/
69 changes: 69 additions & 0 deletions examples/multimodal-search/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Search images with text

This example embeds the text query `a red leather handbag` and six photographs
with the same SigLIP model. It computes cosine similarity locally and returns
the image ranking.

The candidate set makes the result meaningful:

- a red leather handbag, the intended match
- red shoes, which match the color but not the object
- a black handbag, which matches the object but not the color
- a green backpack, black camera, and blue running shoe

Every photograph is real. Three are CC0 files; the red handbag has a
No Copyright, United States rights statement. The black handbag is CC BY 4.0,
and the red shoes are CC BY 3.0. Exact attribution, source links, dimensions,
byte lengths, and checksums are in `data/sources.json`.

## Run it

Start public SIE with `google/siglip-so400m-patch14-384` enabled, then install
the SDK:

```sh
uv sync
```

Run against a local server:

```sh
uv run python run.py --output run-output/results.json
```

Run against a hosted cluster:

```sh
SIE_BASE_URL=https://your-cluster.example \
SIE_API_KEY=your-key \
uv run python run.py --output run-output/results.json
```

The script verifies every image byte before making a request. It requires one
1,152-dimensional embedding for the query and each image, rejects non-finite
vectors, recomputes cosine similarity, and fails if the red leather handbag
does not rank first.

## Inspect the recorded run

`verified-run/raw/` contains all seven recorded vectors from public SIE. The
saved evaluation is derived from those vectors. It is never substituted for a
live response.

`verified-run/requests/` contains audit envelopes. They record the SDK inputs
plus local filenames, byte lengths, and hashes used for provenance. The SDK
encodes the image bytes in the SIE wire payload.

`verified-run/manifest.json` pins public SIE `v0.6.23`, commit
`9d6ca6b00f788b6ab19f8d6dc9506e1b31dad2f0`, model revision
`9fdffc58afc957d1a03a25b10dba0329ab15c2a3`, NVIDIA L4 hardware, input
checksums, and artifact checksums. The recorded query latency includes model
loading, so it is not a benchmark.

## Test offline

```sh
uv run python -m unittest discover -s tests -v
```

The tests recompute the six recorded cosine scores from the full vectors.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
100 changes: 100 additions & 0 deletions examples/multimodal-search/data/sources.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
{
"schema_version": "1.0",
"query": {
"text": "a red leather handbag",
"provenance": "authored retrieval query"
},
"synthetic_or_generated_images": false,
"images": [
{
"file": "images/black-camera.png",
"sha256": "cfb7740d0569f1c3cd2dc462ab2e75ff0b580782f35be7726baaa2c354724316",
"byte_length": 87916,
"width": 384,
"height": 384,
"format": "PNG",
"mode": "P",
"subject": "Black Nikon FE SLR camera",
"source": "https://www.flickr.com/photos/10456228@N00/7972305222",
"license": "CC0 1.0",
"license_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"creator": "Akairom",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
},
{
"file": "images/blue-running-sneaker.png",
"sha256": "0be82d701632164af8aabde05d04cf129976420044428bd16b5c3eb556fa41da",
"byte_length": 77302,
"width": 384,
"height": 384,
"format": "PNG",
"mode": "P",
"subject": "Blue Diadora running sneaker",
"source": "https://commons.wikimedia.org/w/index.php?curid=186982438",
"license": "CC0 1.0",
"license_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"creator": "WorldTravleerAndPhotoTaker",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
},
{
"file": "images/green-backpack.png",
"sha256": "0ccafd77f1dabbc20e871b01395f778af54d7ca2c74c933cca4f6db1b9e42a8f",
"byte_length": 85291,
"width": 384,
"height": 384,
"format": "PNG",
"mode": "P",
"subject": "Olive-green backpack",
"source": "https://stocksnap.io/photo/bag-backpack-THRCTPZSJP",
"license": "CC0 1.0",
"license_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"creator": "Jakob Owens",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
},
{
"file": "images/red-leather-handbag.png",
"sha256": "535c76fbb350e171b3a150b96661954010527bb562f9c22230a120ab5c94893f",
"byte_length": 59394,
"width": 384,
"height": 384,
"format": "PNG",
"mode": "P",
"subject": "Red leather barrel purse",
"source": "https://commons.wikimedia.org/wiki/File:Red_Leather_Barrel-Shaped_Purse_Used_by_Bonnie_Blake_-_DPLA_-_e849136953ed7fb210ad4532b5fef93e_(page_1).jpg",
"license": "No Copyright, United States",
"license_url": "https://rightsstatements.org/page/NoC-US/1.0/",
"creator": "Missouri Historical Society via DPLA",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
},
{
"file": "images/black-handbag.jpg",
"sha256": "75cf60b0cc0c43b2d916a75cf0747ac0a15ecc42b6d2eccbd3c4de7b1e809632",
"byte_length": 369633,
"width": 960,
"height": 1280,
"format": "JPEG",
"mode": "RGB",
"subject": "Black handbag",
"source": "https://commons.wikimedia.org/wiki/File:Black_handbag.jpg",
"license": "CC BY 4.0",
"license_url": "https://creativecommons.org/licenses/by/4.0/",
"creator": "Tahrirchiqiz",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
},
{
"file": "images/red-shoes.jpg",
"sha256": "9d8819824d27486352afd83f5d1fe9e443a621747ba51a27b49a6321b96fa898",
"byte_length": 136109,
"width": 960,
"height": 640,
"format": "JPEG",
"mode": "RGB",
"subject": "Red shoes",
"source": "https://commons.wikimedia.org/wiki/File:Red_Shoes_(34075396).jpeg",
"license": "CC BY 3.0",
"license_url": "https://creativecommons.org/licenses/by/3.0/",
"creator": "Benjamin Caviezel",
"derivative": "Exact evaluated bytes. Source image was resized or cropped for this example without semantic edits."
}
]
}
20 changes: 20 additions & 0 deletions examples/multimodal-search/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[project]
name = "superlinked-sie-multimodal-search-example"
version = "0.1.0"
description = "Search licensed images with text and public SIE"
requires-python = ">=3.12,<3.13"
dependencies = [
"sie-sdk>=0.6.19,<0.7.0",
]

[tool.uv]
package = false

[dependency-groups]
dev = [
"ruff>=0.14",
]

[tool.ruff]
line-length = 120
target-version = "py312"
Loading