Skip to content

Feature/qdrant ltm foundation#5

Merged
matstech merged 9 commits into
mainfrom
feature/qdrant-ltm-foundation
Jul 18, 2026
Merged

Feature/qdrant ltm foundation#5
matstech merged 9 commits into
mainfrom
feature/qdrant-ltm-foundation

Conversation

@matstech

@matstech matstech commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Qdrant LTM Backend

Summary

This PR adds Qdrant as an optional long-term memory vector backend alongside the existing Chroma implementation.

The main goal is to make DMF backend-neutral at the LTM boundary without changing the default runtime behavior: Chroma remains the default, while Qdrant can be enabled explicitly with storage_type = "qdrant" and qdrant_mode = "memory".

Why

DMF previously coupled long-term memory persistence and vector search to Chroma-specific APIs. That made adding another vector store risky because filtering, raw-record codecs, card lookup, and hook construction were spread across backend-specific assumptions.

This implementation introduces Qdrant through the same public LTM contract used by Chroma, keeping raw records canonical and preserving the existing card-to-raw resolution semantics. Qdrant is intentionally limited to Local Mode in this PR so the integration can be validated without introducing server lifecycle, auth, or persistence concerns.

What Changed

  • Added backend-neutral LTM primitives:

    • shared raw/card payload codecs;
    • shared vector hit types;
    • RecallFilter for role, record id, interaction range, timestamp range, and card kind filtering;
    • runtime factory wiring for Chroma, Qdrant, File, and Null hooks.
  • Added Qdrant Local Mode:

    • optional qdrant-client dependency via the qdrant extra;
    • in-memory QdrantClient(":memory:") connection mode;
    • deterministic UUIDv5 point IDs for raw records and cards;
    • COSINE scoring mapped back to DMF distance/threshold semantics;
    • raw search, card search, count, read-all, clear, and collection validation.
  • Preserved existing Chroma behavior:

    • default config remains Chroma embedded;
    • legacy import shims remain available;
    • Chroma card filtering still resolves card hits through source raw records.
  • Extended benchmark tooling:

    • benchmark runner can select --backend chroma|qdrant;
    • Qdrant benchmark does not require Docker;
    • final console summary is computed from the produced JSON report;
    • summary now separates completed-case retrieval rate from overall dataset retrieval rate;
    • Ollama requests disable model thinking to avoid empty message.content with reasoning models.
  • Updated docs for Qdrant install, config, Local Mode limits, raw/card collections, score conversion, filters, errors, and API references.

Test Coverage

  • Unit coverage for Qdrant client setup, optional dependency behavior, filters, codecs, hook operations, cards, malformed payloads, collection validation, and concurrency basics.
  • Backend contract tests comparing Chroma and Qdrant behavior for raw archival, recall, thresholding, count/read/clear, and framework-level recall flows.
  • Regression coverage for Chroma, File, Null, factory wiring, legacy imports, config parsing, benchmark utilities, and docs build.

Validation run:

poetry check --lock
make compile
make test
make docs-build
make build
git diff --check
make chroma-up
make test-integration
make chroma-down

Results recorded during final gate:

  • make test: 745 passed, 1 deselected
  • Chroma integration test: 1 passed
  • docs and wheel build: passed

@matstech matstech self-assigned this Jul 13, 2026
@matstech matstech added the wip label Jul 13, 2026
@matstech
matstech marked this pull request as ready for review July 13, 2026 20:58
@matstech
matstech requested a review from ezimuel July 14, 2026 07:47
@matstech matstech removed the wip label Jul 14, 2026
@matstech
matstech enabled auto-merge (rebase) July 17, 2026 20:02

@ezimuel ezimuel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Comment thread dmf/memory/ltm_hooks/qdrant_hook.py
@matstech
matstech merged commit 455a6c3 into main Jul 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants