feat(kv): add native key-value store to hyperdb-api by StefanSteiner · Pull Request #3 · StefanSteiner/hyper-api-rust

StefanSteiner · 2026-07-09T07:52:38Z

Summary

Adds a string-native key-value store to hyperdb-api (Milestone 1 of the KV
feature; the MCP surfacing is a separate follow-up). Connection::kv_store(name)
and AsyncConnection::kv_store(name) return a KvStore / AsyncKvStore handle
scoped to a store name, all namespaced within a single fixed backing table
_hyperdb_kv_store(store_name, key, value).

API surface

Per handle (sync + async twins):

get / set — plain-string values (upsert)
get_as::<T> / set_as::<T> — any serde-(de)serializable type, stored as JSON
delete, exists, size, keys (ordered), clear
pop — destructive get-of-lowest-key, in one transaction
set_batch — write many entries atomically in one transaction

Plus Connection::kv_list_stores() / AsyncConnection::kv_list_stores() — a
cross-store discovery op returning the names of all non-empty stores.

New error variant: Error::Serialization (JSON round-trip failures in
get_as/set_as).

Design notes

No PRIMARY KEY on the backing table. Hyper rejects PRIMARY KEY at
CREATE TABLE (0A000: Index support is disabled, verified live). Uniqueness
is enforced application-side by an UPDATE-then-conditional-INSERT … SELECT … WHERE NOT EXISTS upsert — the same idiom this repo already uses for
_table_catalog. hyperd serializes statements per connection, so no duplicate
rows result.
Parameterized throughout. store_name and key/value are always bound
as $N params via the real extended-query protocol (Parse/Bind/Execute); only
the bare table name is interpolated (trusted, construction-time).
All DDL flows through one kv_create_table_sql() helper so the sync/async
twins can't diverge.

Docs

hyperdb-api/README.md — overview bullet + a Key-Value Store section
Crate-level rustdoc — KV types in the lifetime-safety hierarchy (with a
compile_fail borrow test) and the "Key Types" list
hyperdb-api/CHANGELOG.md — ### Added entry
hyperdb-api/DEVELOPMENT.md — internals
Design spec + implementation plan under docs/superpowers/

Testing

cargo fmt --check — clean
cargo clippy -p hyperdb-api --all-targets -- -D warnings — clean
RUSTDOCFLAGS="-D warnings" cargo doc -p hyperdb-api --no-deps — 0 warnings
(also fixes pre-existing broken intra-doc links in the pool module docs)
make test-api (core + api) — 918 passed / 0 failed against a real hyperd
(14 dedicated KV tests: 12 sync + 4 async, covering upsert, serde round-trip,
ordering, destructive pop, empty-store pop, batch atomicity, per-method
invalid-name rejection, and store isolation)

Notes for reviewers

This branch went through the full plan → adversarial-plan-review → execute →
final-adversarial-sweep loop. The final sweep's 6 confirmed findings (all
test-coverage / doc polish; no correctness or security defects) are folded into
the last two commits. hyperdb-mcp is intentionally untouched — surfacing
the store through MCP tools is Milestone 2, a separate PR.

…runnng tests

Single-table KvStore abstraction over _hyperdb_kv_store, sync + async twins, serde get_as/set_as, upsert emulation (Hyper has no ON CONFLICT). Covers both milestones; M1 (feat) planned next, M2 (fix) later.

…, test gate)

Reconcile findings from both plan reviewers (Phase 2/3): - Drop PRIMARY KEY from all backing-table DDL (empirically rejected by Hyper: 0A000 Index support is disabled). Single kv_create_table_sql helper is the sole DDL source for both twins + both kv_list_stores guards. - Replace invented Connection::query_count with execute_query + scalar. - Uniform .as_str() on all bound params across sync + async twins. - Benchmark: #[expect]->#[allow(cast_precision_loss)] to avoid unfulfilled_lint_expectations; drop unused mod common.

Scaffolds the KV module: KV_TABLE, KV_MAX_NAME_BYTES, KV_CHARSET consts, validate_kv_name(), and the kv_create_table_sql() DDL helper (no PRIMARY KEY; Hyper rejects one). These are consumed by KvStore in the next commit; the transient dead_code warnings resolve there.

KvStore<'conn> borrows &Connection (mirrors Catalog/Inserter). open() creates the PK-less backing table via kv_create_table_sql; new() uses the default location, with_target() is the M2 seam. Connection gains kv_store() and kv_list_stores(). PK-probe test empirically confirms (pinned hyperd): a raw duplicate INSERT is ACCEPTED (1 row) because the table has no PRIMARY KEY -- which is exactly why set() must use the app-side conditional-INSERT upsert.

set() is an UPDATE-then-conditional-INSERT upsert (distinct $4/$5 placeholders, extended-query-safe); get_as/set_as JSON-(de)serialize via serde_json, mapping failures to Error::Serialization. Empirically verified against real hyperd (7 integration tests pass; upsert overwrite confirmed). Closes the per-iteration reviewer's transient doctest finding: the KvStore struct example calling set/get now compiles (cargo test --doc green).

delete returns whether a row was removed; size uses COUNT(*) scalar (always one row); keys streams ORDER BY key ASC; clear removes only this store's rows from the shared table. Verified against real hyperd, incl. same-key/different-store isolation and empty kv_list_stores on fresh db.

pop peeks the lowest-ordered key and deletes it in one transaction (first_row consumes the Rowset, releasing its statement guard before the DELETE). set_batch validates every key up front, then upserts all pairs in one transaction; both roll back on error preserving the original error.

Add a Key-Value Store README section, CHANGELOG Added entry, and a DEVELOPMENT design note. Convert public-doc intra-doc links to the crate-private KV_TABLE / kv_create_table_sql into inline code (they tripped rustdoc's private_intra_doc_links lint). Also correct the stale Parameterized Queries note to reflect the real extended-query protocol (prepare_typed + binary params), not escaped literals.

The `pool` module's `//!` header links (`[`Pool`]`, `[`create_pool`], `[`PoolConfig`]`, `[`ConnectionPool`]`, `[`SyncPoolConfig`]`, `[`RecycleStrategy`]`, `[`SyncRecycleStrategy`]`, and their methods) all failed to resolve under `RUSTDOCFLAGS="-D warnings"` with "no item named X in scope" — 9 errors that broke `cargo doc` on the crate. Root cause: `pub mod pool;` (and `pub mod copy;`) each carried a redundant *outer* `///` doc line in lib.rs *in addition to* the module file's own *inner* `//!` header. When a module declaration has both, rustdoc merges the two doc strings and resolves the whole block in the parent (crate-root) scope. In crate-root scope, extern crates (`deadpool`) and root re-exports resolve, but items reachable only as `crate::pool::*` do not — hence the failures. `pool` was the only module whose inner `//!` links pointed at module-only items, so it alone surfaced the break; `copy` had the same latent misconfiguration without triggering it. Fix: drop both redundant outer `///` lines. Each module's `//!` header already carries the same one-line summary (it renders identically on the crate index), and every other module in lib.rs already uses inner-only docs. Restores pure module-scope link resolution. `cargo doc -p hyperdb-api --no-deps` now builds clean with `RUSTDOCFLAGS="-D warnings"` (0 warnings); fmt + clippy unaffected.

Adversarial Phase-5 sweep found the key-validation tests only exercised set/get, and the async suite lacked isolation + empty-pop coverage. - sync: validates_key_on_every_entry_point now asserts InvalidName from set/get/delete/exists/get_as/set_as (was set+get only) - async: add async_validates_key_on_every_entry_point mirroring sync - async: add async_store_isolation (same key, two stores, distinct values) - async: tighten clear() assertion to == 2 and assert empty-store pop == None

- lib.rs: add KvStore/AsyncKvStore to the crate-level "Key Types" list (they were already in the lifetime-safety hierarchy but missing here) - async_kv_store: add the `&str`-coercion bind comment present in the sync KvStore::get twin, so the async path documents the same rationale

StefanSteiner · 2026-07-09T15:25:40Z

Superseded by the upstream PR against tableau/hyper-api-rust (which has the CI runners): tableau#182. Closing this fork-targeted duplicate.

StefanSteiner added 19 commits July 8, 2026 14:15

chore(docs): Add more guidance in AGENTS.md for accessing hyperd and …

099c5dd

…runnng tests

docs: add design spec for native key-value store (M1 API + M2 MCP)

efc0e0d

Single-table KvStore abstraction over _hyperdb_kv_store, sync + async twins, serde get_as/set_as, upsert emulation (Hyper has no ON CONFLICT). Covers both milestones; M1 (feat) planned next, M2 (fix) later.

docs: add guidelines-compliance + Harness review sections to KV spec

07dcfa8

docs: fold DEVELOPMENT.md facts into KV spec (param facade, lifetimes…

2cf8c7b

…, test gate)

docs(kv): add M1 (Rust API) implementation plan

98cd06f

feat(kv): add Error::Serialization variant for get_as/set_as

94948e2

test(kv): add compile-fail lifetime doc test for KvStore

f7080bb

feat(kv): add AsyncKvStore async twin

c32a308

feat(kv): add KV write benchmark (single vs batched commit)

f62ebb7

StefanSteiner closed this Jul 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kv): add native key-value store to hyperdb-api#3

feat(kv): add native key-value store to hyperdb-api#3
StefanSteiner wants to merge 19 commits into
mainfrom
feat/add-native-kv-support

StefanSteiner commented Jul 9, 2026

Uh oh!

StefanSteiner commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

StefanSteiner commented Jul 9, 2026

Summary

API surface

Design notes

Docs

Testing

Notes for reviewers

Uh oh!

StefanSteiner commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant