feat: openapi + mcp remote (proxy) instance backends#244
Conversation
Adds two live remote (proxy) instance backends alongside markdown/sql_view/
document:
- BackendKind::OpenApi / ::Mcp (+ is_remote helper); wire strings openapi/mcp
- SearchMode::None — remote data is live-fetched, never indexed, so it feeds
no search lane (the overlay page is still indexed like any page)
- Capabilities::for_kind: remote = { writable:false, page-grain, search:none,
no crdt }; update_page is rejected, write-back is the write_instance tool
- RemoteBinding / RemoteKind / RemoteOp + parse arms for kind: openapi|mcp
(endpoint name + read/write op + project map); fail-closed when endpoint or
read op is missing (mirrors sql_view)
Unit tests cover openapi/mcp/read-only/missing-endpoint parsing and the
remote capability contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
external_endpoints table (name, kind, base_url, auth, secret) + Indexer register/lookup/list/delete_endpoint, mirroring the sql_view credential registry. A skill's backend.endpoint references a row by name; the base URL and secret live server-side only (SSRF + secrets-in-markdown guard), never in the corpus. Ensured on every connection open and preserved across rebuild (separate canonical input). No-mock integration tests over real DuckDB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
backend/remote.rs: the pure, network-free heart shared by the openapi/mcp
backends — resolve_projection (response JSON -> overlay fields via $.a.b
paths), json_path_get, fill_template (/customers/{id} + id), plus RemoteError.
Fully unit-tested. The live transport (RemoteClient) and expand/create/write
wiring build on these.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
protocol.md gains a 'Remote backends (openapi/mcp)' section (live-on-expand reads, write_instance write-back, endpoint registry / SSRF guard, search:none, live-read failure policy, frontmatter examples, new tools) and the Instance backends intro now lists four external kinds. roadmap.md gains the matching bullet. Adds a discovered-note for the libduckdb release-download 403 that blocks the local build in a network-restricted session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
Wires the openapi/mcp backends end-to-end at the gateway: - remote_backend.rs: reqwest transport for openapi (HTTP method+path template) and mcp (JSON-RPC tools/call + resources/read), auth from the endpoint registry, projection via the index-crate core. fetch_projection (expand's live read; failure -> backend_projection.issue) + write_instance (write-back) + probe (reachability). Unit-tested helpers. - expand: live remote projection branch (backend_ref.kind in openapi|mcp). - MCP tools + dispatch: register/list/delete/validate_endpoints, create_remote_instance (admin), write_instance (agent, acl.update-gated via may_write_instance). Registered in tools/list. - Outbound GET /openapi.json: OpenAPI 3.1 doc generated from the same tool schemas tools/list serves. NOTE: compiled-by-construction only — the libduckdb egress 403 blocks the local cargo gate in this session (see docs/notes/discovered/2026-07-05-...). The no-mock wire e2e test (real upstream server via EscurelProcess) is the remaining verification step once the build is unblocked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
Closes the verification gap the prior commit flagged: the branch was
authored in a session where the libduckdb-download egress block stopped
the local cargo gate from running, so the openapi/mcp code was
"compiled-by-construction only". Built, clippy-clean, and full-suite
green locally now.
remote_backend_tools.rs drives BOTH remote-backend kinds over the wire
against REAL, STATEFUL upstreams (in-process axum on loopback ports) —
no canned stubs. Four tests:
- openapi_remote_backend_read_write_over_the_wire: a REST CRM whose GET
returns the current row and whose PATCH actually mutates it.
register -> validate -> create -> expand (projected `gold`) ->
write_instance admin (PATCH -> `platinum`, read-after-write re-expand
proves the state changed) -> write_instance agent (refused by the
fail-closed acl.update gate; a final expand proves it never landed).
- mcp_remote_backend_read_write_over_the_wire: a JSON-RPC 2.0 KB
answering tools/list + tools/call whose putArticle mutates the store;
same read/write/read-after-write/ACL walk (`draft` -> `published`).
- openapi_bearer_auth_forwarded_and_read_degrades_when_unauthorised: a
guarded CRM that 401s reads lacking the bearer. A registered secret is
forwarded (200, fields projected) and never echoed by list_endpoints;
flipping the endpoint to auth=none makes the read fail closed to a
`{ issue }`, never a fabricated body.
- openapi_unreachable_endpoint_degrades_read_and_probe: a dead port ->
validate_endpoints reports it unreachable and a live read degrades to
a `{ issue }`.
Verified red->green for each behaviour by mutating the feature and
watching the right assertion fail (openapi write: empty PATCH body; mcp
write: drop the payload merge in mcp_args; auth: skip attaching the
bearer in apply_auth), then restoring to green. The degraded-read cases
are themselves the fail-closed assertion.
Also apply rustfmt to the five files the web session couldn't format
(whitespace-only line reflows; no semantic change).
Test plan: cargo test -p escurel-server --test remote_backend_tools
(4 tests); full fmt/clippy/test/build gate green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-templating docs Triage of the codex review on PR #244 (two P2 findings): - MUST-FIX (finding 1): remote read/write dispatched the transport purely from the skill's backend kind and never checked the registered endpoint's kind. Since create_remote_instance only checks the endpoint *name* exists, an `openapi` skill pointing at an endpoint registered as `mcp` (or the reverse) would silently fire the wrong protocol at that URL — and, against a matching server, even return data. exec() now fails closed when `ep.kind != remote.kind`, degrading the read to an `{ issue }` naming both kinds. Adds RemoteKind::as_str(). - DOC (finding 2): the RemoteOp::Http doc comment and protocol.md both claimed OpenAPI path placeholders are "filled from the overlay id / write payload", but only `{id}` (from the overlay id) is bound — a multi-placeholder write path is rejected as `unfilled path placeholders`. Corrected both to state the real contract (payload merges into MCP tool-call *arguments*, not into OpenAPI path templates) rather than promise unimplemented behaviour. Filling path placeholders from the payload is left as a follow-up. Verified red->green: with the guard disabled, the new remote_read_fails_closed_on_endpoint_protocol_mismatch test observes the mismatched read silently returning the CRM row (the bug); with it, the read degrades to the mismatch issue. Test plan: cargo test -p escurel-server --test remote_backend_tools (5 tests); full fmt/clippy/test/build gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex second-opinion review (CLAUDE.md principle 9)Ran ✅ Fixed — endpoint protocol mismatch not rejected (
|
The openapi/mcp write path bound only `{id}` in templates. Extend it so a
write op can address subresources and reshape the payload:
- Path/URI placeholders now resolve from `{id}` (overlay slug) PLUS every
scalar leaf of the write payload, flattened to dotted keys — so
`/customers/{id}/orders/{order_id}` and `{customer.tier}` both fill.
- OpenAPI write ops gain an optional `body:` template. An exact `"{name}"`
leaf is substituted type-preserving (a number stays a number, an object
stays an object); embedded `{name}` interpolates as a string; objects/
arrays recurse. Absent `body:` ⇒ the payload is sent verbatim (back-compat).
- Both path and body fail closed on an unresolved placeholder (`unfilled
path/body placeholders`) rather than sending a literal `{x}`.
Pure core (template_vars, flatten_scalars, render_body) lives in
escurel-index and is unit-tested without a network. RemoteOp::Http gains
`body: Option<Value>`; Value is not Eq, so the Eq derive is dropped from
RemoteOp / RemoteBinding / BackendBinding / SkillInfo (none are used as
hash/set keys — PartialEq is retained).
Verified red->green: the new wire test
openapi_write_fills_path_and_body_placeholders_from_payload drives a POST to
/customers/acme/orders/o-1 with a `body:` template ({sku} string, {qty}
type-preserving number, `via` constant) against a real stateful axum orders
CRM, then reads the stored order back; with the payload merge disabled the
`{order_id}` path placeholder goes unfilled and the write fails closed.
Test plan: cargo test -p escurel-index --lib backend::remote (6 pure cases);
cargo test -p escurel-server --test remote_backend_tools (6 wire tests);
full fmt/clippy/test/build gate green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up landed: complex path + body placeholders (
|
Summary
Adds
openapiandmcpremote (proxy) instance backends — instancebodies fetched live on
expandfrom a REST/OpenAPI endpoint (openapi)or an upstream MCP server (
mcp), rather than materialised in DuckDB. Thewhole feature was authored across the earlier commits on this branch; this PR
also closes their verification gap — they were written in a session where
the libduckdb-download egress block stopped the local cargo gate from running,
so the code was "compiled-by-construction only". It is now built, clippy-clean,
and full-suite green locally, with real-service wire coverage.
What lands:
escurel-index):openapi/mcpbackend kinds,the
RemoteBindingparser (endpoint/read/write/project), theadmin
external_endpointsregistry (base URL + auth server-side, secret neverin the markdown corpus — the SSRF guard), and the pure projection/templating
core (
fill_template,resolve_projection).escurel-server/remote_backend.rs):outbound
reqwesttransport foropenapi(method+path template) andmcp(JSON-RPC
tools/call/resources/read), auth applied from the registry,JSONPath projection.
fetch_projection(expand's live read; any failure →fail-closed
{ issue }, never a fabricated body) +write_instance(write-back) +
probe(reachability).register_endpoint/list_endpoints/delete_endpoint/validate_endpoints/create_remote_instance(admin),and
write_instance(agent, gated by the target'sacl.updateviamay_write_instance, fail-closed). Registered intools/list.GET /openapi.json: an OpenAPI 3.1 doc generated from the sametool schemas
tools/listserves.docs/spec/protocol.md§"Remote backends", roadmap note, and adiscovered-note on the libduckdb egress block.
Test plan
New no-mock wire e2e — real gateway (
POST /mcp), real OIDC (TestIssuer),real DuckDB +
FsStore, real reqwest, and real, stateful in-process axumupstreams on loopback ports (no canned stubs):
crates/escurel-server/tests/remote_backend_tools.rs(4 tests):openapi_remote_backend_read_write_over_the_wire— REST CRM; register →validate → create → expand (
gold) → write admin (PATCH →platinum,read-after-write proves the upstream mutated) → write agent (refused, never
landed).
mcp_remote_backend_read_write_over_the_wire— JSON-RPC KB (tools/call,stateful
putArticle); same walk (draft→published).openapi_bearer_auth_forwarded_and_read_degrades_when_unauthorised— aguarded upstream: registered bearer is forwarded and never echoed by
list_endpoints; flipping toauth=nonemakes the read fail closed to{ issue }.openapi_unreachable_endpoint_degrades_read_and_probe— dead port →validate_endpointsunreachable + read degrades to{ issue }.Each behaviour was verified red→green by mutating the feature and watching
the right assertion fail, then restoring. Plus the pre-existing
escurel-index/tests/endpoints.rs(registry roundtrip / secret-not-in-corpus /survives-rebuild).
Full local gate green:
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace --all-targets,cargo build --workspace --release.🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.