Skip to content

feat: openapi + mcp remote (proxy) instance backends#244

Merged
jrosskopf merged 8 commits into
mainfrom
claude/instance-types-mcp-openapi-ojp70z
Jul 6, 2026
Merged

feat: openapi + mcp remote (proxy) instance backends#244
jrosskopf merged 8 commits into
mainfrom
claude/instance-types-mcp-openapi-ojp70z

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds openapi and mcp remote (proxy) instance backends — instance
bodies fetched live on expand from a REST/OpenAPI endpoint (openapi)
or an upstream MCP server (mcp), rather than materialised in DuckDB. The
whole feature was authored across the earlier commits on this branch; this PR
also closes their verification gap — they were written in a session where
the libduckdb-download egress block stopped the local cargo gate from running,
so the code was "compiled-by-construction only". It is now built, clippy-clean,
and full-suite green locally, with real-service wire coverage.

What lands:

  • Backend model + registry (escurel-index): openapi/mcp backend kinds,
    the RemoteBinding parser (endpoint / read / write / project), the
    admin external_endpoints registry (base URL + auth server-side, secret never
    in the markdown corpus — the SSRF guard), and the pure projection/templating
    core (fill_template, resolve_projection).
  • Live execution at the gateway (escurel-server/remote_backend.rs):
    outbound reqwest transport for openapi (method+path template) and mcp
    (JSON-RPC tools/call / resources/read), auth applied from the registry,
    JSONPath projection. fetch_projection (expand's live read; any failure →
    fail-closed { issue }, never a fabricated body) + write_instance
    (write-back) + probe (reachability).
  • MCP tools + dispatch: register_endpoint / list_endpoints /
    delete_endpoint / validate_endpoints / create_remote_instance (admin),
    and write_instance (agent, gated by the target's acl.update via
    may_write_instance, fail-closed). Registered in tools/list.
  • Outbound GET /openapi.json: an OpenAPI 3.1 doc generated from the same
    tool schemas tools/list serves.
  • Spec: docs/spec/protocol.md §"Remote backends", roadmap note, and a
    discovered-note on the libduckdb egress block.

Test plan

New no-mock wire e2e — real gateway (POST /mcp), real OIDC (TestIssuer),
real DuckDB + FsStore, real reqwest, and real, stateful in-process axum
upstreams
on loopback ports (no canned stubs):

crates/escurel-server/tests/remote_backend_tools.rs (4 tests):

  • openapi_remote_backend_read_write_over_the_wire — REST CRM; register →
    validate → create → expand (gold) → write admin (PATCH → platinum,
    read-after-write proves the upstream mutated) → write agent (refused, never
    landed).
  • mcp_remote_backend_read_write_over_the_wire — JSON-RPC KB (tools/call,
    stateful putArticle); same walk (draftpublished).
  • openapi_bearer_auth_forwarded_and_read_degrades_when_unauthorised — a
    guarded upstream: registered bearer is forwarded and never echoed by
    list_endpoints; flipping to auth=none makes the read fail closed to
    { issue }.
  • openapi_unreachable_endpoint_degrades_read_and_probe — dead port →
    validate_endpoints unreachable + read degrades to { issue }.

Each behaviour was verified red→green by mutating the feature and watching
the right assertion fail, then restoring. Plus the pre-existing
escurel-index/tests/endpoints.rs (registry roundtrip / secret-not-in-corpus /
survives-rebuild).

Full local gate green: cargo fmt --check,
cargo clippy --workspace --all-targets -- -D warnings,
cargo test --workspace --all-targets, cargo build --workspace --release.

🤖 Generated with Claude Code


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

claude and others added 7 commits July 5, 2026 12:00
Adds two live remote (proxy) instance backends alongside markdown/sql_view/
document:

- BackendKind::OpenApi / ::Mcp (+ is_remote helper); wire strings openapi/mcp
- SearchMode::None — remote data is live-fetched, never indexed, so it feeds
  no search lane (the overlay page is still indexed like any page)
- Capabilities::for_kind: remote = { writable:false, page-grain, search:none,
  no crdt }; update_page is rejected, write-back is the write_instance tool
- RemoteBinding / RemoteKind / RemoteOp + parse arms for kind: openapi|mcp
  (endpoint name + read/write op + project map); fail-closed when endpoint or
  read op is missing (mirrors sql_view)

Unit tests cover openapi/mcp/read-only/missing-endpoint parsing and the
remote capability contract.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
external_endpoints table (name, kind, base_url, auth, secret) + Indexer
register/lookup/list/delete_endpoint, mirroring the sql_view credential
registry. A skill's backend.endpoint references a row by name; the base URL
and secret live server-side only (SSRF + secrets-in-markdown guard), never in
the corpus. Ensured on every connection open and preserved across rebuild
(separate canonical input). No-mock integration tests over real DuckDB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
backend/remote.rs: the pure, network-free heart shared by the openapi/mcp
backends — resolve_projection (response JSON -> overlay fields via $.a.b
paths), json_path_get, fill_template (/customers/{id} + id), plus RemoteError.
Fully unit-tested. The live transport (RemoteClient) and expand/create/write
wiring build on these.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
protocol.md gains a 'Remote backends (openapi/mcp)' section (live-on-expand
reads, write_instance write-back, endpoint registry / SSRF guard, search:none,
live-read failure policy, frontmatter examples, new tools) and the Instance
backends intro now lists four external kinds. roadmap.md gains the matching
bullet. Adds a discovered-note for the libduckdb release-download 403 that
blocks the local build in a network-restricted session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
Wires the openapi/mcp backends end-to-end at the gateway:

- remote_backend.rs: reqwest transport for openapi (HTTP method+path template)
  and mcp (JSON-RPC tools/call + resources/read), auth from the endpoint
  registry, projection via the index-crate core. fetch_projection (expand's
  live read; failure -> backend_projection.issue) + write_instance (write-back)
  + probe (reachability). Unit-tested helpers.
- expand: live remote projection branch (backend_ref.kind in openapi|mcp).
- MCP tools + dispatch: register/list/delete/validate_endpoints,
  create_remote_instance (admin), write_instance (agent, acl.update-gated via
  may_write_instance). Registered in tools/list.
- Outbound GET /openapi.json: OpenAPI 3.1 doc generated from the same tool
  schemas tools/list serves.

NOTE: compiled-by-construction only — the libduckdb egress 403 blocks the
local cargo gate in this session (see docs/notes/discovered/2026-07-05-...).
The no-mock wire e2e test (real upstream server via EscurelProcess) is the
remaining verification step once the build is unblocked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8tWZR44iboGqp2uHb1ec4
Closes the verification gap the prior commit flagged: the branch was
authored in a session where the libduckdb-download egress block stopped
the local cargo gate from running, so the openapi/mcp code was
"compiled-by-construction only". Built, clippy-clean, and full-suite
green locally now.

remote_backend_tools.rs drives BOTH remote-backend kinds over the wire
against REAL, STATEFUL upstreams (in-process axum on loopback ports) —
no canned stubs. Four tests:

- openapi_remote_backend_read_write_over_the_wire: a REST CRM whose GET
  returns the current row and whose PATCH actually mutates it.
  register -> validate -> create -> expand (projected `gold`) ->
  write_instance admin (PATCH -> `platinum`, read-after-write re-expand
  proves the state changed) -> write_instance agent (refused by the
  fail-closed acl.update gate; a final expand proves it never landed).
- mcp_remote_backend_read_write_over_the_wire: a JSON-RPC 2.0 KB
  answering tools/list + tools/call whose putArticle mutates the store;
  same read/write/read-after-write/ACL walk (`draft` -> `published`).
- openapi_bearer_auth_forwarded_and_read_degrades_when_unauthorised: a
  guarded CRM that 401s reads lacking the bearer. A registered secret is
  forwarded (200, fields projected) and never echoed by list_endpoints;
  flipping the endpoint to auth=none makes the read fail closed to a
  `{ issue }`, never a fabricated body.
- openapi_unreachable_endpoint_degrades_read_and_probe: a dead port ->
  validate_endpoints reports it unreachable and a live read degrades to
  a `{ issue }`.

Verified red->green for each behaviour by mutating the feature and
watching the right assertion fail (openapi write: empty PATCH body; mcp
write: drop the payload merge in mcp_args; auth: skip attaching the
bearer in apply_auth), then restoring to green. The degraded-read cases
are themselves the fail-closed assertion.

Also apply rustfmt to the five files the web session couldn't format
(whitespace-only line reflows; no semantic change).

Test plan: cargo test -p escurel-server --test remote_backend_tools
(4 tests); full fmt/clippy/test/build gate green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-templating docs

Triage of the codex review on PR #244 (two P2 findings):

- MUST-FIX (finding 1): remote read/write dispatched the transport purely
  from the skill's backend kind and never checked the registered endpoint's
  kind. Since create_remote_instance only checks the endpoint *name* exists,
  an `openapi` skill pointing at an endpoint registered as `mcp` (or the
  reverse) would silently fire the wrong protocol at that URL — and, against a
  matching server, even return data. exec() now fails closed when
  `ep.kind != remote.kind`, degrading the read to an `{ issue }` naming both
  kinds. Adds RemoteKind::as_str().

- DOC (finding 2): the RemoteOp::Http doc comment and protocol.md both claimed
  OpenAPI path placeholders are "filled from the overlay id / write payload",
  but only `{id}` (from the overlay id) is bound — a multi-placeholder write
  path is rejected as `unfilled path placeholders`. Corrected both to state
  the real contract (payload merges into MCP tool-call *arguments*, not into
  OpenAPI path templates) rather than promise unimplemented behaviour. Filling
  path placeholders from the payload is left as a follow-up.

Verified red->green: with the guard disabled, the new
remote_read_fails_closed_on_endpoint_protocol_mismatch test observes the
mismatched read silently returning the CRM row (the bug); with it, the read
degrades to the mismatch issue.

Test plan: cargo test -p escurel-server --test remote_backend_tools (5 tests);
full fmt/clippy/test/build gate green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jrosskopf

Copy link
Copy Markdown
Contributor Author

Codex second-opinion review (CLAUDE.md principle 9)

Ran codex exec review --base main focused on design / security / stability / missing functions, with SSRF, secret handling, the write_instance ACL gate, and the fail-closed degraded read called out for scrutiny. Nothing flagged on the security surface (registry-gated egress, secret-never-echoed, ACL gate, fail-closed reads). Two P2 correctness findings, both now addressed in 0cea744:

✅ Fixed — endpoint protocol mismatch not rejected (remote_backend.rs)

exec chose the transport purely from the skill's remote.kind and never checked the registered endpoint's kind; create_remote_instance only checks the endpoint name exists. So an openapi skill pointing at an endpoint registered as mcp (or vice-versa) would dispatch the wrong protocol — and against a matching server, silently return data instead of failing closed. exec now rejects ep.kind != remote.kind, degrading the read to an { issue } naming both kinds. New test remote_read_fails_closed_on_endpoint_protocol_mismatch, verified red→green (with the guard disabled it observes the mismatched read silently returning the CRM row).

✅ Fixed (docs) — write-path placeholder templating overstated

The RemoteOp::Http doc comment and protocol.md both claimed OpenAPI path placeholders are "filled from the overlay id / write payload", but only {id} (from the overlay id) is bound — a multi-placeholder write path (e.g. /customers/{id}/orders/{order_id}) is rejected as unfilled path placeholders. Corrected both to state the real contract (the payload merges into MCP tool-call arguments, not into OpenAPI path templates). Actually filling path placeholders from the payload is left as a follow-up — not documented as supported, so this is an enhancement rather than a regression.

Full local gate green after the fix: fmt --check, clippy --workspace --all-targets -D warnings, test --workspace --all-targets (remote-backend suite now 5 tests), build --workspace --release.

The openapi/mcp write path bound only `{id}` in templates. Extend it so a
write op can address subresources and reshape the payload:

- Path/URI placeholders now resolve from `{id}` (overlay slug) PLUS every
  scalar leaf of the write payload, flattened to dotted keys — so
  `/customers/{id}/orders/{order_id}` and `{customer.tier}` both fill.
- OpenAPI write ops gain an optional `body:` template. An exact `"{name}"`
  leaf is substituted type-preserving (a number stays a number, an object
  stays an object); embedded `{name}` interpolates as a string; objects/
  arrays recurse. Absent `body:` ⇒ the payload is sent verbatim (back-compat).
- Both path and body fail closed on an unresolved placeholder (`unfilled
  path/body placeholders`) rather than sending a literal `{x}`.

Pure core (template_vars, flatten_scalars, render_body) lives in
escurel-index and is unit-tested without a network. RemoteOp::Http gains
`body: Option<Value>`; Value is not Eq, so the Eq derive is dropped from
RemoteOp / RemoteBinding / BackendBinding / SkillInfo (none are used as
hash/set keys — PartialEq is retained).

Verified red->green: the new wire test
openapi_write_fills_path_and_body_placeholders_from_payload drives a POST to
/customers/acme/orders/o-1 with a `body:` template ({sku} string, {qty}
type-preserving number, `via` constant) against a real stateful axum orders
CRM, then reads the stored order back; with the payload merge disabled the
`{order_id}` path placeholder goes unfilled and the write fails closed.

Test plan: cargo test -p escurel-index --lib backend::remote (6 pure cases);
cargo test -p escurel-server --test remote_backend_tools (6 wire tests);
full fmt/clippy/test/build gate green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jrosskopf

Copy link
Copy Markdown
Contributor Author

Follow-up landed: complex path + body placeholders (5f2216c)

Closes the enhancement flagged in the codex review (payload-derived path placeholders). The write path now binds far more than {id}:

  • Path/URI placeholders resolve from {id} (overlay slug) plus every scalar leaf of the write payload, flattened to dotted keys — /customers/{id}/orders/{order_id} and {customer.tier} both fill.
  • OpenAPI body: template (optional on a write op): an exact "{name}" leaf substitutes type-preserving (a number stays a number, an object stays an object); embedded {name} interpolates as a string; objects/arrays recurse. Absent body: → payload sent verbatim (back-compat).
  • Both fail closed on an unresolved placeholder (unfilled path/body placeholders), never sending a literal {x}.

Pure core (template_vars / render_body) lives in escurel-index with 6 unit cases; new wire test openapi_write_fills_path_and_body_placeholders_from_payload drives a POST /customers/acme/orders/o-1 with a body template against a real stateful axum orders CRM and reads the order back (verified red→green — disabling the payload merge leaves {order_id} unfilled and the write fails closed). Spec updated in protocol.md.

Full gate green: fmt, clippy --workspace --all-targets, test --workspace --all-targets (remote-backend suite now 6 tests), release build.

@jrosskopf jrosskopf merged commit b1e3538 into main Jul 6, 2026
1 check passed
@jrosskopf jrosskopf deleted the claude/instance-types-mcp-openapi-ojp70z branch July 6, 2026 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants