serve: a web console that browses the URI data tree#4
Merged
Conversation
ant serve now answers a browser as well as a curl. The same routes a script reads as JSON render as a shadcn-styled console when the request asks for HTML, so there is one URL surface, not two. The console is cache-first. A resource view reads the record already materialized under the data tree and only fetches over the network on a miss or when refresh is set; a live fetch is written back (JSON always, Markdown when the record has a body) so the next read is offline. A "cached" or "live" badge says which path served, and a refresh control pulls a fresh copy. Browsing follows the on-disk tree like directories: the root lists every registered domain, a scheme lists its record types, and a type lists its records, with breadcrumbs back up. Domains that expose a search op get a search box; a hit that is a preview shape is resolved back to its canonical URI so it is one click from its record. The web package serves from go:embed (templates and assets, no Node), SSR through html/template, Markdown through goldmark, and sets a per-response CSP nonce with no secrets in the page. Every page and the JSON negotiation are covered by web/console_test.go against a network-free fake, and serve_test.go covers the wiring. Requires any-cli v0.3.3 for Host.Search/Host.Searchable.
The render path reads the request nonce inline; the helper was dead and the linter flagged it (and its lone context import).
The dashboard and browse-root were listing the entire data root to count
and group records. $HOME/data is shared with many other tools, so that
walk crossed hundreds of thousands of unrelated files: the dashboard took
~7s and browse-root ~15s on a real tree, and it was wrong too, surfacing
other tools' files as ant records.
Scope both pages to ant's own domains: the dashboard sums per-domain
counts and browse-root lists the registered domains, never LL("").
Back LL with an in-memory index. The first call for a prefix walks the
domain's subtree; the result is held in memory, and every cache-write
folds the new URI into the matching listings, so a record fetched
mid-session shows up in browse with no re-walk. serve warms the index for
each domain in the background at startup, off the request path.
Measured on the real tree: dashboard 7438ms -> ~1ms, browse 14993ms ->
~1ms; every page well under 25ms. With a synthetic 50k-record domain the
dashboard and browse-root stay ~0.5ms (cached counts) and the deepest
grouping is ~11ms.
Tests: the engine index is cached and write-through (a file written
behind the Engine is not seen; one written through it appears at once);
the dashboard and browse-root never list the whole root.
The resource page called Dereference synchronously with a 30s request timeout, so a slow upstream (x://status/... was the reported case) blocked the request until the deadline tripped and the page died with a raw "context deadline exceeded". Move the network off the request's critical path so a page renders the data, a loading screen, or a clean error, but never hangs and never times out. Add a small job manager (web/jobs.go): background fetches deduplicated by key so N viewers of one slow URI cost a single upstream call, results retained for a few minutes so a reload reads memory, workers on a background context so navigating away does not abort a fetch others want. Reads resolve cache-first: a materialized record renders from disk with no goroutine and no network (Engine.Lookup, the read-only half of Dereference). On a miss, a browser waits a short grace window and either renders inline (fast) or hands off to a self-reloading loading screen (slow) that polls /status and reloads when ready; JS-off falls back to a meta refresh. Scripts still wait for the data and get a structured 202 pending instead of a hang. Warm the next click: viewing a record fire-and-forgets a bounded prefetch of the records it links to, so the likely next navigation is a cache hit. Prefetch is capped per page and globally and never queues ahead of an interactive click. The only synchronous, timeout-bounded path left is the deliberate export button. Tests cover dedup, error capture, deadline-leaves-it-running, prefetch bounding, and the cold loading-then-status path; race-clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ant servenow answers a browser as well as a curl. The same routes a script reads as JSON render as a shadcn-styled console when the request asks for HTML (Accept: text/html, or?format=html), so there is one URL surface, not two. JSON stays the default for tools;/api/...forces it.Cache-first
A resource view reads the record already materialized under the data tree (
$HOME/data,ANT_DATA-overridable) and only fetches over the network on a miss or whenrefreshis set. A live fetch is written back (JSON always, Markdown when the record has a body) so the next read is offline. A cached or live badge shows which path served, and a refresh control pulls a fresh copy.Browse like directories
The console walks the on-disk tree as folders: the root lists every registered domain, a scheme lists its record types, a type lists its records, with breadcrumbs back up.
Search
Domains that expose a search op get a search box (driven by
Host.Searchable/Host.Search, new in any-cli v0.3.3). A hit that is a preview shape is resolved back to its canonical URI, so a result is one click from its record. Verified live across goodreads, wikipedia, x and youtube.How
web/package serves fromgo:embed(templates + assets, no Node), SSR throughhtml/template, Markdown through goldmark.web/negotiate.go; a hand-rolled first-segment router preserves//in raw-URI paths.Tests
web/console_test.gorenders every page and checks JSON negotiation against a network-free fake (status + shell + marker + content-type + CSP nonce).cli/serve_test.gocovers the wiring (named endpoints, HTML negotiation, raw-URI path not redirected).go build/go test/go vet/gofmt -lall clean; binary built withCGO_ENABLED=0.Depends on any-cli v0.3.3 (#12 there); the path
replaceused during development has been removed sogo installand goreleaser stay clean.