refactor: rewrite cuttle in Go + remote backends + local-canonical profiles#4
Merged
Conversation
…ure) Phase 1 of the Go rewrite plan (docs/plans/2607-16-cuttle-go-rewrite.md): the entire Python implementation moves verbatim to packages/cuttle-py/ where it stays building as the parity oracle. Root keeps shared docs, LICENSE, CHANGELOG, .github. New root AGENTS.md orients agents to the monorepo.
…rmalization with Python parity tests Port build_args, proxy URL normalization, WebRTC exit-IP resolution, geoip tz/locale/exit-IP lookup, and CLOAKBROWSER_BINARY_PATH resolution from the vendored cloakbrowser subset into internal/fingerprint. A CPython urllib.parse reimplementation (pyurl.go) preserves byte-exact proxy-URL semantics. Parity is enforced by a golden matrix dumped from the Python oracle (dump_parity_golden.py) covering seed set/unset x proxy absent/http/socks5/ with-auth x timezone+locale set/unset x webrtc auto/none, plus the COUNTRY_LOCALE_MAP. Exit-IP resolution is injectable so tests stay hermetic.
Fold the in-container daemon into a cuttle serve subcommand: per-seed Chrome pool with connection refcounting and idle reap (only when idle-timeout > 0), client<->Chrome CDP WebSocket proxying via coder/websocket, /json/version and /json endpoints with webSocketDebuggerUrl host rewritten to the request Host so they are correct behind port-forward/ssh -L, a loopback Origin allow-list, exact container detection (dockerenv/containerenv/KUBERNETES_SERVICE_HOST/cgroup) for the 0.0.0.0-vs-loopback bind, and clean SIGTERM drain. Ports the three upstream patches: proxy-auth-over-CDP injection, the service_worker browserContextId stamp, and the fork launch-parity flags. Adds a server-level default proxy (--proxy / CUTTLESERVE_PROXY, inherited when a connection omits ?proxy=, with geoip keyed off the effective proxy) and ephemeral profile-dir support (--ephemeral / CUTTLESERVE_EPHEMERAL). Hermetic table-driven tests cover CLI/idle-timeout parsing, seed validation, bind-host detection via injected fs/env, connect-URL parsing, the Origin allow-list, ws-URL host rewrite and default-proxy inheritance, and a fake CDP server proving frames pipe both ways with refcounting and idle reaping.
… CDP + locking internal/cdp connects to the running seed via chromedp RemoteAllocator (never launches Chrome) and extracts/injects Playwright storageState: cookies via CDP Storage/Network, per-origin localStorage via Runtime.Evaluate, resolving the seed browser ws through the multiplexer's /json/version?fingerprint routing. internal/profile keeps auth state canonical under $XDG_DATA_HOME/cuttle/profiles/<name>/storage_state.json: single-writer PID lockfile with stale-lock detection, checkout injects on session start, periodic crash-safe checkpoints and checkin extract back with atomic writes; storage=remote is an inert durable-remote handle.
internal/mcp installs a CDP driver (default browser-use via uv tool install) and generates its MCP client config as JSON pointing at the active context's CDP endpoint with the profile seed (BU_CDP_URL=...?fingerprint=<name>). cli: new mcp command; --profile on login/connect/mcp threads the seed into every attach line as ?fingerprint=<name>; login/connect run profile checkout before handing over and check in on exit including Ctrl-C via signal-aware cleanup.
Static Go builder + fontbuilder (build-time fontTools, never in the final image) + debian:trixie-slim runtime with verbatim clark/clearcote engine and KasmVNC/noVNC stages. A cuttleserve shim execs `cuttle serve` so the host CLI's docker run keeps working.
Deployment (cuttle serve, hard amd64 nodeSelector, CUTTLESERVE_HOST/PROXY, resources/tolerations from values), ClusterIP Service (CDP 9222 + VNC 6080), deny-all-ingress NetworkPolicy, and an emptyDir scratch by default / durable RWO PVC (Recreate + SingletonLock clear + resource-policy keep) when profileStorage=remote.
Point the Python ci/smoke workflows and release-please config at packages/cuttle-py after the monorepo move; add go-ci.yml (golangci-lint v2.11 + gotestsum + govulncheck, path-filtered to packages/cuttle-go).
Reshape the root README into a monorepo overview (what lives where, Go is the future, Python is the shipped oracle until parity) and add packages/cuttle-go/README.md covering install, contexts/backends, profiles, mcp, and serve.
Tune .golangci.yml (wrapcheck ignores our own internal packages, revive unhandled-error/unchecked-type-assertion noise, goconst ignore-tests, test-path exclusions) and clean the residual production findings: static sentinel errors, checked type assertions, ListenConfig/CommandContext for noctx, hoisted repeated literals to constants, closed websocket.Dial response bodies, and removed dead nolint directives.
…parity SplitProxyAuth moves to internal/fingerprint and is reimplemented on the CPython urlsplit/urlunsplit primitives so it byte-matches the oracle: proxy host is lowercased and inline credentials keep their raw percent- encoded form (net/url decoded them and preserved host case), which is what gets answered to the proxy 407 over CDP. ForkParityArgs likewise moves to fingerprint. The parity golden now dumps split_proxy_auth and fork_parity_args sections and credential-strips --proxy-server in compose_argv, and new table tests assert Go == oracle across mixed-case hosts and special-character credentials.
ResolveProxyGeoWithIP now mirrors the oracle: when every IP-echo service is unreachable but the proxy host still resolves, it DNS-resolves the proxy hostname and returns that gateway IP for tz/locale and WebRTC, instead of dropping tz/locale/webrtc-ip from the Chrome argv. The resolver step is injectable so the degrade paths stay hermetic.
resolve() now threads the --name flag (default "cuttle", matching the Python CLI's DEFAULT_NAME) as the docker container name for the local and ssh backends, instead of silently using the context name (so the default local container was created as "local"). k8s/direct still identify by context. Fixes the dead --name flag and the container-name regression.
…ntexts Reach now takes explicit local cdp/vnc ports: cuttle connect pins them to --cdp-port/--vnc-port (default 9222/6080) so a held k8s/ssh forward is deterministic and the mcp config (which already targets 127.0.0.1:cdpPort) points at it, while the ephemeral status/login/up forwards still pass 0 to auto-pick a free port and avoid colliding with a local container on 9222.
The load-bearing STEALTH-VERIFICATION check (docs/STEALTH-VERIFICATION.md) was not automated: a SwiftShader/llvmpipe/Mesa WebGL renderer is the classic automation tell the fork masks, but the smoke harness only checked webdriver + UA/platform coherence. Extend PROBE_JS to read UNMASKED_VENDOR/RENDERER and fail any cold cycle whose renderer is software-rendered or not via ANGLE. Verified GREEN against the Go image (cuttle:go-local): AMD Radeon RX 7600 / NVIDIA RTX 3060 via ANGLE Direct3D11 across seeds.
…code Quality cleanup from a parallel simplify review (reuse/simplification/efficiency/altitude), behavior-neutral: - Hoist the seed/profile-name grammar (regex + reserved sentinel + validity) into fingerprint.ValidSeed/ReservedSeed; serve and profile now share one source instead of two correctness-coupled copies. - Collapse 11 copy-pasted run-and-check-exit blocks across the local/k8s/ssh backends into one backend.runOK helper. - Delete the dead proxy-dict cluster (ProxySettings/ExtractProxyURL/IsSocksProxy/ReconstructSocksURL) and its parity test/golden/dumper section - cuttle only ever handles proxy strings. - wsproxy: prefilter the per-frame proxy-auth path with bytes.Contains before the full JSON decode (matching the sibling patches), skipping the decode for the vast majority of CDP frames. - pool: reuse the exit IP already resolved by resolveGeo for --fingerprint-webrtc-ip=auto instead of re-hitting the echo services on the launch path. - navigate: reuse withFingerprint and strings.HasPrefix instead of hand-rolled copies.
From a 4-lens polish review (correctness/cleanliness/design), all validated: - Correctness: profile checkpoint no longer silently drops localStorage on a transient per-origin load failure. cdp.Extract now reports origins that failed to LOAD (vs loaded-but-empty); checkpoint carries forward the prior canonical localStorage for exactly those, so a blip preserves state while a real logout still clears it. Added a regression test. - Cleanliness: replace an em-dash in a geoip doc-comment with ASCII; drop two rotting (see plan 8.x) comment references. - Design: extract the triplicated XDG base-dir fallback into internal/xdg (ConfigDir/DataDir), used by config/mcp/profile; drop the redundant K8s.kubeContext field (read k.ctx.KubeContext); share the docker-status->State mapping between the local and ssh backends. - Hardening: install the SIGINT handler before profile checkout in connect/login so a Ctrl-C in that window still runs checkin + lock release instead of exiting with the profile locked. Left as-is (documented): caching the geoip mmdb reader would require making the hermetic value-type GeoResolver stateful for a negligible, opt-in, per-launch cost - a net design regression, so skipped.
- Replace the Python release pipeline with GoReleaser: cross-platform CLI binaries, checksums, and a Homebrew cask (the deprecated brews formula section is gone) published to the tap on release. - release-please switches to the go release type and now owns CHANGELOG.md and the release notes, so git-cliff (cliff.toml) is dropped. - The image job builds from the repo-root context and stamps the binary version via the VERSION build-arg. - Promote go-ci.yml to ci.yml (repo-root module, no path filters); smoke.yml builds the root image and runs the Go smoke harness instead of the Python one.
The metric-compatible free fonts (Liberation/Carlito/Caladea) that report the Windows family names are now committed under winfonts/ (see winfonts/README.md for provenance; no Microsoft font software is included). The Dockerfile copies them directly instead of renaming at build time, so the python:3.12-slim fontbuilder stage and scripts/rename-fonts.py are removed - the build is now fully Python-free.
- test/smoke: a neutral, self-contained CDP smoke harness (per-seed canvas isolation, stealth coherence, WebGL-via-ANGLE, cold-cycle stability) ported from the deleted Python harness.py, over a minimal raw-CDP client. - internal/fingerprint: a -update golden test regenerates testdata/golden.json from the Go primitives, replacing dump_parity_golden.py. The regenerated golden is byte-identical to the prior Python-oracle golden.
…tree The Go rewrite reached parity (fingerprint golden, STEALTH-VERIFICATION, live challenge), so the coexistence phase ends: packages/cuttle-py is deleted and the Go module moves from packages/cuttle-go to the repo root. The module path shortens to github.com/glim-sh/cuttle, so go install now targets github.com/glim-sh/cuttle/cmd/cuttle@latest. AGENTS.md, README, .gitignore/.dockerignore, and the golangci wrapcheck glob are updated for the single-project layout; the rewrite plan is marked completed and kept as the historical design record. BREAKING CHANGE: the Go module path is now github.com/glim-sh/cuttle.
Move the Dockerfile, its dockerignore (now the co-located ops/docker/Dockerfile.dockerignore), the winfonts pack, and bin/ (entrypoint + VNC viewer) under ops/docker/ to declutter the repo root. The build context stays the repo root, so the Dockerfile COPY paths become ops/docker/... and the dockerignore uses 'ops' + '!ops/docker' (default-deny anything else added to ops/ later). Add a 'just build-image' target for the now-non-default 'docker build -f ops/docker/Dockerfile .' invocation.
Fold smoke.yml and release.yml into one ci.yml with two jobs: build-and-test (lint + test + vuln + image build + smoke, always) and release (release-please then GoReleaser + GHCR image, gated on release_created), which runs only on push to main after build-and-test is green. Per-job concurrency keeps builds cancellable while the release job is non-cancellable mid-publish.
The published image bakes the clark (MIT) and clearcote (BSD-3) stealth-Chromium binaries; BSD-3 requires reproducing clearcote's notice in binary redistributions, so restore THIRD-PARTY.md (dropped with the Python tree). The cloakbrowser section is trimmed to a courtesy line since the Go fingerprint/serve code is an authored port, not vendored source. README links to it.
Restore the agent-facing guide and fix gaps found reviewing the Go rewrite before merge. - cli: re-add `cuttle skill`, embedding internal/cli/SKILL.md via go:embed (root SKILL.md is a relative symlink to it); briefing no longer points at a nonexistent command; release-please bumps the embedded frontmatter version. Modernized the guide (go/brew/docker install, cuttle serve, contexts/backends, profiles, mcp) and dropped the repo-file "See also" list that a skill-loaded agent cannot read. - docs: recover STEALTH-VERIFICATION.md and UPGRADE.md (probe checklist, benign-log table, swiftshader warning, exit-IP rotation rule, engine-bump runbook + real-amd64 release gate) and RELEASING.md (release-please + GoReleaser flow, one-time admin setup); delete the completed rewrite plan doc, which also removes the last maintainer-identity leak. - fingerprint: reframe the golden as a regression tripwire, not parity-against-the-removed-Python-oracle, in AGENTS.md and code comments. - build/ci: DOCKER_BUILDKIT=1 for the per-Dockerfile .dockerignore, a helm lint step, a brew install line in the README, a fixed .goreleaser comment, and toolchain go1.26.5 so the govulncheck gate is clean and deterministic.
- golangci-lint action pinned tool v2.11 -> v2.12; v2.11.4's `config verify` rejected `goconst.ignore-tests`, failing every CI run on this branch. v2.12 accepts the existing config; local `run` + `config verify` are clean. - actions/setup-go v6 -> v7, goreleaser/goreleaser-action v6 -> v7; the rest (checkout, buildx, build-push, release-please, login, metadata) already latest. - go deps: maxminddb-golang/v2 2.3.0 -> 2.4.1, pflag 1.0.9 -> 1.0.10 (direct deps already latest).
tenequm
changed the title
…root - Reduce the cloakbrowser/cloakserve credit to a single MIT attribution in THIRD-PARTY.md (no source or binary from them is redistributed) and drop the "port of CloakHQ / CloakBrowser Manager" framing everywhere else: AGENTS.md, README.md, args.go/geoip.go comments, the Dockerfile/entrypoint/vnc-viewer design notes, and UPGRADE/STEALTH-VERIFICATION docs. - serve: route the bare-metal data dir through internal/xdg like the rest of cuttle (config/profiles/mcp) - XDG_DATA_HOME or ~/.local/share/cuttle/serve, container /tmp/cuttle. Was the only path still hardcoding ~/.cloakbrowser. - Move .goreleaser.yaml -> ops/config/goreleaser.yaml (ci.yml --config + a `just release-check` recipe) and THIRD-PARTY.md -> docs/ to unclutter the repo root; lefthook.yml stays at root (git hooks only auto-discover root/.config).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Full rewrite of cuttle from Python to Go. The Go binary folds together what used
to be three things - the host CLI (
cuttle/cli.py), the in-container daemon(
bin/cuttleserve), and the vendoredcloakbrowserPython subset - into oneauthored codebase we own outright. It also turns cuttle into a universal
browser-lifecycle CLI that can run the stealth browser local / k8s / ssh /
direct, adds local-canonical profiles (auth state lives on your machine,
injected per-session over CDP), and a
cuttle mcpthat wires a driver to theactive context.
The Chromium fork itself (clark / clearcote) is unchanged - we still consume it
as a SHA-pinned prebuilt binary. Only the thin orchestration around it moved to
Go.
Structure
The Go module is the repo root; the Python tree is gone. This PR includes the
final cutover (module promoted to
github.com/glim-sh/cuttle,cuttle-pydeleted) - it is no longer deferred.
What's implemented
internal/fingerprint- the stealth-critical core. Portsbuild_args, allproxy-URL normalization helpers, geoip (tz/locale/exit-IP resolution + GeoLite2
mmdb download/cache), WebRTC exit-IP resolution, and
CLOAKBROWSER_BINARY_PATHselection. Includes a byte-for-byte reimplementation of CPython's
urllib.parseso proxy URLs normalize identically.
internal/serve- thecuttle serveCDP multiplexer (the containerentrypoint). One Chrome per fingerprint seed on one port, connection
refcounting + idle reap, containerd-safe bind-host detection, SIGTERM drain, and
the three load-bearing upstream patches (proxy-auth-over-CDP, service_worker
browserContextIdstamp, fork launch-parity flags). New: a server-level defaultproxy (
CUTTLESERVE_PROXY) and ephemeral profile dirs. ThewebSocketDebuggerUrlhost is rewritten to the request Host, so it is correctbehind
kubectl port-forward/ssh -Lby construction.internal/config+internal/backend- TOML contexts (strict mode) and thefour backends (local / k8s / ssh / direct) behind one
Backendinterface, allgoing through an injected exec seam (unit-testable without docker/kubectl/ssh).
internal/cli-up / down / status / login / connect / mcp / skill / context ls+ the agent-facing driver briefing.internal/cdp+internal/profile- local-canonical storage_statecheckout/checkin over CDP, single-writer lockfile, crash-safe periodic
checkpoints, remote-persistent opt-in.
internal/mcp- installs a CDP driver and writes its MCP config pointed atthe active context + profile.
KasmVNC / noVNC / Windows-font stages are kept verbatim from the Python image.
ops/helm/cuttle/) - Deployment (cuttle serve, hardamd64nodeSelector), emptyDir scratch by default / RWO PVC for remote-persistent
profiles, ClusterIP service, deny-all-ingress NetworkPolicy.
ci.yml(golangci-lint v2, gotestsum-race, govulncheck,helm lint, image build + Go smoke) plus a release-please + GoReleaser release
job that publishes binaries, a Homebrew cask, and the GHCR image.
Stealth fidelity - the safety net
The one real risk of this rewrite is silently degrading stealth. Three guards, all
passing:
internal/fingerprint/testdata/golden.jsonpins argv / proxy-normalization / geoip output (regenerate with
just parity-golden). Table-driven tests assert the committed golden still matcheswhat the Go primitives emit, so any change to that output surfaces as a diff a
human must consciously regenerate and review - never a silent drift. (It was
originally captured byte-for-byte from the Python oracle, before that tree was
removed.)
smoke harness against it. GREEN:
navigator.webdriver=false,platform=Win32coherent with the Windows UA, per-seed canvas isolation, andthe load-bearing WebGL check - real GPUs via ANGLE/Direct3D11, no SwiftShader.
The harness asserts the WebGL renderer so every future bump enforces it.
real escalated challenge (reached the cleared page, not the interstitial),
webdriver=false, over real TLS, on a clean residential exit.Validation
go build ./...,gofumpt -lclean,golangci-lint run0 issues,go test -race ./...176 pass / 11 packages,govulncheckclean (toolchainpinned to go1.26.5), linux/amd64 cross-compile,
helm lint+helm templateall pass.
the image build + Go smoke on every PR.
Not in this PR
included.