Skip to content

ops(enoomian-staging): codify sslip stream authority audit and runbook#134

Open
rndrntwrk wants to merge 4 commits into
mainfrom
enoomian/staging-sslip-authority-topology
Open

ops(enoomian-staging): codify sslip stream authority audit and runbook#134
rndrntwrk wants to merge 4 commits into
mainfrom
enoomian/staging-sslip-authority-topology

Conversation

@rndrntwrk

@rndrntwrk rndrntwrk commented Apr 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

This draft PR codifies the enoomian staging stream-authority outage class that blocked Hyperbet playback even while the Cloudflare live input and manifest remained healthy.

The root problem was authority drift: public capture truth and internal bet-sync truth were sourced from different authorities, which allowed keeper readiness to fail closed before the Cloudflare player mounted.

Scope

  1. authority audit tooling
  2. runbook documentation for this outage class
  3. discoverability of the runbook from the existing docs index

Key Changes

1. Add an authority audit command

  • adds enoomian:authority:audit to package.json
  • adds scripts/enoomian-staging-authority-audit.ts
  • verifies public capture truth, keeper public truth, and playback manifest reachability
  • uses direct bet-sync truth when readable
  • falls back to keeper truth when direct bet-sync is intentionally private
  • preserves partial endpoint evidence when one probe fails so the operator can still see which upstream authority dropped truth

2. Add a runbook for the exact incident class

  • adds docs/runbooks/enoomian-staging-stream-authority.md
  • documents symptoms, authoritative endpoints, expected truth relationships, common failure split between capture and bet-sync authority, and the operator validation sequence

3. Link the runbook from the docs index

  • updates docs/runbooks/README.md

4. Track the AWS GPU deploy rail used by enoomian staging

  • adds docs/runbooks/enoomian-aws-gpu-migration.md
  • adds scripts/enoomian-staging/deploy.sh targets for hyperscapes-aws-gpu-*
  • adds scripts/enoomian-staging/aws-gpu.sh for AWS runtime env generation, code sync, and API/source deployment controls
  • adds scripts/enoomian-staging/deploy.test.sh coverage for AWS activation defaults, opt-in Railway syncs, RTMPS URL normalization, bearer-token propagation, and AWS-specific delivery-provider overrides
  • tracks scripts/enoomian-staging-secret-manifest.sh so a fresh checkout has the variable manifest required by the deploy rail

Validation

  • ran bun scripts/enoomian-staging-authority-audit.ts against enoomian staging
  • confirmed capture/status returned 200
  • confirmed keeper public readiness returned true
  • confirmed the Cloudflare playback manifest returned 200
  • audit concluded PASS
  • ran ENOOMIAN_ALLOW_OFFBRANCH_DEPLOY=1 bash scripts/enoomian-staging/deploy.test.sh on the PR branch after rebase
  • live enoomian AWS validation showed Hyperbet mounting https://3.129.45.146.sslip.io/live/stream.m3u8 with visible fighters/avatars

Review Guidance

  1. scripts/enoomian-staging-authority-audit.ts
  2. docs/runbooks/enoomian-staging-stream-authority.md
  3. scripts/enoomian-staging/
  4. scripts/enoomian-staging-secret-manifest.sh
  5. docs/runbooks/enoomian-aws-gpu-migration.md
  6. package.json
  7. docs/runbooks/README.md

Explicit Non-Goals

This draft PR does not include Hyperbet app behavior changes, imported upstream feature work, Hyperscapes /stream spectator fixes, local-only Caddy snapshots, or secrets/cookies/machine-specific runtime artifacts.

`bunx playwright install --with-deps chromium` resolves the latest
Playwright CLI from npm at each run. When npm republished the CLI on
Apr 22 the bundled Chromium moved from v1208 to v1217, but every app's
`@playwright/test` pin is still `1.58.2`, which expects
`~/.cache/ms-playwright/chromium_headless_shell-1208/...`. The test
runner crashed on every chain with `Executable doesn't exist`.

Install from the app package's own `node_modules/.bin/playwright` so
the CLI matches the pinned test runner version. `bun run dev:bootstrap`
already populates the app package's `node_modules`, so this is a
zero-cost swap.

Restores: Cross-Chain E2E (solana|bsc|avax).
@rndrntwrk rndrntwrk marked this pull request as ready for review April 24, 2026 09:40
@rndrntwrk rndrntwrk marked this pull request as draft April 28, 2026 17:06
@rndrntwrk rndrntwrk marked this pull request as ready for review April 28, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant