AUT-1462: Playwright harness to reproduce the intermittent stg redirect loop

.npmignore

@@ -1,2 +1,8 @@
 node_modules
 .idea
+
+# Playwright repro harness (test/playwright-repro/README.md) - local dev/QA
+# tooling only, not part of the published package. Also guards against a
+# real .env with stg credentials ever being included in a publish.
+test/playwright-repro
+.env

package.json

@@ -22,6 +22,7 @@
     "middleware",
     "stores",
     "test",
+    "!test/playwright-repro",
     "index.js",
     "uuid.js"
   ],

test/playwright-repro/.env.example

@@ -0,0 +1,71 @@
+# Copy this file to .env and fill in real values. .env is gitignored - never commit it.
+#
+# These are STAGING credentials only. This harness is scoped to stg for now
+# (see the plan: prod is out of scope until the harness proves useful there).
+
+# --- Keycloak hostnames (stg) ---
+# These are TWO DIFFERENT hostnames, confirmed by direct testing - do not
+# collapse them into one:
+#   KC_BASE_URL       - public, serves OIDC login endpoints only
+#                       (/realms/*/protocol/*). This is what real users and
+#                       the fixture app's browser flow talk to.
+#   KC_ADMIN_BASE_URL - internal, the ONLY place the Admin REST API
+#                       (/admin/realms/*) is reachable. The public hostname
+#                       returns a network-level 403 for every /admin/* path
+#                       regardless of credentials. Only provision.js uses this.
+KC_BASE_URL=https://sso.stg.smartling.net/auth
+KC_ADMIN_BASE_URL=https://admin-keycloak-ha-v26.inception.dev.smartling.net/auth
+KC_REALM=Smartling
+
+# Ask a current Inception team member for admin credentials with rights to
+# create/update clients and users in the Smartling realm (not a master-realm
+# account - that realm isn't reachable from the public hostname, and master
+# users don't exist in the Smartling realm's own user store).
+KC_ADMIN_USERNAME=
+KC_ADMIN_PASSWORD=
+KC_ADMIN_CLIENT_ID=admin-cli
+# Realm the admin account itself lives in. Usually the same as KC_REALM if
+# it's a realm-scoped admin (the common case here) - only set this to
+# "master" if your account is specifically a master-realm superadmin.
+KC_ADMIN_REALM=Smartling
+
+# --- Throwaway client/user this harness manages ---
+# provision.js creates/updates these. Change KC_CLIENT_ID if you want your own
+# isolated copy alongside a teammate's (e.g. playwright-repro-<yourname>), so
+# you're not fighting over the same client's session overrides mid-run.
+KC_CLIENT_ID=playwright-repro-stg
+# Leave blank - provision.js creates this client as confidential (not
+# public; see provision.js for why) and fetches its auto-generated secret,
+# writing it here for you. repro.js refuses to start without it.
+KC_CLIENT_SECRET=
+# Must be email-shaped: Smartling's stg login theme's first step is an HTML
+# type="email" field with native browser validation, so a plain non-email
+# username can never even be submitted. ".invalid" is an IANA-reserved TLD
+# guaranteed to never resolve or match a real SSO domain-routing rule, so it
+# won't accidentally get redirected to Google/Okta federated login.
+KC_TEST_USERNAME=playwright-repro-test-user@stg-repro.invalid
+# Leave blank to have provision.js generate a random password and print it once.
+KC_TEST_PASSWORD=
+
+# --- Fixture app (runs under Node 16, see README) ---
+FIXTURE_APP_PORT=3987
+# Optional: absolute path to a Node 16 binary, e.g. output of `nvm which 16`.
+# If unset, the harness tries `nvm exec 16 node ...` itself and fails loudly
+# with setup instructions if that doesn't resolve on your machine.
+FIXTURE_NODE_BINARY=
+
+# --- Session/token overrides applied ONLY to KC_CLIENT_ID ---
+# These are per-client "Advanced > Fine Grain OpenID Connect Configuration"
+# overrides in the Keycloak admin console - they do NOT touch the realm-wide
+# SSO Session Max, so nobody else's stg session is affected. Verify the exact
+# attribute keys provision.js sets (client.session.max.lifespan,
+# client.session.idle.timeout, access.token.lifespan) still match what the
+# stg KC26 admin console's Advanced tab shows before relying on results -
+# this hasn't been confirmed against a live server yet.
+# Keep these ordered: SESSION_MAX > ACCESS_TOKEN_LIFESPAN > TOKEN_MIN_TTL,
+# so the session-cap window actually has room to trigger before the session
+# itself ends.
+KC_CLIENT_SESSION_MAX_SECONDS=90
+KC_CLIENT_SESSION_IDLE_SECONDS=120
+KC_ACCESS_TOKEN_LIFESPAN_SECONDS=30
+KC_TOKEN_MIN_TTL_SECONDS=20

test/playwright-repro/.gitignore

@@ -0,0 +1,5 @@
+node_modules/
+.env
+results/*.json
+results/traces/
+results/har/

test/playwright-repro/README.md

@@ -0,0 +1,216 @@
+# AUT-1462 redirect-loop repro harness
+
+Playwright harness that tries to trigger the intermittent "Too Many Redirects"
+loop on stg, many times in a row, without needing a human to manually shrink
+Keycloak session timeouts and babysit browser tabs each time.
+
+Background: see `AUT-1461` / `AUT-1462` in Jira and the `#keycloak-dev` Slack
+channel. Even with the fixes in `middleware/grant-attacher.js`,
+`middleware/logout.js` and `middleware/post-auth.js`, the team can still
+intermittently reproduce the loop on stg - this harness exists to make that
+reproducible on demand instead of by luck.
+
+## Scope: what this DOES and DOES NOT test
+
+**Does:** exercises this repo's own `keycloak-connect` adapter code
+(`index.js`, `middleware/*`) directly, end to end - real login against real
+stg Keycloak, real session/token handling, real grant storage, through the
+actual middleware chain (`Setup → PostAuth → Admin → GrantAttacher →
+Logout → Protect`). The session-cap/redirect-loop bug we're chasing lives
+entirely in this code, so testing it in isolation is sufficient to reproduce
+and diagnose it.
+
+**Does NOT (yet):** touch `tms-dashboard-app`, `ti-next`, `wa`, or any other
+real Smartling product surface. `fixture-app/` is a bare-bones Express app
+with one protected JSON route - it never calls TMS's backend, never checks
+account/permission data, and has no dependency on whatever additional
+provisioning (beyond the Keycloak identity itself) a real product login
+would need. The throwaway test user this harness creates exists **only**
+in Keycloak; it is not, and does not need to be, provisioned in TMS. This
+is a deliberate choice to keep the reproduction focused on the adapter code
+under test, not a limitation to work around - but it does mean this harness
+cannot tell you whether the bug (or its fix) behaves the same way inside the
+real dashboard app. If that ever becomes necessary, see
+[`ROADMAP-dashboard.md`](./ROADMAP-dashboard.md) for a brief note on what
+would need to change.
+
+## How it works, briefly
+
+1. `provision.js` creates a **throwaway OIDC client** in the stg Smartling
+   realm (not the shared `tms-dashboard-app`/`ti-next`/`wa` clients) and
+   overrides *only that client's* session/token lifetimes to ~60-120 seconds
+   via the Advanced settings Keycloak exposes per-client. This forces the
+   KC26 "session-cap" condition (see `grant-manager.js`'s `ensureFreshness`)
+   to happen quickly, without touching the realm-wide SSO Session Max that
+   every other stg user/client relies on.
+2. `repro.js` spins up a small fixture Express app (`fixture-app/`) built on
+   **this repo's own `index.js`/`middleware/*`** - the actual code being
+   tested - logs in as a dedicated throwaway test user, waits past the
+   session-cap window, and reloads a deep-linked protected URL repeatedly
+   (optionally from two concurrent tabs, to target the multi-tab race shape
+   both AUT-1461 and the suspected residual AUT-1462 issue share). It records
+   a result row per iteration and keeps a Playwright trace + HAR only for
+   iterations that actually reproduce the loop.
+
+## One-time setup
+
+Prerequisite: run `npm install` at the **repo root** first, under Node 16
+(`nvm use`), if you haven't already for other work in this repo. The fixture
+app requires `../../../index.js` directly and reuses `express`/
+`express-session` from the root `node_modules` (both already root
+devDependencies) rather than duplicating them here.
+
+```bash
+cd test/playwright-repro
+npm install
+cp .env.example .env
+```
+
+`repro.js` drives real installed **Google Chrome** (via Playwright's
+`channel: 'chrome'`), not Playwright's bundled Chromium - the original bug
+report is specifically about Chrome, so that's what this reproduces against.
+This requires Chrome to already be installed on your machine (normal for a
+dev laptop); if it isn't, either install it or run
+`npx playwright install chromium` and switch `repro.js`'s `chromium.launch()`
+call back to no `channel` option.
+
+Fill in `.env`:
+- `KC_ADMIN_USERNAME` / `KC_ADMIN_PASSWORD`: must be an admin account that
+  exists **inside the `Smartling` realm itself** (i.e. shows up under
+  Users when you have the Smartling realm selected in the admin console),
+  with realm-management rights to create/update clients and users there -
+  **not** a `master`-realm superadmin account. A master-realm account will
+  fail with a generic "Invalid user credentials" error here even when the
+  password is correct, because it doesn't exist in the Smartling realm's own
+  user store (confirmed by direct testing - see `KC_ADMIN_REALM` below).
+  Ask the team for the shared 1Password entry - don't use a personal stg
+  login. **This harness never asks for or stores prod credentials or access.**
+- `KC_TEST_PASSWORD`: leave blank and `provision.js` will generate one and
+  print it once (it's also written to `.env` for you on first run).
+- Everything else has a reasonable default; only change `KC_CLIENT_ID` if you
+  want your own isolated throwaway client alongside a teammate's.
+
+The fixture app needs to run under **Node 16** (this repo's pinned version,
+see the root `CLAUDE.md`/`.nvmrc`), independent of whatever Node version you
+use to run Playwright itself. If you have `nvm` with Node 16 installed
+(`nvm install 16`), you don't need to do anything else - the harness resolves
+it automatically. If that fails on your machine, set `FIXTURE_NODE_BINARY` in
+`.env` to the absolute path of a Node 16 binary (e.g. `nvm which 16`).
+
+## Safety model - read this before running anything
+
+`provision.js` is **dry-run by default**. Without `--apply`, it authenticates
+(read-only) and prints exactly what it *would* create/update/delete without
+making any change. Nothing in the stg realm is touched until you explicitly
+add `--apply`.
+
+Every client/user this tool creates is tagged with a
+`smartling.playwright-repro` attribute. If `KC_CLIENT_ID` or
+`KC_TEST_USERNAME` happens to collide with something that already exists and
+doesn't carry that tag, `provision.js` refuses to touch it (create, update,
+*or* delete) and errors out instead - so a naming collision can never
+silently modify or delete something real.
+
+Even in dry-run mode, `provision.js` still makes one real (but read-only)
+authenticated call to stg Keycloak with the admin credentials in `.env`, to
+look up whether the client/user already exist. That's the only network
+activity dry-run performs.
+
+Both `provision.js` and `repro.js` also hard-fail immediately if
+`KC_BASE_URL` doesn't look like a staging URL (i.e. doesn't contain "stg") -
+this harness is scoped to stg only, and this check exists so a `.env` typo
+can't point it at prod.
+
+## Running it
+
+```bash
+# 1. See exactly what would be created, without touching stg
+npm run provision:plan
+
+# 2. Review that output, then actually create it
+npm run provision:apply
+
+# 3. Open the stg admin console (Clients -> playwright-repro-stg -> Advanced)
+#    and cross-check the attributes provision.js printed against what's
+#    actually there. Do this before running repro.js at scale.
+
+# 4. Start small: one iteration, one tab, headed so you can watch it
+node repro.js --iterations=1 --tabs=1 --headed
+
+# 5. Once that looks right, run the real loop
+node repro.js --iterations=50 --tabs=2
+```
+
+Results land in `results/results.json` (one JSON object per line - iteration
+number, tab count, reproduced true/false, redirect-chain length, elapsed
+time, final URL/error). Traces and HARs for iterations that **did** reproduce
+the bug are saved under `results/traces/` and `results/har/` respectively;
+non-reproducing iterations don't leave artifacts behind.
+
+When you're done:
+
+```bash
+npm run teardown:plan    # see what would be deleted
+npm run teardown:apply   # actually delete the throwaway client + user
+```
+
+## Sanity-checking the harness itself
+
+Before trusting a run of "0/50 reproduced" against current `master`, first
+confirm the harness can actually catch the bug: check out the commit just
+before the AUT-1462 fix (`git log --oneline` to find the parent of `cbef5ad`)
+in a separate worktree, point the fixture app at that checkout, and confirm
+`repro.js` reliably reports `reproduced: true`. If it doesn't reproduce the
+*known* old bug, don't trust it reporting "no repro" on current code either.
+
+## Known defect found while building this harness
+
+While getting a single iteration working, this harness surfaced a real,
+confirmed bug in this repo's own code - independent of whether the redirect
+loop itself reproduces:
+
+**`ensureFreshness()` in `middleware/auth-utils/grant-manager.js` (around
+line 173-176) omits `client_id` from the `refresh_token` grant request
+body.** `postOptions()` only adds an `Authorization: Basic` header for
+confidential clients (`if (!manager.public)`), so a **public** client's
+refresh request carries neither that header nor `client_id` anywhere in the
+payload - Keycloak has no way to identify the calling client and correctly
+rejects it with `invalid_client`. Confirmed live: our throwaway harness
+client failed every refresh with exactly this error while public; switching
+it to confidential (see `provision.js`) made the identical flow work
+immediately. `obtainDirectly()` elsewhere in the same file already includes
+`client_id` in its params, so the fix is a one-line addition of
+`client_id: this.clientId` to the `refresh_token` grant's `params` object,
+matching that existing pattern.
+
+This most likely hasn't surfaced in production because `tms-dashboard-app`/
+`ti-next`/`wa` are presumably confidential clients, but it's a real
+correctness bug in the library for anyone using this adapter with a public
+client. Tracked as [AUT-1468](https://smartling.atlassian.net/browse/AUT-1468).
+
+## Known unknowns (flagging rather than guessing)
+
+Confirmed against the live stg KC26 admin console:
+- The client attribute keys (`client.session.max.lifespan`,
+  `client.session.idle.timeout`, `access.token.lifespan`) are correct -
+  `provision.js`'s read-back after apply matched what the admin console
+  showed under Clients → `<client>` → Advanced.
+- The admin account must exist inside the `Smartling` realm itself, not
+  `master` (see the setup section above) - confirmed by testing both.
+- The Admin REST API (`/admin/realms/*`) is only reachable via the internal
+  `KC_ADMIN_BASE_URL` hostname; the public `KC_BASE_URL` hostname returns a
+  network-level 403 for every `/admin/*` path regardless of credentials.
+- The login form is a two-step identifier-first flow, confirmed by live
+  inspection (screenshot + DOM dump): the email field (`#username`, HTML
+  `type="email"` with native browser validation) is submitted first via
+  `#kc-login`, which reveals the password field - also submitted via the
+  same `#kc-login` selector. `login()` handles this now. This is also why
+  `KC_TEST_USERNAME` must be email-shaped (see `.env.example`) - a plain
+  username can never even be submitted past step one, regardless of what
+  Keycloak itself would accept.
+
+Still open:
+- Whether a per-client session override actually reaches the same
+  "remaining session time" calculation `ensureFreshness()` reads, versus only
+  a realm-wide SSO Session Max doing so. The sanity-check section above is
+  what confirms or refutes this once a full iteration completes end-to-end.

test/playwright-repro/ROADMAP-dashboard.md

@@ -0,0 +1,46 @@
+# Roadmap: extending this harness to `tms-dashboard-app` / `ti-next`
+
+Brief notes on what would change if this harness needed to test the real
+product surface instead of the isolated `fixture-app/` (see README.md
+"Scope" section for why it doesn't today). Not a commitment to build this -
+just what's known so it doesn't need to be re-derived later.
+
+1. **Target app.** Point the Playwright browser flow at a running
+   `tms-dashboard-app` (or `ti-next`) instance instead of `fixture-app/`.
+   Either a local dev instance of that app pointed at stg Keycloak, or the
+   real stg deployment - these have different tradeoffs (see point 3).
+
+2. **TMS-side account provisioning.** A real dashboard login needs more than
+   a Keycloak identity - per the checkpoint discussion in this session, the
+   product likely needs a corresponding account record in TMS's own backend
+   (accountUid, permissions, etc.) that a Keycloak-only user created by
+   `provision.js` won't have. Cheapest path: reuse an existing, already-
+   working stg test account (Keycloak + TMS both provisioned) rather than
+   trying to script that provisioning here. Whoever owns that provisioning
+   flow (ask in `#keycloak-dev`) would know if there's a supported way to
+   create one from scratch.
+
+3. **Session/token overrides can't be isolated the same way.**
+   `provision.js`'s whole approach depends on a throwaway client whose
+   session/token lifetimes we can shrink without affecting anyone else. The
+   real `tms-dashboard-app`/`ti-next`/`wa` clients are shared production
+   configuration - shrinking their overrides would affect every real stg
+   user, the exact coordination cost this harness was built to avoid (see
+   the 6/29-6/30 Slack history in `#keycloak-dev` about restoring shared
+   session settings afterward). Two options, not mutually exclusive:
+   - Run a local dev instance of the target app registered as its own
+     throwaway client (same isolation model as `fixture-app/`), or
+   - Coordinate a short, announced window to shrink the real shared client's
+     overrides, as the team has done manually before.
+
+4. **Login flow should mostly transfer.** The two-step identifier-first
+   login theme and the Tab-blur requirement discovered while building this
+   harness (see `repro.js`'s `login()`) are realm-level (Smartling realm),
+   not client-specific - they should apply the same way against
+   `tms-dashboard-app`/`ti-next`'s login pages.
+
+5. **Deep-linked protected routes.** `fixture-app/`'s `/app/:accountId`
+   route already loosely mirrors the real shape
+   (`/app/<accountId>?locale=fr&start=0&...`) reported in the original bug.
+   Swapping in the actual dashboard route(s) that showed the loop should be
+   straightforward once points 1-3 are settled.