Skip to content

test(core): fix flaky image-sanitization logger assertion#571

Open
lmorchard wants to merge 1 commit into
mainfrom
fix/flaky-logger-image-sanitization-test
Open

test(core): fix flaky image-sanitization logger assertion#571
lmorchard wants to merge 1 commit into
mainfrom
fix/flaky-logger-image-sanitization-test

Conversation

@lmorchard

Copy link
Copy Markdown
Collaborator

Problem

The core CI job intermittently fails on:

packages/core/test/loggers.test.ts > JSONConsoleLogger > JSON output > should sanitize image Buffer data in AI_GENERATION messages

AssertionError: expected '{"event":"ai:generation","data":{"tim…' not to contain '255'

This has surfaced on unrelated PRs (e.g. dependabot bumps) where it has nothing to do with the change — most recently #568.

Root cause

The test verifies that a fake JPEG buffer (whose first byte is 0xff = 255) gets sanitized out of the logged output. To do so it asserted that the entire serialized JSON output did not contain the bare string "255":

timestamp: Date.now(),          // 13-digit ms value, serialized into the output
...
expect(output).not.toContain("255"); // 0xff byte value

But the output also includes the Date.now() timestamp. Whenever that millisecond value happens to contain the substring 255, the assertion fails — regardless of whether sanitization worked. That's a ~0.5–1% chance per run, so it flakes randomly across CI.

Verified locally: forcing Date.now() to return 1782552000000 (contains 255) reproduces the failure; the current real timestamp passes.

Fix

Assert against the serialized raw byte sequence (255,216,255) instead of a bare 255. This matches the actual leak format (an unsanitized buffer serializes as a comma-separated byte array) and cannot collide with the timestamp, since a numeric timestamp has no commas. The check still fails loudly if sanitization regresses.

Testing

pnpm --filter pilo-core run test — all 900 tests pass.

🤖 Generated with Claude Code

The "should sanitize image Buffer data in AI_GENERATION messages" test
asserted that the serialized log output did not contain the bare string
"255" (the 0xff byte value). But the output also includes a Date.now()
millisecond timestamp, which intermittently contains the substring
"255", failing the test at random (~0.5-1% of CI runs) regardless of
whether sanitization worked.

Assert against the serialized raw byte sequence ("255,216,255") instead.
This matches the actual leak format (a raw byte array) and cannot collide
with the timestamp, since a numeric timestamp contains no commas.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a flaky assertion in the core logger tests by making the “image bytes are sanitized” check target the actual leaked serialization pattern instead of a substring that can coincidentally appear in timestamps.

Changes:

  • Updates the JSONConsoleLogger sanitization test to assert the logged JSON output does not contain the byte sequence 255,216,255 (JPEG header) rather than the bare string "255".
  • Adds an explanatory comment documenting why the prior assertion was flaky (timestamp collisions).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants