test(core): fix flaky image-sanitization logger assertion#571
Open
lmorchard wants to merge 1 commit into
Open
Conversation
The "should sanitize image Buffer data in AI_GENERATION messages" test
asserted that the serialized log output did not contain the bare string
"255" (the 0xff byte value). But the output also includes a Date.now()
millisecond timestamp, which intermittently contains the substring
"255", failing the test at random (~0.5-1% of CI runs) regardless of
whether sanitization worked.
Assert against the serialized raw byte sequence ("255,216,255") instead.
This matches the actual leak format (a raw byte array) and cannot collide
with the timestamp, since a numeric timestamp contains no commas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a flaky assertion in the core logger tests by making the “image bytes are sanitized” check target the actual leaked serialization pattern instead of a substring that can coincidentally appear in timestamps.
Changes:
- Updates the
JSONConsoleLoggersanitization test to assert the logged JSON output does not contain the byte sequence255,216,255(JPEG header) rather than the bare string"255". - Adds an explanatory comment documenting why the prior assertion was flaky (timestamp collisions).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
coreCI job intermittently fails on:packages/core/test/loggers.test.ts > JSONConsoleLogger > JSON output > should sanitize image Buffer data in AI_GENERATION messagesThis has surfaced on unrelated PRs (e.g. dependabot bumps) where it has nothing to do with the change — most recently #568.
Root cause
The test verifies that a fake JPEG buffer (whose first byte is
0xff= 255) gets sanitized out of the logged output. To do so it asserted that the entire serialized JSON output did not contain the bare string"255":But the output also includes the
Date.now()timestamp. Whenever that millisecond value happens to contain the substring255, the assertion fails — regardless of whether sanitization worked. That's a ~0.5–1% chance per run, so it flakes randomly across CI.Verified locally: forcing
Date.now()to return1782552000000(contains255) reproduces the failure; the current real timestamp passes.Fix
Assert against the serialized raw byte sequence (
255,216,255) instead of a bare255. This matches the actual leak format (an unsanitized buffer serializes as a comma-separated byte array) and cannot collide with the timestamp, since a numeric timestamp has no commas. The check still fails loudly if sanitization regresses.Testing
pnpm --filter pilo-core run test— all 900 tests pass.🤖 Generated with Claude Code