feat(utils): add WriteAheadLog classes #1210

BioPhoton · 2026-01-09T21:39:57Z

Related: #1197

This PR includes:

types for source sink flows
wal.ts - general WriteAheadLog and sharded WAL inc format and filenames
wal-trace-json.ts traceEvent specific login for a WAL
profiler constants to control WAL leader
moved trace utils into profiler

Note

This PR addresses a comment related to logging.
It could also be used as file output for the logger

Warning

All filesystem operations in WAL—and indirectly in the profiler—are synchronous by design.
The eslint rules are disables over file name pattern

Why synchronous fs is used for WAL code?

Profilers are side-channels, not business logic
Data must never be lost, reordered, or partially written
process.on('exit') cannot safely run async I/O
Sync writes guarantee ordering + durability
Profiler I/O volume is small → overhead is negligible
Crash-safety > throughput

nx-cloud · 2026-01-09T21:41:17Z

View your CI Pipeline Execution ↗ for commit f34ef73

Command	Status	Duration	Result
`nx run ci:code-pushup -- merge-diffs --files=/h...`	✅ Succeeded	5s	View ↗
`nx run-many --targets=code-pushup --parallel=fa...`	✅ Succeeded	1m 39s	View ↗
`nx run-many --targets=code-pushup --parallel=fa...`	✅ Succeeded	13m 23s	View ↗
`nx run-many -t unit-test,int-test`	✅ Succeeded	52s	View ↗

☁️ Nx Cloud last updated this comment at 2026-01-19 21:06:30 UTC

pkg-pr-new · 2026-01-09T21:41:34Z

Open in StackBlitz

@code-pushup/ci

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/ci@1210

@code-pushup/cli

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/cli@1210

@code-pushup/core

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/core@1210

@code-pushup/create-cli

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/create-cli@1210

@code-pushup/nx-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/nx-plugin@1210

@code-pushup/axe-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/axe-plugin@1210

@code-pushup/models

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/models@1210

@code-pushup/coverage-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/coverage-plugin@1210

@code-pushup/eslint-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/eslint-plugin@1210

@code-pushup/js-packages-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/js-packages-plugin@1210

@code-pushup/jsdocs-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/jsdocs-plugin@1210

@code-pushup/lighthouse-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/lighthouse-plugin@1210

@code-pushup/typescript-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/typescript-plugin@1210

@code-pushup/utils

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/utils@1210

commit: f34ef73

github-actions · 2026-01-09T21:48:09Z

Code PushUp

🤨 Code PushUp report has both improvements and regressions – compared current commit d5f2a33 with previous commit 270b474.

🕵️ See full comparison in Code PushUp portal 🔍

🏷️ Categories

🏷️ Category	⭐ Previous score	⭐ Current score	🔄 Score change
Documentation	🔴 42	🔴 46
Performance	🔴 37	🔴 36
Code style	🟢 99	🟢 100
Code coverage	🟢 93	🟢 93
Bug prevention	🟡 75	🟡 75
Axe Accessibility	🟡 88	🟡 88
Security	🟡 51	🟡 51	–
Updates	🟡 78	🟡 78	–
Type Safety	🟡 67	🟡 67	–
Miscellaneous	🟡 67	🟡 67	–
Accessibility	🟢 92	🟢 92	–
Best Practices	🟢 100	🟢 100	–
SEO	🟢 92	🟢 92	–

👍 4 groups improved, 👎 1 group regressed, 👍 11 audits improved, 👎 2 audits regressed, 23 audits changed without impacting score

🗃️ Groups

🔌 Plugin	🗃️ Group	⭐ Previous score	⭐ Current score
ESLint	Style guide (import)	🟡 80	🟢 100
JSDocs coverage	Documentation coverage	🔴 42	🔴 46
Lighthouse	Performance	🔴 37	🔴 36
ESLint	Suggestions	🟢 99	🟢 100
Code coverage	Code coverage metrics	🟢 93	🟢 93

29 other groups are unchanged.

🛡️ Audits

🔌 Plugin	🛡️ Audit	📏 Previous value	📏 Current value
ESLint	Forbid unassigned imports	🟥 1 warning	🟩 passed
JSDocs coverage	Classes coverage	🟥 6 undocumented classes	🟨 6 undocumented classes
JSDocs coverage	Properties coverage	🟥 30 undocumented properties	🟥 40 undocumented properties
Lighthouse	Total Blocking Time	🟥 1,890 ms	🟥 2,280 ms
Lighthouse	Speed Index	🟥 6.0 s	🟥 5.9 s
JSDocs coverage	Functions coverage	🟥 577 undocumented functions	🟥 580 undocumented functions
JSDocs coverage	Types coverage	🟥 268 undocumented types	🟥 258 undocumented types
JSDocs coverage	Variables coverage	🟥 268 undocumented variables	🟥 276 undocumented variables
Lighthouse	Time to Interactive	🟥 13.0 s	🟥 12.9 s
JSDocs coverage	Methods coverage	🟨 13 undocumented methods	🟨 15 undocumented methods
Code coverage	Branch coverage	🟨 89 %	🟨 89.2 %
Code coverage	Function coverage	🟩 94.4 %	🟩 94.5 %
Code coverage	Line coverage	🟩 92.7 %	🟩 92.6 %
Lighthouse	Uses efficient cache policy on static assets	🟨 31 resources found	🟨 31 resources found
Lighthouse	Avoids enormous network payloads	🟩 Total size was 2,104 KiB	🟩 Total size was 2,103 KiB
Lighthouse	Minimizes main-thread work	🟥 9.2 s	🟥 10.1 s
Lighthouse	JavaScript execution time	🟥 4.3 s	🟥 5.1 s
Lighthouse	Server Backend Latencies	🟩 1,200 ms	🟩 510 ms
Lighthouse	Largest Contentful Paint	🟥 11.5 s	🟥 10.9 s
Lighthouse	Reduce unused JavaScript	🟥 Potential savings of 224 KiB	🟥 Potential savings of 226 KiB
Lighthouse	Max Potential First Input Delay	🟥 980 ms	🟥 1,200 ms
Lighthouse	Reduce unused CSS	🟥 Potential savings of 95 KiB	🟥 Potential savings of 95 KiB
Lighthouse	Remove duplicate modules in JavaScript bundles	🟥 Potential savings of 93 KiB	🟥 Potential savings of 93 KiB
Lighthouse	Initial server response time was short	🟩 Root document took 540 ms	🟩 Root document took 410 ms
Lighthouse	Metrics	🟩 100%	🟩 100%
Lighthouse	First Contentful Paint	🟥 3.1 s	🟥 3.1 s
Lighthouse	Avoids an excessive DOM size	🟥 2,403 elements	🟥 2,405 elements
Axe	Elements must only use supported ARIA attributes	🟩 100 elements	🟩 102 elements
Axe	ARIA attributes must be used as specified for the element's role	🟩 100 elements	🟩 102 elements
Axe	Elements must only use permitted ARIA attributes	🟩 100 elements	🟩 102 elements
Axe	ARIA attributes must conform to valid names	🟩 100 elements	🟩 102 elements
Lighthouse	Network Round Trip Times	🟩 10 ms	🟩 10 ms
JS packages	Outdated npm prod dependencies.	🟨 16 outdated package versions (5 major, 8 minor, 3 patch)	🟨 15 outdated package versions (5 major, 8 minor, 2 patch)
TypeScript	Semantic errors	🟥 25 errors	🟥 26 errors
Axe	Inline text spacing must be adjustable with custom stylesheets	🟩 43 elements	🟩 42 elements
Axe	Interactive controls must not be nested	🟩 61 elements	🟩 60 elements

643 other audits are unchanged.

# Conflicts: # packages/utils/src/lib/sink-source.types.ts

# Conflicts: # testing/test-utils/src/lib/utils/perf-hooks.mock.ts

github-actions · 2026-01-16T22:26:47Z

Code PushUp

🥳 Code PushUp report has improved – compared current commit f523c36 with previous commit 270b474.

💼 Project `utils`

🥳 Code PushUp report has improved.

🕵️ See full comparison in Code PushUp portal 🔍

🏷️ Category	⭐ Previous score	⭐ Current score	🔄 Score change
Documentation	🔴 45	🟡 51
Code coverage	🟢 95	🟢 95

4 other categories are unchanged.

👍 2 groups improved, 👍 9 audits improved

🗃️ Groups

🔌 Plugin	🗃️ Group	⭐ Previous score	⭐ Current score	🔄 Score change
JSDocs coverage	Documentation coverage	🔴 45	🟡 51
Code coverage	Code coverage metrics	🟢 95	🟢 95

13 other groups are unchanged.

🛡️ Audits

🔌 Plugin	🛡️ Audit	📏 Previous value	📏 Current value
JSDocs coverage	Classes coverage	🟨 3 undocumented classes	🟨 3 undocumented classes
JSDocs coverage	Properties coverage	🟥 27 undocumented properties	🟥 37 undocumented properties
JSDocs coverage	Types coverage	🟥 66 undocumented types	🟨 56 undocumented types
JSDocs coverage	Functions coverage	🟥 239 undocumented functions	🟥 242 undocumented functions
JSDocs coverage	Variables coverage	🟥 49 undocumented variables	🟥 57 undocumented variables
JSDocs coverage	Methods coverage	🟨 13 undocumented methods	🟨 15 undocumented methods
Code coverage	Function coverage	🟩 96.2 %	🟩 96.6 %
Code coverage	Branch coverage	🟩 91.8 %	🟩 92.2 %
Code coverage	Line coverage	🟩 97.8 %	🟩 98 %

435 other audits are unchanged.

13 other projects are unchanged.

hanna-skryl

The justification for sync file operations is well-reasoned 👍 I've added a few minor suggestions and comments.

hanna-skryl · 2026-01-19T16:07:54Z

packages/utils/src/lib/wal.ts

+  SHARD_ID: /^\d{8}-\d{6}-\d{3}(?:\.\d+){3}$/,
+} as const;
+
+export function soratebleReadableDateString(timestampMs: string): string {


Suggested change

export function soratebleReadableDateString(timestampMs: string): string {

export function sortableReadableDateString(timestampMs: string): string {

hanna-skryl · 2026-01-19T16:23:14Z

packages/utils/src/lib/wal.ts

+   * @throws Error if recovery encounters decode errors
+   */
+  repack(out = this.#file) {


The JSDoc says that repack() throws if recovery encounters decode errors, but the implementation logs to the console and continues without throwing. Am I missing something?

hanna-skryl · 2026-01-19T16:27:56Z

packages/utils/mocks/sink.mock.ts

+import type { Codec } from '../src/lib/wal.js';

-export class MockSink implements Sink<string, string> {
+export class MockFileSink implements WriteAheadLogFile<string> {


The TypeScript plugin reports an error here. Maybe MockFileSink should implement AppendableSink<string> instead?

hanna-skryl · 2026-01-19T16:31:46Z

packages/utils/src/lib/profiler/wal-json-trace.unit.test.ts

+  it('should create WAL format with custom directory', () => {
+    const format = traceEventWalFormat({ dir: '/custom/path' });
+
+    expect(format.baseName).toBe('trace');
+    expect(format.walExtension).toBe('.jsonl');
+    expect(format.finalExtension).toBe('.json');
+  });
+
+  it('should create WAL format with groupId', () => {
+    const format = traceEventWalFormat({ groupId: 'session-123' });
+
+    expect(format.baseName).toBe('trace');
+    expect(format.walExtension).toBe('.jsonl');
+    expect(format.finalExtension).toBe('.json');
+  });


traceEventWalFormat doesn't take any parameters 🤔

hanna-skryl · 2026-01-19T16:40:04Z

packages/utils/src/lib/wal.ts

+    const recordsToWrite = hasInvalidEntries
+      ? (r.records as T[])
+      : filterValidRecords(r.records);
+    fs.mkdirSync(path.dirname(out), { recursive: true });


There's an existing ensureDirectoryExists utility, but it's async. Maybe worth adding a sync variant (ensureDirectoryExistsSync) since this pattern repeats multiple times?

Yes also thought about it. For me I'm pretty soled on going sync in only this edge case of WAL and profiling. Let's wait on @matejchalk opinion here.

hanna-skryl · 2026-01-19T16:46:56Z

packages/utils/src/lib/wal.ts

+   * @throws Error if any shard contains decode errors
+   */
+  finalize(opt?: Record<string, unknown>) {


Same question as in repack() - the JSDoc says it throws on decode errors, but there is no explicit Error.

hanna-skryl · 2026-01-19T16:48:55Z

packages/utils/src/lib/wal.ts

+  format: Partial<WalFormat<T>>,
+): WalFormat<T> {
+  const {
+    baseName = 'trace',


There is a mismatch between the JSDoc and the actual implementation - should baseName default to 'trace' or Date.now().toString()?

packages/utils/src/lib/wal.ts

Co-authored-by: Hanna Skryl <80118140+hanna-skryl@users.noreply.github.com>

John Doe added 2 commits January 8, 2026 20:11

refactor: add general file sink logic

c03b2c4

feat: add file sink classes

756f8c0

github-actions bot assigned BioPhoton Jan 9, 2026

github-actions bot added 🔬 testing writing tests 🧩 utils labels Jan 9, 2026

BioPhoton mentioned this pull request Jan 9, 2026

feat(utils): add performance observer #1206

Merged

BioPhoton mentioned this pull request Jan 10, 2026

Implement performance profiling and trace file generation #1197

Open

John Doe and others added 16 commits January 10, 2026 01:51

Merge branch 'main' into feat/utils/file-sink

4c95897

# Conflicts: # packages/utils/src/lib/sink-source.types.ts

refactor: add trace json file

b0c9cc4

refactor: wip

1f6e326

refactor: wip

6bcb73b

Merge remote-tracking branch 'origin/main' into feat/utils/file-sink

dfb43be

refactor: wip arc

3fe6871

refactor: fix lint

67e004c

Merge branch 'main' into feat/utils/file-sink

a4a9f10

# Conflicts: # testing/test-utils/src/lib/utils/perf-hooks.mock.ts

refactor: wip

0b1fd3a

refactor: wip

bf0ecb9

refactor: wip

c88ffe9

refactor: wip

d6e75e6

refactor: wip

21575b3

refactor: wip

09437a9

refactor: wip

4705394

refactor: wip

e0c210e

BioPhoton added 3 commits January 17, 2026 20:34

refactor: wip

29016b0

refactor: wip

e60ea2b

refactor: wip

5ea2ac4

BioPhoton added 8 commits January 17, 2026 21:39

refactor: wip

73ff371

refactor: wip

34b0eb8

refactor: fix lint

dd2e959

refactor: fix tests

64501ec

refactor: add tests

eb559f5

refactor: wip

e20e804

refactor: wip

8f812c5

refactor: fix lint

a839b9b

BioPhoton marked this pull request as ready for review January 17, 2026 21:44

BioPhoton requested a review from hanna-skryl January 17, 2026 21:44

BioPhoton added 2 commits January 17, 2026 22:53

refactor: fix lint

27b1506

refactor: wip

592fb89

BioPhoton requested a review from matejchalk January 17, 2026 22:34

BioPhoton added 3 commits January 18, 2026 00:35

refactor: wip

a19070e

refactor: wip

4f8cd2d

refactor: wip

bc86d08

BioPhoton changed the title ~~feat(utils): add file sink classes~~ feat(utils): add WriteAheadLog classes Jan 18, 2026

BioPhoton added 4 commits January 18, 2026 23:46

refactor: wip

a530134

refactor: remove path logic

d9eb84b

refactor: fix lint

67f0c1e

refactor: fix unit tests on windows

a3641d3

hanna-skryl reviewed Jan 19, 2026

View reviewed changes

BioPhoton and others added 3 commits January 19, 2026 21:48

Update packages/utils/src/lib/wal.ts

f7bf8e8

Co-authored-by: Hanna Skryl <80118140+hanna-skryl@users.noreply.github.com>

Update packages/utils/src/lib/wal.ts

d658b0f

Co-authored-by: Hanna Skryl <80118140+hanna-skryl@users.noreply.github.com>

Update packages/utils/src/lib/wal.ts

f34ef73

Co-authored-by: Hanna Skryl <80118140+hanna-skryl@users.noreply.github.com>

	export function soratebleReadableDateString(timestampMs: string): string {
	export function sortableReadableDateString(timestampMs: string): string {

feat(utils): add WriteAheadLog classes #1210

Are you sure you want to change the base?

feat(utils): add WriteAheadLog classes #1210

Conversation

BioPhoton commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nx-cloud bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code PushUp

🏷️ Categories

🗃️ Groups

🛡️ Audits

Uh oh!

github-actions bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code PushUp

💼 Project utils

🗃️ Groups

🛡️ Audits

Uh oh!

hanna-skryl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BioPhoton commented Jan 9, 2026 •

edited

Loading

nx-cloud bot commented Jan 9, 2026 •

edited

Loading

pkg-pr-new bot commented Jan 9, 2026 •

edited

Loading

github-actions bot commented Jan 9, 2026 •

edited

Loading

github-actions bot commented Jan 16, 2026 •

edited

Loading

💼 Project `utils`