logs_2.sqlite-wal grows without bound into tens of GB

[th317erd]

What version of Codex CLI is running?

codex-cli 0.140.0

codex doctor reports 0.141.0 is available, but this was observed on 0.140.0.

Also observed locally:

codex-cli 0.140.0
codex-cli 0.138.0  # codext wrapper

What platform is your computer?

Linux x86_64, Ubuntu 24.04 noble.

Runtime/install from codex doctor:

Codex Doctor v0.140.0 · linux-x86_64
runtime: npm
Node path: ~/.nvm/versions/node/v24.13.0/...
terminal: Alacritty
auth: ChatGPT auth

Which model were you using?

gpt-5.5

What issue are you seeing?

Codex SQLite log WAL files are growing without bound into many GB/tens of GB during normal local use. The main symptom is enormous logs_2.sqlite-wal files, while the base logs_2.sqlite database remains comparatively small.

This appears to be a default Codex log/state behavior, not something caused by a custom log setting. My setup uses multiple CODEX_HOME profile directories (~/.codex-work and ~/.codex-personal) in addition to the default ~/.codex, but that only relocates Codex state. The default config in ~/.codex/config.toml has not changed any log or SQLite settings. It only contains a trusted project entry. No log_dir or sqlite_home override is configured there.

The Codex docs say local state is stored under CODEX_HOME, defaulting to ~/.codex, and mention logs/caches as per-user state. They also document optional log_dir and sqlite_home settings. So this should affect a standard ~/.codex setup the same way; the profile dirs here just made the issue easier to isolate.

Concrete evidence

Earlier in this session, before cleanup, the two active Codex profile/session folders had roughly 65 GB of SQLite log WAL files between them:

~/.codex-work/logs_2.sqlite-wal      tens of GB
~/.codex-personal/logs_2.sqlite-wal  tens of GB
combined log WAL sidecars            about 65 GB
logs_2.sqlite-shm sidecars           tens of MB
logs_2.sqlite base DB                about 115,650,560 bytes per active profile

After deleting/moving aside WAL/SHM sidecars and restarting Codex, the WALs reappeared and began growing again. A later post-cleanup check showed:

/home/<user>/.codex-personal/logs_2.sqlite      115,650,560 bytes
/home/<user>/.codex-personal/logs_2.sqlite-wal  1,693,608,432 bytes
/home/<user>/.codex-personal/logs_2.sqlite-shm  3,309,568 bytes

/home/<user>/.codex-work/logs_2.sqlite          115,650,560 bytes
/home/<user>/.codex-work/logs_2.sqlite-wal      19,184,359,792 bytes
/home/<user>/.codex-work/logs_2.sqlite-shm      37,257,216 bytes

/home/<user>/.codex/logs_2.sqlite               14,266,368 bytes
/home/<user>/.codex/logs_2.sqlite-wal           8,207,072 bytes
/home/<user>/.codex/logs_2.sqlite-shm           32,768 bytes

At the same time, profile directory sizes were dominated by these SQLite sidecars:

~/.codex-work      23G
~/.codex-personal  2.8G
~/.codex            104M

codex doctor reported the log DB itself as healthy:

state: databases healthy
log DB ~/.codex-personal/logs_2.sqlite (file) · integrity ok

But it did not warn about the enormous WAL file.

Impact

• Normal backup/rsync of Codex state attempted to copy tens of GB of volatile SQLite WAL files.
• The huge WAL files consumed significant disk space and made simple profile synchronization unexpectedly expensive.
• Deleting the WAL while Codex processes are live does not actually reclaim disk until those processes exit because the file remains open.
• Attempting to truncate open deleted WAL file descriptors caused active Codex instances to crash immediately.
• The current workaround is to exclude *.sqlite-wal and *.sqlite-shm from rsync/backup and only clean/checkpoint these files when Codex is not running.

Steps to reproduce

1. Use Codex CLI/app normally for an extended local coding session, especially with substantial tool output.

2. Inspect the Codex state directory:

   find "$CODEX_HOME" -maxdepth 1 \( -name '*sqlite-wal' -o -name '*sqlite-shm' -o -name 'logs_2.sqlite' \) -printf '%p %s bytes\n' | sort

3. Compare logs_2.sqlite-wal size with logs_2.sqlite.

Expected result: WAL remains bounded, periodically checkpointed, or is truncated on clean shutdown.

Actual result: logs_2.sqlite-wal can grow into tens of GB while the base log DB remains around ~100 MB.

Expected behavior

• Codex should not allow logs_2.sqlite-wal to grow without a practical bound.
• The logging/state layer should periodically checkpoint/truncate SQLite WAL files or configure WAL autocheckpointing appropriately.
• Large tool output should be capped, summarized, or rotated before it can accumulate unboundedly in persistent local log state.
• codex doctor should warn when logs_2.sqlite-wal or other Codex SQLite sidecars exceed a reasonable threshold.
• Ideally, Codex should expose documented retention/size controls for local logs and SQLite-backed runtime state.

Related issues

This may be related to, but is more specific than:

• Desktop launch can fail when logs_2.sqlite grows large: app-server SQLite pool times out during startup #27741, where a large logs_2.sqlite causes Desktop startup failures.
• Heavy I/O activity from idle codex processes. #20563, which reports heavy I/O from idle Codex processes.

This issue is specifically about unbounded logs_2.sqlite-wal growth under default-style Codex state behavior.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• Codex SQLite feedback logs can write ~640 TB/year and rapidly consume SSD endurance #28224
• Desktop launch can fail when logs_2.sqlite grows large: app-server SQLite pool times out during startup #27741

Powered by Codex Action

[th317erd]

Correction to the original evidence section: the peak pre-cleanup observation was roughly 65 GB of SQLite log WAL files across the two active Codex profile/session folders. The smaller per-file numbers originally listed there were from a later post-cleanup/regrowth check and should not have been presented as the pre-cleanup peak. I edited the issue body to separate the peak observation from the later regrowth evidence.

[th317erd]

Additional observation: a closed/stale Codex work-profile log WAL just measured at 219G / 234,157,858,816 bytes: /home/<user>/.codex-work/logs_2.sqlite-wal. At that point the user's /home filesystem was full: 656G size, 623G used, 40M free, 100% used. The file was not held open by currently running Codex processes, so removing only the closed .codex-work/logs_2.sqlite-wal and .codex-work/logs_2.sqlite-shm restored free space to 219G available. Separately, currently running codext app-server/remotes were attached to .codex-personal/logs_2.sqlite-wal; three remote worker processes each showed roughly 170GB cumulative writes in /proc//io while the active personal WAL was about 1.6G at inspection time. This confirms the issue can become disk-filling, not just inconvenient for backups.

[th317erd]

Follow-up DB-content scan from the surviving base log DBs found a likely self-amplifying source. The schema is a small log DB: logs(id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes), with WAL journaling enabled. The base DBs are only about 111 MB each, but they contain many TRACE rows where Codex logs file watcher events for its own log DB/WAL changes.\n\nExamples from .codex-work/logs_2.sqlite:\n\n- total logs rows: 42,534\n- TRACE rows: 38,537\n- rows with inotify event: 35,571\n- rows mentioning logs_2.sqlite-wal: 26,401\n- repeated recent sample: inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some(\"logs_2.sqlite-wal\") }\n\nExamples from .codex-personal/logs_2.sqlite:\n\n- total logs rows: 48,872\n- TRACE rows: 44,153\n- rows with inotify event: 38,580\n- rows mentioning logs_2.sqlite-wal: 25,397\n- recent samples include repeated inotify event ... name: Some(\"logs_2.sqlite\") rows.\n\nThis looks like a feedback loop: Codex logs that its own log DB/WAL changed; that insert changes the WAL again; the watcher observes another modify event; then another TRACE row is written. Because the DB is in WAL mode and checkpoint/truncation is not keeping the WAL bounded, a small 111 MB base DB can generate a 219 GB logs_2.sqlite-wal. There are also large TRACE payload rows from websocket/frame logging, but the inotify self-logging loop is the clearest disk-filling mechanism found so far.

[th317erd]

Additional sanitized findings from a deeper investigation:

The strongest evidence now points to a self-watch / self-log loop around $CODEX_HOME.

Likely loop

1. Codex writes a diagnostic row to logs_2.sqlite.
2. SQLite updates logs_2.sqlite-wal.
3. A Codex inotify watcher sees logs_2.sqlite-wal modified.
4. Codex records a TRACE row for that inotify event into the same logs_2.sqlite.
5. That write modifies logs_2.sqlite-wal again.
6. Repeat.

Important environment note

The newest evidence was collected while the affected session was being run through the codext fork, which vendors/runs a codex binary. That means this may be fork-specific, fork-amplified, or caused by an interaction between upstream Codex code and codext's app-server/remote process model. I am posting this here because the DB/schema/logging/watch behavior is Codex-derived, but I am also filing the same sanitized report against Loongphy/codext.

Observed state

Sanitized paths:

CODEX_HOME=/home/<user>/.codex-work
SQLite DB=/home/<user>/.codex-work/logs_2.sqlite
SQLite WAL=/home/<user>/.codex-work/logs_2.sqlite-wal

The current post-cleanup WAL still grew back to roughly 12 GB:

11,884,873,392  /home/<user>/.codex-work/logs_2.sqlite-wal
115,650,560     /home/<user>/.codex-work/logs_2.sqlite
23,101,440      /home/<user>/.codex-work/logs_2.sqlite-shm

Earlier growth samples during active sessions showed severe WAL growth:

2,492,175,672 -> 2,842,709,392 bytes in 10 seconds
5,953,070,432 -> 6,316,977,672 bytes in 10 seconds
8,921,312,072 -> 9,101,376,672 bytes in 5 seconds

This is after a previous stale/closed WAL reached 219 GB and filled the /home filesystem.

SQLite log-table evidence

Counts from the affected logs_2.sqlite:

total_rows=47,565
TRACE=42,794
INFO=3,533
DEBUG=864
WARN=356
ERROR=18
inotify_rows=39,532
logs_2.sqlite-wal mentions=28,727

Top repeated inotify messages:

28,699  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite-wal") }
 8,049  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite") }
   147  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("state_5.sqlite-wal") }

There are also unrelated file-open watcher events for system files such as ld.so.cache, locale.alias, and passwd, which appear to come from a separate /etc watch. The disk-filling loop is the one involving logs_2.sqlite*.

Kernel inotify evidence

The active affected process had an inotify file descriptor with:

FD 30 anon_inode:inotify
  inotify wd:1 ino:c6860b ...

The inode maps to $CODEX_HOME:

hex c6860b == decimal 13010443
13010443 /home/<user>/.codex-work

The repeated SQLite log rows also use WatchDescriptor { id: 1, ... }, and the row names are logs_2.sqlite-wal / logs_2.sqlite. That ties the logged file events directly to a watcher rooted at $CODEX_HOME, not just to a random project directory.

Open file holders

The affected codext/vendored codex process held open handles to:

/home/<user>/.codex-work/logs_2.sqlite
/home/<user>/.codex-work/logs_2.sqlite-wal
/home/<user>/.codex-work/logs_2.sqlite-shm

Other active codext app-server/remote processes held handles to a separate profile's logs_2.sqlite-wal.

Process model clue

The process tree included:

node .../bin/codext ... app-server --listen ws://127.0.0.1:<port>
.../codex ... app-server --listen ws://127.0.0.1:<port>
node .../bin/codext ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
.../codex ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
node .../bin/codext ... --sandbox danger-full-access --ask-for-approval never
.../codex ... --sandbox danger-full-access --ask-for-approval never

The exact project names have been omitted intentionally. Non-Codex dev-server processes running in one project did not hold handles to logs_2.sqlite*; only Codex/codext processes did.

Trust/root clue

The current Git repo root resolved correctly to a project directory under /home/<user>/Projects/<repo>, but the UI reportedly displayed a trust warning for /home/<user>, not the project root. If a broad home directory becomes a trusted/watch surface, it can include $CODEX_HOME and therefore Codex's own SQLite state.

Even without that clue, the kernel inotify evidence above shows a watcher rooted at /home/<user>/.codex-work.

Relevant source areas

These upstream files look relevant:

codex-rs/file-watcher/src/lib.rs
codex-rs/app-server/src/fs_watch.rs
codex-rs/app-server/src/skills_watcher.rs
codex-rs/tui/src/onboarding/onboarding_screen.rs

Specifically:

• file-watcher can watch requested paths and can fall back to the nearest existing ancestor.
• app-server exposes fs/watch.
• skills roots can be watched recursively.
• trust onboarding falls back to cwd if no Git root is resolved.

Expected behavior

Codex should not log file watcher events for its own SQLite diagnostic/state files into the same SQLite log sink.

At minimum, file watcher trace logging should suppress:

logs_2.sqlite
logs_2.sqlite-wal
logs_2.sqlite-shm
state_5.sqlite
state_5.sqlite-wal
state_5.sqlite-shm
goals_1.sqlite*
memories_1.sqlite*

Better fixes:

1. Never watch $CODEX_HOME as a filesystem watch root unless explicitly required.
2. Never log watcher events for Codex's own SQLite files into the SQLite log sink.
3. Respect log-level filtering before inserting TRACE rows into logs_2.sqlite.
4. Add WAL size limits, checkpointing, rotation, or emergency safeguards so diagnostic logs cannot consume hundreds of GB.
5. If a broad home directory trust target is selected, exclude Codex state directories from any watch surfaces.

Local mitigation being tested

The local mitigation is to move SQLite-backed runtime state outside the watched CODEX_HOME path:

sqlite_home = "/home/<user>/.local/share/codex-sqlite/work"

This was added after the issue was observed. Already-running Codex/codext processes can continue holding the old logs_2.sqlite* files until restarted.

[th317erd]

Cross-filed the sanitized fork-specific report against Loongphy/codext because the latest evidence was collected from codext/vendored-codex processes: Loongphy#11

[insilications]

This kind of program behavior is disrespectful in general, especially toward users with SSDs.

[th317erd]

Follow-up after comparing the official OpenAI repo against the Codext fork and testing a fix:

I now think the concrete trigger I hit is not directly an OpenAI/Codex bug. The direct self-watch loop appears fork-specific in Loongphy/codext: that fork adds an AuthWatch that watches the top-level CODEX_HOME directory for auth.json changes. Since the SQLite log DB normally also lives under CODEX_HOME, WAL writes can be observed by that watcher, and raw notify TRACE logs can be written back into the same SQLite log DB.

Fork-specific issue:

Loongphy#11

Fix PR:

Loongphy#12

That said, I think the underlying risk is still present in OpenAI/Codex's implementation: the SQLite log sink persists TRACE logs by default, and the shared log_db.rs did not filter raw notify/inotify compatibility logs. I did not find an official OpenAI/Codex production path that automatically watches CODEX_HOME itself, so the immediate trigger seems absent upstream. But if any current or future official watcher targets the SQLite directory, or a client asks app-server fs/watch to watch that location, the same class of feedback loop could become user-visible.

So this can probably be closed as "not directly reproduced in upstream OpenAI/Codex" if maintainers agree, but I would still recommend hardening the SQLite log sink so it does not persist raw file-watcher backend TRACE/DEBUG noise in a very damaging leaking fs-notify loop.