Skip to content

fix(nats): propagate object store init errors#1361

Open
corylanou wants to merge 2 commits into
mainfrom
issue-1349-panic-nil-pointer-dereference-in-nats-replica-when-object-st
Open

fix(nats): propagate object store init errors#1361
corylanou wants to merge 2 commits into
mainfrom
issue-1349-panic-nil-pointer-dereference-in-nats-replica-when-object-st

Conversation

@corylanou

@corylanou corylanou commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Propagates replica client initialization failures from Store.Open() before any database or compaction monitors start. All replica clients complete initialization as a separate pass before any database opens. NATS initialization now considers the object store, rather than only the NATS connection, to be the completed initialization state and reuses an existing connection when retrying object store access.

Motivation and Context

With origin/main at c96c0f4 and a local nats:latest -js server that did not contain the configured object store, this command reproduced issue #1349:

litestream replicate /tmp/test.db nats://127.0.0.1:4222/missing-bucket

Before this change, the first access returned nats: bucket not found after setting ReplicaClient.nc and ReplicaClient.js. A concurrent startup access then saw the non-nil connection, treated initialization as complete, and called objectStore.List() on a nil object store:

panic: runtime error: invalid memory address or nil pointer dereference
nats.(*ReplicaClient).LTXFiles
nats/replica_client.go:285

The focused regression test reproduced the same partial state and panicked before the fix. After the fix, the CLI returns the initialization error once and exits with status 1:

Error: cannot open store: initialize replica client for "/tmp/test.db": nats: failed to initialize object store: failed to access object store bucket "missing-bucket" (bucket must be created beforehand): nats: bucket not found

Fixes #1349

Scope

In scope:

  • Initialize replica clients before Store starts database and maintenance monitors.
  • Require the complete client initialization pass to succeed before opening any database.
  • Return initialization failures to the caller.
  • Keep NATS clients retryable after object store initialization fails.
  • Cover partial NATS initialization, Store error propagation, and multi-database startup atomicity.

Not in scope:

  • Creating NATS object store buckets automatically.
  • Changing NATS authentication or TLS configuration.
  • Changing background retry behavior after successful startup.

Behavior Change

Scenario Before After
NATS bucket is missing Logs an initialization error, then panics Returns the initialization error and exits 1
Replica client initialization fails Starts background work that encounters the error Store.Open() returns the error before monitors start
NATS connection exists but object store init failed Treats the client as initialized Retries object store initialization on the existing connection

How Has This Been Tested?

  • go test ./nats -run TestReplicaClient_LTXFiles_InitError -count=1 -v
  • go test . -run TestStore_Open_InitError -count=1 -v
  • go test -race . -run TestStore_Open_InitErrorDoesNotOpenDBs -count=20
  • go test ./... -count=1
  • go test -race ./... -count=1
  • go vet ./...
  • pre-commit run --all-files
  • Docker NATS JetStream reproduction with a missing bucket; verified one initialization error, no panic, and exit status 1.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (would cause existing functionality to not work as expected)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes (go test ./...)
  • I have updated the documentation accordingly (if needed)

Treat the object store as the NATS initialization invariant and initialize replica clients before Store starts background work. This returns missing-bucket errors from startup instead of leaving a partial client that panics.

Fixes #1349
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

PR Build Metrics

All clear — no issues detected

Check Status Summary
Binary size 36.95 MB (0.0 KB / 0.00%)
Dependencies No changes
Vulnerabilities None detected
Go toolchain 1.25.12 (latest)
Module graph 1240 edges (0)

Binary Size

Size Change
Base (c96c0f4) 36.95 MB
PR (597d4e1) 36.95 MB 0.0 KB (0.00%)

Dependency Changes

No dependency changes.

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 1
vulnerability in modules you require, but your code doesn't appear to call these
vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric Value
Build time 42s
Go version go1.25.12
Commit 597d4e1

History (1 previous)

Commit Updated Status Summary
c68e90d 2026-07-10 20:00 UTC 36.95 MB (0.0 KB / 0.00%)

🤖 Updated on each push.

Run replica client initialization as a complete pass before any database opens. This prevents a later initialization error from returning with an earlier database and its monitor still active.
@corylanou corylanou marked this pull request as ready for review July 15, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic: nil pointer dereference in NATS replica when object store bucket does not exist

2 participants