SQL cluster refresh can hang indefinitely and run the same entity concurrently on two runners

[jan-wilhelm]

What version of Effect is running?

effect@3.21.2 / @effect/cluster@0.58.2 / @effect/sql-pg@0.52.1; also reproduced with effect@4.0.0-beta.99 / @effect/platform-node@4.0.0-beta.99 / @effect/sql-pg@4.0.0-beta.99

What steps can reproduce the bug?

Minimal reproductions:

• Effect v3: https://github.com/jan-wilhelm/effect-cluster-stuck-refresh-loop-reproduction
• Effect v4: https://github.com/jan-wilhelm/effect-cluster-stuck-refresh-loop-reproduction/tree/v4

To reproduce:

1. Clone the repository, create a throwaway PostgreSQL database, and configure it:

   npm install
   cp .env.example .env

2. Start two runners in separate terminals:

   RUNNER_NAME=runner-1 RPC_PORT=3011 RUNNER_PORT=34431 npm run runner

   RUNNER_NAME=runner-2 RPC_PORT=3012 RUNNER_PORT=34432 npm run runner

3. Enqueue one 120-second entity call:

   RPC_HOST=ws://localhost:3011 npm run enqueue

4. Once one runner prints START, wait about five seconds and target that runner. For example:

   TARGET_RUNNER=runner-1 npm run terminate

The terminate command kills that runner's Effect SQL PostgreSQL sessions and creates a local fault marker. The custom pg.Client then leaves subsequent SQL calls unresolved, deterministically simulating a network partition / blackholed connection rather than depending on OS TCP recovery timing.

The connection termination query is:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE application_name = '@effect/sql-pg:runner-1';

The reproduction uses shardLockDisableAdvisory: true, a one-second refresh interval, and a six-second lock expiration.

What is the expected behavior?

At most one runner should execute a given entity at a time.

If a runner can no longer refresh its ownership within a bounded amount of time, it should stop or interrupt its local entities before the lease can expire and be acquired by another runner. Reconnecting or shutting down are both plausible recovery strategies, but the old runner must not continue executing the entity concurrently with the new owner.

What do you see instead?

The affected runner's refresh operation remains suspended indefinitely while its local entity handler continues running. Once the SQL lease expires, the other runner acquires the shard and starts the same entity, producing split-brain concurrent execution.

A verified Effect v4 run produced:

19:02:38 runner-1 START entity=demo-1784588558578
19:03:11 terminated 2 connections and partitioned runner-1
19:03:17 runner-2 START entity=demo-1784588558578

Runner 1 had not emitted STOP when runner 2 started. Its handler sleeps for 120 seconds, so both runners were executing the same entity at that point. The same behavior occurs on the v3 reproduction.

Additional information

This appears to come from the SQL runner refresh path using a reserved/fixed SQL connection without a timeout.

Even when advisory locks are disabled, SqlRunnerStorage reserves a PostgreSQL connection and routes runner heartbeat / shard refresh queries through it. In v4 the connection is held through a ResourceRef and rebuilt after an error, but a query that never completes produces no error, so neither rebuilding nor lease-loss handling occurs:

https://github.com/Effect-TS/effect/blob/main/packages/effect/src/unstable/cluster/SqlRunnerStorage.ts

The explicit local fault marker makes the failure deterministic; it models the important failure mode where the database path is blackholed and an in-flight or subsequent query never resolves. It is not intended to claim that pg_terminate_backend alone always hangs.

This violates the cluster's single-owner assumption and can cause duplicate external side effects or conflicting state updates from two live handlers for the same entity.