Add resilience testing plan and 26 E2E tests

[pinodeca]

Resilience Testing Plan + 26 E2E Tests for pg_durable

Comprehensive plan to systematically break pg_durable the way a real user might — finding resource limits, edge-case bugs, and failure modes not covered by existing E2E tests.

What's in this PR

Test plan: docs/resilience-testing.md — six testing categories with prioritized phased rollout.

26 new E2E tests (tests 38–63) covering Phases 1–5:

Test	Plan Item	What it tests
38	B1/B2	Infinite loop cancellation
39	B3	Truthiness edge cases (NULL, 0, "false", "no", etc.)
40	B5/B6	Empty/DML result $var substitution (bug found)
41	B10	df.break() outside a loop
42	B11	Recursive df.start() inside workflow (no guard found)
43	C1	Empty/whitespace/invalid SQL
44	C7	Manually crafted JSON bypassing DSL
45	A1	20 concurrent instances burst
46	A2	50-level deep graph nesting
47	A7	20 rapid start/cancel cycles
48	B8	RACE where both branches fail
49	B9	JOIN where one branch fails
50	B12/B13	Signal edge cases (non-existent, completed, duplicate)
51	A3	9-branch wide parallel graph (3×3 join3)
52	A4	100-iteration loop history
53	A5	10K-row result set
54	A6	~10KB query text
55	A8	5KB variable payload
56	C5	500K rapid df.status() polls
57	B4/B14	Variable name conflicts and result shadowing
58	D1	pg_terminate_backend on BGW → PG auto-restarts in ≤5s; in-flight instance reaches terminal state
59	E2/E3	Signal-waiting instance stays running indefinitely — no idle timeout; df.cancel() is the only exit
60	E1	Deleting df.instances row leaves df.nodes rows intact (no FK cascade); df.status() returns gracefully
61	E4/E5	Instance and node row counts grow proportionally after 10 runs; confirms zero automatic GC
62	F1	10 independent dblink connections each call df.start(); all produce distinct IDs and complete
63	F4	Variable snapshot-at-start semantics verified; cross-session last-writer-wins on df.vars demonstrated

Test runner improvements:

• --keep-going / -k flag for test-e2e-local.sh to continue past failures with summary at end.
• Tests 58, 60, 62, 63 added to the superuser (postgres) override list — they require pg_terminate_backend, direct RLS-bypassing table access, or dblink with postgres credentials.

Key findings

ID	Finding	Severity	Test
F1	$var substitution of empty/0-row results produces unquoted JSON → syntax error	Bug	40
F2	No iteration limit on df.loop() — infinite loops run forever	Design gap	38
F3	No recursion guard on df.start() — can spawn unbounded child instances	Design gap	42
F4	df.break() outside loop returns break sentinel as result (not an error)	Quirk	41
F5	Serde ignores unknown JSON fields in crafted Durofut payloads	Quirk	44
F6	Empty/whitespace SQL accepted by DSL, fails at execution time	Quirk	43
F7	Signal to non-existent/completed instance silently succeeds	Quirk	50
F8	No FK between df.instances / df.nodes — orphaned nodes accumulate silently	Design gap	60
F9	No GC for completed instances or duroxide history — unbounded table growth	Design gap	61
F10	df.vars is per-owner-global — concurrent sessions race on the same key (last writer wins)	Design gap	63
F11	No idle timeout for signal-waiting instances — df.cancel() is the only escape	Design gap	59

Remaining gaps (not addressed here)

• D2 (PostgreSQL crash recovery), F2/F3 (signal/cancel vs. completion races), and F5 (concurrent df.status() lock contention) require a shell-level or timing-sensitive test harness beyond what SQL-only E2E tests support.
• D3 (Disk full simulation — infeasible in SQL)
• D4 (Network partition to remote database)
• D5 (Clock skew / time jumps)
• D6 (Extension drop+recreate while instances are in-flight — partial: test 28 covers drop+recreate with no in-flight instances)
• E6 (Tampering with df.nodes mid-execution)

[pinodeca]

@copilot go ahead and implement the resilience testing plan proposed in this PR.

[Copilot]

@pinodeca I've opened a new pull request, #56, to work on those changes. Once the pull request is ready, I'll request review from you.

[pinodeca]

@copilot #56 implemented some of the resilience test plan but wasn't tested and then was merged into branch pinodeca/breakit, but this PR's description wasn't updated. Can you update it?

Regardless, continue implementing the resilience test plan. You may want to check the status of #56 - that's where we left off.

[Copilot]

@pinodeca I've opened a new pull request, #62, to work on those changes. Once the pull request is ready, I'll request review from you.

[pinodeca]

@copilot continue executing the plan in docs/resilience-testing.md

[Copilot]

@pinodeca I've opened a new pull request, #66, to work on those changes. Once the pull request is ready, I'll request review from you.