feat: back off auto top-up card declines

[pjb157]

Summary

• pause auto top-up for 24 hours after the first soft card decline and send a retry/update-card email
• disable auto top-up after the second consecutive soft decline, or immediately after a hard decline, and send a manual-recharge email
• reset the consecutive soft-decline state after a successful charge or when auto top-up is reconfigured
• classify Stripe card failures using decline/advice codes and keep provider, network, and API errors outside the decline counter
• use atomic state transitions to avoid duplicate scheduler instances escalating or notifying twice

Testing

• just lint rust
• cargo test -p dwctl --all-features auto_topup -- --test-threads=4 (36 passed)
• full Rust workspace with bounded concurrency (passed; nine organization-email tests were excluded because the local macOS resolver configuration contains a scoped IPv6 nameserver that Hickory cannot parse)
• the API-key group (41 tests) and traffic-routing test that initially hit local concurrency limits were rerun separately and passed

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	cfa6c21
Status:	✅  Deploy successful!
Preview URL:	https://8431b7b6.control-layer.pages.dev
Branch Preview URL:	https://peter-auto-topup-decline-bac.control-layer.pages.dev

View logs

[Copilot]

Pull request overview

Implements Stripe decline classification and stateful backoff for automatic top-ups, including user-facing emails and DB-backed, atomic decline-state transitions to prevent duplicate scheduler escalation/notifications.

Changes:

• Introduces AutoTopupDeclineKind and maps Stripe card errors into soft vs hard declines.
• Adds persistent auto top-up decline/backoff state (auto_topup_soft_failure_count, auto_topup_retry_after) with atomic update helpers and scheduler handling + new emails.
• Updates tests, templates, and SQLx metadata to cover the new decline/backoff flows.

Reviewed changes

Copilot reviewed 12 out of 44 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
dwctl/src/payment_providers/stripe.rs	Classifies Stripe card errors into soft/hard declines and maps auto-topup charge errors accordingly.
dwctl/src/payment_providers/mod.rs	Adds AutoTopupDeclineKind and a new PaymentError::AutoTopupDeclined variant + status mapping.
dwctl/src/payment_providers/dummy.rs	Extends dummy provider to simulate soft/hard declines in tests via special customer IDs.
dwctl/src/notifications.rs	Applies decline backoff/disable logic during auto-topup processing and sends appropriate emails; resets state on success.
dwctl/src/email.rs	Adds new email templates and send helpers for “retry tomorrow” and “auto top-up disabled”.
dwctl/src/db/handlers/users.rs	Adds decline/backoff fields, filters candidates by retry-after, and implements atomic decline-state transitions/reset.
dwctl/src/api/handlers/payments.rs	Adjusts tests to assert decline state resets when auto top-up is reconfigured.
dwctl/migrations/124_add_auto_topup_decline_backoff.sql	Adds DB columns for soft-decline counter and retry-after timestamp.
dwctl/default_templates/auto_topup_retry.html	New email template for first soft-decline retry/backoff notification.
dwctl/default_templates/auto_topup_disabled.html	New email template for hard-decline or second soft-decline disable notification.
.sqlx/query-f5df0b9ebb99e1e1bba4dbedfe2cf858b9057cfa1bab29f2a9eefdea1423a209.json	SQLx metadata update for resetting auto-topup decline state.
.sqlx/query-f3388664b9d07ca787a450e25b7559c92d8b2ed917ff8313ab75fa411f23ffe2.json	SQLx metadata update (underway task_queue insert).
.sqlx/query-df59ccd8080d8eb0bd1e11ab6e5a2b8fcdfbb3ae2c95a05ccd9f76042320f41a.json	SQLx metadata update for expanded users row shape.
.sqlx/query-de50515666fa9da0b24b0d8db50bd1b03e099ea946c151d821520d1687ad6881.json	SQLx metadata update (underway task_attempt state update).
.sqlx/query-dae0ef841b214a8f0d3981f27aa490d09dd5db2962112316ae892a59fc7f152c.json	SQLx metadata update (underway task insert).
.sqlx/query-d1e63e3a8d4d9be55cb5665760152c590e19fba33ccfde199808971b62e9557a.json	SQLx metadata update (underway task lease/lock query).
.sqlx/query-ccf314910c001d4933326838f507380d36a50d63778ed5589d7ffe0fe9f60db3.json	SQLx metadata update (set search_path to underway).
.sqlx/query-b7f2624732084aa92ef7b7c5b06b71a8284eeb8f735f31af7c038b2cb50a8680.json	SQLx metadata update for applying first soft-decline transition.
.sqlx/query-b211278db4a4733dc2a16dfa56ea390053421a45c8254c3cac118f704810f74f.json	SQLx metadata update (underway task heartbeat).
.sqlx/query-98f5535ef035447135c267ffb61e3ee1eb5c42a9c3038ccbe7b93c92d393c54f.json	SQLx metadata update (underway task completion).
.sqlx/query-98e88f5fc48233b957f267d42a4ea13343d22cfb096ff366d1996cd7894743e8.json	SQLx metadata update (underway task_attempt error_message).
.sqlx/query-98aa8d56e8fb6d28480105370b0b6a6d3e68e0e64bdeeed97a04e7597aa43853.json	SQLx metadata update (pg_try_advisory_xact_lock).
.sqlx/query-9731fc6ba31c2cc1bd00b8e177c44ca85285a2606f9485a9f38a3585ce48b00b.json	SQLx metadata update (underway task heartbeat update).
.sqlx/query-96a0895113d5e00a565fe1ad6e9eb1205b4d36e5eef7e5b096fc707ac107c42e.json	SQLx metadata update (underway task queue rename).
.sqlx/query-9639647f76c40eedcd0649cc5dc0f22809203a0ac8bb32802c7f4db73215a805.json	SQLx metadata update (underway task_schedule select).
.sqlx/query-9158f9abdc7c3f5c2ffaa876bf3871bbdd3fd31b61869e3ccd9e84aa2e058cfc.json	SQLx metadata update (underway task_attempt insert).
.sqlx/query-8c5868495ee0e66b4b6869f74b06e1e3930ef6ab49a5255816564262943ae5a9.json	SQLx metadata update (underway task state update w/ ordering guard).
.sqlx/query-8a6beb878ad5d685281a3eb393e7ef4d30b78f14dafe23916ad08b28c4ccc858.json	SQLx metadata update (underway task_schedule delete).
.sqlx/query-7513f95eb46920c91e32cf7dac34155bbaac4ee4a5011c110df702eab4c8e03f.json	SQLx metadata update (underway task select by job_id).
.sqlx/query-65bf99c8e1eff69c58dbb62e8737abcb9ccd1a3feb5f3be6b4d69801c7e5d900.json	SQLx metadata update for user update query resetting decline backoff state.
.sqlx/query-63da920fcf11b0f6fc6cdb39e0dbcc61c3d70196c12097f6609e405ab8a4e38a.json	SQLx metadata update (underway task TTL cleanup).
.sqlx/query-616e1ecc744bf9e743aaed13db27cc77cc74e7f9c164ca0f8dee9091fd955d83.json	SQLx metadata update (underway task_attempt count).
.sqlx/query-596a1dcc593b0dc55002a47cea8a2e6fa1328977f2a41d01ba092b0854853365.json	SQLx metadata update (underway task delay/state update).
.sqlx/query-5929146ea62ec166d52a248b29116e1cb087e6d93f103478f9c4a87056c0d271.json	SQLx metadata update for disabling auto-topup after second soft decline.
.sqlx/query-56a0f6c70e41d1b89d0b16a4cbbb989aa3da718f76f4f4abf8794e538c4ed2b8.json	SQLx metadata update (underway task_attempt state transition).
.sqlx/query-54d124a54b2bb28f85b3ee9882f1e103d8e690ea0cb5189411834b9d8b246fc4.json	SQLx metadata update (pg_notify).
.sqlx/query-3a4f75221e74290abdb289495f6d35a63e7b1d76b839f2ac9d4aa31d02143866.json	SQLx metadata update for expanded users row shape.
.sqlx/query-336285bd86388c578e7c1bb68f24a3153f9f4af9fd1bfd56660a05195a97efb7.json	SQLx metadata update for disabling auto-topup after hard decline.
.sqlx/query-2effb4373a8f9073e94b51971f1fede91325b4662fb2f9d1f183bda5b307c2f8.json	SQLx metadata update for users_with_auto_topup_enabled filtering/shape.
.sqlx/query-2d6600e95898c5baf3e10780b8110fe0002e6c1056d44cb2534cc47f6e48876e.json	SQLx metadata update (underway bulk task insert via unnest).
.sqlx/query-29dbbb71782841c85e29f1b9830629b5163848c8e2281e17926c22a89fce3428.json	SQLx metadata update for expanded users row shape.
.sqlx/query-27475eb9bcb6ae1ebd88ba22d68d7536bb47555e1f5b7c22db8b25552619e6a8.json	SQLx metadata update (underway task_schedule upsert).
.sqlx/query-0dece29c8db5fc229c7e3ac6ef549526161e370c453d459ddc717ac8c263ec05.json	SQLx metadata update (underway task_attempt completion w/ ordering guard).
.sqlx/query-03c2ac38e6ec2801eed7e5b19076eafcc589f32e2424460b0c345e6c477a1b28.json	SQLx metadata update (create schema underway).

Files not reviewed (32)

• .sqlx/query-03c2ac38e6ec2801eed7e5b19076eafcc589f32e2424460b0c345e6c477a1b28.json: Generated file
• .sqlx/query-0dece29c8db5fc229c7e3ac6ef549526161e370c453d459ddc717ac8c263ec05.json: Generated file
• .sqlx/query-27475eb9bcb6ae1ebd88ba22d68d7536bb47555e1f5b7c22db8b25552619e6a8.json: Generated file
• .sqlx/query-29dbbb71782841c85e29f1b9830629b5163848c8e2281e17926c22a89fce3428.json: Generated file
• .sqlx/query-2d6600e95898c5baf3e10780b8110fe0002e6c1056d44cb2534cc47f6e48876e.json: Generated file
• .sqlx/query-336285bd86388c578e7c1bb68f24a3153f9f4af9fd1bfd56660a05195a97efb7.json: Generated file
• .sqlx/query-3a4f75221e74290abdb289495f6d35a63e7b1d76b839f2ac9d4aa31d02143866.json: Generated file
• .sqlx/query-54d124a54b2bb28f85b3ee9882f1e103d8e690ea0cb5189411834b9d8b246fc4.json: Generated file
• .sqlx/query-56a0f6c70e41d1b89d0b16a4cbbb989aa3da718f76f4f4abf8794e538c4ed2b8.json: Generated file
• .sqlx/query-5929146ea62ec166d52a248b29116e1cb087e6d93f103478f9c4a87056c0d271.json: Generated file
• .sqlx/query-596a1dcc593b0dc55002a47cea8a2e6fa1328977f2a41d01ba092b0854853365.json: Generated file
• .sqlx/query-616e1ecc744bf9e743aaed13db27cc77cc74e7f9c164ca0f8dee9091fd955d83.json: Generated file
• .sqlx/query-63da920fcf11b0f6fc6cdb39e0dbcc61c3d70196c12097f6609e405ab8a4e38a.json: Generated file
• .sqlx/query-7513f95eb46920c91e32cf7dac34155bbaac4ee4a5011c110df702eab4c8e03f.json: Generated file
• .sqlx/query-8a6beb878ad5d685281a3eb393e7ef4d30b78f14dafe23916ad08b28c4ccc858.json: Generated file
• .sqlx/query-8c5868495ee0e66b4b6869f74b06e1e3930ef6ab49a5255816564262943ae5a9.json: Generated file
• .sqlx/query-9158f9abdc7c3f5c2ffaa876bf3871bbdd3fd31b61869e3ccd9e84aa2e058cfc.json: Generated file
• .sqlx/query-9639647f76c40eedcd0649cc5dc0f22809203a0ac8bb32802c7f4db73215a805.json: Generated file
• .sqlx/query-96a0895113d5e00a565fe1ad6e9eb1205b4d36e5eef7e5b096fc707ac107c42e.json: Generated file
• .sqlx/query-9731fc6ba31c2cc1bd00b8e177c44ca85285a2606f9485a9f38a3585ce48b00b.json: Generated file
• .sqlx/query-98aa8d56e8fb6d28480105370b0b6a6d3e68e0e64bdeeed97a04e7597aa43853.json: Generated file
• .sqlx/query-98e88f5fc48233b957f267d42a4ea13343d22cfb096ff366d1996cd7894743e8.json: Generated file
• .sqlx/query-98f5535ef035447135c267ffb61e3ee1eb5c42a9c3038ccbe7b93c92d393c54f.json: Generated file
• .sqlx/query-b211278db4a4733dc2a16dfa56ea390053421a45c8254c3cac118f704810f74f.json: Generated file
• .sqlx/query-b7f2624732084aa92ef7b7c5b06b71a8284eeb8f735f31af7c038b2cb50a8680.json: Generated file
• .sqlx/query-ccf314910c001d4933326838f507380d36a50d63778ed5589d7ffe0fe9f60db3.json: Generated file
• .sqlx/query-d1e63e3a8d4d9be55cb5665760152c590e19fba33ccfde199808971b62e9557a.json: Generated file
• .sqlx/query-dae0ef841b214a8f0d3981f27aa490d09dd5db2962112316ae892a59fc7f152c.json: Generated file
• .sqlx/query-de50515666fa9da0b24b0d8db50bd1b03e099ea946c151d821520d1687ad6881.json: Generated file
• .sqlx/query-df59ccd8080d8eb0bd1e11ab6e5a2b8fcdfbb3ae2c95a05ccd9f76042320f41a.json: Generated file
• .sqlx/query-f3388664b9d07ca787a450e25b7559c92d8b2ed917ff8313ab75fa411f23ffe2.json: Generated file
• .sqlx/query-f5df0b9ebb99e1e1bba4dbedfe2cf858b9057cfa1bab29f2a9eefdea1423a209.json: Generated file

Comments suppressed due to low confidence (1)

dwctl/src/db/handlers/users.rs:376

• Same issue as get(): get_bulk() hard-codes auto_topup_soft_failure_count/auto_topup_retry_after to 0/None, which can hide the real decline backoff state present in the DB row. Prefer selecting these columns (and updating GROUP BY) so the returned User struct is accurate.

[pjb157]

get_by_id() and get_bulk() return UserDBResponse, which deliberately does not expose either decline-backoff field. The private User value here is consumed immediately by From<(Vec<Role>, User)> for UserDBResponse, and that conversion drops both fields, so no internal caller can observe these placeholder values. The auto-top-up scheduler reads the persisted counter through the dedicated AutoTopupUser query in users_with_auto_topup_enabled(), where auto_topup_soft_failure_count is selected from the row and auto_topup_retry_after is applied as an eligibility predicate. Keeping these explicit user queries unchanged avoids widening their SELECT/GROUP BY for state outside their response contract.