Skip to content

fix activity webhook returning 200 before chain work finishes #990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mooncitydev wants to merge 1 commit into
base: main
Choose a base branch
from
+145 −137
Open

fix activity webhook returning 200 before chain work finishes #990

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 145 additions & 137 deletions server/hooks/activity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ export default new Hono().post(
),
}),
}),
validatorHook({ code: "bad alchemy", status: 200, debug }),
validatorHook({ code: "bad alchemy", debug }),
Copy link
Copy Markdown

@devin-ai-integration devin-ai-integration Bot Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Removed status: 200 from webhook validatorHook causes Alchemy to retry on validation failures

The status: 200 parameter was removed from the validatorHook call, so it now defaults to status: 400 (server/utils/validatorHook.ts:17). When Alchemy sends a webhook payload that fails validation, the endpoint now returns 400 instead of 200. Alchemy retries webhooks on non-200 responses, so malformed payloads will be retried repeatedly until the retry limit is exhausted — wasting resources and generating duplicate Sentry errors.

Every other webhook hook in the codebase explicitly passes status: 200 to acknowledge the webhook even when validation fails:

  • server/hooks/bridge.ts:92 — status: 200
  • server/hooks/block.ts:108 — status: 200
  • server/hooks/manteca.ts:147 — status: 200
  • server/hooks/persona.ts:207 — status: 200
Suggested change
validatorHook({ code: "bad alchemy", debug }),
validatorHook({ code: "bad alchemy", status: 200, debug }),
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

),
async (c) => {
const payload = c.req.valid("json");
Expand Down Expand Up @@ -172,149 +172,157 @@ export default new Hono().post(
},
});
const client = createPublicClient({ chain, transport, rpcSchema: rpcSchema<RpcSchema>() }).extend(trace);
Promise.allSettled(
[...pokes].map(([account, { publicKey, factory, source, assets }]) =>
continueTrace({ sentryTrace, baggage }, () =>
withScope((scope) =>
startSpan(
{ name: "account activity", op: "exa.activity", attributes: { account }, forceTransaction: true },
async (span) => {
scope.setUser({ id: account });
const isDeployed = !!(await client.getCode({ address: account }));
scope.setTag("exa.new", !isDeployed);
if (!isDeployed) {
try {
await createWalletClient({ chain, transport, account: keeper.account })
.extend((wallet) => extender(wallet, { publicClient: client, traceClient: client }))
.exaSend(
{ name: "create account", op: "exa.account", attributes: { account } },
{
address: factory,
functionName: "createAccount",
args: [0n, [decodePublicKey(publicKey, bytesToBigInt)]],
abi: exaAccountFactoryAbi,
},
chain.id === exaChain.id ? undefined : { fees: "auto" },
);
track({ event: "AccountFunded", userId: account, properties: { source } });
} catch (error: unknown) {
span.setStatus({ code: SPAN_STATUS_ERROR, message: "account_failed" });
throw error;
try {
const activityResults = await Promise.allSettled(
[...pokes].map(([account, { publicKey, factory, source, assets }]) =>
continueTrace({ sentryTrace, baggage }, () =>
withScope((scope) =>
startSpan(
{ name: "account activity", op: "exa.activity", attributes: { account }, forceTransaction: true },
async (span) => {
scope.setUser({ id: account });
const isDeployed = !!(await client.getCode({ address: account }));
scope.setTag("exa.new", !isDeployed);
if (!isDeployed) {
try {
await createWalletClient({ chain, transport, account: keeper.account })
.extend((wallet) => extender(wallet, { publicClient: client, traceClient: client }))
.exaSend(
{ name: "create account", op: "exa.account", attributes: { account } },
{
address: factory,
functionName: "createAccount",
args: [0n, [decodePublicKey(publicKey, bytesToBigInt)]],
abi: exaAccountFactoryAbi,
},
chain.id === exaChain.id ? undefined : { fees: "auto" },
);
track({ event: "AccountFunded", userId: account, properties: { source } });
} catch (error: unknown) {
span.setStatus({ code: SPAN_STATUS_ERROR, message: "account_failed" });
throw error;
}
}
}
if (chain.id !== exaChain.id) {
span.setStatus({ code: SPAN_STATUS_OK });
return;
}
if (assets.has(ETH)) assets.delete(WETH);
const results = await Promise.allSettled(
[...assets]
.filter((asset) => marketsByAsset.has(asset) || asset === ETH)
.map(async (asset) =>
withRetry(
() =>
keeper
.exaSend(
{ name: "poke account", op: "exa.poke", attributes: { account, asset } },
{
address: account,
abi: [...exaPluginAbi, ...upgradeableModularAccountAbi, ...auditorAbi, ...marketAbi],
...(asset === ETH
? { functionName: "pokeETH" }
: {
functionName: "poke",
args: [marketsByAsset.get(asset)!], // eslint-disable-line @typescript-eslint/no-non-null-assertion
}),
},
{ ignore: ["NoBalance()"] },
)
.then((receipt) => {
if (receipt) return receipt;
throw new Error("NoBalance()");
}),
{
delay: 2000,
retryCount: 5,
shouldRetry: ({ error }) => {
if (error instanceof Error && error.message === "NoBalance()") return true;
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
return true;
if (chain.id !== exaChain.id) {
span.setStatus({ code: SPAN_STATUS_OK });
return;
}
if (assets.has(ETH)) assets.delete(WETH);
const pokeResults = await Promise.allSettled(
[...assets]
.filter((asset) => marketsByAsset.has(asset) || asset === ETH)
.map(async (asset) =>
withRetry(
() =>
keeper
.exaSend(
{ name: "poke account", op: "exa.poke", attributes: { account, asset } },
{
address: account,
abi: [...exaPluginAbi, ...upgradeableModularAccountAbi, ...auditorAbi, ...marketAbi],
...(asset === ETH
? { functionName: "pokeETH" }
: {
functionName: "poke",
args: [marketsByAsset.get(asset)!], // eslint-disable-line @typescript-eslint/no-non-null-assertion
}),
},
{ ignore: ["NoBalance()"] },
)
.then((receipt) => {
if (receipt) return receipt;
throw new Error("NoBalance()");
}),
{
delay: 2000,
retryCount: 5,
shouldRetry: ({ error }) => {
if (error instanceof Error && error.message === "NoBalance()") return true;
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
return true;
},
},
},
),
),
),
);
for (const result of results) {
if (result.status === "fulfilled") continue;
if (result.reason instanceof Error && result.reason.message === "NoBalance()") {
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureScope.addEventProcessor((event) => {
if (event.exception?.values?.[0]) event.exception.values[0].type = "NoBalance";
return event;
);
for (const result of pokeResults) {
if (result.status === "fulfilled") continue;
if (result.reason instanceof Error && result.reason.message === "NoBalance()") {
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureScope.addEventProcessor((event) => {
if (event.exception?.values?.[0]) event.exception.values[0].type = "NoBalance";
return event;
});
captureException(result.reason, {
level: "warning",
fingerprint: ["{{ default }}", "NoBalance"],
});
});
captureException(result.reason, {
level: "warning",
fingerprint: ["{{ default }}", "NoBalance"],
});
});
continue;
continue;
}
span.setStatus({ code: SPAN_STATUS_ERROR, message: "poke_failed" });
throw result.reason;
}
span.setStatus({ code: SPAN_STATUS_ERROR, message: "poke_failed" });
throw result.reason;
}
autoCredit(account)
.then(async (auto) => {
span.setAttribute("exa.autoCredit", auto);
if (!auto) return;
const credential = await database.query.credentials.findFirst({
where: eq(credentials.account, account),
columns: {},
with: {
cards: {
columns: { id: true, mode: true },
where: inArray(cards.status, ["ACTIVE", "FROZEN"]),
autoCredit(account)
.then(async (auto) => {
span.setAttribute("exa.autoCredit", auto);
if (!auto) return;
const credential = await database.query.credentials.findFirst({
where: eq(credentials.account, account),
columns: {},
with: {
cards: {
columns: { id: true, mode: true },
where: inArray(cards.status, ["ACTIVE", "FROZEN"]),
},
},
},
});
if (!credential || credential.cards.length === 0) return;
const card = credential.cards[0];
span.setAttribute("exa.card", card?.id);
if (card?.mode !== 0) return;
await database.update(cards).set({ mode: 1 }).where(eq(cards.id, card.id));
span.setAttribute("exa.mode", 1);
sendPushNotification({
userId: account,
headings: t("Card mode changed"),
contents: t("Credit mode activated"),
}).catch((error: unknown) => captureException(error));
})
.catch((error: unknown) => captureException(error));
span.setStatus({ code: SPAN_STATUS_OK });
},
});
if (!credential || credential.cards.length === 0) return;
const card = credential.cards[0];
span.setAttribute("exa.card", card?.id);
if (card?.mode !== 0) return;
await database.update(cards).set({ mode: 1 }).where(eq(cards.id, card.id));
span.setAttribute("exa.mode", 1);
sendPushNotification({
userId: account,
headings: t("Card mode changed"),
contents: t("Credit mode activated"),
}).catch((error: unknown) => captureException(error));
})
.catch((error: unknown) => captureException(error));
Comment on lines +270 to +296
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The autoCredit promise chain is not awaited, which means the parent function can resolve before the logic finishes. Additionally, when an expected database record (like a card) is not found, the code should throw an error rather than returning silently. This indicates a system or data integrity issue, making a 5xx-level error more appropriate.

                  await autoCredit(account)
                    .then(async (auto) => {
                      span.setAttribute("exa.autoCredit", auto);
                      if (!auto) return;
                      const credential = await database.query.credentials.findFirst({
                        where: eq(credentials.account, account),
                        columns: {},
                        with: {
                          cards: {
                            columns: { id: true, mode: true },
                            where: inArray(cards.status, ["ACTIVE", "FROZEN"]),
                          },
                        },
                      });
                      if (!credential || credential.cards.length === 0) {
                        throw new Error("Expected card record not found for account");
                      }
                      const card = credential.cards[0];
                      span.setAttribute("exa.card", card.id);
                      if (card.mode !== 0) return;
                      await database.update(cards).set({ mode: 1 }).where(eq(cards.id, card.id));
                      span.setAttribute("exa.mode", 1);
                      sendPushNotification({
                        userId: account,
                        headings: t("Card mode changed"),
                        contents: t("Credit mode activated"),
                      }).catch((error: unknown) => captureException(error));
                    })
                    .catch((error: unknown) => captureException(error));
References
  1. Throw an error when an expected database record (like a card) is not found. This indicates a system or data integrity issue, not a client-side error that can be fixed by the user, so a 5xx-level error is more appropriate than a 4xx.

span.setStatus({ code: SPAN_STATUS_OK });
},
),
),
),
).catch((error: unknown) => {
withScope((scope) => {
scope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
throw error;
}),
),
)
.then((results) => {
getActiveSpan()?.setStatus(
results.every((result) => result.status === "fulfilled")
? { code: SPAN_STATUS_OK }
: { code: SPAN_STATUS_ERROR, message: "activity_failed" },
);
})
.catch((error: unknown) => captureException(error));
return c.json({});
).catch((error: unknown) => {
withScope((scope) => {
scope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
throw error;
}),
),
);
getActiveSpan()?.setStatus(
activityResults.every((result) => result.status === "fulfilled")
? { code: SPAN_STATUS_OK }
: { code: SPAN_STATUS_ERROR, message: "activity_failed" },
);
const rejected = activityResults.filter(
(result): result is PromiseRejectedResult => result.status === "rejected",
);
if (rejected.length > 0) {
return c.json({ code: "activityFailed", failed: rejected.length }, 500);
}
Comment on lines +318 to +320
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid retrying whole activity batch after partial success

Returning 500 when any account pipeline rejects causes Alchemy to retry the entire webhook payload, not just the failed account. In this handler, side effects (e.g., sendPushNotification in the transfer loop and successful exaSend calls for other accounts) can already be committed before this response is decided, and there is no webhook-event idempotency check, so a single failing account can repeatedly duplicate notifications/transactions for accounts that already succeeded.

Useful? React with 👍 / 👎.

return c.json({});
} catch (error: unknown) {
captureException(error);
return c.json({ code: "activityError" }, 500);
}
Comment on lines +175 to +325
Copy link
Copy Markdown

@devin-ai-integration devin-ai-integration Bot Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Behavioral shift: webhook now blocks on processing and returns 500 on failure

The old code was fire-and-forget: Promise.allSettled(...) was not awaited, the handler immediately returned c.json({}) with a 200 status, and processing happened in the background. The new code awaits Promise.allSettled and returns 500 if any account processing fails (server/hooks/activity.ts:318-319). This means:

  1. The HTTP response is delayed until all account creation, poke retries (up to 5 retries × 2s delay each), and error handling complete.
  2. Alchemy receives 500 on processing failures and will retry the entire webhook, potentially re-triggering account creation and poke operations for accounts that already succeeded.

This may be intentional (the commit message says "await alchemy activity hook work before returning 200"), but the retry implications are worth considering — especially since Promise.allSettled collects both fulfilled and rejected results, so partial failures (e.g., 3/5 accounts succeed, 2 fail) will return 500 and Alchemy will retry all 5.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

},
);

Expand Down