AI Agent Execution & Governance Layer
Exactly-once execution + runtime safety + agent control plane.
Website β’ Docs β’ API Reference β’ Examples
The problem: AI agents are non-deterministic. They retry failed calls, re-run tools, crash mid-execution, and replay events. This causes duplicate payments, repeated emails, and inconsistent state.
The solution: OnceOnly sits between your AI and the real world, guaranteeing:
- Exactly-once execution. Same input = same result.
- Crash safety. Worker dies? Resume safely.
- Retry safety. Replays are deduplicated.
- Budget control. Cap spend per agent/hour/day.
- Permission enforcement. Allowlist and blocklist tools.
- Kill switch. Disable rogue agents instantly.
- Forensic audit. Full action history.
This is not just idempotency. This is an AI agent control plane.
npm i @onceonly/onceonly-sdkimport { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({
apiKey: process.env.ONCEONLY_API_KEY!
});
const result = await client.checkLock({
key: "webhook:stripe:evt_123",
ttl: 3600
});
if (result.duplicate) {
return { status: "already_processed" };
}
// Process webhook exactly once.That is it. Your webhook is now idempotent.
import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
export async function stripeWebhook(eventId: string) {
const lock = await client.checkLock({
key: `stripe:${eventId}`,
ttl: 7200
});
if (lock.duplicate) {
return { status: "ok" };
}
await handlePaymentSucceeded(eventId);
return { status: "processed" };
}import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
await client.gov.upsertPolicy({
agent_id: "billing-agent",
max_actions_per_hour: 200,
max_spend_usd_per_day: 50,
allowed_tools: ["stripe.charge", "send_email"],
blocked_tools: ["delete_user"]
});
const result = await client.ai.runTool({
agentId: "billing-agent",
tool: "stripe.charge",
args: { amount: 9999, currency: "usd" },
spendUsd: 0.5
});
if (result.allowed) {
console.log("Charged", result.result);
} else {
console.log("Blocked", result.policyReason);
}import { OnceOnly, idempotentAi } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
const sendWelcomeEmail = idempotentAi(
client,
async (userId: string) => {
await emailService.send({
to: getUserEmail(userId),
template: "welcome"
});
return { sent: true };
},
{
keyFn: (userId) => `welcome:email:${userId}`,
ttl: 86400
}
);
await sendWelcomeEmail("user_123");
await sendWelcomeEmail("user_123");
await sendWelcomeEmail("user_123");import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
const runId = `run_demo_${Math.floor(Date.now() / 1000)}`;
const key = `ai:job:debug:${runId}`;
await client.postEvent({
runId,
type: "sdk_debug",
status: "start",
message: "run debug demo started from ts sdk"
});
await client.aiRun({ key, runId });
const timeline = await client.getRunTimeline(runId, 200, 0);
console.log(timeline);I want...
- Idempotent webhook/cron/job:
checkLock({ key, ttl, meta }) - Long-running server job:
ai.runAndWait({ key, ttl, metadata }) - Start/attach run without polling:
aiRun({ key, runId }) - Governed tool call:
ai.runTool({ agentId, tool, args, spendUsd, runId }) - Local side-effect exactly once:
ai.runFn(key, fn, opts) - Add custom run event:
postEvent({ runId, type, ... }) - Fetch run timeline:
getRunTimeline(runId, limit, offset) - Update notification preferences:
updateNotifications({ ... }) - Decorator version:
idempotent(...)oridempotentAi(...)
Async alias methods also exist
checkLockAsyncaiRunAsyncaiRunAndWaitAsyncpostEventAsyncgetRunTimelineAsyncupdateNotificationsAsync
// examples/ai/agent_full_flow_no_onceonly.ts
const decision = await llmDecide();
const payload = { tool: decision.tool, args: decision.args };
await callTool(payload);
await callTool(payload); // duplicate charge on retry// examples/ai/agent_full_flow_onceonly.ts
const res = await client.ai.runTool({
agentId: "billing-agent",
tool: "stripe.charge",
args: { amount: 9999, currency: "usd", user_id: "u_42" },
spendUsd: 0.5
});
if (res.allowed) {
console.log(res.result);
} else {
console.log("Blocked", res.policyReason);
}Why this matters:
- Prevents duplicate external side effects on retries.
- Enforces budgets and permissions at runtime.
- Produces auditable decision and execution traces.
| Feature | Description | Typical Use Case |
|---|---|---|
checkLock() |
Fast idempotency primitive | Webhooks, cron jobs, workers |
ai.runAndWait() |
Server-side long-running jobs | Reports, generation pipelines |
ai.runTool() |
Governed tool execution | Agent tool calls with budgets/permissions |
ai.runFn() |
Local exactly-once side effects | Email, billing, write operations |
idempotent() / idempotentAi() |
Decorator API | Simple function-level dedup |
gov.upsertPolicy() |
Policy management | Budget caps, tool allow/block |
gov.disableAgent() |
Kill switch | Emergency stop |
gov.agentLogs() |
Audit trail | Forensics and compliance |
postEvent() + getRunTimeline() |
Run debugging timeline | Operational debugging and support |
OnceOnly provides 5 layers of safety:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β L5: Agent Governance (policies, kill switch, audit) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β L4: Decorator Runtime (`idempotentAi`) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β L3: Local Side-Effects (`ai.runFn`) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β L2: AI Job Orchestration (`ai.runAndWait`) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β L1: Idempotency Primitive (`checkLock`) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The TypeScript SDK now supports two debug-first methods:
postEvent(...)->POST /v1/eventsgetRunTimeline(runId, ...)->GET /v1/runs/{run_id}
runId is automatically propagated by the SDK:
- Key mode: into
metadata.run_id - Tool mode: into
args.run_id
await client.ai.run({
key: "ai:job:report:42",
runId: "run_report_42"
});
await client.ai.run({
agentId: "debug-agent",
tool: "send_email",
args: { to: "ops@example.com" },
runId: "run_report_42"
});import { ApiError } from "@onceonly/onceonly-sdk";
try {
await client.ai.run({
agentId: "debug-agent",
tool: "this_tool_must_not_exist",
runId: "run_fail_demo_123"
});
} catch (err) {
if (err instanceof ApiError) {
console.error(err.statusCode, err.detail);
}
}
console.log(await client.getRunTimeline("run_fail_demo_123", 200, 0));Look for tool_result and run_finished events to identify failure reason quickly.
import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
async function chargeOrder(orderId: string, userId: string, amountCents: number) {
const key = `payment:${orderId}:charge`;
const lock = await client.checkLock({
key,
ttl: 3600,
meta: { orderId, userId, amountCents, source: "checkout" }
});
if (lock.duplicate) {
return { status: "already_processed", key };
}
const governed = await client.ai.runTool({
agentId: "billing-agent",
tool: "stripe.charge",
args: { order_id: orderId, user_id: userId, amount: amountCents, currency: "usd" },
spendUsd: 0.02
});
if (!governed.allowed) {
return { status: "blocked", reason: governed.policyReason };
}
return { status: "charged", result: governed.result };
}OnceOnly governance controls are available under client.gov:
- Policy management:
upsertPolicy,policyFromTemplate,getPolicy,listPolicies - Tool controls:
createTool,toggleTool,deleteTool,listTools - Agent controls:
disableAgent,enableAgent - Observability:
agentLogs,agentMetrics
await client.gov.upsertPolicy({
agent_id: "billing-agent",
max_actions_per_hour: 200,
max_spend_usd_per_day: 50,
allowed_tools: ["stripe.charge", "send_email"],
blocked_tools: ["delete_user"]
});
await client.gov.disableAgent("billing-agent", "manual safety stop");
await client.gov.enableAgent("billing-agent", "resume operations");const policy = await client.gov.policyFromTemplate(
"new-agent",
"moderate", // strict|moderate|permissive|read_only|support_bot
{
max_actions_per_hour: 300,
blocked_tools: ["delete_user"]
}
);
console.log(policy);await client.gov.disableAgent("rogue-agent", "suspicious behavior detected");
await client.gov.enableAgent("rogue-agent", "re-enabled after investigation");const logs = await client.gov.agentLogs("billing-agent", 100);
for (const log of logs) {
console.log(`${String(log.ts)} ${log.tool} ${log.decision}`);
console.log(` reason=${log.policyReason ?? log.reason}`);
console.log(` risk=${log.riskLevel} spend=${log.spendUsd}`);
}
const metrics = await client.gov.agentMetrics("billing-agent", "day");
console.log(metrics.totalActions, metrics.blockedActions, metrics.totalSpendUsd, metrics.topTools);import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
export async function idempotentSendEmailTool(input: { to: string; subject: string; body: string }) {
return client.ai.runFn(
`agent:email:${input.to}:${input.subject}`,
async () => {
await emailService.send(input);
return { sent: true, to: input.to };
},
{ ttl: 3600, metadata: { tool: "send_email", to: input.to } }
);
}import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({ apiKey: process.env.ONCEONLY_API_KEY! });
app.post("/webhooks/stripe", async (req, res) => {
const event = req.body as { id: string; type?: string };
const lock = await client.checkLock({
key: `stripe:${event.id}`,
ttl: 7200,
meta: { type: event.type ?? "unknown" }
});
if (lock.duplicate) {
return res.send({ status: "duplicate" });
}
await processStripeEvent(event);
return res.send({ status: "processed" });
});Integration examples in this repo:
import { OnceOnly } from "@onceonly/onceonly-sdk";
const client = new OnceOnly({
apiKey: process.env.ONCEONLY_API_KEY!,
baseUrl: "https://api.onceonly.tech/v1", // optional
timeoutMs: 5000, // HTTP timeout
failOpen: true, // graceful degradation for checkLock
maxRetries429: 3, // auto-retry on 429/5xx/network
retryBackoffSec: 0.5, // initial backoff (seconds)
retryMaxBackoffSec: 10.0 // max backoff (seconds)
});- Core:
GET /v1/me,POST /v1/me/notifications,GET /v1/usage,GET /v1/usage/all,GET /v1/events,GET /v1/metrics,POST /v1/events,GET /v1/runs/{run_id} - Idempotency:
POST /v1/check-lock - AI Jobs:
POST /v1/ai/run,GET /v1/ai/status,GET /v1/ai/result - AI Lease (local side-effects):
POST /v1/ai/lease,POST /v1/ai/extend,POST /v1/ai/complete,POST /v1/ai/fail,POST /v1/ai/cancel - Governance (policies):
POST /v1/policies/{agent_id},POST /v1/policies/{agent_id}/from-template,GET /v1/policies,GET /v1/policies/{agent_id} - Governance (agents):
POST /v1/agents/{agent_id}/disable,POST /v1/agents/{agent_id}/enable,GET /v1/agents/{agent_id}/logs,GET /v1/agents/{agent_id}/metrics - Tools Registry:
POST /v1/tools,GET /v1/tools,GET /v1/tools/{tool},POST /v1/tools/{tool}/toggle,DELETE /v1/tools/{tool}
const prefs = await client.updateNotifications({
emailNotificationsEnabled: true,
toolErrorNotificationsEnabled: true,
runFailureNotificationsEnabled: false
});
console.log(prefs);const result = await client.checkLock({
key: "order:12345",
ttl: 3600,
meta: { user_id: 42 }
});
if (result.duplicate) {
console.log("Duplicate", result.firstSeenAt);
} else {
console.log("First time - proceed");
}// Long-running server job
const job = await client.ai.runAndWait({
key: "report:monthly:2026-04",
ttl: 1800,
timeout: 120,
pollMin: 1,
pollMax: 10,
metadata: { month: "2026-04" },
runId: "run_report_2026_04"
});
console.log(job);
// Governed tool runner
const toolRes = await client.ai.runTool({
agentId: "billing-agent",
tool: "stripe.charge",
args: { amount: 9999, currency: "usd" },
runId: "run_charge_001",
spendUsd: 0.5
});
if (toolRes.allowed) console.log(toolRes.result);
else console.log("Blocked:", toolRes.policyReason);
// Local function execution
const fnRes = await client.ai.runFn(
"email:welcome:user123",
async () => sendEmail(),
{
ttl: 300,
waitOnConflict: true,
timeout: 60,
errorCode: "email_failed"
}
);
console.log(fnRes);
// Run debug APIs
await client.postEvent({ runId: "run_report_2026_04", type: "note", message: "manual investigation marker" });
console.log(await client.getRunTimeline("run_report_2026_04", 200, 0));run({ key, ttl, metadata, runId })run({ agentId, tool, args, spendUsd, runId })status(key),result(key),wait(key, opts),runAndWait(opts)runTool({ agentId, tool, args, spendUsd, runId })runFn(key, fn, opts)- Low-level lease lifecycle:
lease,extend,complete,fail,cancel
| Mode | Use When | Call | Result Type |
|---|---|---|---|
| Job (server-side) | Long-running tasks | ai.runAndWait({ key: ... }) |
AiResult |
| Tool (governed) | Agent tool execution | ai.runTool({ agentId, tool, ... }) |
AiToolResult |
| Local side-effects | Your code does the work | ai.runFn(key, fn, ...) |
AiResult |
type AiToolResult = {
ok: boolean;
allowed: boolean;
decision: string; // "executed" | "blocked" | "dedup"
policyReason?: string;
riskLevel?: string;
result?: Record<string, unknown>;
};
type AiResult = {
ok: boolean;
status: string; // "completed" | "failed" | "in_progress"
key: string;
result?: Record<string, unknown>;
errorCode?: string;
doneAt?: string;
};const res = await client.ai.runTool({
agentId: "billing-agent",
tool: "stripe.refund",
args: { charge_id: "ch_123", amount: 500 },
spendUsd: 0.2
});
if (res.allowed) {
console.log("OK", res.result);
} else {
console.log("BLOCKED", res.policyReason);
}import { idempotent, idempotentAi } from "@onceonly/onceonly-sdk";
const safePayment = idempotent(client, async (orderId: string) => {
await charge(orderId);
return { ok: true };
}, { keyPrefix: "payment", ttl: 3600 });
const onboardUser = idempotentAi(client, async (userId: string) => {
await createAccount(userId);
await sendWelcome(userId);
return { onboarded: true };
}, {
keyFn: (userId) => `onboard:${userId}`,
ttl: 600,
metadataFn: (userId) => ({ userId })
});- Policies:
upsertPolicy,policyFromTemplate,getPolicy,listPolicies - Tools registry:
createTool,listTools,getTool,toggleTool,deleteTool - Agent controls:
disableAgent,enableAgent - Observability:
agentLogs,agentMetrics
const tool = await client.gov.createTool({
name: "send_email",
url: "https://example.com/tools/send_email",
scope_id: "global",
auth: { type: "hmac_sha256", secret: "shared_secret" },
timeout_ms: 15000,
max_retries: 2,
enabled: true
});
await client.gov.toggleTool("send_email", false, "global");
console.log(tool);Tools registry limits by plan:
- Free: 1 tool
- Starter: 20 tools
- Pro: 100 tools
- Agency: 1000 tools
Rules and expectations:
namemust be unique perscope_idand match^[a-zA-Z0-9_.:-]+$scope_idnamespaces tools (for exampleglobaloragent:billing-agent)auth.typecurrently supportshmac_sha256- Your tool endpoint should verify HMAC and be idempotent on its side
Typed errors mirror backend semantics:
UnauthorizedError(401 and auth-related 403)OverLimitError(402)RateLimitError(429)ValidationError(422)ApiError(other non-2xx, including feature gating and business 403)
import { OverLimitError, RateLimitError } from "@onceonly/onceonly-sdk";
try {
await client.checkLock({ key: "order:123", ttl: 3600 });
} catch (err) {
if (err instanceof OverLimitError) {
console.error("Upgrade required", err.detail);
} else if (err instanceof RateLimitError) {
console.error("Retry after seconds", err.retryAfterSec);
} else {
throw err;
}
}export ONCEONLY_API_KEY="once_live_..."
export ONCEONLY_BASE_URL="https://api.onceonly.tech/v1" # optionalconst client = new OnceOnly({
apiKey: process.env.ONCEONLY_API_KEY!,
baseUrl: "https://api.onceonly.tech/v1",
timeoutMs: 5000,
failOpen: true,
maxRetries429: 0,
retryBackoffSec: 0.5,
retryMaxBackoffSec: 5.0
});failOpen: true applies only to checkLock network/timeouts/5xx paths.
Fail-open never applies to:
- 401/403 auth errors
- 402 usage limit
- 422 validation errors
- 429 rate limits (retry/backoff path)
If your app already has custom fetch instrumentation, pass fetchImpl to reuse it:
const client = new OnceOnly({
apiKey: process.env.ONCEONLY_API_KEY!,
fetchImpl: globalThis.fetch
});Cause: invalid/disabled API key.
// Wrong
const client = new OnceOnly({ apiKey: "sk_test_..." });
// Correct
const client = new OnceOnly({ apiKey: "once_live_..." });Cause: exceeded monthly quota for plan limits.
const usage = await client.usage("make");
console.log(`Used: ${usage.usage} / ${usage.limit}`);Cause: too many requests per second.
const client = new OnceOnly({
apiKey: process.env.ONCEONLY_API_KEY!,
maxRetries429: 3,
retryBackoffSec: 0.5,
retryMaxBackoffSec: 10
});Cause: non-deterministic key.
// Wrong
const key1 = `order:${crypto.randomUUID()}`;
// Correct
const key2 = `order:${orderId}`;Cause: policy restrictions.
const logs = await client.gov.agentLogs("my-agent", 10);
for (const log of logs) {
if (log.decision === "blocked") {
console.log(`Blocked ${log.tool}: ${log.policyReason ?? log.reason}`);
}
}
await client.gov.upsertPolicy({
agent_id: "my-agent",
allowed_tools: ["tool_a", "tool_b", "tool_c"]
});If timeline contains only your custom debug event and no run_started / run_finished, your worker likely has not picked this run yet.
Check:
- Queue and worker are running.
- Worker is subscribed to the correct environment/namespace.
- The key/run_id pair is reused when you expect the same run context.
If you get ApiError with detail.error = "tool_not_found":
- Verify tool is registered in expected scope (
agent:{id}orglobal). - Verify tool name is exactly the one registered.
- Inspect timeline
tool_resultevent for final resolution details.
runId is validated client-side and cannot be blank or whitespace.
// β
Use deterministic, business-meaningful keys
const key = `payment:${orderId}:${userId}`;
// β
Pick TTL by retry window and processing latency
const ttl = 3600;
// β
Attach metadata for traceability
const meta = { userId, source: "web", traceId };
// β
Handle duplicate path explicitly
if (lock.duplicate) return cachedResponse;
// β
Use decorators for simple function wrappers
const safeFn = idempotent(client, myFn, { keyPrefix: "task" });// β Non-deterministic key prevents dedup
const key = `order:${crypto.randomUUID()}`;
// β Very short TTL may let retries through
const ttl = 1;
// β Ignore duplicate result and run side-effect anyway
await client.checkLock({ key, ttl });
await chargeCard();
// β Swallow all errors silently
try { await client.checkLock({ key, ttl }); } catch {}| Feature | Free | Starter | Pro | Agency |
|---|---|---|---|---|
| Core Idempotency | ||||
checkLock() |
1K/mo | 20K/mo | 200K/mo | 2M/mo |
ai.runAndWait() |
3K/mo | 100K/mo | 1M/mo | 10M/mo |
| Agent Governance | ||||
gov.upsertPolicy() |
β Entitlement-based | β Entitlement-based | β Entitlement-based | β Entitlement-based |
gov.agentLogs() |
β | β | β | β |
gov.agentMetrics() |
β | β | β | β |
gov.disableAgent() (Kill switch) |
β | β | β | β |
gov.enableAgent() |
β | β | β | β |
| Policy Features | ||||
Budget/action limits (max_spend_usd_per_day, max_actions_per_hour) |
β | β | β | β |
Tool blocklist (blocked_tools) |
β | β | β | β |
Tool whitelist (allowed_tools) |
β | β | β | β |
Per-tool limits (max_calls_per_tool) |
β | β | β | β |
Pricing rules (pricing_rules) |
β | β | β | β |
| Tools Registry | ||||
gov.createTool() + tool CRUD |
β | β | β | β |
| Max tools per account | 1 | 20 | 100 | 1000 |
Note: limits are enforced by backend plan entitlements, independent of SDK language.
| Plan | checkLock (make) |
ai (runs) |
Default TTL | Max TTL | Tools Registry Limit |
|---|---|---|---|---|---|
| Free | 1K / month | 3K / month | 60s | 1h | 1 tool |
| Starter | 20K / month | 100K / month | 1h | 24h | 20 tools |
| Pro | 200K / month | 1M / month | 6h | 7d | 100 tools |
| Agency | 2M / month | 10M / month | 24h | 30d | 1000 tools |
Notes:
- Free plan enforces hard monthly stop.
- Starter/Pro/Agency continue after threshold with notifications.
Before going live:
- Use production API key (
once_live_...) - Set stable keys and correct TTLs
- Enable 429 retries (
maxRetries429) - Add metadata for debugging/correlation
- Configure governance policies for each agent
- Test fail-open path for
checkLock - Review timeline/audit logs regularly
- Add alerts for blocked actions and failures
- Main index: examples/README.md
- Debug timeline: examples/ai/run_debug_timeline.ts
- Debug failure: examples/ai/run_debug_failure.ts
- Full flow without OnceOnly: examples/ai/agent_full_flow_no_onceonly.ts
- Full flow with OnceOnly: examples/ai/agent_full_flow_onceonly.ts
Run with tsx:
npm run example:debug:timeline
npm run example:debug:failure
npm run example:basicnpm install
npm run check
npm test
npm run release:check- Website: onceonly.tech
- Documentation: docs.onceonly.tech
- API Reference: docs.onceonly.tech/reference/idempotency
- TypeScript SDK Repo: github.com/OnceOnly-Tech/onceonly-typescript
- npm Package: @onceonly/onceonly-sdk
- Support: support@onceonly.tech
MIT License - see LICENSE for details.
Contributions are welcome. Open an issue or pull request in the TypeScript repository.
npm testRelease validation:
npm run release:checkIf OnceOnly helps your project, star the repo on GitHub.
Questions? Open an issue or email support@onceonly.tech.
