feat(sdk): read host-delivered worker secrets from Task.runtimeMetadata (all 4 SDKs)#313
Open
v1r3n wants to merge 24 commits into
Open
feat(sdk): read host-delivered worker secrets from Task.runtimeMetadata (all 4 SDKs)#313v1r3n wants to merge 24 commits into
v1r3n wants to merge 24 commits into
Conversation
…AgentspanSecretsDAO Server-side of host-delivered worker secrets when embedded (agentspan.embedded=true): - Declare each worker tool's credential names on TaskDef.runtimeMetadata at registration (AgentService + AgentCompiler.collectToolCredentials/collectAgentCredentials), extended to the non-worker user-code task kinds (guardrail/callback/stop_when/ gate/instructions/router/graph) via agent-level creds. Gated on EmbeddedMode. - AgentspanSecretsDAO: bridges conductor's SecretsDAO to AgentSpan's encrypted CredentialStoreProvider (scoped to the anonymous/system user), selected by conductor.secrets.type=agentspan; the embedded conductor's RuntimeMetadataResolver resolves declared names at task poll onto the wire-only Task.runtimeMetadata. - Store beans (datasource, master key, schema migrator, store provider) re-gated on conductor.secrets.type=agentspan (defaulted on in application.properties). - Native secret mechanism gated on agentspan.embedded; LLM keys via host AI integration. Tests: WorkerRuntimeMetadataTest, AgentspanSecretsDAOTest, NativeSecretGatingTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ta (all 4 SDKs)
SDK-side of host-delivered worker secrets. Each worker prefers the host-resolved
map on Task.runtimeMetadata (embedded), else falls back to the native
execution-token pull (standalone); the resolved values feed the existing
get_secret()/getCredential accessors.
- Java — WorkerManager.readRuntimeMetadata; create-only worker TaskDef registration
so host-declared runtimeMetadata isn't clobbered.
- Python — _dispatch.py reads task.runtime_metadata; tool_registry/runtime register
create-only.
- TypeScript — worker.ts reads task.runtimeMetadata (structural cast).
- C# — WorkerManager.ReadRuntimeMetadata.
- Client dep bumps + demo_secret_resolution.py.
Tests: ReadRuntimeMetadataTest / EmbeddedTaskDefRegistrationTest (Java),
test_runtime_metadata / test_embedded_taskdef_registration (Python),
RuntimeMetadataReadTests (C#), worker/credentials tests (TS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…module It is a standalone-server-only @ControllerAdvice (its only consumers are the server-module masking tests). Co-locating it in conductor-agentspan-server removes it from the embeddable library, where it never applied. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lobal) store Remove userId from the CredentialStoreProvider SPI (get/set/delete/list). The EncryptedDbCredentialStoreProvider keeps its credentials_store schema (the user_id column stays, part of the (user_id, name) PK) but always reads/writes under the fixed 00000000-0000-0000-0000-000000000000 owner, so lookups are effectively by name. - SPI: drop userId from all four methods. - EncryptedDbCredentialStoreProvider: DEFAULT_USER_ID constant in the SQL. - Consumers drop the userId argument: CredentialResolutionService, SecretController (currentUserId() retained for audit logging), CredentialEnvSeeder, AgentspanSecretsDAO. - Per-user isolation is gone; delete the obsolete isolation tests (WorkerCredentialsIntegrationTest, AgentspanAIModelProviderPerUserTest) and update the remaining tests to the no-userId signatures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follows the CredentialStoreProvider single-scope change: resolve is now resolve(name), and the userId plumbing that fed it is removed. - CredentialResolutionService.resolve(name) — drop userId (logs no longer reference it). - WorkerController / CredentialAwareHttpTask / CredentialAwareMcpService: call resolve(name). HTTP/MCP keep the execution-token presence check as an auth gate, but the header resolution methods no longer take userId (resolveHeadersForUser -> resolveHeaders on HTTP). - AgentspanAIModelProvider: resolveUserApiKey/resolveUserCredential resolve globally; the now-dead extractUserIdFromTaskContext + the ExecutionTokenService dependency (field + constructor param) are removed (ExecutionTokenService itself is unchanged and still used by WorkerController et al.). - Tests updated to the no-userId signatures; drop dead per-user constants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The execution-token mechanism gated the native out-of-process worker credential pull, but provides no real auth/tenancy in agentspan today (the credential store is global, userId already removed). Remove it; real security will be added separately. Credential delivery relies on host-resolved TaskDef.runtimeMetadata. DELETE: - ExecutionTokenService, WorkerController (/api/workers/secrets), ResolveRequest - Token-only tests: ExecutionTokenServiceTest, AgentEventListenerTokenRevocationTest, WorkerCredentialsTest MODIFY: - AgentService: drop the mint block + extractDeclaredCredentials/collectCredentialsRecursive; keep preallocatedExecutionId/setWorkflowId (host time-based IDs). The Lombok @requiredargsconstructor now serves tests too. - AgentEventListener: drop the token-revocation path. - AgentExceptionHandler: drop the 401 token handler. - Remove __agentspan_ctx__ end-to-end (only carried the token): AgentService writer, the forwarders in AgentCompiler/MultiAgentCompiler/ToolCompiler/JavaScriptBuilder/ PlanAndCompileTask, and the strips in AgentChatCompleteTaskMapper/PlanAndCompileTask. - Update NativeSecretGatingTest, AgentServiceTokenTest, MultiAgentCompilerTest, PlanAndCompileTaskTest. Survives: CredentialResolutionService (+ CredentialNotFoundException), CredentialAware Http/Mcp (resolve directly), credentialMasterKey/MasterKeyConfig. Consequence (server-only now): no worker credential-pull path remains; delivery is via TaskDef.runtimeMetadata. Coordinated follow-up in SDK PR #313 must remove the native token-pull fetchers and rely on the runtimeMetadata read-path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….secrets.NAME}
Rewrite credential placeholders in HTTP/MCP/planner-context headers from ${NAME} to
conductor's native ${workflow.secrets.NAME}, resolved by ParametersUtils.substituteSecrets
wire-only at task hand-off (poll / async+sync system tasks) against the configured
SecretsDAO — AgentspanSecretsDAO by default. Plaintext is never persisted.
- ToolCompiler.rewriteCredentialPlaceholders + MultiAgentCompiler planner-context: always
emit ${workflow.secrets.NAME} (drop the embedded/#{ split; remove EmbeddedMode usage).
- Delete the now-redundant #{NAME} resolvers: CredentialAwareHttpTask +
CredentialAwareHttpTaskConfig (reverts to conductor's built-in @component("HTTP") HttpTask)
and CredentialAwareMcpService (reverts to conductor-ai's base @component MCPService). Their
tests are removed.
- CredentialResolutionService stays (still used by AgentspanAIModelProvider for LLM keys).
- Bonus: planner-context secrets now resolve in standalone too (previously only embedded),
since AgentspanSecretsDAO is the default SecretsDAO.
Tests updated: ToolCompilerTest, MultiAgentCompilerTest, EnrichToolsScriptTest assert the
${workflow.secrets.NAME} form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (plaintext leak)
The ${workflow.secrets.NAME} rewrite baked the reference into the enrich INLINE task's
'expression' input; conductor's substituteSecrets resolves any such reference in an INLINE
task's input to PLAINTEXT at that task's hand-off, and the plaintext then persisted via the
script's dynamicTasks output into the forked HTTP/MCP tasks' inputParameters — defeating the
wire-only guarantee. A secret containing a quote/backslash also broke the GraalJS script.
Two-form design:
- Transport form #{NAME} (dots allowed for JSONPath): inert to BOTH ParametersUtils passes,
rides safely through every INLINE hop (script text, $.mcpConfig/$.apiConfig data, toolSpecs).
ToolCompiler.rewriteCredentialPlaceholders now emits this everywhere headers are escaped.
- Real reference ${workflow.secrets.NAME}: applied ONLY at final placement into a real task's
inputParameters, where conductor defers it at binding and resolves wire-only at that task's
own hand-off:
* Java secretRefHeaders() for statically built tasks (LIST_MCP_TOOLS inputs, api spec-fetch
HTTP tasks — both variants).
* JS _sec() at dynamic-task emission in both enrich scripts (HTTP, MCP, API headers). The
'$' + '{workflow.secrets.' split keeps the contiguous reference literal out of the script
SOURCE so it can never be early-resolved.
Also widens both CREDENTIAL_PLACEHOLDER patterns to allow dotted names (${GCP_SVC.project_id}
-> ${workflow.secrets.GCP_SVC.project_id}, supported by conductor's resolveWithOptionalJsonPath),
fixing the dotted-JSONPath regression.
Tests (fail-first validated): leak tripwires assert the enrich script source never contains
'${workflow.secrets.' (ToolCompilerTest, EnrichToolsScriptTest); GraalJS emission tests assert
#{NAME} configs emit ${workflow.secrets.NAME} on dynamic HTTP/MCP task inputs, incl. dotted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…re module They test ToolCompiler and JavaScriptBuilder, which live in conductor-agentspan — tests belong with the module of the class under test. The core module's test classpath already carries GraalJS for the script-execution harness. No content changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…odule Tests belong with the module of the class under test. Moves the 29 pure unit tests that exercise conductor-agentspan classes (compilers, normalizers, models, services, utils, ai providers, ProviderController) plus SkillNormalizerTest's /skills JSON fixtures (used by it alone). Stays in conductor-agentspan-server: the @SpringBootTest/MockMvc integration tests, everything exercising server-module classes (encrypted store, seeder, migrator, master key, AgentspanSecretsDAO, CredentialMaskingResponseAdvice, skill package store), and ServerSmokeTest. Pure git mv — no content changes. Suites: core 523, server 159, all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ci-server.yml (Server CI): agentspan library + server module tests and the Go CLI tests. Triggers on server/**, cli/**. - ci-sdk.yml (SDK CI): the four SDK unit-test jobs plus all e2e suites — e2e is part of the SDK pipeline. Builds its own server JAR (-x test; server correctness is Server CI's job) so the workflows share no artifacts. Triggers on sdk/**, server/**, cli/** — server changes still prove the SDKs keep working via e2e. Job bodies are unchanged from the old ci.yml; this is a reorganization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
runtimeMetadata is now the only worker credential-delivery path (the native execution-token pull was removed). Standalone resolves it end-to-end already: the built-in conductor core's RuntimeMetadataResolver + AgentspanSecretsDAO (the default SecretsDAO) resolve declared names at poll from the encrypted store. Gating the declaration on EmbeddedMode left standalone with no credential delivery at all (review finding #3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…anch The endpoint-free SDK read-path requires the new server (TaskDef.runtimeMetadata declaration in both modes + AgentspanSecretsDAO + wire-only resolution; the native /api/workers/secrets pull is deleted). Stacking on the server branch makes this PR's e2e run against that server.
The server resolved-secret delivery is now exclusively the wire-only Task.runtimeMetadata (the native POST /api/workers/secrets pull and the execution token were removed server-side). The SDK no longer calls any endpoint for secrets. - _task_compat.py shim: registers runtime_metadata on conductor-python's Task model (swagger_types/attribute_map/ctor/property) — published client releases drop the unregistered JSON key at deserialization. Remove once the client ships the field. - _dispatch.py: _read_runtime_metadata() is the only source; deleted WorkerCredentialFetcher wiring + _extract_execution_token; declared-but- undelivered names log a warning and surface via get_secret(). - Framework adapters (langchain/langgraph/claude_agent_sdk): same swap. - resolve_credentials() public helper: reads the task's runtime metadata instead of calling the removed endpoint. - Deleted credentials/fetcher.py + test_fetcher.py; reworked delivery tests to runtime_metadata; new shim regression test proves the wire value survives ApiClient.deserialize. 1689 unit tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The wire-only Task.runtimeMetadata (resolved by the conductor core at poll) is the only credential-delivery path — the native POST /workers/secrets pull and the execution token were removed server-side. - credentials.ts: context reduced to the resolved map; getCredential() reads it directly (missing name -> CredentialNotFoundError); deleted extractExecutionToken + resolveCredentials (endpoint pull). runWithCredentialContext/setCredentialContext now take the resolved map. - worker.ts: reads task.runtimeMetadata (structural cast — JS keeps unknown JSON keys, so no client-lib change needed); declared-but-undelivered names log a warning; the context wraps every execution. - Tests reworked around runtimeMetadata delivery; the concurrency-isolation regression test drops its handler barrier (delivered secrets serialize handler bodies via the env-injection mutex by design). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eflective) The wire-only Task.runtimeMetadata (resolved by the conductor core at poll) is the only credential-delivery path — the native POST /workers/secrets pull and the execution token were removed server-side. - WorkerManager.readRuntimeMetadata reads the field REFLECTIVELY: the published conductor-client (5.1.0, now pinned from Maven Central — mavenLocal dropped) does not carry Task.runtimeMetadata yet, so the read compiles today, returns empty against the current client, and lights up automatically once conductor-oss/java-sdk ships the accessor. (Older clients also drop the unregistered JSON key at deserialization, so no data is lost by degrading.) - Deleted WorkerCredentialFetcher + extractExecutionToken + the terminal-error fallback; declared-but-undelivered names log a warning and surface via ToolContext.getCredential. - ReadRuntimeMetadataTest reworked around a Task subclass exposing the accessor + a published-client degradation case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops ci-sdk.yml (SDK unit tests + all e2e suites). Server CI (ci-server.yml) remains as the only workflow, covering the agentspan library/server modules and the Go CLI. SDK test coverage moves out of per-PR CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix docs build breakage from the submodule swap: the new repo nests the agent docs under docs/agents/ instead of docs/, so retarget the docs/python-sdk symlink there to keep mkdocs --strict passing. Also fetch submodules in the docs CI/deploy checkouts, since the docs symlink now resolves into submodule content.
Plain git clone now leaves sdk/python empty since it moved to a submodule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SDK-side of host-delivered worker secrets. Split out from #307 (SDK half). Server half: companion PR.
Each worker prefers the host-resolved secret map on
Task.runtimeMetadata(embedded mode), else falls back to the native execution-token pull (standalone). Resolved values feed the existingget_secret()/getCredentialaccessors unchanged.WorkerManager.readRuntimeMetadata; workerTaskDefregistration is create-only so the host-declaredruntimeMetadataisn't clobbered._dispatch.pyreadstask.runtime_metadata;tool_registry/runtimeregister create-only.worker.tsreadstask.runtimeMetadata(structural cast).WorkerManager.ReadRuntimeMetadata.demo_secret_resolution.py.Tests
ReadRuntimeMetadataTest/EmbeddedTaskDefRegistrationTest(Java),test_runtime_metadata/test_embedded_taskdef_registration(Python),RuntimeMetadataReadTests(C#), worker/credentials tests (TS).Known follow-up
The
conductor-pythonclient model does not yet carryTask.runtime_metadata, so the wire field is dropped on deserialize and the Python read-path is inert until the client ships the field (a compat shim is the interim fix). Java/TS/C# read via model additions / structural casts.🤖 Split from #307 with Claude Code