-
Notifications
You must be signed in to change notification settings - Fork 161
fix: set final statusMessage before storeTaskResult (#638) #677
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
6002585
a7a89a1
29884d2
44074f9
a9974e9
08dc039
66f4238
92acf50
74eb5db
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| name: Smoke test npm package | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| npm_tag: | ||
| description: NPM dist-tag to install (e.g. "beta", "latest"). Ignored when version is set. | ||
| required: false | ||
| type: string | ||
| default: latest | ||
| version: | ||
| description: Exact version to install (e.g. "1.2.3"). Takes precedence over npm_tag. | ||
| required: false | ||
| type: string | ||
| default: "" | ||
|
|
||
| jobs: | ||
| smoke_test: | ||
| name: Smoke test published package | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Use Node.js | ||
| uses: actions/setup-node@v6 | ||
| with: | ||
| node-version: 22 | ||
|
|
||
| - name: Install published package | ||
| uses: nick-fields/retry@v4 | ||
| env: | ||
| PKG_VERSION: ${{ inputs.version || inputs.npm_tag }} | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 3 | ||
| retry_wait_seconds: 30 | ||
| command: | | ||
|
Check warning on line 35 in .github/workflows/_smoke_test_npm_package.yaml
|
||
| mkdir -p smoke-test && cd smoke-test | ||
| npm init -y | ||
| npm install "@apify/actors-mcp-server@${PKG_VERSION}" @apify/mcpc | ||
|
|
||
| - name: Smoke test MCP server via mcpc | ||
| working-directory: smoke-test | ||
| run: | | ||
| echo '{"mcpServers":{"dev":{"command":"npx","args":["actors-mcp-server","--tools","docs,search-actors,fetch-actor-details","--telemetry-enabled","false"]}}}' > .mcp.json | ||
| npx mcpc connect .mcp.json:dev @dev | ||
| npx mcpc @dev ping | ||
| npx mcpc --json @dev tools-list | jq -e 'length > 0' | ||
| npx mcpc --json @dev tools-call search-actors keywords:="web scraper" | jq -e '.content[0].text' | ||
| npx mcpc --json @dev tools-call fetch-actor-details actor:="apify/web-scraper" | jq -e '.content[0].text' | ||
| npx mcpc --json @dev tools-call search-apify-docs query:="web scraping" docSource:="apify" | jq -e '.content[0].text' | ||
| npx mcpc --json @dev tools-call fetch-apify-docs url:="https://docs.apify.com/platform/actors" | jq -e '.content[0].text' | ||
| npx mcpc @dev close | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -119,9 +119,16 @@ | |
| "tag": "latest" | ||
| } | ||
|
|
||
| smoke_test: | ||
| name: Smoke test published package | ||
| needs: [ release_metadata, publish_to_npm ] | ||
| uses: ./.github/workflows/_smoke_test_npm_package.yaml | ||
| with: | ||
| version: ${{ needs.release_metadata.outputs.version_number }} | ||
|
|
||
| bump_dependency_in_internal_repo: | ||
| name: Bump dependency in apify-mcp-server-internal | ||
| needs: [ release_metadata, publish_to_npm ] | ||
|
Check failure on line 131 in .github/workflows/manual_release_stable.yaml
|
||
|
Comment on lines
129
to
131
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 The new smoke_test job was added to gate releases, but bump_dependency_in_internal_repo still has needs: [release_metadata, publish_to_npm], so it runs in parallel with the smoke test rather than after it. If the smoke test fails (broken npm package), the internal-repo dependency bump has already started and will create a PR pinning the broken version. Adding smoke_test to the needs list of bump_dependency_in_internal_repo fixes this. Extended reasoning...What the bug is and how it manifests The PR introduces a new smoke_test job (lines 122-127 in manual_release_stable.yaml) that installs the just-published npm package and runs a series of mcpc commands to verify it is functional. The intent is that this acts as a quality gate before the release is propagated to the internal repository. However, bump_dependency_in_internal_repo (line 129) has the same needs: [release_metadata, publish_to_npm] as smoke_test, so GitHub Actions schedules both jobs to run concurrently as soon as publish_to_npm finishes. The specific code path that triggers it In .github/workflows/manual_release_stable.yaml, after publish_to_npm completes, GitHub Actions fans out to any job whose needs list is fully satisfied. Both smoke_test and bump_dependency_in_internal_repo only require [release_metadata, publish_to_npm], so they both start simultaneously. If the smoke test catches a broken release and fails, the bump job is already mid-execution: checking out the internal repo, running npm install, and opening a PR. Why existing code does not prevent it There is no needs: smoke_test dependency on bump_dependency_in_internal_repo. GitHub Actions job ordering is determined solely by the needs graph; there is no implicit sequencing. The smoke_test job cannot cancel a sibling job that is already running. What the impact would be If a broken version is caught by the smoke test, bump_dependency_in_internal_repo will have already opened a pull request in apify/apify-mcp-server-internal pinning the broken version. Although this is only a PR (not an auto-merge), it creates noise, risks accidental merges by reviewers who trust CI to gate quality, and defeats the entire purpose of adding the smoke test quality gate. Step-by-step proof
How to fix it Change the needs list of bump_dependency_in_internal_repo to include smoke_test: bump_dependency_in_internal_repo:
name: Bump dependency in apify-mcp-server-internal
needs: [ release_metadata, publish_to_npm, smoke_test ]This ensures the bump only runs after the smoke test has confirmed the published package is functional. Note that on_master.yaml correctly adds smoke_test as a leaf job with no downstream dependents, so that workflow does not have this issue. |
||
| runs-on: ubuntu-latest | ||
| env: | ||
| GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }} | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,12 +23,60 @@ | |
| } | ||
|
|
||
| /** | ||
| * Removes newlines, trims whitespace, and strips surrounding quotes from env values. | ||
| * CI secrets often include trailing newlines or quotes that break HTTP headers and URLs. | ||
| * Strips control characters, trims whitespace, and removes surrounding double quotes. | ||
| * CI secrets often contain trailing newlines or invisible control chars that break HTTP headers. | ||
| */ | ||
| export function sanitizeEnvValue(value?: string): string | undefined { | ||
| if (value == null) return value; | ||
| return value.replace(/[\r\n]/g, '').trim().replace(/^"|"$/g, ''); | ||
| // eslint-disable-next-line no-control-regex | ||
| return value.replace(/[\x00-\x08\x0a-\x1f\x7f]/g, '').trim().replace(/^"|"$/g, ''); | ||
| } | ||
|
|
||
| /** | ||
| * Env vars used in HTTP headers (API keys, tokens, URLs). | ||
| * | ||
| * Why in-place? The phoenix-otel OTel exporter reads PHOENIX_API_KEY directly | ||
| * from process.env (inside getEnvApiKey()) and passes it to node:http, which | ||
| * throws ERR_INVALID_CHAR on any control characters. We can't intercept that | ||
| * read, so we sanitize process.env itself before any library loads. | ||
| */ | ||
| const ENV_KEYS_TO_SANITIZE = [ | ||
| 'OPENROUTER_API_KEY', | ||
| 'OPENROUTER_BASE_URL', | ||
| 'PHOENIX_API_KEY', | ||
| 'PHOENIX_BASE_URL', | ||
| ]; | ||
|
|
||
| /** | ||
| * Redact a value for safe logging: shows first 4 and last 4 chars, masks the rest. | ||
| * Fully masks short values (≤ 8 chars) to prevent reconstruction from the log line. | ||
| * Returns '(empty)' for empty strings, '(unset)' for undefined/null. | ||
| */ | ||
| function redact(value?: string | null): string { | ||
| if (value == null) return '(unset)'; | ||
| if (value.length === 0) return '(empty)'; | ||
| if (value.length <= 6) return `*** (${value.length} chars)`; | ||
| return `${value.slice(0, 3)}***${value.slice(-3)} (${value.length} chars)`; | ||
| } | ||
|
Check warning on line 60 in evals/shared/config.ts
|
||
|
Comment on lines
+50
to
+60
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 The JSDoc for the new Extended reasoning...What the bug is: The JSDoc comment for the newly-introduced The specific code path: The function is called inside Why existing code doesn't prevent it: There are no automated tests that assert the format of the redacted output (only that Impact: No functional impact—the actual redaction logic is self-consistent and secrets are still protected. However, the incorrect documentation could mislead a security reviewer auditing information-leakage guarantees of the CI log output. For example: a 7-character API key fragment would expose 6 of its 7 characters (3+3), whereas the JSDoc implies it would be fully masked (≤ 8 chars threshold). Step-by-step proof: Take an 8-character value like How to fix: Correct the JSDoc to match the implementation: change "first 4 and last 4 chars" to "first 3 and last 3 chars" and change "≤ 8 chars" to "≤ 6 chars". |
||
|
|
||
| /** | ||
| * Sanitize env vars in-place on process.env and log redacted values for CI debugging. | ||
| * Must be called before any library reads these values. | ||
| */ | ||
| export function sanitizeProcessEnv(): void { | ||
| for (const key of ENV_KEYS_TO_SANITIZE) { | ||
| const raw = process.env[key]; | ||
| if (raw != null) { | ||
| const sanitized = sanitizeEnvValue(raw)!; | ||
| const changed = raw !== sanitized; | ||
| process.env[key] = sanitized; | ||
| // eslint-disable-next-line no-console | ||
| console.log(`env ${key}: ${redact(sanitized)}${changed ? ' (sanitized)' : ''}`); | ||
| } else { | ||
| // eslint-disable-next-line no-console | ||
| console.log(`env ${key}: ${redact(raw)}`); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /** | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 The smoke test installs
@apify/mcpcwithout a version pin (npm install "@apify/actors-mcp-server@${PKG_VERSION}" @apify/mcpc), so it always installs the latest published version. If@apify/mcpcreleases a breaking change (renamed CLI flags, changed output format), the smoke test would fail even whenactors-mcp-serveritself is perfectly healthy, creating false CI failures that could block releases. Pinning to a known-good version (e.g.@apify/mcpc@0.2.0or@apify/mcpc@^0.2.0) would make the test deterministic.Extended reasoning...
What the bug is and how it manifests
In
.github/workflows/_smoke_test_npm_package.yaml(line 35), the install command is:@apify/mcpchas no version constraint, so npm always resolves and installs the latest published version. This means the smoke-test environment is non-deterministic: different CI runs can silently pull in different versions of the test harness.The specific code path that triggers it
The smoke test creates a fresh
smoke-test/directory, runsnpm init -y, and installs both packages from scratch. Because@apify/mcpcis not pinned (unlike the main package under test, which is pinned to${PKG_VERSION}), any new release of@apify/mcpcpublished between two CI runs will be picked up automatically. The subsequent smoke commands—npx mcpc connect,npx mcpc @dev ping,npx mcpc --json @dev tools-list, etc.—all depend on the CLI API of whichever version was installed.Why existing code does not prevent it
The project's own
package.jsonpins@apify/mcpcat^0.2.0indevDependencies, demonstrating that the team already values version stability for this tool. However, the smoke-test workflow bypassespackage.jsonentirely: it sets up a standalone directory with its ownpackage.json(vianpm init -y) and installs packages directly. The retry mechanism (max_attempts: 3) mitigates transient npm registry failures but does nothing about determinism across versions.What the impact would be
If
@apify/mcpc0.3.0 (hypothetically) renames theconnectsubcommand, changes the--jsonflag semantics, or alters thetools-listoutput format, the smoke test would fail at step 2 or 3—even though@apify/actors-mcp-serveritself is perfectly healthy. This would block the beta or stable release pipeline. Because the failure comes from the test harness rather than the package under test, it could be confusing to diagnose, especially in time-sensitive release windows.How to fix it
Pin
@apify/mcpcto the same range already used indevDependencies:npm install "@apify/actors-mcp-server@${PKG_VERSION}" "@apify/mcpc@^0.2.0"Or use an exact version for maximum reproducibility:
npm install "@apify/actors-mcp-server@${PKG_VERSION}" "@apify/mcpc@0.2.0"The refutation argument—that
@apify/mcpcis an Apify-owned package and won't introduce uncoordinated breaking changes—is reasonable and lowers the practical risk. However, the same logic would argue against pinning anything, which contradicts the team's existing practice. Even for internal tools, a pinned test harness version is a best-practice that prevents CI surprise failures and makes version-upgrade decisions explicit and intentional.