Skip to content

feat: add integration tests#3

Merged
skevetter merged 4 commits into
mainfrom
feat/e2e-suite
Jul 25, 2026
Merged

feat: add integration tests#3
skevetter merged 4 commits into
mainfrom
feat/e2e-suite

Conversation

@skevetter

@skevetter skevetter commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Adds a Ginkgo/Gomega e2e suite and CI (macOS, since the provider binary is darwin-only).

  • integration label: manifest generation + provider-binary error handling; no OrbStack needed. Runs in CI on macos-15 (validated green).
  • vm label: full workspace lifecycle on a real OrbStack machine; validated locally 3/3. Not in CI — OrbStack can't start on GitHub-hosted macOS runners (orbctl start panics: vm config get max ipa size: unsupported), so it's local/self-hosted only (task test:e2e:vm).

Single squashed commit; supersedes #1/#2 (repo ruleset blocks pushing follow-ups to an existing PR branch).

Summary by CodeRabbit

  • New Features

    • Added end-to-end integration testing for provider setup, workspace creation, command execution, and cleanup.
    • Added convenient tasks for running integration and full virtual-machine test suites.
  • Documentation

    • Added guidance for running end-to-end tests locally, including environment requirements.
  • Chores

    • Added automated macOS integration test execution for pull requests and manual runs.
    • Updated the Go toolchain and test dependencies.

Add a Ginkgo/Gomega e2e suite and a Tests workflow. Both labels run on macOS
since the provider binary is darwin-only:
- integration: manifest generation and provider-binary error handling; no
  OrbStack required. Runs in CI on macos-15.
- vm: full workspace lifecycle on a real OrbStack machine. Not run in CI
  (OrbStack cannot start on GitHub-hosted runners, which lack the Apple
  Virtualization capability it needs); run it locally via task test:e2e:vm.

Adds Taskfile test:e2e / test:e2e:vm targets and an alpine workspace fixture.
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@skevetter, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 30 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 86e76898-e778-4c87-ac2d-71660f34cec4

📥 Commits

Reviewing files that changed from the base of the PR and between 5d94726 and 3146591.

📒 Files selected for processing (4)
  • .github/workflows/tests.yaml
  • Taskfile.yml
  • e2e/e2e_suite_test.go
  • e2e/tests/integration/integration.go
📝 Walkthrough

Walkthrough

Adds a macOS-focused Ginkgo E2E suite for provider configuration, failure validation, and OrbStack workspace lifecycle tests, with Taskfile commands, documentation, fixtures, dependency updates, and pull-request CI automation.

Changes

OrbStack E2E testing

Layer / File(s) Summary
E2E test foundation
go.mod, e2e/e2e_suite_test.go, e2e/tests/integration/integration.go
Adds Ginkgo/Gomega dependencies, the E2E runner, provider artifact generation, isolated Devsy state, and platform-specific CLI installation helpers.
Provider and workspace validation
e2e/tests/integration/integration.go
Validates generated provider configuration and provider failure cases, then exercises provider registration, workspace creation, SSH command execution, and forced deletion.
Local and CI E2E execution
Taskfile.yml, .github/workflows/tests.yaml, e2e/README.md, e2e/fixtures/..., .gitignore
Adds labeled local test tasks, macOS pull-request automation, execution documentation, a workspace devcontainer fixture, and an ignored E2E binary directory.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2ETests
  participant DevsyCLI
  participant OrbStackProvider
  participant OrbStack
  E2ETests->>DevsyCLI: add provider from provider.yaml
  DevsyCLI->>OrbStackProvider: create workspace
  OrbStackProvider->>OrbStack: provision machine workspace
  E2ETests->>DevsyCLI: execute workspace command
  DevsyCLI->>OrbStackProvider: connect through workspace SSH
  E2ETests->>DevsyCLI: delete workspace
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: adding integration and e2e tests, plus CI support.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@skevetter
skevetter marked this pull request as ready for review July 25, 2026 19:42
@skevetter skevetter changed the title test: e2e suite + macOS CI feat: add integration tests Jul 25, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/tests.yaml:
- Around line 20-23: Update the setup Go step using actions/setup-go to prevent
cache writes for pull_request events by setting cache-write to evaluate false
for pull requests and true otherwise, while preserving caching for trusted
workflow events.

In `@e2e/e2e_suite_test.go`:
- Around line 1-13: Add an e2e build constraint to the TestRunE2ETests runner in
e2e_suite_test.go so it is excluded from default go test ./... execution. Update
the E2E test commands or scripts to pass -tags=e2e, preserving the existing
Ginkgo runner behavior when explicitly running the E2E suite.

In `@e2e/tests/integration/integration.go`:
- Around line 116-123: The integration suite currently hardcodes workspaceID as
"devsy-provider-orbstack", risking collisions and unintended deletion. Update
the BeforeAll setup and all related workspace operations to generate a unique
per-run workspace ID, then consistently reuse that value in provider setup,
assertions, and cleanup, including any later --force deletion.
- Around line 95-99: Update the “should fail init when orbctl is missing” test
to verify providerBinary() exists and is executable before running it, then
validate that cmd.Run() reaches the provider and produces the expected
missing-orbctl failure rather than accepting any execution error. Preserve the
ORBSTACK_PATH=/nonexistent/orbctl setup and use the test’s existing assertion
conventions.
- Around line 55-67: Update the resource handling in the integration test around
the HTTP response and binary file creation: defer resp.Body.Close immediately
after confirming the response was obtained, before status assertions, and defer
out.Close immediately after os.OpenFile succeeds, before io.Copy. Preserve the
existing assertions while ensuring cleanup runs when any later assertion or I/O
operation aborts execution.
- Around line 51-68: Update setupDevsyCLI to download a specific immutable
release artifact instead of the mutable releases/latest URL, then verify the
downloaded bytes against the repository’s pinned checksum or signature before
writing bin/devsy. Abort on verification failure and only execute the binary
after successful verification, preserving the existing platform-specific
artifact naming and execution flow.
- Around line 125-127: Update the DeferCleanup callback around the workspace
deletion command to report failures instead of discarding the error returned by
exec.Command(...).Run(). Ensure the cleanup remains idempotent by avoiding a
second delete attempt or treating an already-deleted workspace as successful.

In `@go.mod`:
- Line 90: Update the module graph entries for github.com/docker/docker and
google.golang.org/grpc to patched, non-vulnerable releases, or remove the
dependencies if they are no longer required. Ensure go.mod and go.sum are
consistent and no vulnerable versions remain in the resolved dependency graph.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5703bd0a-dcd8-4c67-a905-2b61cf2e71c2

📥 Commits

Reviewing files that changed from the base of the PR and between b783c94 and 5d94726.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (8)
  • .github/workflows/tests.yaml
  • .gitignore
  • Taskfile.yml
  • e2e/README.md
  • e2e/e2e_suite_test.go
  • e2e/fixtures/workspace/.devcontainer/devcontainer.json
  • e2e/tests/integration/integration.go
  • go.mod

Comment on lines +20 to +23
- name: setup Go
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6
with:
go-version-file: go.mod

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow =="
cat -n .github/workflows/tests.yaml | sed -n '1,80p'

echo
echo "== package manifests =="
for f in go.mod package.json pyproject.toml; do
  if [ -f "$f" ]; then
    echo "-- $f --"
    sed -n '1,40p' "$f"
  fi
done

echo
echo "== search for cache settings in workflows =="
rg -n "setup-go|cache:|pull-request|pull_request|workflow_dispatch|permissions:" .github/workflows || true

Repository: devsy-org/devsy-provider-orbstack

Length of output: 4225


🌐 Web query:

actions/setup-go v6 cache input default true pull request cache poisoning

💡 Result:

In actions/setup-go v6, the cache input defaults to true, which enables automatic caching of Go modules and build outputs [1][2]. Regarding cache poisoning concerns, while the default cache: true configuration enables both restoration and saving of the cache, you can mitigate risks using the following mechanisms: 1. New Read-Only Cache Controls: The actions/setup-go action has introduced a cache-write input (defaulting to true) [3]. By setting cache-write: ${{ github.event_name != 'pull_request' }}, you can allow the action to restore the cache while preventing it from saving (and potentially overwriting) the cache during pull request workflows [3]. This is a recommended best practice for untrusted PR environments [3][4]. 2. Platform-Level Security: As of June 2026, GitHub now automatically issues read-only cache tokens to the default branch for workflow events that can be triggered without repository write permissions (such as certain pull_request and workflow_run contexts) [5]. This provides a native layer of protection against cache poisoning by restricting the ability of untrusted workflows to modify the cache used by more privileged, trusted workflows [5]. If you are concerned about security in sensitive workflows, explicitly setting cache: false or using the cache-write input to disable writes for pull request events are effective ways to prevent cache poisoning [3][4].

Citations:


Disable setup-go cache writes for PRs.

This workflow runs untrusted pull_request events, and setup-go v6 enables caching by default. Set cache-write: ${{ github.event_name != 'pull_request' }}, or disable setup-go caching entirely with cache: false.

🧰 Tools
🪛 zizmor (1.26.1)

[error] 21-21: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): enables caching by default

(cache-poisoning)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/tests.yaml around lines 20 - 23, Update the setup Go step
using actions/setup-go to prevent cache writes for pull_request events by
setting cache-write to evaluate false for pull requests and true otherwise,
while preserving caching for trusted workflow events.

Source: Linters/SAST tools

Comment thread e2e/e2e_suite_test.go
Comment on lines +51 to +68
url := fmt.Sprintf(
"https://github.com/devsy-org/devsy/releases/latest/download/devsy-%s-%s",
goruntime.GOOS, goruntime.GOARCH,
)
resp, err := client.Get(url) //nolint:gosec // fixed release URL
framework.ExpectNoError(err)
framework.ExpectEqual(resp.StatusCode, http.StatusOK)
defer func() { _ = resp.Body.Close() }()

framework.ExpectNoError(os.MkdirAll("bin", 0o750))
binPath := filepath.Join("bin", "devsy")
//nolint:gosec // fixed path; the devsy binary needs the executable bit
out, err := os.OpenFile(binPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0o755)
framework.ExpectNoError(err)
_, err = io.Copy(out, resp.Body)
framework.ExpectNoError(err)
framework.ExpectNoError(out.Close())
framework.ExpectNoError(exec.Command(binPath, "--version").Run()) //nolint:gosec // fixed path

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file excerpt =="
sed -n '1,120p' e2e/tests/integration/integration.go

echo
echo "== related files/usages =="
git ls-files | rg 'e2e/tests/integration|integration\.go|Makefile|go.mod|go\.sum'
echo
rg -n "releases/latest/download|github.com/devsy-org/devsy|exec\\.Command\\(|os\\.OpenFile|Checksum|SHA|signature|Pinning|PIN" .

Repository: devsy-org/devsy-provider-orbstack

Length of output: 7169


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== repository references to devsy integration binary download =="
python3 - <<'PY'
from pathlib import Path
terms = [
    "github.com/devsy-org/devsy/releases/latest/download/devsy-",
    "releases/latest/download/devsy-",
    "devsy",
]
for p in Path(".").glob("**/*.go"):
    if not p.is_file():
        continue
    try:
        text = p.read_text(errors="replace")
    except Exception:
        continue
    lines = text.splitlines()
    for term in terms:
        for i,l in enumerate(lines, 1):
            if term in l:
                start=max(1,i-8); end=min(len(lines),i+12)
                print(f"\n--- {p}:{i} ---")
                for j in range(start,end+1):
                    print(f"{j:5}: {lines[j-1]}")
                print()
PY

echo "== file excerpt =="
sed -n '1,120p' e2e/tests/integration/integration.go

echo
echo "== candidate checksum/signature/download pins =="
rg -n "releases/latest/download|checksumsum|sha256|sha1|integrity|provenance|cosign|cosign\.verify|gh verify|sigstore|gpg|gpgv|signed|Signature|checksum" e2e tests || true

Repository: devsy-org/devsy-provider-orbstack

Length of output: 33668


Pin and verify the CLI binary before executing it.

In setupDevsyCLI() (lines 50-68), the test resolves releases/latest/download/devsy-..., writes the response body to bin/devsy, and immediately executes it. releases/latest is mutable and HTTPS only protects transport; use a pinned release artifact and verify a pinned checksum or signature before writing/running the binary.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/tests/integration/integration.go` around lines 51 - 68, Update
setupDevsyCLI to download a specific immutable release artifact instead of the
mutable releases/latest URL, then verify the downloaded bytes against the
repository’s pinned checksum or signature before writing bin/devsy. Abort on
verification failure and only execute the binary after successful verification,
preserving the existing platform-specific artifact naming and execution flow.

Comment on lines +55 to +67
resp, err := client.Get(url) //nolint:gosec // fixed release URL
framework.ExpectNoError(err)
framework.ExpectEqual(resp.StatusCode, http.StatusOK)
defer func() { _ = resp.Body.Close() }()

framework.ExpectNoError(os.MkdirAll("bin", 0o750))
binPath := filepath.Join("bin", "devsy")
//nolint:gosec // fixed path; the devsy binary needs the executable bit
out, err := os.OpenFile(binPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0o755)
framework.ExpectNoError(err)
_, err = io.Copy(out, resp.Body)
framework.ExpectNoError(err)
framework.ExpectNoError(out.Close())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Register resource cleanup before assertions can abort.

resp.Body.Close() is deferred only after the status assertion, and out.Close() is reached only after a successful copy. Failed assertions or I/O can therefore leave descriptors open. Defer each close immediately after the corresponding successful open.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/tests/integration/integration.go` around lines 55 - 67, Update the
resource handling in the integration test around the HTTP response and binary
file creation: defer resp.Body.Close immediately after confirming the response
was obtained, before status assertions, and defer out.Close immediately after
os.OpenFile succeeds, before io.Copy. Preserve the existing assertions while
ensuring cleanup runs when any later assertion or I/O operation aborts
execution.

Comment on lines +95 to +99
ginkgo.It("should fail init when orbctl is missing", func() {
cmd := exec.Command(providerBinary(), "init") //nolint:gosec // built provider path
cmd.Env = append(cmd.Environ(), "ORBSTACK_PATH=/nonexistent/orbctl")
framework.ExpectError(cmd.Run())
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Ensure this test actually executes the provider binary.

cmd.Run() returns an error when providerBinary() does not exist, so this test can pass without invoking the provider at all. Assert that the binary exists first, and/or validate the expected missing-orbctl error output.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/tests/integration/integration.go` around lines 95 - 99, Update the
“should fail init when orbctl is missing” test to verify providerBinary() exists
and is executable before running it, then validate that cmd.Run() reaches the
provider and produces the expected missing-orbctl failure rather than accepting
any execution error. Preserve the ORBSTACK_PATH=/nonexistent/orbctl setup and
use the test’s existing assertion conventions.

Comment thread e2e/tests/integration/integration.go Outdated
Comment on lines +125 to +127
ginkgo.DeferCleanup(func() {
_ = exec.Command("bin/devsy", "workspace", "delete", "--force", workspaceID).Run()
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the relevant e2e integration test and surrounding cleanup/workspace lifecycle.
if [ -f e2e/tests/integration/integration.go ]; then
  wc -l e2e/tests/integration/integration.go
  sed -n '1,220p' e2e/tests/integration/integration.go
else
  echo "e2e/tests/integration/integration.go not found"
  fd -a 'integration\.go$' .
fi

# Search for workspace creation/deletion and DeferCleanup usages.
rg -n "workspace|DeferCleanup|devsy|workspaceID|delete|--force|Create|Delete|idempoten|exists" e2e tests . --glob '!vendor/**' --glob '!node_modules/**' || true

Repository: devsy-org/devsy-provider-orbstack

Length of output: 16093


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== e2e/README.md cleanup context =="
sed -n '1,80p' e2e/README.md

echo
echo "== pkg/orb/orbctl.go delete implementation =="
sed -n '90,125p' pkg/orb/orbctl.go

echo
echo "== cmd/lifecycle.go delete implementation =="
sed -n '30,55p' cmd/lifecycle.go

echo
echo "== Ginkgo DeferCleanup handling patterns in go modules =="
python3 - <<'PY'
# Read-only check of local ginkgo source availability and the relevant DeferCleanup implementation, if vendored/module cache is present.
import glob, os, re, pathlib
candidates = []
for root in ["/root/go/pkg/mod/github.com/onsi/ginkgo*"]:
    candidates += glob.glob(root + "/core/deferpanic.go")
    candidates += glob.glob(root + "/core/spec_runner.go")
print("ginkgo candidate files:", candidates[:10])
for path in candidates[:10]:
    print(f"\n--- {path} ---")
    text = pathlib.Path(path).read_text()
    for needle in ["DeferCleanup", "AfterEach", "Cleanup", "CleanupFailed"]:
        idx = text.find(needle)
        if idx != -1:
            start=max(0, idx-400); end=min(len(text), idx+1400)
            print(text[start:end])
PY

Repository: devsy-org/devsy-provider-orbstack

Length of output: 2584


🌐 Web query:

Ginkgo DeferCleanup runs after specs in spec group fail DeferCleanup returns non-nil error

💡 Result:

Yes, Ginkgo runs DeferCleanup functions even if a spec fails [1]. When a failure occurs during the execution of a test (such as in a BeforeEach, JustBeforeEach, or It closure), Ginkgo halts the current spec but proceeds to execute all registered cleanup nodes, including AfterEach and DeferCleanup nodes, to ensure proper state cleanup [1]. Regarding the return value, if a function passed to DeferCleanup returns a non-nil error, Ginkgo will treat this as a failure [1]. Specifically, DeferCleanup ignores all return values from the passed function except for the last one; if that last value is a non-nil error, Ginkgo will mark the spec as having failed [1]. If the spec had already failed before the cleanup node ran, the error returned by the DeferCleanup function is typically tracked as an additional failure associated with that spec [2][3].

Citations:


🌐 Web query:

GitHub Ginkgo Ginkgo DeferCleanup failed cleanup spec

💡 Result:

In the Ginkgo testing framework, DeferCleanup is a convenience function used to register cleanup callbacks that run automatically when a spec completes [1][2]. If a DeferCleanup callback function returns an error, Ginkgo handles it by failing the spec [1]. Key behaviors regarding DeferCleanup and spec failures include: Error Handling: When DeferCleanup is passed a function that returns an error as its last return value, Ginkgo checks this value [1]. If the returned error is non-nil, Ginkgo treats it as a spec failure [1]. Execution Order: DeferCleanup callbacks are executed in a first-in-last-out (FILO) order [3]. They are triggered after other standard cleanup nodes (like AfterEach) have finished [3][4]. Placement Restrictions: DeferCleanup must be called within setup or subject nodes (e.g., BeforeEach, It, BeforeSuite) [2][5]. It cannot be called at the top level or within container nodes (e.g., Describe, Context) [2][5]. Attempting to call it in an invalid location will result in an error during test execution [5][6]. Context: If a test fails in a BeforeEach, JustBeforeEach, or It closure, Ginkgo halts the current spec execution and proceeds to run registered cleanup nodes, including DeferCleanup callbacks, to ensure the environment is cleaned up properly [1]. If you are experiencing a failed cleanup spec, ensure that: 1. Your cleanup function is correctly returning an error (if intended) or nil (if successful) [1]. 2. You are not attempting to call DeferCleanup from an invalid location, such as directly inside a Describe or Context container [2][5]. 3. You are aware of the execution order; if your AfterAll block relies on state that a DeferCleanup is responsible for clearing, remember that DeferCleanups typically run after the standard After* family of nodes [4].

Citations:


Do not discard VM cleanup failures.

A failed DeferCleanup runs even when a later spec fails, and swallowing the delete error can leave the OrbStack workspace/machine behind. Report cleanup failures from DeferCleanup, and make cleanup idempotent or avoid deleting the workspace twice.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/tests/integration/integration.go` around lines 125 - 127, Update the
DeferCleanup callback around the workspace deletion command to report failures
instead of discarding the error returned by exec.Command(...).Run(). Ensure the
cleanup remains idempotent by avoiding a second delete attempt or treating an
already-deleted workspace as successful.

Comment thread go.mod
github.com/devsy-org/ssh v1.1.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/cli v29.4.0+incompatible // indirect
github.com/docker/docker v28.5.2+incompatible // indirect

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== go.mod relevant lines =="
nl -ba go.mod | sed -n '80,95p;200,212p;228,236p'

echo
echo "== module graph references for suspected packages =="
go mod why -m github.com/docker/docker || true
echo "--- otel ---"
go mod why -m go.opentelemetry.io/otel || true
go mod why -m go.opentelemetry.io/otel/sdk || true
echo "--- grpc ---"
go mod why -m google.golang.org/grpc || true

echo
echo "== direct imports/usages =="
rg -n --glob '!go.mod' --glob '!go.sum' 'github\.com/docker/docker|go\.opentelemetry\.io/otel|google\.golang\.org/grpc|opentelemetry|otel' .

Repository: devsy-org/devsy-provider-orbstack

Length of output: 239


🌐 Web query:

GitHub OSV database vulnerability docker/docker v28.5.2+incompatible

💡 Result:

The version v28.5.2+incompatible of the Go module github.com/docker/docker (also known as Moby) is a legitimate release published on November 5, 2025 [1][2]. While this version includes several security patches—specifically addressing high-severity vulnerabilities in runc (CVE-2025-31133, CVE-2025-52565, and CVE-2025-52881) [1][3]—it is itself listed as an affected version for subsequent security vulnerabilities [4][5]. Specifically, the GitHub Advisory Database and the National Vulnerability Database (NVD) identify version 28.5.2 and all prior versions as affected by the following vulnerabilities: CVE-2026-41567: A vulnerability in the PUT /containers/{id}/archive API endpoint where a malicious container can cause the Docker daemon to execute arbitrary code with host root privileges by exploiting incorrect binary resolution when handling compressed archives [4]. CVE-2026-42306: A race condition during docker cp mount setup that allows a malicious container to redirect a bind mount target to an arbitrary path on the host, potentially leading to unauthorized file overwrites or denial of service [5]. Users of this version are advised to review these security advisories and check for updates provided by the Moby/Docker project to mitigate these risks [4][5]. As of late 2025, the github.com/docker/docker Go module has been deprecated in favor of newer release patterns [6].

Citations:


🌐 Web query:

GitHub OSV database vulnerability go.opentelemetry.io/otel grpc google.golang.org/grpc version vulnerabilities

💡 Result:

Both go.opentelemetry.io/otel and google.golang.org/grpc have recorded security vulnerabilities in the GitHub OSV database. Below is a summary of significant recent vulnerabilities for both ecosystems. OpenTelemetry Go (go.opentelemetry.io/otel) Several vulnerabilities have been identified and addressed in the OpenTelemetry Go SDK and related components: 1. Arbitrary Code Execution (CVE-2026-24051 / GHSA-9h8m-3fm2-qjrq): Versions 1.21.0 to 1.39.0 are vulnerable to PATH hijacking on macOS/Darwin because the SDK executes the ioreg command using an untrusted search path [1][2]. This was patched in version 1.40.0 [1]. 2. PATH Hijacking on BSD/Solaris (GHSA-hfvc-g4fc-pqhx): Similar to the above, versions up to 1.42.0 are vulnerable to PATH hijacking via the kenv command on BSD-based systems and Solaris [3]. 3. Denial of Service via File Descriptor Leak (CVE-2026-45287 / GHSA-995v-fvrw-c78m): In package go.opentelemetry.io/otel/schema/v1.0 and v1.1 (versions <= 0.0.16), the ParseFile function fails to close file descriptors, potentially leading to denial of service in long-running processes [4]. 4. Unbounded Response Body Memory Exhaustion (GHSA-w8rr-5gcm-pp58): OTLP HTTP exporters (traces/metrics/logs) read full HTTP response bodies without a size limit, making them vulnerable to memory exhaustion attacks if communicating with an untrusted collector [5]. 5. Unbound Cardinality Metrics (CVE-2023-47108): In opentelemetry-go-contrib, versions prior to 0.46.0 could suffer from memory exhaustion due to unbound cardinality in grpc Unary Server Interceptor labels [6][7]. gRPC-Go (google.golang.org/grpc) gRPC-Go has also addressed several critical and high-severity issues: 1. Authorization Bypass via Missing Leading Slash (CVE-2026-33186 / GHSA-p77j-4mvh-x3m3): In versions before 1.79.3, servers were too lenient when routing requests with:path pseudo-headers missing a leading slash, which could allow attackers to bypass path-based authorization rules [8][9]. 2. xDS RBAC and HTTP/2 Vulnerabilities (GHSA-hrxh-6v49-42gf): Recent findings (July 2026) include an authorization bypass in xDS RBAC policies, a denial of service (high CPU) due to an HTTP/2 Rapid Reset mitigation bypass, and a denial of service (panic) in the xDS RBAC engine. These are fixed in version 1.82.1 [10][11]. It is recommended that users of these packages regularly monitor the GitHub Advisory Database and the official Go vulnerability database (vuln.go.dev) to stay informed of patched versions [12][13].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== go.mod relevant lines =="
awk 'NR>=80 && NR<=95 || NR>=200 && NR<=212 || NR>=228 && NR<=236 { printf "%6d\t%s\n", NR, $0 }' go.mod

echo
echo "== lock versions =="
for pkg in "github.com/docker/docker" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/sdk" "google.golang.org/grpc" "google.golang.org/grpc/internal/xds/rbac"; do
  echo "--- $pkg ---"
  rg -n "^\t*$pkg" go.mod go.sum || true
done

echo
echo "== module graph references for suspected packages =="
go mod why -m github.com/docker/docker || true
go mod why -m go.opentelemetry.io/otel || true
go mod why -m go.opentelemetry.io/otel/sdk || true
go mod why -m google.golang.org/grpc || true

echo
echo "== direct imports/usages in tracked Go files =="
rg -n --glob '*.go' 'github\.com/docker/docker|go\.opentelemetry\.io/otel|google\.golang\.org/grpc' .

Repository: devsy-org/devsy-provider-orbstack

Length of output: 14817


🌐 Web query:

docker docker Go module API endpoint vulnerability CVE-2026-41567 v28.5.2 docker daemon /containers/{id}/archive

💡 Result:

CVE-2026-41567 is a high-severity security vulnerability in Docker (Moby) involving the PUT /containers/{id}/archive API endpoint [1][2][3]. In affected versions, which include all versions up to and including 28.5.2, the Docker daemon incorrectly resolves external decompression binaries (such as xz or unpigz) from the container's filesystem instead of the host's when processing compressed archives [1][2][4]. If a user uploads a specifically crafted compressed archive to a container created from a malicious image that contains a trojanized decompression binary, the daemon may execute that binary [1][5][6]. Because this execution occurs within the daemon process, the malicious binary runs with full host root privileges and unrestricted capabilities, enabling arbitrary code execution and a complete container-to-host trust boundary breakout [1][3][4]. This vulnerability only impacts scenarios where xz or gzip-compressed archives are explicitly passed to the PUT /containers/{id}/archive API or piped via docker cp - [1][5][4]. Standard docker cp usage is not affected because the CLI defaults to uncompressed tar [5][6]. Decompression formats utilizing pure Go implementations—such as bzip2, zstd, and gzip (when no unpigz binary is present in the container)—are also not affected [1][4]. The issue was addressed in Docker Engine 29.5.1 and moby/moby v2.0.0-beta.14 [2]. Recommended workarounds for affected systems include: - Only running containers from trusted images [1][3]. - Using authorization plugins to restrict access to the PUT /containers/{id}/archive endpoint [1][4]. - Avoiding the piping of compressed archives into containers created from untrusted images [5][3].

Citations:


Resolve the vulnerable module pins before merging.

github.com/docker/docker v28.5.2+incompatible and google.golang.org/grpc v1.80.0 are still in the module graph and carry high-severity advisories for versions up to/including those picks. Upgrade to patched releases or remove them from the dependency graph; indirect status does not reduce the exploit surface.

🧰 Tools
🪛 OSV Scanner (2.4.0)

[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Moby has an Off-by-one error in its plugin privilege validation in github.com/docker/docker

(GO-2026-4883)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Moby has AuthZ plugin bypass when provided oversized request bodies in github.com/docker/docker

(GO-2026-4887)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: Race condition in docker cp allows bind mount redirection to host path in github.com/docker/docker

(GO-2026-5617)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: Race condition in docker cp allows creation of arbitrary empty files on the host via symlink swap in github.com/docker/docker

(GO-2026-5668)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: 'PUT /containers/{id}/archive' executes container binary on the host in github.com/docker/docker

(GO-2026-5746)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Moby has an Off-by-one error in its plugin privilege validation

(GHSA-pxq6-2prw-chj9)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: Race condition in docker cp allows bind mount redirection to host path

(GHSA-rg2x-37c3-w2rh)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: Race condition in docker cp allows creation of arbitrary empty files on the host via symlink swap

(GHSA-vp62-88p7-qqf5)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Moby has AuthZ plugin bypass when provided oversized request bodies

(GHSA-x744-4wpc-v9h2)


[HIGH] 90-90: github.com/docker/docker 28.5.2+incompatible: Docker: PUT /containers/{id}/archive executes container binary on the host

(GHSA-x86f-5xw2-fm2r)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@go.mod` at line 90, Update the module graph entries for
github.com/docker/docker and google.golang.org/grpc to patched, non-vulnerable
releases, or remove the dependencies if they are no longer required. Ensure
go.mod and go.sum are consistent and no vulnerable versions remain in the
resolved dependency graph.

Source: Linters/SAST tools

@skevetter
skevetter marked this pull request as draft July 25, 2026 19:48
- Gate the e2e suite behind a build tag so 'go test ./...' no longer runs the
  VM specs; run it explicitly with -tags=e2e (Taskfile + CI updated).
- Use a unique per-run workspace ID in the vm suite to avoid colliding with or
  deleting an existing workspace.
- Assert the init-missing spec actually reaches the provider (checks for the
  'orbctl not found' message) rather than accepting any exec error.
- Close the download response body via defer before the status assertion.
@skevetter
skevetter marked this pull request as ready for review July 25, 2026 20:11
@skevetter
skevetter merged commit 26b680d into main Jul 25, 2026
6 checks passed
@skevetter
skevetter deleted the feat/e2e-suite branch July 25, 2026 20:12
@devsy-app devsy-app Bot mentioned this pull request Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant