Tentacle can detect Powershell scripts that don't start, and can abandon them. by LukeButters · Pull Request #1200 · OctopusDeploy/OctopusTentacle

LukeButters · 2026-03-13T00:12:28Z

Background

When powershell.exe is invoked to run a script, it can sometimes start the process but never actually begin executing the script body. Additionally the script could never be terminated. This results in hung deployments where we may wait forever for a script that will never complete.

Refs: EFT-365
Refs: EFT-3145
Refs: HPY-1295, HPY-1296

Fixes: #1208

#investigation-stuck-tasks-blocking-deployments

Results

This PR introduces a startup detection mechanism that lets Tentacle detect and report when PowerShell never executes the script content.

Scripts that opt in include a special marker comment:

# TENTACLE-POWERSHELL-STARTUP-DETECTION-AND-GUARD-MUST-BE-AT-THE-START-OF-THE-SCRIPT

When Tentacle bootstraps the script, it replaces this marker with generated detection code that:

Attempts to exclusively create a .octopus_powershell_started sentinel file
Checks for a .octopus_powershell_should_run file (written by Tentacle before execution)
Exits early if either check fails

Concurrently, RunningScript runs a monitoring task alongside the script execution task. If the started file isn't created within the timeout window (default: 5 minutes), the monitor concludes PowerShell never executed the script and returns exit code -47 (PowerShellNeverStartedExitCode). Additionally tentacle will ensure the power shell body is never executed.

Notes

Detection is opt-in — scripts without the marker comment are unaffected
The startup timeout is configurable via the RunningScript constructor (tests use a shorter value)
Currently scoped to Windows only (powershell.exe); pwsh support on Linux/Mac is possible as a follow-up

Fixes https://github.com/OctopusDeploy/OctopusTentacle/issues/... (optional public issue)

When Octopus feature toggle is enabled

Message written to server task log when PowerShell startup detection is enabled

12:46:27   Verbose  |       Executable name or full path: C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe
12:46:27   Verbose  |       Starting C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Tentacle-002\Work\08de894f-87c0-33c4-b71f-356080ed09bf' using 'OEM United States' encoding running as 'DOMAIN\user'
12:46:27   Info     |       PowerShell startup detection: Checks passed, continuing script execution
...
12:46:27   Verbose  |       Executing 'C:\Octopus\Tentacle-002\Work\08de894f-87c0-33c4-b71f-356080ed09bf\Script.ps1'
...
12:46:27   Verbose  |       Invoking target script 'C:\Octopus\Tentacle-002\Work\08de894f-87c0-33c4-b71f-356080ed09bf\Script.ps1'.
12:46:27   Info     |       Hello

Warning written to Tentacle logs when it takes longer than the configured timeout to start powershell:

2026-03-24 12:46:05.8801   3172      1  INFO  WorkspaceCleanerTask.Start(): Starting
...
2026-03-24 12:47:46.8117   3172      5  WARN  PowerShell startup detection: PowerShell did not start within <n> minutes for task T4neCG
...
2026-03-24 12:47:59.6891   3172      1  INFO  WorkspaceCleanerTask.Stop(): Stopping
2026-03-24 12:47:59.6891   3172      1  INFO  WorkspaceCleanerTask.Stop(): Stopped
2026-03-24 12:47:59.6891   3172      1  INFO  WorkspaceCleanerTask.Dispose(): Disposing

How to review this PR

PowerShellStartupDetection.cs (new) — Generates and injects the detection code; manages sentinel file paths
PowerShellStartupStatus.cs (new) — Enum for monitoring outcomes: NotMonitored, Started, NeverStarted
RunningScript.cs — Made Execute async; added concurrent startup monitoring task
ScriptWorkspace.cs — Injects detection code during bootstrap when the marker comment is present; creates the should_run file
ScriptExitCodes.cs — Added PowerShellNeverStartedExitCode = -47
StartScriptCommandV2.cs / ExecuteShellScriptCommand.cs — Added DurationToWaitForPowerShellToStartup parameter to allow configurable timeout
PowerShellStartupDetectionTests.cs (new) — Integration tests covering successful detection, failure detection, monitoring timeout, and scripts without the marker

Quality ✔️

Pre-requisites

I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
I have considered informing or consulting the right people, according to the ownership map.
I have considered appropriate testing for my change.

…ssue so no need to have it for *nix

…ent comment exist

…ring task

This reverts commit 4300b37.

…nt in the script body

hnrkndrssn

LGTM 👍

rhysparry

Some initial comments

rhysparry · 2026-04-01T03:56:37Z

+    /// but silently stall before executing any script content. This was seen happening because
+    /// CrowdStrike prevented the script body from running.
+    /// 
+    /// When this happens, we get no output from the script AND the script is un-killable.


Is this because it deviates from what we expect or can we literally not kill the powershell process?

Tentacle is unable to kill the script, with whatever the standard kill command dotnet uses.

We have taken dumps and seen crowdstrike is in the dump, I never saw the dump myself though.

I suspect the issue is something like powershell calls something that enters the kernel which hangs. Since it is in the kernel it can never be killed.

rhysparry

I'm a little unclear on the cancellation behaviour.

rhysparry · 2026-04-01T23:19:52Z

        readonly IOctopusFileSystem fileSystem;
        readonly IHomeDirectoryProvider home;
        readonly SensitiveValueMasker sensitiveValueMasker;
+        readonly bool useBashWorkspace;


Should this be an enum to select a specific workspace type?

rhysparry · 2026-04-01T23:28:33Z

+    [IntegrationTestTimeout]
+    public class PowerShellStartupDetectionTests : IntegrationTest
+    {
+        static (ScriptServiceV2 service, ScriptWorkspaceFactory workspaceFactory, ScriptStateStoreFactory stateStoreFactory, TemporaryDirectory tempDir) CreateScriptService(


Can we replace this with a nested class? It should implement IDisposable for the TemporaryDirectory.

Yes and done.

rhysparry

💚 Just a few minor nits. Otherwise, LGTM.

rhysparry · 2026-04-09T03:19:28Z

+        Bash,
+        PowerShell
+    }
+}


nit: missing newline at end of file

rhysparry · 2026-04-09T03:24:02Z

+
+            await DeletePotentiallyInUseFile(stillRunning);
+            await Task.Delay(TimeSpan.FromSeconds(5));
+            File.Exists(stillRunning).Should().BeFalse("Otherwise the script is still running and we made not effort to cancel it.");


nit:

Suggested change

File.Exists(stillRunning).Should().BeFalse("Otherwise the script is still running and we made not effort to cancel it.");

File.Exists(stillRunning).Should().BeFalse("Otherwise the script is still running and we made no effort to cancel it.");

rhysparry · 2026-04-09T03:27:26Z

+                        $"{shellPath} process did not start within {powerShellStartupTimeout.TotalMinutes} minutes. Script execution aborted.");
+
+                    // The script has not started, and the files on disk have been arranged, so it will never meaningfully progress.
+                    // We will now abandon the script, as we do we will cancell its cancellation token. Which will result in


nit:

Suggested change

// We will now abandon the script, as we do we will cancell its cancellation token. Which will result in

// We will now abandon the script, as we do we will cancel its cancellation token. Which will result in

rhysparry · 2026-04-09T03:28:02Z

+
+                    // The script has not started, and the files on disk have been arranged, so it will never meaningfully progress.
+                    // We will now abandon the script, as we do we will cancell its cancellation token. Which will result in
+                    // the script possibly dieing, although from what we have seen, the script will never die.


nit:

Suggested change

// the script possibly dieing, although from what we have seen, the script will never die.

// the script possibly dying, although from what we have seen, the script will never die.

…time we expect a powershell to start in

LukeButters requested a review from hnrkndrssn March 13, 2026 01:30