ntfs-du: find large files on NTFS partitions on Windows

.github/workflows/fuzz.yml

@@ -97,6 +97,13 @@ jobs:
             # in df. Fuzzing it directly is much faster than going
             # through the shell runner.
             corpus_path: builtins/internal/diskstats
+          - pkg: ./builtins/internal/ntfsmft/
+            name: ntfsmft
+            # The $MFT record parser consumes untrusted on-disk binary data
+            # (raw MFT records read straight off the volume). Fuzzing it
+            # directly is far faster than the Windows-only volume scan; the
+            # parser is platform-neutral (see parser.go) so it runs on Linux.
+            corpus_path: builtins/internal/ntfsmft
           - pkg: ./builtins/tests/xargs/
             name: xargs
             corpus_path: builtins/tests/xargs

AGENTS.md

@@ -34,6 +34,8 @@ The shell is supported on Linux, Windows and macOS.
 
 - **`journalctl` bypasses `AllowedPaths` for trusted systemd target access.** The builtin delegates journal discovery and reads, disk-usage scans, vacuuming, machine-ID reads, and journald control-socket access to `internal/systemd`, which opens the configured paths directly with `os` APIs. `SystemdTargetConfig.JournalDirs`, `SystemdTargetConfig.MachineIDPath`, and `SystemdTargetConfig.JournalControlSocket` are fixed by the embedding application when constructing the runner and cannot be supplied or changed by shell scripts. Operators therefore cannot use `AllowedPaths` to block these accesses; authorization is enforced separately by `AllowedCommands`, exact `AllowedSystemServices` grants, and remediation mode for mutations. All configured target paths are trusted and must refer to the same host. See the trusted systemd target exception in `docs/RULES.md` for the backend requirements.
 
+- **`ntfs-du` bypasses `AllowedPaths` and reads the raw volume device (Windows only).** `ntfs-du` opens the raw NTFS volume `\\.\<drive>:` directly via `windows.CreateFile` (delegated to `builtins/internal/ntfsmft`) and streams the entire Master File Table (`$MFT`). It never routes through `callCtx.OpenFile`, so `AllowedPaths` restrictions do not apply — the same trade-off as `ss` / `df` / `ip route`, except that the volume device path is derived from the scan target's drive letter rather than being fully hardcoded. Two consequences follow: (1) operators cannot use `AllowedPaths` to constrain which volume `ntfs-du` reads or to hide individual files from it; and (2) because the scan enumerates every record in the MFT, `ntfs-du` can surface file names and sizes across the **entire volume**, regardless of the configured sandbox roots. This is intentional — whole-disk usage analysis is the command's purpose — but it means `ntfs-du` exposes more filesystem metadata than any sandbox-scoped builtin. The command is read-only (no writes, no execution), reads only NTFS metadata (record parents, sizes, names — never file *contents*), and requires Administrator privileges to open the volume device, so it is unavailable to non-elevated callers.
+
 ## CRITICAL: Bug Fixes and Bash Compatibility
 
 - **ALWAYS prioritise fixing the shell implementation to match bash behaviour over changing tests to match the current (incorrect) shell output.** Never "fix" a failing test by updating its expected output to match broken shell behaviour — fix the shell instead.
@@ -53,6 +55,7 @@ The shell is supported on Linux, Windows and macOS.
 
 - **Prefer scenario tests (`tests/scenarios/`) over Go tests.** Scenario tests are declarative YAML files that are automatically validated against both the shell and bash, making them easier to write, review, and maintain. Only use Go tests when scenario tests cannot express the required behaviour (e.g. testing Go APIs directly, complex programmatic assertions).
 - **`ip route show`/`ip route get` happy-path scenario tests cannot be added.** The scenario test framework has no platform-skip mechanism, and `ip route` reads `/proc/net/route` which is Linux-only — the command exits 1 with "not supported" on macOS and Windows. Happy-path coverage lives in `builtins/tests/ip/ip_linux_test.go` instead.
+- **`ntfs-du` has no scenario tests, and its scan is not validated by CI.** Two independent reasons: (1) the scenario framework has no way to run a scenario only on Windows, and `ntfs-du` is registered only on Windows (see `interp/register_builtins_windows.go`), so any scenario would fail on the Linux/macOS CI jobs where the command doesn't exist; and (2) even on Windows, the interesting behavior — the raw `$MFT` scan — cannot run in CI. It needs a *genuine NTFS volume* opened with elevation, whereas CI containers expose `C:` as a filesystem layer that rejects raw volume reads (`ERROR_NOT_SUPPORTED`/`ERROR_INVALID_FUNCTION`), independent of privilege (containers already run as `ContainerAdministrator`). Flag/`--help`/validation coverage therefore lives in the Windows-gated Go tests under `builtins/ntfsdu/`, and `TestScanTempDirJSON` exercises the scan only opportunistically (it skips when raw MFT access is unavailable). **Real validation of scan correctness requires a VM-based E2E test** (a provisioned Windows VM with a real NTFS volume) — not yet built; rshell has no E2E harness today.
 - In test scenarios, use `expect.stderr` when possible instead of `stderr_contains`.
 - Always use the YAML `|+` block scalar for `input.script`, `expect.stdout`, and `expect.stderr` values, even single-line ones.
 - Test scenarios are asserted against bash by default. Only set `skip_assert_against_bash: true` for features that intentionally diverge from standard bash behavior (e.g. blocked commands, restricted redirects, readonly enforcement).

SHELL_FEATURES.md

@@ -29,6 +29,7 @@ The in-shell `help` command mirrors these feature categories: run `help` for a c
 - ✅ `sort [-rnhubfds] [-k KEYDEF] [-t SEP] [-c|-C] [FILE]...` — sort lines of text files; `-h`/`--human-numeric-sort` orders by SI suffix (none < K/k < M < G < T < P < E < Z < Y < R < Q) then by numeric value (single-letter suffixes only — `Ki`, `Mi`, etc. are not recognised); `-o`, `--compress-program`, and `-T` are rejected (filesystem write / exec)
 - ✅ `ss [-tuaxlans4689Hoehs] [OPTION]...` — display network socket statistics; reads kernel socket state directly via `os.Open` (bypassing `AllowedPaths`) from: Linux: `/proc/net/`; macOS: sysctl; Windows: iphlpapi.dll; `-F`/`--filter` (GTFOBins file-read), `-p`/`--processes` (PID disclosure), `-K`/`--kill`, `-E`/`--events`, and `-N`/`--net` are rejected
 - ✅ `ls [-1aAdFhlpRrSt] [--offset N] [--limit N] [FILE]...` — list directory contents; `--offset`/`--limit` are non-standard pagination flags (single-directory only, silently ignored with `-R` or multiple arguments, capped at 1,000 entries per call); offset operates on filesystem order (not sorted order) for O(n) memory
+- ✅ `ntfs-du [--apparent-size] [--top-files N] [--top-ext N] [-d N] [--min SIZE] [--exclude PATH]... [--find-ext CSV|--find-glob PAT|--find-regex RE]... [--find-limit N] [--output json] [FOLDER]` — quickly find large folders, files, and file extensions across an NTFS volume by reading the raw `$MFT`, so runtime scales with the volume's file count rather than the starting folder (use `du` for a small subtree); `--find-ext`/`--find-glob`/`--find-regex` locate specific files (**Windows only** — registered only on Windows, absent elsewhere; **requires Administrator**; NTFS volumes only); opens the raw volume device `\\.\<drive>:` directly, bypassing `AllowedPaths` (same trade-off as `ss`/`df`/`ip route`) and reporting names/sizes across the whole volume regardless of sandbox roots; scans the current drive root by default (depth 1, top-10 files and extensions, `--min` 100M); emits JSON only (`--output json`; a human-readable format is planned)
 - ✅ `ping [-c N] [-W DURATION] [-i DURATION] [-q] [-4|-6] [-h] HOST` — send ICMP echo requests to a network host and report round-trip statistics; `-f` (flood), `-b` (broadcast), `-s` (packet size), `-I` (interface), `-p` (pattern), and `-R` (record route) are blocked; count/wait/interval are clamped to safe ranges with a warning; multicast, unspecified (`0.0.0.0`/`::`), and broadcast addresses (IPv4 last-octet `.255`) are rejected — note: directed broadcasts on non-standard subnets (e.g. `.127` on a `/25`) are not blocked without subnet-mask knowledge
 - ✅ `ps [-e|-A] [-f] [-p PIDLIST]` — report process status; default shows current-session processes; `-e`/`-A` shows all; `-f` adds UID/PPID/STIME columns; `-p` selects by PID list; `CMD` shows only the process comm/executable name, never argv
 - ✅ `printf FORMAT [ARGUMENT]...` — format and print data to stdout; supports `%s`, `%b`, `%c`, `%d`, `%i`, `%o`, `%u`, `%x`, `%X`, `%e`, `%E`, `%f`, `%F`, `%g`, `%G`, `%%`; format reuse for excess arguments; `%n` rejected (security risk); `-v` rejected

analysis/symbols_builtins.go

@@ -239,6 +239,19 @@ var builtinPerCommandSymbols = map[string][]string{
 		"syscall.Stat_t",                     // 🟢 Unix file stat struct for extracting UID/GID/nlink; read-only type, no I/O.
 		"time.Time",                          // 🟢 time value type; pure data, no side effects.
 	},
+	"ntfsdu": {
+		"context.Context",             // 🟢 deadline/cancellation plumbing; pure interface, no side effects.
+		"encoding/json.MarshalIndent", // 🟢 serialises the scan result to indented JSON; pure function, no I/O.
+		"fmt.Errorf",                  // 🟢 error formatting for --min parsing; pure function, no I/O.
+		"path/filepath.Join",          // 🟢 builds tree-node paths from parent path + basename; pure function, no I/O.
+		"strconv.ParseInt",            // 🟢 parses the numeric part of a --min SIZE value; pure function, no I/O.
+		"strings.TrimSpace",           // 🟢 trims a --min value before parsing; pure function, no I/O.
+		"strings.TrimSuffix",          // 🟢 strips B/i/b suffixes from a --min value; pure function, no I/O.
+		"time.RFC3339",                // 🟢 layout constant for formatting file created/modified times; pure constant.
+		"time.Time",                   // 🟢 created/modified timestamp type carried from the engine into JSON; pure data type, no I/O.
+		// Note: builtins/internal/ntfsmft symbols are exempt from this allowlist
+		// (internal packages are not checked by the builtinAllowedSymbols test).
+	},
 	"ps": {
 		"context.Context",    // 🟢 deadline/cancellation plumbing; pure interface, no side effects.
 		"fmt.Errorf",         // 🟢 error formatting; pure function, no I/O.
@@ -597,6 +610,7 @@ var builtinPerCommandCallContextFields = map[string][]string{
 	"continue": {},
 	"df":       {},
 	"echo":     {},
+	"ntfsdu":   {"WorkDir"},
 	"exit":     {},
 	"false":    {},
 	"ping":     {},
@@ -731,25 +745,26 @@ var builtinPerCommandCallContextFields = map[string][]string{
 }
 
 var builtinAllowedSymbols = []string{
-	"bufio.NewReaderSize",      // 🟢 buffered reader with caller-supplied size; pure wrapper, no I/O capability of its own.
-	"bufio.NewScanner",         // 🟢 line-by-line input reading (e.g. head, cat); no write or exec capability.
-	"bufio.Reader",             // 🟢 buffered reader type; pure data, no side effects.
-	"bufio.Scanner",            // 🟢 scanner type for buffered input reading; no write or exec capability.
-	"bufio.SplitFunc",          // 🟢 type for custom scanner split functions; pure type, no I/O.
-	"bytes.Buffer",             // 🟢 in-memory buffer to capture command output; no I/O side effects.
-	"bytes.Equal",              // 🟢 compares two byte slices for equality; pure function, no I/O.
-	"bytes.IndexByte",          // 🟢 finds a byte in a byte slice; pure function, no I/O.
-	"bytes.NewReader",          // 🟢 wraps a byte slice as an io.Reader; pure in-memory, no I/O.
-	"context.CancelFunc",       // 🟢 cancellation function returned by context.WithTimeout/WithCancel; pure type, no side effects beyond context tree.
-	"context.Context",          // 🟢 deadline/cancellation plumbing; pure interface, no side effects.
-	"context.DeadlineExceeded", // 🟢 sentinel error value for context deadline expiry; pure constant.
-	"context.WithTimeout",      // 🟢 creates a child context with a deadline; no filesystem or network I/O itself.
-	"errors.As",                // 🟢 error type assertion; pure function, no I/O.
-	"errors.Is",                // 🟢 error comparison; pure function, no I/O.
-	"errors.New",               // 🟢 creates a simple error value; pure function, no I/O.
-	"fmt.Errorf",               // 🟢 error formatting; pure function, no I/O.
-	"fmt.Fprint",               // 🟠 writes to a writer (e.g. callCtx.Stderr for read -p prompts); no filesystem access, delegates to Write.
-	"fmt.Sprintf",              // 🟢 string formatting; pure function, no I/O.
+	"bufio.NewReaderSize",         // 🟢 buffered reader with caller-supplied size; pure wrapper, no I/O capability of its own.
+	"bufio.NewScanner",            // 🟢 line-by-line input reading (e.g. head, cat); no write or exec capability.
+	"bufio.Reader",                // 🟢 buffered reader type; pure data, no side effects.
+	"bufio.Scanner",               // 🟢 scanner type for buffered input reading; no write or exec capability.
+	"bufio.SplitFunc",             // 🟢 type for custom scanner split functions; pure type, no I/O.
+	"bytes.Buffer",                // 🟢 in-memory buffer to capture command output; no I/O side effects.
+	"bytes.Equal",                 // 🟢 compares two byte slices for equality; pure function, no I/O.
+	"bytes.IndexByte",             // 🟢 finds a byte in a byte slice; pure function, no I/O.
+	"bytes.NewReader",             // 🟢 wraps a byte slice as an io.Reader; pure in-memory, no I/O.
+	"context.CancelFunc",          // 🟢 cancellation function returned by context.WithTimeout/WithCancel; pure type, no side effects beyond context tree.
+	"context.Context",             // 🟢 deadline/cancellation plumbing; pure interface, no side effects.
+	"context.DeadlineExceeded",    // 🟢 sentinel error value for context deadline expiry; pure constant.
+	"context.WithTimeout",         // 🟢 creates a child context with a deadline; no filesystem or network I/O itself.
+	"encoding/json.MarshalIndent", // 🟢 ntfsdu: serialises the scan result to indented JSON; pure function, no I/O.
+	"errors.As",                   // 🟢 error type assertion; pure function, no I/O.
+	"errors.Is",                   // 🟢 error comparison; pure function, no I/O.
+	"errors.New",                  // 🟢 creates a simple error value; pure function, no I/O.
+	"fmt.Errorf",                  // 🟢 error formatting; pure function, no I/O.
+	"fmt.Fprint",                  // 🟠 writes to a writer (e.g. callCtx.Stderr for read -p prompts); no filesystem access, delegates to Write.
+	"fmt.Sprintf",                 // 🟢 string formatting; pure function, no I/O.
 	"github.com/DataDog/rshell/internal/version.Version",  // 🟢 build version string; read-only package-level variable, no I/O.
 	"github.com/prometheus-community/pro-bing.NewPinger",  // 🔴 creates an ICMP pinger by resolving host; network I/O is the explicit purpose of the ping builtin.
 	"github.com/prometheus-community/pro-bing.NoopLogger", // 🟢 no-op logger that discards pro-bing internal messages; no side effects.
@@ -857,6 +872,7 @@ var builtinAllowedSymbols = []string{
 	"strings.ToLower",                                     // 🟢 converts string to lowercase; pure function, no I/O.
 	"strings.TrimPrefix",                                  // 🟢 removes a leading prefix from a string; pure function, no I/O.
 	"strings.TrimSpace",                                   // 🟢 removes leading/trailing whitespace; pure function.
+	"strings.TrimSuffix",                                  // 🟢 ntfsdu: strips B/i/b suffixes from a --min SIZE value; pure function, no I/O.
 	"syscall.ByHandleFileInformation",                     // 🟢 Windows file info struct for extracting nlink; read-only type, no I/O.
 	"syscall.EACCES",                                      // 🟢 POSIX errno constant for permission denied; pure constant, no I/O.
 	"syscall.EISDIR",                                      // 🟢 error number constant for "is a directory"; pure constant, no I/O.
@@ -874,6 +890,7 @@ var builtinAllowedSymbols = []string{
 	"time.Parse",                                          // 🟢 parses timestamps according to a caller-supplied layout; pure function, no I/O.
 	"time.ParseDuration",                                  // 🟢 parses Go duration strings (e.g. "1s"); pure function, no I/O.
 	"time.ParseInLocation",                                // 🟢 parses timestamps in a caller-supplied location; pure function, no I/O.
+	"time.RFC3339",                                        // 🟢 layout constant for formatting ntfs-du file created/modified times; pure constant.
 	"time.RFC3339Nano",                                    // 🟢 standard RFC3339 timestamp layout with optional fractional seconds; pure constant.
 	"time.Second",                                         // 🟢 constant representing one second; no side effects.
 	"time.Time",                                           // 🟢 time value type; pure data, no side effects.