diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index 21924e93..064f3921 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -97,6 +97,13 @@ jobs: # in df. Fuzzing it directly is much faster than going # through the shell runner. corpus_path: builtins/internal/diskstats + - pkg: ./builtins/internal/ntfsmft/ + name: ntfsmft + # The $MFT record parser consumes untrusted on-disk binary data + # (raw MFT records read straight off the volume). Fuzzing it + # directly is far faster than the Windows-only volume scan; the + # parser is platform-neutral (see parser.go) so it runs on Linux. + corpus_path: builtins/internal/ntfsmft - pkg: ./builtins/tests/xargs/ name: xargs corpus_path: builtins/tests/xargs diff --git a/AGENTS.md b/AGENTS.md index f9e90d12..6426f0a6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,6 +34,8 @@ The shell is supported on Linux, Windows and macOS. - **`journalctl` bypasses `AllowedPaths` for trusted systemd target access.** The builtin delegates journal discovery and reads, disk-usage scans, vacuuming, machine-ID reads, and journald control-socket access to `internal/systemd`, which opens the configured paths directly with `os` APIs. `SystemdTargetConfig.JournalDirs`, `SystemdTargetConfig.MachineIDPath`, and `SystemdTargetConfig.JournalControlSocket` are fixed by the embedding application when constructing the runner and cannot be supplied or changed by shell scripts. Operators therefore cannot use `AllowedPaths` to block these accesses; authorization is enforced separately by `AllowedCommands`, exact `AllowedSystemServices` grants, and remediation mode for mutations. All configured target paths are trusted and must refer to the same host. See the trusted systemd target exception in `docs/RULES.md` for the backend requirements. +- **`ntfs-du` bypasses `AllowedPaths` and reads the raw volume device (Windows only).** `ntfs-du` opens the raw NTFS volume `\\.\:` directly via `windows.CreateFile` (delegated to `builtins/internal/ntfsmft`) and streams the entire Master File Table (`$MFT`). It never routes through `callCtx.OpenFile`, so `AllowedPaths` restrictions do not apply — the same trade-off as `ss` / `df` / `ip route`, except that the volume device path is derived from the scan target's drive letter rather than being fully hardcoded. Two consequences follow: (1) operators cannot use `AllowedPaths` to constrain which volume `ntfs-du` reads or to hide individual files from it; and (2) because the scan enumerates every record in the MFT, `ntfs-du` can surface file names and sizes across the **entire volume**, regardless of the configured sandbox roots. This is intentional — whole-disk usage analysis is the command's purpose — but it means `ntfs-du` exposes more filesystem metadata than any sandbox-scoped builtin. The command is read-only (no writes, no execution), reads only NTFS metadata (record parents, sizes, names — never file *contents*), and requires Administrator privileges to open the volume device, so it is unavailable to non-elevated callers. + ## CRITICAL: Bug Fixes and Bash Compatibility - **ALWAYS prioritise fixing the shell implementation to match bash behaviour over changing tests to match the current (incorrect) shell output.** Never "fix" a failing test by updating its expected output to match broken shell behaviour — fix the shell instead. @@ -53,6 +55,7 @@ The shell is supported on Linux, Windows and macOS. - **Prefer scenario tests (`tests/scenarios/`) over Go tests.** Scenario tests are declarative YAML files that are automatically validated against both the shell and bash, making them easier to write, review, and maintain. Only use Go tests when scenario tests cannot express the required behaviour (e.g. testing Go APIs directly, complex programmatic assertions). - **`ip route show`/`ip route get` happy-path scenario tests cannot be added.** The scenario test framework has no platform-skip mechanism, and `ip route` reads `/proc/net/route` which is Linux-only — the command exits 1 with "not supported" on macOS and Windows. Happy-path coverage lives in `builtins/tests/ip/ip_linux_test.go` instead. +- **`ntfs-du` has no scenario tests, and its scan is not validated by CI.** Two independent reasons: (1) the scenario framework has no way to run a scenario only on Windows, and `ntfs-du` is registered only on Windows (see `interp/register_builtins_windows.go`), so any scenario would fail on the Linux/macOS CI jobs where the command doesn't exist; and (2) even on Windows, the interesting behavior — the raw `$MFT` scan — cannot run in CI. It needs a *genuine NTFS volume* opened with elevation, whereas CI containers expose `C:` as a filesystem layer that rejects raw volume reads (`ERROR_NOT_SUPPORTED`/`ERROR_INVALID_FUNCTION`), independent of privilege (containers already run as `ContainerAdministrator`). Flag/`--help`/validation coverage therefore lives in the Windows-gated Go tests under `builtins/ntfsdu/`, and `TestScanTempDirJSON` exercises the scan only opportunistically (it skips when raw MFT access is unavailable). **Real validation of scan correctness requires a VM-based E2E test** (a provisioned Windows VM with a real NTFS volume) — not yet built; rshell has no E2E harness today. - In test scenarios, use `expect.stderr` when possible instead of `stderr_contains`. - Always use the YAML `|+` block scalar for `input.script`, `expect.stdout`, and `expect.stderr` values, even single-line ones. - Test scenarios are asserted against bash by default. Only set `skip_assert_against_bash: true` for features that intentionally diverge from standard bash behavior (e.g. blocked commands, restricted redirects, readonly enforcement). diff --git a/SHELL_FEATURES.md b/SHELL_FEATURES.md index dfa53fc5..ebaf3242 100644 --- a/SHELL_FEATURES.md +++ b/SHELL_FEATURES.md @@ -29,6 +29,7 @@ The in-shell `help` command mirrors these feature categories: run `help` for a c - ✅ `sort [-rnhubfds] [-k KEYDEF] [-t SEP] [-c|-C] [FILE]...` — sort lines of text files; `-h`/`--human-numeric-sort` orders by SI suffix (none < K/k < M < G < T < P < E < Z < Y < R < Q) then by numeric value (single-letter suffixes only — `Ki`, `Mi`, etc. are not recognised); `-o`, `--compress-program`, and `-T` are rejected (filesystem write / exec) - ✅ `ss [-tuaxlans4689Hoehs] [OPTION]...` — display network socket statistics; reads kernel socket state directly via `os.Open` (bypassing `AllowedPaths`) from: Linux: `/proc/net/`; macOS: sysctl; Windows: iphlpapi.dll; `-F`/`--filter` (GTFOBins file-read), `-p`/`--processes` (PID disclosure), `-K`/`--kill`, `-E`/`--events`, and `-N`/`--net` are rejected - ✅ `ls [-1aAdFhlpRrSt] [--offset N] [--limit N] [FILE]...` — list directory contents; `--offset`/`--limit` are non-standard pagination flags (single-directory only, silently ignored with `-R` or multiple arguments, capped at 1,000 entries per call); offset operates on filesystem order (not sorted order) for O(n) memory +- ✅ `ntfs-du [--apparent-size] [--top-files N] [--top-ext N] [-d N] [--min SIZE] [--exclude PATH]... [--find-ext CSV|--find-glob PAT|--find-regex RE]... [--find-limit N] [--output json] [FOLDER]` — quickly find large folders, files, and file extensions across an NTFS volume by reading the raw `$MFT`, so runtime scales with the volume's file count rather than the starting folder (use `du` for a small subtree); `--find-ext`/`--find-glob`/`--find-regex` locate specific files (**Windows only** — registered only on Windows, absent elsewhere; **requires Administrator**; NTFS volumes only); opens the raw volume device `\\.\:` directly, bypassing `AllowedPaths` (same trade-off as `ss`/`df`/`ip route`) and reporting names/sizes across the whole volume regardless of sandbox roots; scans the current drive root by default (depth 1, top-10 files and extensions, `--min` 100M); emits JSON only (`--output json`; a human-readable format is planned) - ✅ `ping [-c N] [-W DURATION] [-i DURATION] [-q] [-4|-6] [-h] HOST` — send ICMP echo requests to a network host and report round-trip statistics; `-f` (flood), `-b` (broadcast), `-s` (packet size), `-I` (interface), `-p` (pattern), and `-R` (record route) are blocked; count/wait/interval are clamped to safe ranges with a warning; multicast, unspecified (`0.0.0.0`/`::`), and broadcast addresses (IPv4 last-octet `.255`) are rejected — note: directed broadcasts on non-standard subnets (e.g. `.127` on a `/25`) are not blocked without subnet-mask knowledge - ✅ `ps [-e|-A] [-f] [-p PIDLIST]` — report process status; default shows current-session processes; `-e`/`-A` shows all; `-f` adds UID/PPID/STIME columns; `-p` selects by PID list; `CMD` shows only the process comm/executable name, never argv - ✅ `printf FORMAT [ARGUMENT]...` — format and print data to stdout; supports `%s`, `%b`, `%c`, `%d`, `%i`, `%o`, `%u`, `%x`, `%X`, `%e`, `%E`, `%f`, `%F`, `%g`, `%G`, `%%`; format reuse for excess arguments; `%n` rejected (security risk); `-v` rejected diff --git a/analysis/symbols_builtins.go b/analysis/symbols_builtins.go index 6f11d710..4b7973a5 100644 --- a/analysis/symbols_builtins.go +++ b/analysis/symbols_builtins.go @@ -239,6 +239,19 @@ var builtinPerCommandSymbols = map[string][]string{ "syscall.Stat_t", // 🟢 Unix file stat struct for extracting UID/GID/nlink; read-only type, no I/O. "time.Time", // 🟢 time value type; pure data, no side effects. }, + "ntfsdu": { + "context.Context", // 🟢 deadline/cancellation plumbing; pure interface, no side effects. + "encoding/json.MarshalIndent", // 🟢 serialises the scan result to indented JSON; pure function, no I/O. + "fmt.Errorf", // 🟢 error formatting for --min parsing; pure function, no I/O. + "path/filepath.Join", // 🟢 builds tree-node paths from parent path + basename; pure function, no I/O. + "strconv.ParseInt", // 🟢 parses the numeric part of a --min SIZE value; pure function, no I/O. + "strings.TrimSpace", // 🟢 trims a --min value before parsing; pure function, no I/O. + "strings.TrimSuffix", // 🟢 strips B/i/b suffixes from a --min value; pure function, no I/O. + "time.RFC3339", // 🟢 layout constant for formatting file created/modified times; pure constant. + "time.Time", // 🟢 created/modified timestamp type carried from the engine into JSON; pure data type, no I/O. + // Note: builtins/internal/ntfsmft symbols are exempt from this allowlist + // (internal packages are not checked by the builtinAllowedSymbols test). + }, "ps": { "context.Context", // 🟢 deadline/cancellation plumbing; pure interface, no side effects. "fmt.Errorf", // 🟢 error formatting; pure function, no I/O. @@ -597,6 +610,7 @@ var builtinPerCommandCallContextFields = map[string][]string{ "continue": {}, "df": {}, "echo": {}, + "ntfsdu": {"WorkDir"}, "exit": {}, "false": {}, "ping": {}, @@ -731,25 +745,26 @@ var builtinPerCommandCallContextFields = map[string][]string{ } var builtinAllowedSymbols = []string{ - "bufio.NewReaderSize", // 🟢 buffered reader with caller-supplied size; pure wrapper, no I/O capability of its own. - "bufio.NewScanner", // 🟢 line-by-line input reading (e.g. head, cat); no write or exec capability. - "bufio.Reader", // 🟢 buffered reader type; pure data, no side effects. - "bufio.Scanner", // 🟢 scanner type for buffered input reading; no write or exec capability. - "bufio.SplitFunc", // 🟢 type for custom scanner split functions; pure type, no I/O. - "bytes.Buffer", // 🟢 in-memory buffer to capture command output; no I/O side effects. - "bytes.Equal", // 🟢 compares two byte slices for equality; pure function, no I/O. - "bytes.IndexByte", // 🟢 finds a byte in a byte slice; pure function, no I/O. - "bytes.NewReader", // 🟢 wraps a byte slice as an io.Reader; pure in-memory, no I/O. - "context.CancelFunc", // 🟢 cancellation function returned by context.WithTimeout/WithCancel; pure type, no side effects beyond context tree. - "context.Context", // 🟢 deadline/cancellation plumbing; pure interface, no side effects. - "context.DeadlineExceeded", // 🟢 sentinel error value for context deadline expiry; pure constant. - "context.WithTimeout", // 🟢 creates a child context with a deadline; no filesystem or network I/O itself. - "errors.As", // 🟢 error type assertion; pure function, no I/O. - "errors.Is", // 🟢 error comparison; pure function, no I/O. - "errors.New", // 🟢 creates a simple error value; pure function, no I/O. - "fmt.Errorf", // 🟢 error formatting; pure function, no I/O. - "fmt.Fprint", // 🟠 writes to a writer (e.g. callCtx.Stderr for read -p prompts); no filesystem access, delegates to Write. - "fmt.Sprintf", // 🟢 string formatting; pure function, no I/O. + "bufio.NewReaderSize", // 🟢 buffered reader with caller-supplied size; pure wrapper, no I/O capability of its own. + "bufio.NewScanner", // 🟢 line-by-line input reading (e.g. head, cat); no write or exec capability. + "bufio.Reader", // 🟢 buffered reader type; pure data, no side effects. + "bufio.Scanner", // 🟢 scanner type for buffered input reading; no write or exec capability. + "bufio.SplitFunc", // 🟢 type for custom scanner split functions; pure type, no I/O. + "bytes.Buffer", // 🟢 in-memory buffer to capture command output; no I/O side effects. + "bytes.Equal", // 🟢 compares two byte slices for equality; pure function, no I/O. + "bytes.IndexByte", // 🟢 finds a byte in a byte slice; pure function, no I/O. + "bytes.NewReader", // 🟢 wraps a byte slice as an io.Reader; pure in-memory, no I/O. + "context.CancelFunc", // 🟢 cancellation function returned by context.WithTimeout/WithCancel; pure type, no side effects beyond context tree. + "context.Context", // 🟢 deadline/cancellation plumbing; pure interface, no side effects. + "context.DeadlineExceeded", // 🟢 sentinel error value for context deadline expiry; pure constant. + "context.WithTimeout", // 🟢 creates a child context with a deadline; no filesystem or network I/O itself. + "encoding/json.MarshalIndent", // 🟢 ntfsdu: serialises the scan result to indented JSON; pure function, no I/O. + "errors.As", // 🟢 error type assertion; pure function, no I/O. + "errors.Is", // 🟢 error comparison; pure function, no I/O. + "errors.New", // 🟢 creates a simple error value; pure function, no I/O. + "fmt.Errorf", // 🟢 error formatting; pure function, no I/O. + "fmt.Fprint", // 🟠 writes to a writer (e.g. callCtx.Stderr for read -p prompts); no filesystem access, delegates to Write. + "fmt.Sprintf", // 🟢 string formatting; pure function, no I/O. "github.com/DataDog/rshell/internal/version.Version", // 🟢 build version string; read-only package-level variable, no I/O. "github.com/prometheus-community/pro-bing.NewPinger", // 🔴 creates an ICMP pinger by resolving host; network I/O is the explicit purpose of the ping builtin. "github.com/prometheus-community/pro-bing.NoopLogger", // 🟢 no-op logger that discards pro-bing internal messages; no side effects. @@ -857,6 +872,7 @@ var builtinAllowedSymbols = []string{ "strings.ToLower", // 🟢 converts string to lowercase; pure function, no I/O. "strings.TrimPrefix", // 🟢 removes a leading prefix from a string; pure function, no I/O. "strings.TrimSpace", // 🟢 removes leading/trailing whitespace; pure function. + "strings.TrimSuffix", // 🟢 ntfsdu: strips B/i/b suffixes from a --min SIZE value; pure function, no I/O. "syscall.ByHandleFileInformation", // 🟢 Windows file info struct for extracting nlink; read-only type, no I/O. "syscall.EACCES", // 🟢 POSIX errno constant for permission denied; pure constant, no I/O. "syscall.EISDIR", // 🟢 error number constant for "is a directory"; pure constant, no I/O. @@ -874,6 +890,7 @@ var builtinAllowedSymbols = []string{ "time.Parse", // 🟢 parses timestamps according to a caller-supplied layout; pure function, no I/O. "time.ParseDuration", // 🟢 parses Go duration strings (e.g. "1s"); pure function, no I/O. "time.ParseInLocation", // 🟢 parses timestamps in a caller-supplied location; pure function, no I/O. + "time.RFC3339", // 🟢 layout constant for formatting ntfs-du file created/modified times; pure constant. "time.RFC3339Nano", // 🟢 standard RFC3339 timestamp layout with optional fractional seconds; pure constant. "time.Second", // 🟢 constant representing one second; no side effects. "time.Time", // 🟢 time value type; pure data, no side effects. diff --git a/analysis/symbols_internal.go b/analysis/symbols_internal.go index 11dd63e3..cdb561a4 100644 --- a/analysis/symbols_internal.go +++ b/analysis/symbols_internal.go @@ -165,6 +165,63 @@ var internalPerPackageSymbols = map[string][]string{ "golang.org/x/sys/windows.GetNumberOfConsoleInputEvents", // 🟠 (windows) reports the count of queued console input events without consuming them; read-only inspection. "golang.org/x/sys/windows.Handle", // 🟢 (windows) opaque file/handle type used to call PeekNamedPipe and GetFileType; pure type. }, + "ntfsmft": { + "bytes.Equal", // 🟢 compares a decoded ASCII extension against a wanted extension; pure function, no I/O. + "container/heap.Fix", // 🟢 re-establishes the top-N min-heap invariant after replacing the root; pure in-memory. + "container/heap.Push", // 🟢 pushes a candidate onto the top-N min-heap; pure in-memory. + "context.Context", // 🟢 deadline/cancellation interface honoured between MFT chunks; no side effects. + "encoding/binary.LittleEndian", // 🟢 decodes little-endian MFT record/attribute fields from in-memory buffers; pure value, no I/O. + "errors.New", // 🟢 creates sentinel parse/volume errors (bad signature, torn write, unsupported layout); pure function, no I/O. + "fmt.Errorf", // 🟢 error formatting; pure function, no I/O. + "path/filepath.Abs", // 🟠 resolves the scan target / exclude paths to absolute form (reads process cwd for relative inputs); no file content read. + "path/filepath.Match", // 🟢 evaluates a --find-glob pattern against a basename; pure function, no I/O. + "regexp.Compile", // 🟢 compiles a --find-regex pattern with the RE2 (linear-time) engine; pure, no I/O. + "regexp.Regexp", // 🟢 compiled-regex type held in a match slot; pure type, no I/O. + "sort.Slice", // 🟢 orders immediate children, top-N files, and find blocks by size/name; pure function, no I/O. + "sort.SliceStable", // 🟢 stably orders buckets and tree children by size then name; pure function, no I/O. + "strings.HasSuffix", // 🟢 checks for a trailing path separator on the target; pure function, no I/O. + "strings.Split", // 🟢 splits a comma-separated --find-ext value; pure function, no I/O. + "strings.ToLower", // 🟢 case-folds child names for stable sort and normalises extensions; pure function, no I/O. + "strings.ToUpper", // 🟢 upcases the drive letter so paths differing only in drive case resolve identically; pure function, no I/O. + "strings.TrimPrefix", // 🟢 strips a leading dot from a normalised extension; pure function, no I/O. + "strings.TrimSpace", // 🟢 trims whitespace from split extension tokens; pure function, no I/O. + "strings.TrimSuffix", // 🟢 strips a trailing backslash when building enumeration patterns; pure function, no I/O. + "time.Duration", // 🟢 pass/wall timing type in Result; pure type, no I/O. + "time.Now", // 🟠 captures scan-phase start times for diagnostics; read-only, no side effects. + "time.Since", // 🟠 computes elapsed pass/wall durations; read-only, no side effects. + "time.Time", // 🟢 created/modified timestamp type on FileEntry; pure data type, no I/O. + "time.Unix", // 🟢 builds a Time from a displayed file's FILETIME nanoseconds during path resolution; pure constructor, no I/O. + "unsafe.Pointer", // 🔴 passes the NTFS-volume-data struct to DeviceIoControl and the file-ID descriptor / path buffer to the kernel32 DLL calls via the syscall ABI. No pointer arithmetic; disk bytes are parsed with encoding/binary. + "unsafe.Sizeof", // 🔴 computes the byte size of the volume-data and file-ID-descriptor structs passed to DeviceIoControl / OpenFileById; compile-time constant. + "unsafe.String", // 🔴 builds a zero-copy string view over a stack name buffer for --fast-name glob/regex matching; view never escapes predicate evaluation (buffer reused next file). + "golang.org/x/sys/windows.ByHandleFileInformation", // 🟢 struct receiving GetFileInformationByHandle output (volume-internal file index); pure data type. + "golang.org/x/sys/windows.CloseHandle", // 🟠 closes the volume / per-file handles after reads; no data read or exec capability. + "golang.org/x/sys/windows.CreateFile", // 🔴 opens the raw volume device \\.\: (GENERIC_READ) and per-path metadata handles; read-only raw-device access, requires Administrator. Deliberately bypasses AllowedPaths (see AGENTS.md). + "golang.org/x/sys/windows.DeviceIoControl", // 🔴 issues FSCTL_GET_NTFS_VOLUME_DATA to read the volume geometry; read-only IOCTL, no write capability. + "golang.org/x/sys/windows.ERROR_NO_MORE_FILES", // 🟢 sentinel ending FindNextFile child enumeration; pure constant. + "golang.org/x/sys/windows.FILE_ATTRIBUTE_DIRECTORY", // 🟢 flag identifying directory entries during child enumeration; pure constant. + "golang.org/x/sys/windows.FILE_ATTRIBUTE_REPARSE_POINT", // 🟢 flag marking junctions / symlinks / mount points; pure constant. + "golang.org/x/sys/windows.FILE_FLAG_BACKUP_SEMANTICS", // 🟢 CreateFile flag required to open directories by path; pure constant. + "golang.org/x/sys/windows.FILE_FLAG_OPEN_REPARSE_POINT", // 🟢 CreateFile flag to resolve a reparse point's own MFT idx rather than its target; pure constant. + "golang.org/x/sys/windows.FILE_READ_ATTRIBUTES", // 🟢 access mask for OpenFileById metadata-only opens; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_DELETE", // 🟢 share mode allowing concurrent delete while scanning; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_READ", // 🟢 share mode allowing concurrent reads while scanning; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_WRITE", // 🟢 share mode allowing concurrent writes while scanning; pure constant. + "golang.org/x/sys/windows.FindClose", // 🟠 closes a directory-enumeration handle; no data read or exec capability. + "golang.org/x/sys/windows.FindFirstFile", // 🟠 begins immediate-child directory enumeration; read-only, no exec capability. + "golang.org/x/sys/windows.FindNextFile", // 🟠 advances immediate-child directory enumeration; read-only, no exec capability. + "golang.org/x/sys/windows.GENERIC_READ", // 🟢 read-only access mask for the volume / path opens; pure constant. + "golang.org/x/sys/windows.GetFileInformationByHandle", // 🟠 reads a file's volume-internal identity (MFT index); read-only metadata, no I/O of content. + "golang.org/x/sys/windows.Handle", // 🟢 opaque volume/file handle type; pure type. + "golang.org/x/sys/windows.InvalidHandle", // 🟢 sentinel for a failed OpenFileById result; pure constant. + "golang.org/x/sys/windows.NewLazySystemDLL", // 🔴 loads kernel32.dll to call OpenFileById / GetFinalPathNameByHandleW for post-scan path resolution; read-only OS loader call. + "golang.org/x/sys/windows.OPEN_EXISTING", // 🟢 CreateFile disposition (open, never create); pure constant. + "golang.org/x/sys/windows.Overlapped", // 🟢 struct carrying the explicit read offset for raw MFT ReadFile calls; pure data type. + "golang.org/x/sys/windows.ReadFile", // 🟠 reads raw MFT bytes from the volume handle at an explicit offset; read-only, no write capability. + "golang.org/x/sys/windows.UTF16PtrFromString", // 🟢 converts a Go path to a NUL-terminated UTF-16 pointer for the Windows API; pure function, no I/O. + "golang.org/x/sys/windows.UTF16ToString", // 🟢 converts a UTF-16 name/path buffer back to a Go string; pure function, no I/O. + "golang.org/x/sys/windows.Win32finddata", // 🟢 struct receiving FindFirstFile / FindNextFile entries; pure data type. + }, } // internalAllowedSymbols lists every "importpath.Symbol" permitted in @@ -263,4 +320,45 @@ var internalAllowedSymbols = []string{ "golang.org/x/sys/windows.ProcessEntry32", // 🟢 procinfo (windows): struct type holding process snapshot entry data; pure data type, no I/O. "golang.org/x/sys/windows.TH32CS_SNAPPROCESS", // 🟢 procinfo (windows): flag constant selecting process entries for CreateToolhelp32Snapshot; pure constant. "golang.org/x/sys/windows.UTF16ToString", // 🟢 procinfo (windows): converts a null-terminated UTF-16 slice to a Go string; pure function, no I/O. + "bytes.Equal", // 🟢 ntfsmft: compares decoded extension bytes; pure function, no I/O. + "container/heap.Fix", // 🟢 ntfsmft: re-establishes a min-heap invariant; pure in-memory. + "container/heap.Push", // 🟢 ntfsmft: pushes onto a top-N min-heap; pure in-memory. + "path/filepath.Abs", // 🟠 ntfsmft: resolves target/exclude paths to absolute form (reads cwd for relative inputs); no file content read. + "path/filepath.Match", // 🟢 ntfsmft: evaluates a --find-glob pattern against a basename; pure function, no I/O. + "regexp.Compile", // 🟢 ntfsmft: compiles a --find-regex pattern with the RE2 engine; pure, no I/O. + "regexp.Regexp", // 🟢 ntfsmft: compiled-regex type held in a match slot; pure type, no I/O. + "sort.Slice", // 🟢 ntfsmft: orders children/top-N/find results by size/name; pure function, no I/O. + "sort.SliceStable", // 🟢 ntfsmft: stably orders buckets/tree children; pure function, no I/O. + "strings.ToLower", // 🟢 ntfsmft: case-folds names and normalises extensions; pure function, no I/O. + "strings.TrimPrefix", // 🟢 ntfsmft: strips a leading dot from an extension; pure function, no I/O. + "strings.TrimSuffix", // 🟢 ntfsmft: strips a trailing backslash from enumeration patterns; pure function, no I/O. + "time.Duration", // 🟢 ntfsmft: pass/wall timing type; pure type, no I/O. + "time.Since", // 🟠 ntfsmft: computes elapsed pass/wall durations; read-only, no side effects. + "time.Time", // 🟢 ntfsmft: created/modified timestamp type on FileEntry; pure data type, no I/O. + "time.Unix", // 🟢 ntfsmft: builds a Time from a displayed file's FILETIME nanoseconds; pure constructor, no I/O. + "unsafe.Sizeof", // 🔴 ntfsmft: byte size of structs passed to DeviceIoControl / OpenFileById; compile-time constant. + "unsafe.String", // 🔴 ntfsmft: zero-copy string view over a stack name buffer for --fast-name matching; view never escapes predicate evaluation. + "golang.org/x/sys/windows.ByHandleFileInformation", // 🟢 ntfsmft (windows): struct receiving GetFileInformationByHandle output; pure data type. + "golang.org/x/sys/windows.CreateFile", // 🔴 ntfsmft (windows): opens the raw volume device \\.\: (GENERIC_READ) and per-path metadata handles; read-only raw-device access, requires Administrator. Bypasses AllowedPaths by design (see AGENTS.md). + "golang.org/x/sys/windows.DeviceIoControl", // 🔴 ntfsmft (windows): FSCTL_GET_NTFS_VOLUME_DATA read-only IOCTL for volume geometry; no write capability. + "golang.org/x/sys/windows.FILE_ATTRIBUTE_DIRECTORY", // 🟢 ntfsmft (windows): directory-entry flag; pure constant. + "golang.org/x/sys/windows.FILE_ATTRIBUTE_REPARSE_POINT", // 🟢 ntfsmft (windows): junction/symlink/mount-point flag; pure constant. + "golang.org/x/sys/windows.FILE_FLAG_BACKUP_SEMANTICS", // 🟢 ntfsmft (windows): CreateFile flag to open directories by path; pure constant. + "golang.org/x/sys/windows.FILE_FLAG_OPEN_REPARSE_POINT", // 🟢 ntfsmft (windows): CreateFile flag to resolve a reparse point's own MFT idx; pure constant. + "golang.org/x/sys/windows.FILE_READ_ATTRIBUTES", // 🟢 ntfsmft (windows): access mask for OpenFileById metadata-only opens; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_DELETE", // 🟢 ntfsmft (windows): share mode allowing concurrent delete; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_READ", // 🟢 ntfsmft (windows): share mode allowing concurrent reads; pure constant. + "golang.org/x/sys/windows.FILE_SHARE_WRITE", // 🟢 ntfsmft (windows): share mode allowing concurrent writes; pure constant. + "golang.org/x/sys/windows.FindClose", // 🟠 ntfsmft (windows): closes a directory-enumeration handle; no data read or exec capability. + "golang.org/x/sys/windows.FindFirstFile", // 🟠 ntfsmft (windows): begins immediate-child directory enumeration; read-only. + "golang.org/x/sys/windows.FindNextFile", // 🟠 ntfsmft (windows): advances immediate-child directory enumeration; read-only. + "golang.org/x/sys/windows.GENERIC_READ", // 🟢 ntfsmft (windows): read-only access mask; pure constant. + "golang.org/x/sys/windows.GetFileInformationByHandle", // 🟠 ntfsmft (windows): reads a file's volume-internal identity (MFT index); read-only metadata. + "golang.org/x/sys/windows.InvalidHandle", // 🟢 ntfsmft (windows): sentinel for a failed OpenFileById result; pure constant. + "golang.org/x/sys/windows.NewLazySystemDLL", // 🔴 ntfsmft (windows): loads kernel32.dll for OpenFileById / GetFinalPathNameByHandleW path resolution; read-only OS loader call. + "golang.org/x/sys/windows.OPEN_EXISTING", // 🟢 ntfsmft (windows): CreateFile disposition (open, never create); pure constant. + "golang.org/x/sys/windows.Overlapped", // 🟢 ntfsmft (windows): struct carrying the explicit read offset for raw MFT reads; pure data type. + "golang.org/x/sys/windows.ReadFile", // 🟠 ntfsmft (windows): reads raw MFT bytes at an explicit offset; read-only, no write capability. + "golang.org/x/sys/windows.UTF16PtrFromString", // 🟢 ntfsmft (windows): converts a Go path to a NUL-terminated UTF-16 pointer; pure function, no I/O. + "golang.org/x/sys/windows.Win32finddata", // 🟢 ntfsmft (windows): struct receiving FindFirstFile / FindNextFile entries; pure data type. } diff --git a/builtins/internal/ntfsmft/counts_windows_test.go b/builtins/internal/ntfsmft/counts_windows_test.go new file mode 100644 index 00000000..b61f23a6 --- /dev/null +++ b/builtins/internal/ntfsmft/counts_windows_test.go @@ -0,0 +1,155 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "path/filepath" + "testing" +) + +// Child file/dir counts follow the same walk-up attribution as byte totals: +// a file anywhere in a child's subtree counts toward that child, and Dirs is +// the number of descendant directories (excluding the child itself). +func TestScan_CountsTreeChildren(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "A", "f1.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "f2.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "sub", "f3.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "B", "f4.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + a := findTreeChild(t, res.Tree, "A") + if a.Files != 3 { + t.Errorf("child A Files = %d, want 3 (f1, f2, sub/f3)", a.Files) + } + if a.Dirs != 1 { + t.Errorf("child A Dirs = %d, want 1 (sub)", a.Dirs) + } + b := findTreeChild(t, res.Tree, "B") + if b.Files != 1 { + t.Errorf("child B Files = %d, want 1", b.Files) + } + if b.Dirs != 0 { + t.Errorf("child B Dirs = %d, want 0", b.Dirs) + } +} + +// In tree mode the counts are cumulative like Size: deep files/dirs roll up into +// their in-tree ancestors, and the root node totals the whole in-scope subtree. +func TestScan_CountsTreeRollup(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "A", "f1.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "sub", "deep", "f2.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "B", "f3.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "loose.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + if res.Tree == nil { + t.Fatal("Tree is nil at TreeDepth 1") + } + + // Root totals: all 4 files (incl. loose) and all 4 descendant dirs + // (A, A/sub, A/sub/deep, B). + if res.Tree.Files != 4 { + t.Errorf("root Files = %d, want 4", res.Tree.Files) + } + if res.Tree.Dirs != 4 { + t.Errorf("root Dirs = %d, want 4 (A, A/sub, A/sub/deep, B)", res.Tree.Dirs) + } + + a := findTreeChild(t, res.Tree, "A") + if a.Files != 2 { + t.Errorf("A Files = %d, want 2 (f1 + rolled-up deep f2)", a.Files) + } + if a.Dirs != 2 { + t.Errorf("A Dirs = %d, want 2 (sub, sub/deep)", a.Dirs) + } + b := findTreeChild(t, res.Tree, "B") + if b.Files != 1 || b.Dirs != 0 { + t.Errorf("B Files/Dirs = %d/%d, want 1/0", b.Files, b.Dirs) + } +} + +// A hardlinked file within a single directory is counted once for that child, +// mirroring the byte-total dedup. +func TestScan_CountsHardlinkSameChild(t *testing.T) { + root := t.TempDir() + primary := filepath.Join(root, "A", "primary.bin") + writeFile(t, primary, make([]byte, 4096)) + createHardLink(t, filepath.Join(root, "A", "alias.bin"), primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + if got := findTreeChild(t, res.Tree, "A").Files; got != 1 { + t.Errorf("child A Files = %d, want 1 (hardlink counted once)", got) + } +} + +// A file hardlinked across two children counts once in each, mirroring how its +// bytes are attributed to both. +func TestScan_CountsHardlinkAcrossChildren(t *testing.T) { + root := t.TempDir() + primary := filepath.Join(root, "A", "primary.bin") + writeFile(t, primary, make([]byte, 4096)) + createHardLink(t, filepath.Join(root, "B", "alias.bin"), primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + if got := findTreeChild(t, res.Tree, "A").Files; got != 1 { + t.Errorf("child A Files = %d, want 1", got) + } + if got := findTreeChild(t, res.Tree, "B").Files; got != 1 { + t.Errorf("child B Files = %d, want 1", got) + } +} + +// Excluded subtrees are omitted from counts exactly as they are from byte totals. +func TestScan_CountsExcludeRespected(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "A", "keep.bin"), make([]byte, 4096)) + drop := filepath.Join(root, "A", "skip") + writeFile(t, filepath.Join(drop, "gone.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1, Exclude: []string{drop}}) + a := findTreeChild(t, res.Tree, "A") + if a.Files != 1 { + t.Errorf("child A Files = %d, want 1 (excluded file omitted)", a.Files) + } + if a.Dirs != 0 { + t.Errorf("child A Dirs = %d, want 0 (excluded 'skip' dir omitted)", a.Dirs) + } +} + +// The fast-path depth-1 tree and the general-path depth-2 scan must agree on +// the depth-1 children's size and counts — the two code paths attribute the +// same way. (Depth 1 takes the fast path; depth >= 2 takes the general path.) +func TestScan_CountsDepth1MatchesDepth2(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "A", "f1.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "sub", "f2.bin"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "B", "sub2", "deep", "f3.bin"), make([]byte, 4096)) + + d1 := scanOrSkip(t, root, Options{TreeDepth: 1}) + d2 := scanOrSkip(t, root, Options{TreeDepth: 2}) + + // Root totals are the headline numbers, and they come from different code + // on each path (subtreeFiles/subtreeDirs on the fast path vs + // anchorFiles/anchorDirs on the general path), so assert they agree. + if d1.Tree.Files != d2.Tree.Files || d1.Tree.Dirs != d2.Tree.Dirs || d1.Tree.Size != d2.Tree.Size { + t.Errorf("root differs: depth1 %d/%d/%d vs depth2 %d/%d/%d", + d1.Tree.Files, d1.Tree.Dirs, d1.Tree.Size, d2.Tree.Files, d2.Tree.Dirs, d2.Tree.Size) + } + + for _, name := range []string{"A", "B"} { + c1 := findTreeChild(t, d1.Tree, name) + c2 := findTreeChild(t, d2.Tree, name) + if c1.Files != c2.Files || c1.Dirs != c2.Dirs || c1.Size != c2.Size { + t.Errorf("child %s differs: depth1 %d/%d/%d vs depth2 %d/%d/%d", + name, c1.Files, c1.Dirs, c1.Size, c2.Files, c2.Dirs, c2.Size) + } + } +} diff --git a/builtins/internal/ntfsmft/du_windows.go b/builtins/internal/ntfsmft/du_windows.go new file mode 100644 index 00000000..2ff5e575 --- /dev/null +++ b/builtins/internal/ntfsmft/du_windows.go @@ -0,0 +1,1442 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "context" + "encoding/binary" + "errors" + "fmt" + "path/filepath" + "sort" + "strings" + "time" + "unsafe" + + "golang.org/x/sys/windows" +) + +// ------------------------------------------------------------------------- +// Public API +// ------------------------------------------------------------------------- + +// Result is the disk-usage breakdown for a target directory: the overall +// subtree total plus, when requested, a depth-limited directory tree, the +// largest files and extensions, any --find matches, and scan diagnostics. +type Result struct { + // Target is the absolute path the scan was asked to analyze. + Target string + + // Subtree is the deduplicated total of all bytes attributed to Target's + // subtree. A file hardlinked into multiple directories counts once toward + // Subtree, but contributes to each directory it is linked into. + Subtree int64 + + // MultiParentFiles counts in-scope files reachable via more than one + // distinct parent directory within the scanned subtree (hardlinks spanning + // multiple in-scope locations). Diagnostic only — not part of the ntfs-du + // output. + MultiParentFiles int + + // Pass diagnostics: parsed/error counts and durations. + // Pass1 builds dirParent + extSize/extParents from a single MFT scan + // (modeAll). Pass2 tallies file bases (modeFileBaseOnly). + RecordsParsed int + ParseErrors int + Pass1, Pass2 time.Duration + Wall time.Duration + + // Volume info reported back for the CLI. + TotalMFTRecords int64 + MFTBytes int64 + + // TopFiles is the N largest in-scope files, sorted descending by Size. + // Populated only when Options.TopFiles > 0. Paths are resolved post-scan + // via OpenFileByID; on resolution failure the entry's Path is + // "?\". + TopFiles []FileEntry + + // TopExtensions is the N largest file extensions by aggregated in-scope + // size, sorted descending. Populated only when Options.TopExtensions > 0. + TopExtensions []ExtensionEntry + + // FindResults is one block per Options.Finds entry, in input order. + // Each block carries the originating FindQuery and the matched files + // sorted by size descending. Populated only when Options.Finds is + // non-empty. + FindResults []FindResultBlock + + // ExcludedDirs is the number of distinct directories that were marked + // out-of-scope by the Exclude option. Reported for diagnostics. + ExcludedDirs int + + // Tree, when Options.TreeDepth > 0, is the depth-limited subtree + // rooted at the scan target. Cumulative sizes (subtree totals) are + // computed for every node at depth 0..TreeDepth from target. + // nil when Options.TreeDepth == 0. + Tree *TreeNode +} + +// TreeNode is one directory entry in the depth-limited tree returned by +// Scan when Options.TreeDepth > 0. Size is the cumulative subtree total +// for everything under this directory on the scanned volume — including +// dirs at depths beyond TreeDepth (their bytes roll up to the deepest +// in-tree ancestor). Children are sorted by Size descending then Name +// ascending; they are filtered by Options.TreeMinSize at the leaves but +// the root and all directories at depth ≤ TreeDepth that have any +// in-scope content are present. +type TreeNode struct { + Name string // target's full path on the root node; basename otherwise + Idx uint64 + Depth int // 0 = target, increasing toward leaves + Size int64 // cumulative subtree total + Files int // files in this subtree (same walk-up / hardlink-dedup rules as Size) + Dirs int // descendant directories in this subtree (excludes this node) + Reparse bool // dir is a reparse point (junction / symlink / mount point) + Children []*TreeNode +} + +// Options configures a scan. +type Options struct { + // ShowApparent reports logical (apparent) sizes instead of disk + // allocation. Default false: report on-disk allocation, which matches + // Windows Explorer "Size on disk" for sparse and compressed files. + ShowApparent bool + + // TopFiles, when > 0, populates Result.TopFiles with the N largest files + // found in the in-scope subtree. Tracked via a min-heap during pass 2; + // hot-path cost is one int comparison per file plus, for the few that + // qualify, basename decode. Path resolution happens once after the scan + // via OpenFileByID. + TopFiles int + + // TopExtensions, when > 0, populates Result.TopExtensions with the top-N + // file extensions ranked by aggregated size. Extensions whose aggregated + // total is below MinFileSize are dropped. Adds ~16 bytes of UTF-16 scanning + // per file in pass 2 — opt-in for that reason. + TopExtensions int + + // MinFileSize is the size floor. Files strictly smaller are not considered + // for the TopFiles heap or for any Finds predicate; extensions whose + // aggregated total is below it are dropped from TopExtensions. Useful to + // focus on large space consumers. It does NOT affect Subtree. 0 = no floor. + MinFileSize int64 + + // Finds is the list of independent file-matching predicates to evaluate + // during the scan. Each entry becomes its own per-query slot with its + // own Limit and result block in Result.FindResults; queries do not + // compete with each other for capacity. See FindQuery for the per-type + // Value syntax (ext / glob / regex). + Finds []FindQuery + + // FindFastNameDecode enables a zero-allocation in-place UTF-16 → ASCII + // decode for glob / regex predicate evaluation, with a fallback to the + // allocating decoder for non-ASCII filenames. Has no effect when every + // configured Find is an "ext" query (extension matching is already + // allocation-free). + FindFastNameDecode bool + + // Exclude is a list of absolute paths whose subtrees should be excluded + // from the scan totals. Each path is resolved to an MFT idx upfront; any + // directory whose ancestor chain includes one of these is treated as + // out-of-scope. Files in excluded subtrees do not count toward the subtree + // total, any node's totals, the top-files heap, or the extension + // aggregation. + Exclude []string + + // TreeDepth controls Result.Tree. 0 leaves Tree nil (only the Subtree total + // and top-files/extensions are produced). 1 returns the root plus its + // immediate children (the fast path). N >= 2 returns the root plus every + // directory at depth 1..N. Files at depths beyond TreeDepth still count — + // their bytes/counts roll up into the deepest in-tree ancestor. + TreeDepth int + + // TreeMinSize hides any tree node whose cumulative size is below this + // threshold from its parent's Children. Has no effect when TreeDepth is 0. + // The root (target) is always included. 0 = show every populated node. The + // threshold only affects which nodes are listed, not the totals on the + // nodes that are. + TreeMinSize int64 +} + +// Scan computes disk usage per immediate child of targetDir on the volume +// containing targetDir. Requires Administrator privileges (raw \\.\: +// open). The context is honored between MFT chunks; cancellation aborts the +// scan with ctx.Err(). +func Scan(ctx context.Context, targetDir string, opts Options) (*Result, error) { + abs, err := filepath.Abs(targetDir) + if err != nil { + return nil, fmt.Errorf("resolve %q: %w", targetDir, err) + } + abs = upcaseDriveLetter(abs) + if !strings.HasSuffix(abs, `\`) { + abs += `\` + } + if len(abs) < 3 || abs[1] != ':' { + return nil, fmt.Errorf("target must be an absolute Windows path: %q", abs) + } + + t0 := time.Now() + res := &Result{Target: abs} + + drive := abs[:1] + hVol, vol, err := openVolume(drive) + if err != nil { + return nil, err + } + defer windows.CloseHandle(hVol) + + res.TotalMFTRecords = vol.mftValidBytes / int64(vol.recordSize) + res.MFTBytes = vol.mftValidBytes + + mftExtents, err := getMFTExtents(hVol, vol) + if err != nil { + return nil, fmt.Errorf("MFT extents: %w", err) + } + + // Resolve target idx + immediate children via Windows API. This is the + // only place names are touched in the entire scan; the bulk MFT walk + // never decodes UTF-16 names. + // Pass abs with its trailing backslash. Stripping it on a drive root + // (e.g. "C:\" → "C:") is fatal: CreateFile("C:") opens the per-process + // current directory on drive C:, not the volume root — every subtree + // rooted at cwd then gets misattributed as "loose" during the C:\ scan. + // CreateFile + FILE_FLAG_BACKUP_SEMANTICS handles "C:\" and "C:\dir\" + // equivalently for non-root paths. + targetIdx, err := getMFTIdxFromPath(abs) + if err != nil { + return nil, fmt.Errorf("resolve target idx: %w", err) + } + children, err := enumerateImmediateChildren(abs) + if err != nil { + return nil, fmt.Errorf("enumerate children: %w", err) + } + sort.Slice(children, func(i, j int) bool { + return strings.ToLower(children[i].name) < strings.ToLower(children[j].name) + }) + + const ( + bucketOutside = -1 + bucketTarget = -2 + ) + + bucketByIdx := make(map[uint64]int, len(children)) + for i, c := range children { + bucketByIdx[c.idx] = i + } + + // Resolve exclusion paths to MFT idxs. We do this BEFORE pass 1 so that + // walkUp can short-circuit excluded subtrees as bucketOutside without + // any per-file cost in passes 2/3. + excludedIdxs := make(map[uint64]struct{}, len(opts.Exclude)) + for _, p := range opts.Exclude { + ap, err := filepath.Abs(p) + if err != nil { + continue + } + ap = upcaseDriveLetter(ap) + // Skip exclusions on a different volume — they cannot be in this MFT. + if len(ap) >= 2 && ap[1] == ':' && ap[0] != abs[0] { + continue + } + idx, err := getMFTIdxFromPath(ap) + if err != nil { + continue + } + excludedIdxs[idx] = struct{}{} + } + + // ===== Pass 1: build dirParent + extSize/extParents in one scan ===== + // Reads every in-use record (modeAll). For each: + // - directory base: record idx → parent in dirParent + // - directory whose $FILE_NAME spilled to an extension: stash via + // pendingExtParent / dirsAwaitingParent and reconcile end-of-pass + // - extension record: accumulate $DATA size into extSize[baseRef] + // and append $FILE_NAME parents into extParents[baseRef] + // + // Folding extension-record accumulation into this pass costs nothing + // extra: modeAll already fully parses every record. It eliminates a + // second full MFT scan that would otherwise re-stream the same bytes. + t1 := time.Now() + + // Map size hint: ~1 directory per ~5 records on typical Windows volumes. + // Overestimating costs nothing (Go's map shrinks unused buckets); under- + // estimating triggers ~lg(N) rehashes during pass 1. + dirHint := int(res.TotalMFTRecords / 5) + dirParent := make(map[uint64]uint64, dirHint) + pendingExtParent := make(map[uint64]uint64) + dirsAwaitingParent := make(map[uint64]struct{}) + + // Hint: typical Windows volumes have ~1.3% of MFT records as extensions. + extHint := int(res.TotalMFTRecords / 70) + extSize := make(map[uint64]int64, extHint) + extParents := make(map[uint64][]uint64, extHint) + + // dirName: populated by a focused post-pass scan after walkUp + // identifies which dirs are at depth ≤ TreeDepth. Captures decoded + // names for the (~10K-50K) displayed dirs only — NOT every dir on + // the volume. The bulk pass 1 walk stays name-free, preserving the + // project's "no UTF-16 decoding in the bulk walk" allocation + // discipline (saves ~25-30 MiB peak vs decoding every dir). + var dirName map[uint64]string + + parsed1, errs1 := streamPipelined(ctx, hVol, mftExtents, vol.recordSize, modeAll, func(idx uint64, e *mftEntry, baseRef uint64) { + // Skip deleted / unallocated MFT slots. + if !e.isInUse { + return + } + // Extension records belong to the base file; use its index for filtering. + check := idx + if baseRef != 0 { + check = baseRef + } + // Skip NTFS system metafiles ($MFT, $Bitmap, root, …) in slots 0–15. + if check <= maxMetafileMFTIndex { + return + } + + if baseRef != 0 { + // Extension record. Accumulate per-base $DATA size and + // $FILE_NAME parents for use by pass 2's tally. + var sz int64 + if opts.ShowApparent { + sz = e.dataSize + } else { + sz = e.allocatedSize + } + if sz > 0 { + extSize[baseRef] += sz + } + extParents[baseRef] = append(extParents[baseRef], e.hardlinkParents...) + + // Reconcile dir parent when $FILE_NAME spilled to an extension record. + if e.primaryParent == 0 { + return // no parent on this extension; nothing to stash or satisfy + } + if _, awaiting := dirsAwaitingParent[baseRef]; awaiting { + // Base dir was seen first without a parent — apply now. + dirParent[baseRef] = e.primaryParent + delete(dirsAwaitingParent, baseRef) + return + } + // Base not seen yet (or not awaiting): stash parent for when base is visited. + if _, exists := pendingExtParent[baseRef]; !exists { + pendingExtParent[baseRef] = e.primaryParent + } + return + } + + // Base record. + if !e.isDir { + delete(pendingExtParent, idx) // wasn't a dir; drop the stash + return + } + if e.primaryParent != 0 { + dirParent[idx] = e.primaryParent + delete(pendingExtParent, idx) + return + } + // Dir base with no $FILE_NAME (overflowed to ext) — recover from + // stash if seen, else mark awaiting. + if p, ok := pendingExtParent[idx]; ok { + dirParent[idx] = p + delete(pendingExtParent, idx) + return + } + dirsAwaitingParent[idx] = struct{}{} + }) + if err := ctx.Err(); err != nil { + return nil, err + } + res.Pass1 = time.Since(t1) + res.RecordsParsed += parsed1 + res.ParseErrors += errs1 + + // End-of-pass reconciliation for dirs whose ext arrived after the base. + for idx := range dirsAwaitingParent { + if p, ok := pendingExtParent[idx]; ok { + dirParent[idx] = p + } + } + pendingExtParent = nil + dirsAwaitingParent = nil + + res.ExcludedDirs = len(excludedIdxs) + + // Two paths from here: + // - TreeDepth <= 1: build dirBucket via walkUp; pass 2 looks up + // dirBucket[parent] in O(1). Depth 0 needs only the subtree totals; + // depth 1 additionally synthesizes a root+children tree from the + // per-child tallies. This is the fast path (modeFileBaseOnly, no + // per-file chain walks, names from the API enumeration). + // - TreeDepth >= 2: retain dirParent, classify each dir's depth, and walk + // dirParent per file in pass 2 to accumulate into anchorTotals at each + // in-tree ancestor (the general, nested path). + var dirBucket map[uint64]int // fast path (depth <= 1) + var bucketDirs []int // fast path: dir count per child (incl. the child itself) + var subtreeDirs int // fast path: total descendant dirs (root total) + var anchorTotals map[uint64]int64 // general path (depth >= 2) + var anchorFiles map[uint64]int // general path: file count per tree node + var treeDirsDepth map[uint64]int16 // idx → depth, only for tree dirs + if opts.TreeDepth <= 1 { + dirBucket = make(map[uint64]int, len(dirParent)) + dirBucket[targetIdx] = bucketTarget + for idx, b := range bucketByIdx { + dirBucket[idx] = b + } + for idx := range excludedIdxs { + if idx == targetIdx { + continue + } + dirBucket[idx] = bucketOutside + } + var walkUp func(idx uint64, depth int) int + walkUp = func(idx uint64, depth int) int { + if depth > 512 { + return bucketOutside + } + if b, ok := dirBucket[idx]; ok { + return b + } + p, ok := dirParent[idx] + if !ok { + dirBucket[idx] = bucketOutside + return bucketOutside + } + b := walkUp(p, depth+1) + dirBucket[idx] = b + return b + } + for idx := range dirParent { + walkUp(idx, 0) + } + // Tally directories per child, mirroring the byte walk-up: every dir + // resolves (via dirBucket) to the top-level child it descends from. + // bucketDirs[i] includes child i itself (assembly subtracts it so a + // child's reported Dirs is descendants only); subtreeDirs is the total + // descendant-dir count for the root. + bucketDirs = make([]int, len(children)) + for idx, b := range dirBucket { + if idx == targetIdx { + continue + } + if b >= 0 { + bucketDirs[b]++ + subtreeDirs++ + } + } + dirParent = nil + } else { + // Tree mode: classify each dir as depth-from-target or -1 (out + // of scope). The classify map is intentionally short-lived — + // it's the same memory shape as dirBucket would have been, + // but we only retain the small subset (tree dirs) afterward + // and free the rest. dirParent stays alive for the per-file + // walks in pass 2. + classify := make(map[uint64]int16, len(dirParent)) + classify[targetIdx] = 0 + for idx := range excludedIdxs { + if idx != targetIdx { + classify[idx] = -1 + } + } + var walkDepth func(idx uint64, recurse int) int16 + walkDepth = func(idx uint64, recurse int) int16 { + if d, ok := classify[idx]; ok { + return d + } + if recurse > 512 { + classify[idx] = -1 + return -1 + } + p, ok := dirParent[idx] + if !ok { + classify[idx] = -1 + return -1 + } + pd := walkDepth(p, recurse+1) + if pd < 0 { + classify[idx] = -1 + return -1 + } + d := pd + 1 + classify[idx] = d + return d + } + for idx := range dirParent { + walkDepth(idx, 0) + } + + // Extract the tree dirs (depth ≤ TreeDepth). Pre-seed + // anchorTotals with zero entries — pass 2's chain walk uses + // map presence to know whether to accumulate. + nTree := 0 + for _, d := range classify { + if d >= 0 && d <= int16(opts.TreeDepth) { + nTree++ + } + } + treeDirsDepth = make(map[uint64]int16, nTree) + anchorTotals = make(map[uint64]int64, nTree) + anchorFiles = make(map[uint64]int, nTree) + dirName = make(map[uint64]string, nTree) + for idx, d := range classify { + if d >= 0 && d <= int16(opts.TreeDepth) { + treeDirsDepth[idx] = d + anchorTotals[idx] = 0 + dirName[idx] = "" + } + } + classify = nil + // dirParent stays alive — pass 2 needs it to walk chains. + } + + // ===== Pass 2: file base records, immediate tally ===== + // modeFileBaseOnly skips dirs and extensions before the attribute walk. + // We immediately add into bucketTotals — no per-file map, no per-file + // slice allocation. + t2 := time.Now() + + // bucketTotals/bucketFiles are the per-child attribution slices, used only + // by the fast path (depth <= 1). The general path (depth >= 2) accumulates + // into anchorTotals/anchorFiles instead and leaves these nil. + var bucketTotals []int64 + var bucketFiles []int + if opts.TreeDepth <= 1 { + bucketTotals = make([]int64, len(children)) + bucketFiles = make([]int, len(children)) + } + var subtree int64 + var subtreeFiles int // fast path: total in-scope files (root total) + var multiParent int // files reachable via >= 2 distinct in-scope parents + + topF := newTopFiles(opts.TopFiles, opts.MinFileSize) + extAgg := newExtAggregator(opts.TopExtensions > 0) + matcher, err := newMatchSet(opts.Finds, opts.FindFastNameDecode, opts.MinFileSize) + if err != nil { + return nil, err + } + + // Pass 2 mode: when TreeDepth > 0 we use modeAll so dir base records + // flow through the callback for opportunistic name capture. + // Otherwise modeFileBaseOnly skips dirs/extensions before the + // attribute walk (saves ~25-30% of pass 2 wall when name capture + // isn't needed). + pass2Mode := modeFileBaseOnly + if opts.TreeDepth >= 2 { + pass2Mode = modeAll + } + + parsed2, errs2 := streamPipelined(ctx, hVol, mftExtents, vol.recordSize, pass2Mode, func(idx uint64, e *mftEntry, baseRef uint64) { + if !e.isInUse || baseRef != 0 || idx <= maxMetafileMFTIndex { + return + } + if e.isDir { + // Dir base record: in tree mode, capture name only for + // tree dirs (dirName has a placeholder entry pre-seeded). + // In depth=0 mode this branch never fires (modeFileBaseOnly + // skips dirs at the parser level). + if dirName != nil { + if _, want := dirName[idx]; want && len(e.nameBytes) > 0 { + dirName[idx] = decodeUTF16Name(e.nameBytes) + } + } + return + } + // Resolve size. Prefer $DATA; fall back to $FILE_NAME cached size + // when $DATA is missing. + var sz int64 + if opts.ShowApparent { + sz = e.dataSize + if sz == 0 { + sz = e.fnDataSize + } + } else { + sz = e.allocatedSize + if sz == 0 { + sz = e.fnAllocSize + } + } + if extra, ok := extSize[idx]; ok { + sz += extra + } + + if opts.TreeDepth <= 1 { + // Fast path: O(1) dirBucket lookups. Collect the file's distinct + // in-scope parents (for the MultiParentFiles diagnostic) and + // distinct in-scope buckets (for per-child size attribution) in one + // pass. Files directly under the target map to bucketTarget: they + // count toward the root totals (subtree/subtreeFiles) but not toward + // any child bucket — there is no separate "loose" concept. + var pInline [8]uint64 + parents := pInline[:0] + var bInline [8]int + buckets := bInline[:0] + visit := func(p uint64) { + b, ok := dirBucket[p] + if !ok || b == bucketOutside { + return + } + seenP := false + for _, x := range parents { + if x == p { + seenP = true + break + } + } + if !seenP { + parents = append(parents, p) + } + seenB := false + for _, x := range buckets { + if x == b { + seenB = true + break + } + } + if !seenB { + buckets = append(buckets, b) + } + } + for _, p := range e.hardlinkParents { + visit(p) + } + if pp, ok := extParents[idx]; ok { + for _, p := range pp { + visit(p) + } + } + if len(buckets) == 0 && e.primaryParent != 0 { + visit(e.primaryParent) + } + if len(buckets) == 0 { + return // not in scope — also skips top-N / extAgg / matcher + } + subtree += sz + subtreeFiles++ + if len(parents) > 1 { + multiParent++ + } + for _, b := range buckets { + if b >= 0 { + bucketTotals[b] += sz + bucketFiles[b]++ + } + } + // Top-N / extAgg / matcher fire only for in-scope files. + topF.consider(idx, e, sz) + extAgg.addFromName(e.nameBytes, sz) + matcher.consider(idx, e, sz) + return + } + + // Tree-mode path: walk dirParent per parent ref, accumulating + // the file's size into every tree-dir ancestor in the last + // TreeDepth+1 chain entries. The walk terminates at target + // (in scope) or by exhausting dirParent (out of scope). Dedup + // across hardlink parents via a small stack-allocated set. + var seenInline [16]uint64 + seen := seenInline[:0] + addUnique := func(a uint64) bool { + for _, x := range seen { + if x == a { + return false + } + } + seen = append(seen, a) + return true + } + var chainScratch [32]uint64 + var parentInline [16]uint64 + inScopeParents := parentInline[:0] // distinct in-scope parents (MultiParentFiles) + anyInScope := false + attribute := func(parentIdx uint64) { + chain := chainScratch[:0] + cur := parentIdx + reached := false + for steps := 0; steps < 512; steps++ { + if _, ex := excludedIdxs[cur]; ex { + return + } + chain = append(chain, cur) + if cur == targetIdx { + reached = true + break + } + p, ok := dirParent[cur] + if !ok { + return + } + cur = p + } + if !reached { + return + } + anyInScope = true + // Track distinct in-scope parents for the MultiParentFiles + // diagnostic (a file with two links in the same directory has one + // distinct parent and is not multi-parent). + seenParent := false + for _, x := range inScopeParents { + if x == parentIdx { + seenParent = true + break + } + } + if !seenParent { + inScopeParents = append(inScopeParents, parentIdx) + } + chainLen := len(chain) + start := chainLen - 1 - opts.TreeDepth + if start < 0 { + start = 0 + } + for i := start; i < chainLen; i++ { + if addUnique(chain[i]) { + anchorTotals[chain[i]] += sz + anchorFiles[chain[i]]++ + } + } + } + for _, p := range e.hardlinkParents { + attribute(p) + } + if parents, ok := extParents[idx]; ok { + for _, p := range parents { + attribute(p) + } + } + if !anyInScope { + return // out of scope — skip top-N / extAgg / matcher + } + subtree += sz + if len(inScopeParents) > 1 { + multiParent++ + } + // Top-N / extAgg / matcher fire only for in-scope files. + topF.consider(idx, e, sz) + extAgg.addFromName(e.nameBytes, sz) + matcher.consider(idx, e, sz) + }) + if err := ctx.Err(); err != nil { + return nil, err + } + res.Pass2 = time.Since(t2) + res.RecordsParsed += parsed2 + res.ParseErrors += errs2 + + // General path (depth >= 2): build Result.Tree from anchorTotals, inverting + // dirParent (over tree dirs) for parent → children. + // Fast path: synthesize a root+children Tree from the per-child tallies at + // depth 1, or leave Tree nil at depth 0 (the subtree total is in Subtree). + if opts.TreeDepth >= 2 { + // Tally directories per tree node, mirroring the per-file byte walk in + // pass 2: each directory contributes to every in-tree ancestor in the + // last TreeDepth+1 chain entries (dirs beyond TreeDepth roll up into the + // deepest in-tree ancestor). Directories have a single parent, so no + // hardlink dedup is needed; a dir counts toward its ancestors, not + // itself, so Dirs reports descendants. dirParent is still alive here. + anchorDirs := make(map[uint64]int, len(treeDirsDepth)) + var dirChain [32]uint64 + for d := range dirParent { + if d == targetIdx { + continue + } + if _, ex := excludedIdxs[d]; ex { + continue // excluded dir itself is out of scope + } + cur, ok := dirParent[d] + if !ok { + continue + } + chain := dirChain[:0] + reached := false + for steps := 0; steps < 512; steps++ { + if _, ex := excludedIdxs[cur]; ex { + break // under an excluded subtree + } + chain = append(chain, cur) + if cur == targetIdx { + reached = true + break + } + p, ok := dirParent[cur] + if !ok { + break + } + cur = p + } + if !reached { + continue + } + chainLen := len(chain) + start := chainLen - 1 - opts.TreeDepth + if start < 0 { + start = 0 + } + for i := start; i < chainLen; i++ { + anchorDirs[chain[i]]++ + } + } + + childrenByParent := make(map[uint64][]uint64, len(treeDirsDepth)) + for idx := range treeDirsDepth { + if idx == targetIdx { + continue + } + parent, ok := dirParent[idx] + if !ok { + continue + } + childrenByParent[parent] = append(childrenByParent[parent], idx) + } + reparseByIdx := make(map[uint64]bool, len(children)) + for _, c := range children { + reparseByIdx[c.idx] = c.reparse + } + var build func(idx uint64, depth int) *TreeNode + build = func(idx uint64, depth int) *TreeNode { + n := &TreeNode{ + Idx: idx, + Depth: depth, + Size: anchorTotals[idx], + Files: anchorFiles[idx], + Dirs: anchorDirs[idx], + Reparse: reparseByIdx[idx], + } + if idx == targetIdx { + n.Name = abs + } else { + n.Name = dirName[idx] + } + kids := childrenByParent[idx] + if len(kids) > 0 { + n.Children = make([]*TreeNode, 0, len(kids)) + for _, kidIdx := range kids { + if anchorTotals[kidIdx] < opts.TreeMinSize { + continue + } + n.Children = append(n.Children, build(kidIdx, depth+1)) + } + sort.SliceStable(n.Children, func(i, j int) bool { + if n.Children[i].Size != n.Children[j].Size { + return n.Children[i].Size > n.Children[j].Size + } + return n.Children[i].Name < n.Children[j].Name + }) + } + return n + } + res.Tree = build(targetIdx, 0) + dirParent = nil + } else if opts.TreeDepth == 1 { + // Fast path, depth 1: synthesize root + immediate-child nodes from the + // per-child tallies. Child names come from the API enumeration; the + // child totals are whole-subtree (walkUp attributes every descendant to + // its top-level child). Apply TreeMinSize to children; the root is + // always present. + root := &TreeNode{ + Idx: targetIdx, + Depth: 0, + Name: abs, + Size: subtree, + Files: subtreeFiles, + Dirs: subtreeDirs, + } + for i, c := range children { + if _, ex := excludedIdxs[c.idx]; ex { + continue + } + if bucketTotals[i] < opts.TreeMinSize { + continue + } + dirs := bucketDirs[i] - 1 // exclude the child directory itself + if dirs < 0 { + dirs = 0 + } + root.Children = append(root.Children, &TreeNode{ + Idx: c.idx, + Depth: 1, + Name: c.name, + Size: bucketTotals[i], + Files: bucketFiles[i], + Dirs: dirs, + Reparse: c.reparse, + }) + } + sort.SliceStable(root.Children, func(i, j int) bool { + if root.Children[i].Size != root.Children[j].Size { + return root.Children[i].Size > root.Children[j].Size + } + return root.Children[i].Name < root.Children[j].Name + }) + res.Tree = root + } + // TreeDepth == 0: res.Tree stays nil (the subtree total is in res.Subtree). + + // Drop the remaining maps before formatting. + extSize = nil + extParents = nil + dirBucket = nil + dirName = nil + anchorTotals = nil + + res.Subtree = subtree + res.MultiParentFiles = multiParent + + // Resolve top-file paths via OpenFileByID. Bounded by Options.TopFiles + // — typically tens to hundreds of syscall pairs. Volume root path is + // "C:\" form (not the raw \\.\C: device); CreateFile + BACKUP_SEMANTICS + // is what OpenFileByID needs as its rootDir. + if topF != nil { + volumeRoot := abs[:3] // "C:\" + res.TopFiles = resolveCandidatePaths(volumeRoot, topF.drained()) + } + if extAgg != nil { + res.TopExtensions = extAgg.topN(opts.TopExtensions, opts.MinFileSize) + } + if matcher != nil { + volumeRoot := abs[:3] + blocks := matcher.drained() + queries := matcher.queries() + res.FindResults = make([]FindResultBlock, len(blocks)) + for i, blk := range blocks { + res.FindResults[i] = FindResultBlock{ + Query: queries[i], + Matches: resolveCandidatePaths(volumeRoot, blk), + } + } + } + + res.Wall = time.Since(t0) + return res, nil +} + +// upcaseDriveLetter uppercases the drive letter on a Windows path, leaving +// the rest unchanged. Match the existing PoC's case-folding so paths that +// differ only in drive case still resolve identically. +func upcaseDriveLetter(p string) string { + if len(p) >= 2 && p[1] == ':' && p[0] >= 'a' && p[0] <= 'z' { + return strings.ToUpper(p[:1]) + p[1:] + } + return p +} + +// ------------------------------------------------------------------------- +// Volume open + NTFS volume data +// ------------------------------------------------------------------------- + +const fsctlGetNTFSVolumeData = 0x00090064 + +type ntfsVolumeData struct { + VolumeSerialNumber int64 + NumberSectors int64 + TotalClusters int64 + FreeClusters int64 + TotalReserved int64 + BytesPerSector uint32 + BytesPerCluster uint32 + BytesPerFileRecordSegment uint32 + ClustersPerFRS uint32 + MftValidDataLength int64 + MftStartLcn int64 + Mft2StartLcn int64 + MftZoneStart int64 + MftZoneEnd int64 +} + +type volumeInfo struct { + recordSize int + bytesPerCluster int64 + mftStartByte int64 + mftValidBytes int64 +} + +func openVolume(drive string) (windows.Handle, *volumeInfo, error) { + volPath := `\\.\` + drive + ":" + pw, err := windows.UTF16PtrFromString(volPath) + if err != nil { + return 0, nil, err + } + h, err := windows.CreateFile( + pw, + windows.GENERIC_READ, + windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE|windows.FILE_SHARE_DELETE, + nil, + windows.OPEN_EXISTING, + 0, + 0, + ) + if err != nil { + return 0, nil, fmt.Errorf("open %s (need admin): %w", volPath, err) + } + + var data ntfsVolumeData + var n uint32 + err = windows.DeviceIoControl( + h, + fsctlGetNTFSVolumeData, + nil, 0, + (*byte)(unsafe.Pointer(&data)), uint32(unsafe.Sizeof(data)), + &n, nil, + ) + if err != nil { + windows.CloseHandle(h) + return 0, nil, fmt.Errorf("FSCTL_GET_NTFS_VOLUME_DATA: %w", err) + } + + // Validate the volume layout against the streamer/parser's assumptions. + // We reject unfamiliar configurations loudly rather than producing + // silently-wrong byte totals — the failure modes are subtle (mis-aligned + // record reads, miscounted fixups) and would not be caught by sanity + // checks downstream. + if err := validateNTFSLayout(&data); err != nil { + windows.CloseHandle(h) + return 0, nil, err + } + + vol := &volumeInfo{ + recordSize: int(data.BytesPerFileRecordSegment), + bytesPerCluster: int64(data.BytesPerCluster), + mftStartByte: data.MftStartLcn * int64(data.BytesPerCluster), + mftValidBytes: data.MftValidDataLength, + } + return h, vol, nil +} + +// validateNTFSLayout rejects volume configurations the scanner cannot +// safely handle. Anything we let through has to behave correctly end to +// end; producing wrong totals on an odd layout is worse than failing. +func validateNTFSLayout(data *ntfsVolumeData) error { + // NTFS multi-sector transfer protection has a fixed 512-byte stride + // per the on-disk format spec — see MULTI_SECTOR_HEADER in MSDN. It is + // not derived from BytesPerSector, so 4Kn / Advanced Format volumes + // use the same stride. We therefore don't validate BytesPerSector; + // applyFixups' hardcoded 512 is correct on every NTFS volume. + const mstpStride = 512 + + if data.BytesPerCluster == 0 { + return errors.New("FSCTL_GET_NTFS_VOLUME_DATA returned BytesPerCluster=0") + } + if data.BytesPerFileRecordSegment == 0 { + return errors.New("FSCTL_GET_NTFS_VOLUME_DATA returned BytesPerFileRecordSegment=0") + } + + // MFT records must be a multiple of the MSTP stride; otherwise + // applyFixups' sector-end positions don't land inside the record. + if data.BytesPerFileRecordSegment%mstpStride != 0 { + return fmt.Errorf( + "unsupported NTFS layout: BytesPerFileRecordSegment=%d is not a multiple of %d", + data.BytesPerFileRecordSegment, mstpStride, + ) + } + + // streamPipelined parses each chunk as a flat array of records and has + // no machinery to carry partial bytes across extent boundaries. That + // assumption holds when each MFT record fits within a single cluster + // (default 4 KiB cluster, 1 KiB record). When clusters are smaller + // than records, a run of an odd number of clusters would split a + // record across extents and the streamer would silently miscount. + if data.BytesPerCluster < data.BytesPerFileRecordSegment { + return fmt.Errorf( + "unsupported NTFS layout: BytesPerCluster (%d) < BytesPerFileRecordSegment (%d); "+ + "MFT records may span data run boundaries and cannot be safely read", + data.BytesPerCluster, data.BytesPerFileRecordSegment, + ) + } + + return nil +} + +// ------------------------------------------------------------------------- +// MFT extents (record 0 + $ATTRIBUTE_LIST chasing) +// ------------------------------------------------------------------------- + +// extent is one contiguous on-disk byte range of the $MFT. +type extent struct { + byteOffset int64 + byteLength int64 +} + +// getMFTExtents reads MFT record 0, decodes its $DATA data runs, and chases +// $ATTRIBUTE_LIST entries to find extension records that hold additional +// $DATA runs (the $MFT itself is typically heavily fragmented). +func getMFTExtents(hVol windows.Handle, vol *volumeInfo) ([]extent, error) { + rec0 := make([]byte, vol.recordSize) + if err := readAt(hVol, rec0, vol.mftStartByte); err != nil { + return nil, fmt.Errorf("read record 0: %w", err) + } + if binary.LittleEndian.Uint32(rec0[0:4]) != mftSignature { + return nil, errors.New("record 0 bad signature") + } + if err := applyFixups(rec0, vol.recordSize); err != nil { + return nil, fmt.Errorf("record 0: %w", err) + } + + firstAttrOff := int(binary.LittleEndian.Uint16(rec0[0x14:0x16])) + var inline []extent + var attrList []attrListEntry + + for off := firstAttrOff; off+8 <= vol.recordSize; { + t := binary.LittleEndian.Uint32(rec0[off : off+4]) + if t == attrEndMarker || t == 0 { + break + } + al := int(binary.LittleEndian.Uint32(rec0[off+4 : off+8])) + if al < 16 || off+al > vol.recordSize { + break + } + switch t { + case attrData: + if rec0[off+8] == 1 && off+0x22 <= vol.recordSize { + drOff := int(binary.LittleEndian.Uint16(rec0[off+0x20 : off+0x22])) + inline = decodeDataRuns(rec0[off+drOff:off+al], vol.bytesPerCluster) + } + case attrAttributeList: + attrList = parseAttributeList(rec0[off : off+al]) + } + off += al + } + + if len(attrList) == 0 { + if len(inline) == 0 { + return nil, errors.New("no $DATA in record 0") + } + return inline, nil + } + + all := append([]extent(nil), inline...) + readByMFTIdx := func(mftIdx uint64) ([]byte, error) { + bo := int64(mftIdx) * int64(vol.recordSize) + var cum int64 + for _, ex := range all { + if bo < cum+ex.byteLength { + disk := ex.byteOffset + (bo - cum) + buf := make([]byte, vol.recordSize) + if err := readAt(hVol, buf, disk); err != nil { + return nil, err + } + return buf, nil + } + cum += ex.byteLength + } + return nil, fmt.Errorf("MFT idx %d not in known extents", mftIdx) + } + + seen := map[uint64]bool{0: true} + for _, e := range attrList { + if e.attrType != attrData || seen[e.mftRef] { + continue + } + seen[e.mftRef] = true + extRec, err := readByMFTIdx(e.mftRef) + if err != nil { + continue + } + if binary.LittleEndian.Uint32(extRec[0:4]) != mftSignature { + continue + } + if err := applyFixups(extRec, vol.recordSize); err != nil { + // Torn-write or malformed extension record — skip it. + // Some MFT extents may be missing from our list, but a wrong + // extent list is preferable to acting on corrupt bytes. + continue + } + + efa := int(binary.LittleEndian.Uint16(extRec[0x14:0x16])) + for off := efa; off+8 <= vol.recordSize; { + t := binary.LittleEndian.Uint32(extRec[off : off+4]) + if t == attrEndMarker || t == 0 { + break + } + al := int(binary.LittleEndian.Uint32(extRec[off+4 : off+8])) + if al < 16 || off+al > vol.recordSize { + break + } + if t == attrData && extRec[off+8] == 1 && off+0x22 <= vol.recordSize { + drOff := int(binary.LittleEndian.Uint16(extRec[off+0x20 : off+0x22])) + more := decodeDataRuns(extRec[off+drOff:off+al], vol.bytesPerCluster) + all = append(all, more...) + } + off += al + } + } + + return all, nil +} + +// readAt is a thin wrapper around ReadFile with explicit OVERLAPPED offset. +func readAt(h windows.Handle, buf []byte, offset int64) error { + var ol windows.Overlapped + ol.Offset = uint32(offset & 0xFFFFFFFF) + ol.OffsetHigh = uint32(offset >> 32) + var n uint32 + if err := windows.ReadFile(h, buf, &n, &ol); err != nil { + return err + } + if int(n) < len(buf) { + return fmt.Errorf("short read: %d < %d", n, len(buf)) + } + return nil +} + +// decodeDataRuns decodes an NTFS data run list into disk extents. +func decodeDataRuns(data []byte, bytesPerCluster int64) []extent { + var ext []extent + var lcn int64 + pos := 0 + for pos < len(data) { + hdr := data[pos] + if hdr == 0 { + break + } + pos++ + lenSz := int(hdr & 0x0F) + offSz := int((hdr >> 4) & 0x0F) + if lenSz == 0 || pos+lenSz+offSz > len(data) { + break + } + var runLen int64 + for i := 0; i < lenSz; i++ { + runLen |= int64(data[pos+i]) << (uint(i) * 8) + } + pos += lenSz + if offSz == 0 { + continue // sparse run, no offset + } + var runOff int64 + for i := 0; i < offSz; i++ { + runOff |= int64(data[pos+i]) << (uint(i) * 8) + } + if data[pos+offSz-1]&0x80 != 0 { + for i := offSz; i < 8; i++ { + runOff |= int64(0xFF) << (uint(i) * 8) + } + } + pos += offSz + lcn += runOff + ext = append(ext, extent{byteOffset: lcn * bytesPerCluster, byteLength: runLen * bytesPerCluster}) + } + return ext +} + +// attrListEntry is one entry from a resident $ATTRIBUTE_LIST attribute. +type attrListEntry struct { + attrType uint32 + mftRef uint64 +} + +// parseAttributeList decodes the resident form. Non-resident $ATTRIBUTE_LIST +// is rare and not handled here; the rest of the chain (extension records +// holding $DATA fragments) usually fits in a resident attr list. +func parseAttributeList(attr []byte) []attrListEntry { + if len(attr) < 24 || attr[8] == 1 { + return nil + } + contentOff := int(binary.LittleEndian.Uint16(attr[0x14:0x16])) + contentLen := int(binary.LittleEndian.Uint32(attr[0x10:0x14])) + if contentOff+contentLen > len(attr) { + return nil + } + c := attr[contentOff : contentOff+contentLen] + + var out []attrListEntry + pos := 0 + for pos+0x18 <= len(c) { + entryType := binary.LittleEndian.Uint32(c[pos : pos+4]) + entryLen := int(binary.LittleEndian.Uint16(c[pos+4 : pos+6])) + if entryLen < 0x18 || pos+entryLen > len(c) { + break + } + mftRef := binary.LittleEndian.Uint64(c[pos+0x10 : pos+0x18]) + out = append(out, attrListEntry{attrType: entryType, mftRef: MFTIndex(mftRef)}) + pos += entryLen + } + return out +} + +// ------------------------------------------------------------------------- +// Pipelined ReadFile streamer +// ------------------------------------------------------------------------- + +// streamPipelined reads MFT bytes via a producer goroutine into one of two +// 4 MiB buffers while the consumer parses the other, then invokes cb for +// each in-buffer record. The pipeline overlaps disk I/O with parsing on cold +// passes (~33% wall reduction in the reflection's measurements). +// +// Single mftEntry reused across all parses; cb MUST NOT retain *mftEntry or +// its hardlinkParents slice past return — copy out anything needed. +func streamPipelined( + ctx context.Context, + h windows.Handle, + extents []extent, + recordSize int, + mode parseMode, + cb func(idx uint64, entry *mftEntry, baseRef uint64), +) (parsed, errs int) { + const chunkRecords = 4096 + chunkBytes := chunkRecords * recordSize + + type chunk struct { + bufIdx int + n int + recordIndex uint64 + err error + } + bufs := [2][]byte{make([]byte, chunkBytes), make([]byte, chunkBytes)} + free := make(chan int, 2) + free <- 0 + free <- 1 + ready := make(chan chunk, 1) + + go func() { + defer close(ready) + recordIndex := uint64(0) + for _, ex := range extents { + extOff := ex.byteOffset + rem := ex.byteLength + for rem > 0 { + if ctx.Err() != nil { + return + } + toRead := int64(chunkBytes) + if toRead > rem { + toRead = rem + } + bi := <-free + buf := bufs[bi][:toRead] + var ol windows.Overlapped + ol.Offset = uint32(extOff & 0xFFFFFFFF) + ol.OffsetHigh = uint32(extOff >> 32) + var n uint32 + rerr := windows.ReadFile(h, buf, &n, &ol) + ready <- chunk{bufIdx: bi, n: int(n), recordIndex: recordIndex, err: rerr} + if rerr != nil { + nr := toRead / int64(recordSize) + recordIndex += uint64(nr) + extOff += toRead + rem -= toRead + } else { + recordIndex += uint64(int64(n) / int64(recordSize)) + extOff += int64(n) + rem -= int64(n) + } + } + } + }() + + var entry mftEntry + for ch := range ready { + if ch.err != nil { + free <- ch.bufIdx + continue + } + nRecs := ch.n / recordSize + buf := bufs[ch.bufIdx] + for i := 0; i < nRecs; i++ { + rb := buf[i*recordSize : (i+1)*recordSize] + idx := ch.recordIndex + uint64(i) + baseRef, perr := parseInto(rb, recordSize, &entry, mode) + if perr != nil { + errs++ + continue + } + parsed++ + cb(idx, &entry, baseRef) + } + free <- ch.bufIdx + } + return parsed, errs +} + +// ------------------------------------------------------------------------- +// Windows API target / child resolution +// ------------------------------------------------------------------------- + +// getMFTIdxFromPath returns the MFT record index of the file or directory at +// path. CreateFile + GetFileInformationByHandle gives us the volume-internal +// identity in FileIndexLow/High; the lower 48 bits match the MFT index the +// raw $FILE_NAME parser would produce. +// +// FILE_FLAG_OPEN_REPARSE_POINT prevents CreateFile from following reparse +// points (junctions, symlinks, volume mount points). Without it, opening a +// volume mount point like C:\d-mount returns the file ID of the *target* +// (the root of the mounted volume), which lives in a different MFT — using +// that index against the source volume's MFT collides with arbitrary +// records and silently misattributes their sizes. We always want the +// placeholder's own idx on the volume being scanned. +func getMFTIdxFromPath(path string) (uint64, error) { + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + return 0, err + } + h, err := windows.CreateFile( + pw, + 0, // metadata only + windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE|windows.FILE_SHARE_DELETE, + nil, + windows.OPEN_EXISTING, + windows.FILE_FLAG_BACKUP_SEMANTICS|windows.FILE_FLAG_OPEN_REPARSE_POINT, + 0, + ) + if err != nil { + return 0, fmt.Errorf("CreateFile(%q): %w", path, err) + } + defer windows.CloseHandle(h) + var info windows.ByHandleFileInformation + if err := windows.GetFileInformationByHandle(h, &info); err != nil { + return 0, err + } + return MFTIndex(uint64(info.FileIndexHigh)<<32 | uint64(info.FileIndexLow)), nil +} + +// childInfo pairs an immediate-child directory's display name with its MFT idx. +type childInfo struct { + name string + idx uint64 + reparse bool +} + +// enumerateImmediateChildren returns the immediate child directories of +// targetDir with their MFT indices, via FindFirstFile + per-child handle +// lookup. ~80 syscalls for a typical Windows root — milliseconds total. +func enumerateImmediateChildren(targetDir string) ([]childInfo, error) { + pattern := strings.TrimSuffix(targetDir, `\`) + `\*` + pw, err := windows.UTF16PtrFromString(pattern) + if err != nil { + return nil, err + } + var fd windows.Win32finddata + h, err := windows.FindFirstFile(pw, &fd) + if err != nil { + return nil, fmt.Errorf("FindFirstFile(%q): %w", pattern, err) + } + defer windows.FindClose(h) + + var out []childInfo + for { + if fd.FileAttributes&windows.FILE_ATTRIBUTE_DIRECTORY != 0 { + name := windows.UTF16ToString(fd.FileName[:]) + if name != "." && name != ".." { + childPath := strings.TrimSuffix(targetDir, `\`) + `\` + name + if idx, err := getMFTIdxFromPath(childPath); err == nil { + reparse := fd.FileAttributes&windows.FILE_ATTRIBUTE_REPARSE_POINT != 0 + out = append(out, childInfo{name: name, idx: idx, reparse: reparse}) + } + } + } + err := windows.FindNextFile(h, &fd) + if err != nil { + if err == windows.ERROR_NO_MORE_FILES { + break + } + return out, err + } + } + return out, nil +} diff --git a/builtins/internal/ntfsmft/du_windows_test.go b/builtins/internal/ntfsmft/du_windows_test.go new file mode 100644 index 00000000..b06f874e --- /dev/null +++ b/builtins/internal/ntfsmft/du_windows_test.go @@ -0,0 +1,1217 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "context" + "errors" + "os" + "path/filepath" + "testing" + "unsafe" + + "golang.org/x/sys/windows" +) + +// ------------------------------------------------------------------------- +// Admin-rights gate +// ------------------------------------------------------------------------- + +// requireAdmin skips the test unless the process is in the local Administrators +// group; Scan opens \\.\:, which requires elevation. +func requireAdmin(t *testing.T) { + t.Helper() + var sid *windows.SID + err := windows.AllocateAndInitializeSid( + &windows.SECURITY_NT_AUTHORITY, + 2, + windows.SECURITY_BUILTIN_DOMAIN_RID, + windows.DOMAIN_ALIAS_RID_ADMINS, + 0, 0, 0, 0, 0, 0, + &sid, + ) + if err != nil { + t.Skipf("AllocateAndInitializeSid failed: %v", err) + } + defer windows.FreeSid(sid) + member, err := windows.Token(0).IsMember(sid) + if err != nil { + t.Skipf("Token.IsMember failed: %v", err) + } + if !member { + t.Skip("requires Administrator privileges (raw \\.\\: open)") + } +} + +// ------------------------------------------------------------------------- +// Test helpers +// ------------------------------------------------------------------------- + +// scanOrSkip runs Scan(target). Requires admin (checked upfront). Skips the +// test if the environment cannot perform raw MFT reads — e.g. Windows +// Server Containers whose C: is a filesystem layer that exposes NTFS-shaped +// volume metadata but rejects raw block reads. CI runs in containers, so +// these tests are skipped rather than failed there. +func scanOrSkip(t *testing.T, target string, opts Options) *Result { + t.Helper() + requireAdmin(t) + res, err := Scan(context.Background(), target, opts) + if err != nil { + if isRawMFTUnsupported(err) { + t.Skipf("raw MFT access not supported on this volume (likely a container filesystem): %v", err) + } + t.Fatalf("Scan: %v", err) + } + return res +} + +// isRawMFTUnsupported reports whether the error indicates the underlying +// volume / sandbox does not permit raw MFT access. Maps to: +// - ERROR_NOT_SUPPORTED (50) — typical for container filesystem layers +// - ERROR_INVALID_FUNCTION (1) — non-NTFS volume (e.g. ReFS) +// - ERROR_ACCESS_DENIED (5) — sandbox blocks the volume handle +func isRawMFTUnsupported(err error) bool { + for _, code := range []windows.Errno{ + windows.ERROR_NOT_SUPPORTED, + windows.ERROR_INVALID_FUNCTION, + windows.ERROR_ACCESS_DENIED, + } { + if errors.Is(err, code) { + return true + } + } + return false +} + +// flushMetadataToDisk forces NTFS to flush the test file's $DATA, $FILE_NAME, +// and parent directory $INDEX entries to the on-disk MFT so the raw-volume +// scan can see them. +func flushMetadataToDisk(t *testing.T, path string) { + t.Helper() + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + t.Fatalf("utf16(%q): %v", path, err) + } + h, err := windows.CreateFile( + pw, windows.GENERIC_WRITE, + windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE, + nil, windows.OPEN_EXISTING, windows.FILE_FLAG_BACKUP_SEMANTICS, 0, + ) + if err != nil { + return + } + defer windows.CloseHandle(h) + _ = windows.FlushFileBuffers(h) +} + +// allocatedSize returns the on-disk allocation for a file via +// GetFileInformationByHandleEx(FileStandardInfo). +func allocatedSize(t *testing.T, path string) int64 { + t.Helper() + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + t.Fatalf("utf16(%q): %v", path, err) + } + h, err := windows.CreateFile(pw, 0, + windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE|windows.FILE_SHARE_DELETE, + nil, windows.OPEN_EXISTING, windows.FILE_FLAG_BACKUP_SEMANTICS, 0) + if err != nil { + t.Fatalf("open(%q): %v", path, err) + } + defer windows.CloseHandle(h) + + var info struct { + AllocationSize int64 + EndOfFile int64 + NumberOfLinks uint32 + DeletePending bool + Directory bool + _ [2]byte + } + const fileStandardInfo = 1 + if err := windows.GetFileInformationByHandleEx(h, fileStandardInfo, + (*byte)(unsafe.Pointer(&info)), uint32(unsafe.Sizeof(info))); err != nil { + t.Fatalf("GetFileInformationByHandleEx(%q): %v", path, err) + } + return info.AllocationSize +} + +// writeFile writes data to path, flushes metadata, and returns the on-disk +// allocated size. +func writeFile(t *testing.T, path string, data []byte) int64 { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir parent of %q: %v", path, err) + } + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + t.Fatalf("utf16(%q): %v", path, err) + } + h, err := windows.CreateFile(pw, + windows.GENERIC_WRITE, + windows.FILE_SHARE_READ, + nil, windows.CREATE_ALWAYS, windows.FILE_ATTRIBUTE_NORMAL, 0) + if err != nil { + t.Fatalf("create(%q): %v", path, err) + } + if len(data) > 0 { + var n uint32 + if err := windows.WriteFile(h, data, &n, nil); err != nil { + windows.CloseHandle(h) + t.Fatalf("write(%q): %v", path, err) + } + } + _ = windows.FlushFileBuffers(h) + windows.CloseHandle(h) + flushMetadataToDisk(t, filepath.Dir(path)) + return allocatedSize(t, path) +} + +func createHardLink(t *testing.T, newPath, existingPath string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(newPath), 0o755); err != nil { + t.Fatalf("mkdir parent of %q: %v", newPath, err) + } + npw, err := windows.UTF16PtrFromString(newPath) + if err != nil { + t.Fatalf("utf16(%q): %v", newPath, err) + } + epw, err := windows.UTF16PtrFromString(existingPath) + if err != nil { + t.Fatalf("utf16(%q): %v", existingPath, err) + } + if err := windows.CreateHardLink(npw, epw, 0); err != nil { + t.Fatalf("CreateHardLink(%q -> %q): %v", newPath, existingPath, err) + } + flushMetadataToDisk(t, filepath.Dir(newPath)) +} + +const fsctlSetSparse = 0x000900C4 +const fsctlSetCompression = 0x0009C040 + +// createSparseFile makes a fully sparse file of the given virtual size. +func createSparseFile(t *testing.T, path string, virtualSize int64) int64 { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir parent of %q: %v", path, err) + } + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + t.Fatalf("utf16(%q): %v", path, err) + } + h, err := windows.CreateFile(pw, + windows.GENERIC_WRITE, windows.FILE_SHARE_READ, + nil, windows.CREATE_ALWAYS, windows.FILE_ATTRIBUTE_NORMAL, 0) + if err != nil { + t.Fatalf("create(%q): %v", path, err) + } + defer windows.CloseHandle(h) + + var n uint32 + if err := windows.DeviceIoControl(h, fsctlSetSparse, nil, 0, nil, 0, &n, nil); err != nil { + t.Fatalf("FSCTL_SET_SPARSE(%q): %v", path, err) + } + + hi := int32(virtualSize >> 32) + if _, err := windows.SetFilePointer(h, int32(virtualSize&0xFFFFFFFF), + &hi, windows.FILE_BEGIN); err != nil { + t.Fatalf("SetFilePointer(%q): %v", path, err) + } + if err := windows.SetEndOfFile(h); err != nil { + t.Fatalf("SetEndOfFile(%q): %v", path, err) + } + if err := windows.FlushFileBuffers(h); err != nil { + t.Fatalf("FlushFileBuffers(%q): %v", path, err) + } + + flushMetadataToDisk(t, filepath.Dir(path)) + return allocatedSize(t, path) +} + +// createCompressedFile creates a file, marks it compressed, then writes +// highly-compressible data (zeros). +func createCompressedFile(t *testing.T, path string, dataSize int) int64 { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir parent of %q: %v", path, err) + } + pw, err := windows.UTF16PtrFromString(path) + if err != nil { + t.Fatalf("utf16(%q): %v", path, err) + } + h, err := windows.CreateFile(pw, + windows.GENERIC_READ|windows.GENERIC_WRITE, windows.FILE_SHARE_READ, + nil, windows.CREATE_ALWAYS, windows.FILE_ATTRIBUTE_NORMAL, 0) + if err != nil { + t.Fatalf("create(%q): %v", path, err) + } + + const compressionFormatDefault uint16 = 1 + cf := compressionFormatDefault + var n uint32 + if err := windows.DeviceIoControl(h, fsctlSetCompression, + (*byte)(unsafe.Pointer(&cf)), 2, nil, 0, &n, nil); err != nil { + windows.CloseHandle(h) + t.Fatalf("FSCTL_SET_COMPRESSION(%q): %v", path, err) + } + + if dataSize > 0 { + zeros := make([]byte, dataSize) + if err := windows.WriteFile(h, zeros, &n, nil); err != nil { + windows.CloseHandle(h) + t.Fatalf("write(%q): %v", path, err) + } + } + _ = windows.FlushFileBuffers(h) + windows.CloseHandle(h) + flushMetadataToDisk(t, filepath.Dir(path)) + return allocatedSize(t, path) +} + +// ------------------------------------------------------------------------- +// Tests +// ------------------------------------------------------------------------- + +func TestScan_BasicDirectories(t *testing.T) { + root := t.TempDir() + + a1 := writeFile(t, filepath.Join(root, "A", "file1.bin"), make([]byte, 4096)) + a2 := writeFile(t, filepath.Join(root, "A", "file2.bin"), make([]byte, 8192)) + b1 := writeFile(t, filepath.Join(root, "B", "file3.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + wantA := a1 + a2 + wantB := b1 + if got := findTreeChild(t, res.Tree, "A").Size; got != wantA { + t.Errorf("child A = %d, want %d", got, wantA) + } + if got := findTreeChild(t, res.Tree, "B").Size; got != wantB { + t.Errorf("child B = %d, want %d", got, wantB) + } + wantSubtree := wantA + wantB + if res.Subtree != wantSubtree { + t.Errorf("Subtree = %d, want %d", res.Subtree, wantSubtree) + } + if res.MultiParentFiles != 0 { + t.Errorf("MultiParentFiles = %d, want 0", res.MultiParentFiles) + } +} + +// A file directly under the target counts toward the root totals but is not a +// child node (only directories are); the child directory reports only its own +// subtree. +func TestScan_FilesDirectlyUnderTarget(t *testing.T) { + root := t.TempDir() + + direct := writeFile(t, filepath.Join(root, "loose.bin"), make([]byte, 4096)) + a1 := writeFile(t, filepath.Join(root, "A", "x.bin"), make([]byte, 8192)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != a1 { + t.Errorf("child A = %d, want %d (must exclude the file directly under target)", got, a1) + } + if res.Subtree != direct+a1 { + t.Errorf("Subtree = %d, want %d", res.Subtree, direct+a1) + } + if res.Tree.Size != direct+a1 { + t.Errorf("root Size = %d, want %d (whole subtree)", res.Tree.Size, direct+a1) + } +} + +func TestScan_NestedDirectories(t *testing.T) { + root := t.TempDir() + + deep := writeFile(t, filepath.Join(root, "A", "sub1", "sub2", "deep.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != deep { + t.Errorf("child A = %d, want %d (file in A/sub1/sub2/)", got, deep) + } + if res.Subtree != deep { + t.Errorf("Subtree = %d, want %d", res.Subtree, deep) + } +} + +func TestScan_HardlinkSameBucket(t *testing.T) { + root := t.TempDir() + + primary := filepath.Join(root, "A", "primary.bin") + link := filepath.Join(root, "A", "secondary.bin") + + sz := writeFile(t, primary, make([]byte, 4096)) + createHardLink(t, link, primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != sz { + t.Errorf("child A = %d, want %d (hard-linked file in same directory)", got, sz) + } + if res.Subtree != sz { + t.Errorf("Subtree = %d, want %d (dedup)", res.Subtree, sz) + } + if res.MultiParentFiles != 0 { + t.Errorf("MultiParentFiles = %d, want 0 (both links share one parent)", res.MultiParentFiles) + } +} + +func TestScan_HardlinkAcrossChildren(t *testing.T) { + root := t.TempDir() + + primary := filepath.Join(root, "A", "shared.bin") + link := filepath.Join(root, "B", "shared.bin") + + sz := writeFile(t, primary, make([]byte, 4096)) + if err := os.MkdirAll(filepath.Join(root, "B"), 0o755); err != nil { + t.Fatal(err) + } + createHardLink(t, link, primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != sz { + t.Errorf("child A = %d, want %d (cross-child hardlink)", got, sz) + } + if got := findTreeChild(t, res.Tree, "B").Size; got != sz { + t.Errorf("child B = %d, want %d (cross-child hardlink)", got, sz) + } + if res.Subtree != sz { + t.Errorf("Subtree = %d, want %d (cross-child should dedup)", res.Subtree, sz) + } + if res.MultiParentFiles != 1 { + t.Errorf("MultiParentFiles = %d, want 1", res.MultiParentFiles) + } +} + +func TestScan_HardlinkTargetAndChild(t *testing.T) { + root := t.TempDir() + + primary := filepath.Join(root, "loose.bin") + link := filepath.Join(root, "A", "linked.bin") + + sz := writeFile(t, primary, make([]byte, 4096)) + createHardLink(t, link, primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != sz { + t.Errorf("child A = %d, want %d", got, sz) + } + if res.Subtree != sz { + t.Errorf("Subtree = %d, want %d", res.Subtree, sz) + } + if res.MultiParentFiles != 1 { + t.Errorf("MultiParentFiles = %d, want 1 (target+child)", res.MultiParentFiles) + } +} + +func TestScan_SparseFile_AllocatedNotApparent(t *testing.T) { + root := t.TempDir() + + const virtual = 64 * 1024 * 1024 + allocated := createSparseFile(t, filepath.Join(root, "A", "sparse.bin"), virtual) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != allocated { + t.Errorf("child A allocated = %d, want %d (sparse: actual on-disk)", got, allocated) + } + if allocated >= virtual/4 { + t.Errorf("sparse file allocated %d is not significantly smaller than virtual %d — sparseness check failed", + allocated, virtual) + } +} + +func TestScan_SparseFile_Apparent(t *testing.T) { + root := t.TempDir() + + const virtual = 64 * 1024 * 1024 + createSparseFile(t, filepath.Join(root, "A", "sparse.bin"), virtual) + + res := scanOrSkip(t, root, Options{ShowApparent: true, TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != virtual { + t.Errorf("child A apparent = %d, want %d (apparent: virtual size)", got, virtual) + } +} + +func TestScan_CompressedFile(t *testing.T) { + root := t.TempDir() + + const dataSize = 256 * 1024 + allocated := createCompressedFile(t, filepath.Join(root, "A", "compressed.bin"), dataSize) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != allocated { + t.Errorf("child A allocated = %d, want %d (compressed)", got, allocated) + } + if allocated >= int64(dataSize) { + t.Errorf("compressed file allocated %d is not smaller than data %d — compression didn't kick in", + allocated, dataSize) + } +} + +func TestScan_ResidentSmallFile(t *testing.T) { + root := t.TempDir() + + sz := writeFile(t, filepath.Join(root, "A", "tiny.bin"), make([]byte, 100)) + + res := scanOrSkip(t, root, Options{ShowApparent: true, TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != 100 { + t.Errorf("child A apparent = %d, want 100 (resident $DATA)", got) + } + _ = sz +} + +func TestScan_TargetWithNoChildDirs(t *testing.T) { + root := t.TempDir() + + direct := writeFile(t, filepath.Join(root, "only.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if res.Tree == nil { + t.Fatal("Tree is nil with TreeDepth=1") + } + if len(res.Tree.Children) != 0 { + t.Errorf("Tree.Children = %+v, want empty (no child dirs)", res.Tree.Children) + } + if res.Tree.Files != 1 { + t.Errorf("root Files = %d, want 1", res.Tree.Files) + } + if res.Subtree != direct { + t.Errorf("Subtree = %d, want %d", res.Subtree, direct) + } +} + +func TestScan_EmptyTarget(t *testing.T) { + root := t.TempDir() + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if res.Tree == nil { + t.Fatal("Tree is nil with TreeDepth=1") + } + if len(res.Tree.Children) != 0 { + t.Errorf("Tree.Children = %+v, want empty", res.Tree.Children) + } + if res.Subtree != 0 { + t.Errorf("Subtree = %d, want 0", res.Subtree) + } +} + +func writeStream(t *testing.T, path, streamName string, data []byte) { + t.Helper() + full := path + if streamName != "" { + full = path + ":" + streamName + } + pw, err := windows.UTF16PtrFromString(full) + if err != nil { + t.Fatalf("utf16(%q): %v", full, err) + } + h, err := windows.CreateFile(pw, + windows.GENERIC_WRITE, windows.FILE_SHARE_READ, + nil, windows.OPEN_ALWAYS, windows.FILE_ATTRIBUTE_NORMAL, 0) + if err != nil { + t.Fatalf("create(%q): %v", full, err) + } + if len(data) > 0 { + var n uint32 + if err := windows.WriteFile(h, data, &n, nil); err != nil { + windows.CloseHandle(h) + t.Fatalf("write(%q): %v", full, err) + } + } + _ = windows.FlushFileBuffers(h) + windows.CloseHandle(h) +} + +func TestScan_FileWithAlternateDataStream(t *testing.T) { + root := t.TempDir() + + const mainBytes = 8192 + const adsBytes = 154 + mainPath := filepath.Join(root, "A", "downloaded.bin") + + if err := os.MkdirAll(filepath.Dir(mainPath), 0o755); err != nil { + t.Fatal(err) + } + writeStream(t, mainPath, "", make([]byte, mainBytes)) + writeStream(t, mainPath, "Zone.Identifier", make([]byte, adsBytes)) + flushMetadataToDisk(t, filepath.Dir(mainPath)) + + if got := allocatedSize(t, mainPath); got != mainBytes { + t.Fatalf("setup: unnamed stream allocated = %d, want %d", got, mainBytes) + } + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + want := int64(mainBytes + adsBytes) + if got := findTreeChild(t, res.Tree, "A").Size; got != want { + t.Errorf("child A = %d, want %d (main+ADS sum)", got, want) + } +} + +func TestEnumerateImmediateChildren_FlagsReparsePoints(t *testing.T) { + root := t.TempDir() + + if err := os.MkdirAll(filepath.Join(root, "A"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(root, "A"), filepath.Join(root, "B")); err != nil { + t.Skipf("cannot create directory symlink: %v", err) + } + + children, err := enumerateImmediateChildren(root + `\`) + if err != nil { + t.Fatalf("enumerateImmediateChildren: %v", err) + } + + got := map[string]bool{} + for _, c := range children { + got[c.name] = c.reparse + } + if r, ok := got["A"]; !ok || r { + t.Errorf("child A: reparse=%v ok=%v, want false/true", r, ok) + } + if r, ok := got["B"]; !ok || !r { + t.Errorf("child B: reparse=%v ok=%v, want true/true", r, ok) + } +} + +func TestGetMFTIdxFromPath_DoesNotFollowReparsePoint(t *testing.T) { + root := t.TempDir() + + if err := os.MkdirAll(filepath.Join(root, "A"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(root, "A"), filepath.Join(root, "B")); err != nil { + t.Skipf("cannot create directory symlink: %v", err) + } + + idxA, err := getMFTIdxFromPath(filepath.Join(root, "A")) + if err != nil { + t.Fatalf("idx A: %v", err) + } + idxB, err := getMFTIdxFromPath(filepath.Join(root, "B")) + if err != nil { + t.Fatalf("idx B: %v", err) + } + if idxA == idxB { + t.Errorf("idx A == idx B (%d) — CreateFile is following the reparse point", idxA) + } +} + +func TestScan_DriveRootFromDeepCwd(t *testing.T) { + sub := t.TempDir() + if len(sub) < 3 || sub[1] != ':' { + t.Skipf("temp dir %q is not on a Windows drive", sub) + } + driveRoot := sub[:2] + `\` + + t.Chdir(sub) + + idx, err := getMFTIdxFromPath(driveRoot) + if err != nil { + t.Fatalf("getMFTIdxFromPath(%q): %v", driveRoot, err) + } + if idx != rootDirMFTIndex { + t.Errorf("getMFTIdxFromPath(%q) = %d, want %d (NTFS volume root) — cwd %q leaked into resolution", + driveRoot, idx, rootDirMFTIndex, sub) + } +} + +// ------------------------------------------------------------------------- +// Top-N files / extensions / find predicates +// ------------------------------------------------------------------------- + +func TestScan_TopFilesAndExtensions(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "A", "small.txt"), make([]byte, 1024)) + writeFile(t, filepath.Join(root, "A", "medium.log"), make([]byte, 16*1024)) + writeFile(t, filepath.Join(root, "B", "large.dat"), make([]byte, 64*1024)) + writeFile(t, filepath.Join(root, "B", "extra.log"), make([]byte, 8*1024)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + TopFiles: 10, + TopExtensions: 10, + }) + + if len(res.TopFiles) < 4 { + t.Fatalf("TopFiles len = %d, want >= 4", len(res.TopFiles)) + } + // Sorted descending by size. + for i := 1; i < len(res.TopFiles); i++ { + if res.TopFiles[i-1].Size < res.TopFiles[i].Size { + t.Errorf("TopFiles not sorted descending: %+v", res.TopFiles) + break + } + } + if res.TopFiles[0].Size != 64*1024 { + t.Errorf("TopFiles[0].Size = %d, want %d (large.dat)", res.TopFiles[0].Size, 64*1024) + } + + extByName := map[string]ExtensionEntry{} + for _, e := range res.TopExtensions { + extByName[e.Ext] = e + } + if e := extByName["dat"]; e.Size != 64*1024 || e.Count != 1 { + t.Errorf("ext dat = %+v, want size=%d count=1", e, 64*1024) + } + if e := extByName["log"]; e.Size != 24*1024 || e.Count != 2 { + t.Errorf("ext log = %+v, want size=%d count=2", e, 24*1024) + } + if e := extByName["txt"]; e.Size != 1024 || e.Count != 1 { + t.Errorf("ext txt = %+v, want size=1024 count=1", e) + } +} + +func TestScan_TopFilesMinFileSize(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "A", "tiny.bin"), make([]byte, 100)) + writeFile(t, filepath.Join(root, "A", "big.bin"), make([]byte, 32*1024)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + TopFiles: 10, + MinFileSize: 16 * 1024, + }) + + if len(res.TopFiles) != 1 { + t.Fatalf("TopFiles len = %d, want 1 (only big.bin qualifies)", len(res.TopFiles)) + } + if res.TopFiles[0].Size != 32*1024 { + t.Errorf("TopFiles[0].Size = %d, want %d", res.TopFiles[0].Size, 32*1024) + } +} + +// MinFileSize drops extensions whose AGGREGATED total is below the threshold +// from TopExtensions (this is an aggregate filter, not a per-file one: many +// small files of one extension can still sum above the floor). +func TestScan_TopExtMinFileSize(t *testing.T) { + root := t.TempDir() + + // .log aggregates to 20 KiB (below the 32 KiB floor); .dat is 64 KiB (above). + writeFile(t, filepath.Join(root, "A", "a.log"), make([]byte, 10*1024)) + writeFile(t, filepath.Join(root, "A", "b.log"), make([]byte, 10*1024)) + writeFile(t, filepath.Join(root, "A", "big.dat"), make([]byte, 64*1024)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + TopExtensions: 10, + MinFileSize: 32 * 1024, + }) + + byExt := map[string]ExtensionEntry{} + for _, e := range res.TopExtensions { + byExt[e.Ext] = e + } + if e, ok := byExt["dat"]; !ok || e.Size != 64*1024 { + t.Errorf("ext dat = %+v (ok=%v), want size=%d (above floor)", e, ok, 64*1024) + } + if e, ok := byExt["log"]; ok { + t.Errorf("ext log present with size %d but 20 KiB aggregate is below the 32 KiB floor", e.Size) + } +} + +func TestScan_FindByExtension(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "A", "crash.dmp"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "trace.etl"), make([]byte, 8192)) + writeFile(t, filepath.Join(root, "A", "ignore.txt"), make([]byte, 1024)) + writeFile(t, filepath.Join(root, "B", "other.dmp"), make([]byte, 2048)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + Finds: []FindQuery{ + {Type: "ext", Value: ".dmp,.etl", Limit: 10}, + }, + }) + + if len(res.FindResults) != 1 { + t.Fatalf("FindResults len = %d, want 1", len(res.FindResults)) + } + matches := res.FindResults[0].Matches + if len(matches) != 3 { + t.Fatalf("matches len = %d, want 3; got %+v", len(matches), matches) + } + for i := 1; i < len(matches); i++ { + if matches[i-1].Size < matches[i].Size { + t.Errorf("matches not sorted descending: %+v", matches) + break + } + } + if matches[0].Size != 8192 { + t.Errorf("matches[0].Size = %d, want 8192 (trace.etl)", matches[0].Size) + } +} + +// MinFileSize floors --find results the same way it floors the top-files list: +// matches strictly smaller than the threshold are excluded, so --find surfaces +// only large hits (the "find large .dmp files" use case). +func TestScan_FindRespectsMinFileSize(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "A", "tiny.dmp"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "big.dmp"), make([]byte, 64*1024)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + MinFileSize: 16 * 1024, + Finds: []FindQuery{ + {Type: "ext", Value: ".dmp", Limit: 10}, + }, + }) + + if len(res.FindResults) != 1 { + t.Fatalf("FindResults len = %d, want 1", len(res.FindResults)) + } + matches := res.FindResults[0].Matches + if len(matches) != 1 { + t.Fatalf("matches len = %d, want 1 (tiny.dmp is below --min); got %+v", len(matches), matches) + } + if matches[0].Size != 64*1024 { + t.Errorf("matches[0].Size = %d, want %d (big.dmp)", matches[0].Size, 64*1024) + } +} + +func TestScan_FindByGlob(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "A", "report-2026.log"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "A", "report-old.log"), make([]byte, 2048)) + writeFile(t, filepath.Join(root, "A", "other.log"), make([]byte, 1024)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + Finds: []FindQuery{ + {Type: "glob", Value: "report-*.log", Limit: 10}, + }, + }) + + if len(res.FindResults) != 1 { + t.Fatalf("FindResults len = %d, want 1", len(res.FindResults)) + } + if got := len(res.FindResults[0].Matches); got != 2 { + t.Fatalf("matches len = %d, want 2; got %+v", got, res.FindResults[0].Matches) + } +} + +func TestScan_FindLimitCapsResults(t *testing.T) { + root := t.TempDir() + + for i, sz := range []int{1024, 2048, 4096, 8192, 16384} { + writeFile(t, filepath.Join(root, "A", "f"+string(rune('a'+i))+".dat"), make([]byte, sz)) + } + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + Finds: []FindQuery{ + {Type: "ext", Value: ".dat", Limit: 3}, + }, + }) + + if len(res.FindResults) != 1 { + t.Fatalf("FindResults len = %d, want 1", len(res.FindResults)) + } + matches := res.FindResults[0].Matches + if len(matches) != 3 { + t.Fatalf("matches len = %d, want 3 (per-query Limit)", len(matches)) + } + // The 3 largest should win (16384, 8192, 4096). + wantSizes := []int64{16384, 8192, 4096} + for i, w := range wantSizes { + if matches[i].Size != w { + t.Errorf("matches[%d].Size = %d, want %d", i, matches[i].Size, w) + } + } +} + +func TestValidateNTFSLayout(t *testing.T) { + // modern-default NTFS layout: 4 KiB cluster, 1 KiB record, 512 sector. + ok := ntfsVolumeData{ + BytesPerSector: 512, + BytesPerCluster: 4096, + BytesPerFileRecordSegment: 1024, + } + if err := validateNTFSLayout(&ok); err != nil { + t.Errorf("ok layout rejected: %v", err) + } + + // equal-size cluster and record (legacy small-volume layout) — still OK. + okEq := ok + okEq.BytesPerCluster = 1024 + if err := validateNTFSLayout(&okEq); err != nil { + t.Errorf("cluster==record layout rejected: %v", err) + } + + // cluster < record — the codex-flagged case. + bad := ok + bad.BytesPerCluster = 512 + if err := validateNTFSLayout(&bad); err == nil { + t.Error("cluster < record was accepted; expected rejection") + } + + // 4Kn drives report BytesPerSector=4096; the MSTP stride is still + // 512 per the NTFS spec, so this configuration must NOT be rejected. + ok4kn := ok + ok4kn.BytesPerSector = 4096 + if err := validateNTFSLayout(&ok4kn); err != nil { + t.Errorf("4Kn (BytesPerSector=4096) layout rejected: %v", err) + } + + // record size not a multiple of 512. + badRec := ok + badRec.BytesPerFileRecordSegment = 1000 + if err := validateNTFSLayout(&badRec); err == nil { + t.Error("non-512-multiple record size was accepted; expected rejection") + } + + // zeroed fields that the validator must reject. + for _, field := range []string{"cluster", "record"} { + z := ok + switch field { + case "cluster": + z.BytesPerCluster = 0 + case "record": + z.BytesPerFileRecordSegment = 0 + } + if err := validateNTFSLayout(&z); err == nil { + t.Errorf("zero %s was accepted; expected rejection", field) + } + } +} + +func TestScan_FindMultipleQueriesIndependentLimits(t *testing.T) { + root := t.TempDir() + + // Two .dmp files (sizes 100, 200) and two .log files (sizes 50, 75). + // If the matcher shared a single heap, the 100/200 dmps could evict + // the smaller logs and leave the log query empty. + writeFile(t, filepath.Join(root, "a.dmp"), make([]byte, 100)) + writeFile(t, filepath.Join(root, "b.dmp"), make([]byte, 200)) + writeFile(t, filepath.Join(root, "c.log"), make([]byte, 50)) + writeFile(t, filepath.Join(root, "d.log"), make([]byte, 75)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + Finds: []FindQuery{ + {Type: "ext", Value: ".dmp", Limit: 10, Label: "dumps"}, + {Type: "ext", Value: ".log", Limit: 10, Label: "logs"}, + }, + }) + + if len(res.FindResults) != 2 { + t.Fatalf("FindResults len = %d, want 2", len(res.FindResults)) + } + dmps := res.FindResults[0] + logs := res.FindResults[1] + if dmps.Query.Label != "dumps" || logs.Query.Label != "logs" { + t.Errorf("labels misordered: got %q, %q", dmps.Query.Label, logs.Query.Label) + } + if len(dmps.Matches) != 2 { + t.Errorf("dumps matches = %d, want 2", len(dmps.Matches)) + } + if len(logs.Matches) != 2 { + t.Errorf("logs matches = %d, want 2 (must not be starved by larger dumps)", len(logs.Matches)) + } +} + +func TestScan_FindLongExtensionNotTruncated(t *testing.T) { + root := t.TempDir() + + // .crdownload (10 chars) and .application (11 chars) — both > 8 chars, + // which would be silently truncated by the old 8-byte buffer. + writeFile(t, filepath.Join(root, "partial.crdownload"), make([]byte, 4096)) + writeFile(t, filepath.Join(root, "installer.application"), make([]byte, 8192)) + writeFile(t, filepath.Join(root, "decoy.crdown"), make([]byte, 100)) + + res := scanOrSkip(t, root, Options{ + ShowApparent: true, + Finds: []FindQuery{ + {Type: "ext", Value: ".crdownload"}, + {Type: "ext", Value: ".application"}, + }, + }) + + if len(res.FindResults) != 2 { + t.Fatalf("FindResults len = %d, want 2", len(res.FindResults)) + } + if got := len(res.FindResults[0].Matches); got != 1 { + t.Errorf("crdownload matches = %d, want 1 (the decoy.crdown must NOT match)", got) + } + if got := len(res.FindResults[1].Matches); got != 1 { + t.Errorf("application matches = %d, want 1", got) + } +} + +func TestScan_ExcludeSubtree(t *testing.T) { + root := t.TempDir() + + keep := writeFile(t, filepath.Join(root, "Keep", "x.bin"), make([]byte, 4096)) + dropDir := filepath.Join(root, "Drop") + if err := os.MkdirAll(dropDir, 0o755); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(dropDir, "y.bin"), make([]byte, 8192)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1, Exclude: []string{dropDir}}) + + if res.Subtree != keep { + t.Errorf("Subtree = %d, want %d (Drop must be excluded)", res.Subtree, keep) + } + for _, c := range res.Tree.Children { + if c.Name == "Drop" { + t.Errorf("Tree.Children includes excluded dir %q with size %d", c.Name, c.Size) + } + } +} + +// ------------------------------------------------------------------------- +// Depth-N tree-mode tests +// ------------------------------------------------------------------------- + +func findTreeChild(t *testing.T, node *TreeNode, name string) *TreeNode { + t.Helper() + for _, c := range node.Children { + if c.Name == name { + return c + } + } + names := make([]string, len(node.Children)) + for i, c := range node.Children { + names[i] = c.Name + } + t.Fatalf("tree child %q not found under %q, have: %v", name, node.Name, names) + return nil +} + +// Depth 1 (the fast path) returns the root plus its immediate children, each +// carrying its whole-subtree total, and the root's Size equals Subtree. +func TestScan_TreeDepth1(t *testing.T) { + root := t.TempDir() + + a1 := writeFile(t, filepath.Join(root, "A", "x.bin"), make([]byte, 4096)) + a2 := writeFile(t, filepath.Join(root, "A", "sub", "y.bin"), make([]byte, 8192)) + b1 := writeFile(t, filepath.Join(root, "B", "z.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + if res.Tree == nil { + t.Fatal("Tree is nil with TreeDepth=1") + } + if res.Tree.Depth != 0 { + t.Errorf("root depth = %d, want 0", res.Tree.Depth) + } + treeA := findTreeChild(t, res.Tree, "A") + if treeA.Size != a1+a2 { + t.Errorf("Tree[A] size = %d, want %d (cumulative, incl. A/sub)", treeA.Size, a1+a2) + } + if treeA.Depth != 1 { + t.Errorf("Tree[A] depth = %d, want 1", treeA.Depth) + } + if got := findTreeChild(t, res.Tree, "B").Size; got != b1 { + t.Errorf("Tree[B] size = %d, want %d", got, b1) + } + if res.Tree.Size != res.Subtree { + t.Errorf("root Size %d != Subtree %d", res.Tree.Size, res.Subtree) + } +} + +func TestScan_TreeDepth2Cumulative(t *testing.T) { + root := t.TempDir() + + a := writeFile(t, filepath.Join(root, "A", "x.bin"), make([]byte, 4096)) + y := writeFile(t, filepath.Join(root, "A", "sub", "y.bin"), make([]byte, 8192)) + z := writeFile(t, filepath.Join(root, "A", "sub", "deeper", "z.bin"), make([]byte, 16384)) + + res := scanOrSkip(t, root, Options{TreeDepth: 2}) + + want := a + y + z + treeA := findTreeChild(t, res.Tree, "A") + if treeA.Size != want { + t.Errorf("Tree[A] cumulative = %d, want %d", treeA.Size, want) + } + subNode := findTreeChild(t, treeA, "sub") + if subNode.Depth != 2 { + t.Errorf("Tree[A/sub] depth = %d, want 2", subNode.Depth) + } + if subNode.Size != y+z { + t.Errorf("Tree[A/sub] cumulative = %d, want %d (y + z)", subNode.Size, y+z) + } + for _, c := range subNode.Children { + if c.Name == "deeper" { + t.Errorf("Tree[A/sub] has child %q at depth %d but TreeDepth=2", c.Name, c.Depth) + } + } +} + +// Files directly under the target roll into the root total (there is no longer +// a separate "loose" bucket), and TreeMinSize filters displayed children +// without changing any total. +func TestScan_TreeRootTotalsIncludeDirectFiles(t *testing.T) { + root := t.TempDir() + + direct1 := writeFile(t, filepath.Join(root, "loose1.bin"), make([]byte, 4096)) + direct2 := writeFile(t, filepath.Join(root, "loose2.bin"), make([]byte, 4096)) + bigChild := writeFile(t, filepath.Join(root, "Big", "x.bin"), make([]byte, 65536)) + smallChild := writeFile(t, filepath.Join(root, "Small", "y.bin"), make([]byte, 4096)) + + wantSubtree := direct1 + direct2 + bigChild + smallChild + + r1 := scanOrSkip(t, root, Options{TreeDepth: 2}) + if r1.Subtree != wantSubtree { + t.Errorf("TreeDepth=2 Subtree = %d, want %d", r1.Subtree, wantSubtree) + } + if r1.Tree.Size != wantSubtree { + t.Errorf("root Size = %d, want %d (incl. files directly under target)", r1.Tree.Size, wantSubtree) + } + if r1.Tree.Files != 4 { + t.Errorf("root Files = %d, want 4 (2 direct + 2 in children)", r1.Tree.Files) + } + + r2 := scanOrSkip(t, root, Options{TreeDepth: 2, TreeMinSize: 32 * 1024}) + if r2.Subtree != wantSubtree { + t.Errorf("TreeMinSize=32K Subtree = %d, want %d (must be unaffected by TreeMinSize)", r2.Subtree, wantSubtree) + } + for _, c := range r2.Tree.Children { + if c.Name == "Small" { + t.Errorf("Tree.Children includes Small but its size %d < TreeMinSize 32768", c.Size) + } + } +} + +func TestScan_TreeMinSizeOnlyAffectsTree(t *testing.T) { + root := t.TempDir() + + direct := writeFile(t, filepath.Join(root, "loose.bin"), make([]byte, 4096)) + big := writeFile(t, filepath.Join(root, "Big", "x.bin"), make([]byte, 65536)) + small := writeFile(t, filepath.Join(root, "Small", "y.bin"), make([]byte, 4096)) + + rNoFilter := scanOrSkip(t, root, Options{TreeDepth: 1}) + rFiltered := scanOrSkip(t, root, Options{TreeDepth: 1, TreeMinSize: 32 * 1024}) + + if rNoFilter.Subtree != rFiltered.Subtree { + t.Errorf("Subtree mismatch: no-filter=%d, filtered=%d", rNoFilter.Subtree, rFiltered.Subtree) + } + if rNoFilter.Subtree != direct+big+small { + t.Errorf("Subtree = %d, want %d", rNoFilter.Subtree, direct+big+small) + } + if len(rFiltered.Tree.Children) != 1 { + names := make([]string, len(rFiltered.Tree.Children)) + for i, c := range rFiltered.Tree.Children { + names[i] = c.Name + } + t.Errorf("filtered Tree.Children = %v, want only [Big]", names) + } +} + +func TestScan_TreeHardlinkAcrossChildren(t *testing.T) { + root := t.TempDir() + + primary := filepath.Join(root, "A", "shared.bin") + link := filepath.Join(root, "B", "shared.bin") + + sz := writeFile(t, primary, make([]byte, 4096)) + if err := os.MkdirAll(filepath.Join(root, "B"), 0o755); err != nil { + t.Fatal(err) + } + createHardLink(t, link, primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != sz { + t.Errorf("child A size = %d, want %d", got, sz) + } + if got := findTreeChild(t, res.Tree, "B").Size; got != sz { + t.Errorf("child B size = %d, want %d", got, sz) + } + if res.Subtree != sz { + t.Errorf("Subtree = %d, want %d (dedup hard-linked file)", res.Subtree, sz) + } + if res.MultiParentFiles != 1 { + t.Errorf("MultiParentFiles = %d, want 1", res.MultiParentFiles) + } +} + +// A file directly under the target that is also hard-linked into a child dir is +// deduped in the root total, attributed to the child, and counted as +// multi-parent (target + child are two distinct in-scope parents). +func TestScan_TreeDirectFileHardlinkedToChild(t *testing.T) { + root := t.TempDir() + + primary := filepath.Join(root, "loose.bin") + link := filepath.Join(root, "A", "shared.bin") + + sz := writeFile(t, primary, make([]byte, 4096)) + if err := os.MkdirAll(filepath.Join(root, "A"), 0o755); err != nil { + t.Fatal(err) + } + createHardLink(t, link, primary) + + res := scanOrSkip(t, root, Options{TreeDepth: 1}) + + if got := findTreeChild(t, res.Tree, "A").Size; got != sz { + t.Errorf("child A = %d, want %d", got, sz) + } + if res.Subtree != sz { + t.Errorf("Subtree = %d, want %d (dedup)", res.Subtree, sz) + } + if res.MultiParentFiles != 1 { + t.Errorf("MultiParentFiles = %d, want 1", res.MultiParentFiles) + } +} + +func TestScan_TreeExcludedSubtree(t *testing.T) { + root := t.TempDir() + + keep := writeFile(t, filepath.Join(root, "Keep", "x.bin"), make([]byte, 4096)) + dropDir := filepath.Join(root, "Drop") + if err := os.MkdirAll(dropDir, 0o755); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(dropDir, "y.bin"), make([]byte, 8192)) + + res := scanOrSkip(t, root, Options{TreeDepth: 1, Exclude: []string{dropDir}}) + + if res.Subtree != keep { + t.Errorf("Subtree = %d, want %d (Drop must be excluded)", res.Subtree, keep) + } + for _, c := range res.Tree.Children { + if c.Name == "Drop" { + t.Errorf("Tree.Children includes excluded dir %q", c.Name) + } + } +} + +// TreeMinSize filters depth-1 children out of the tree at depth >= 2 as well. +func TestScan_TreeMinSizeFiltersChildrenAtDepth2(t *testing.T) { + root := t.TempDir() + + writeFile(t, filepath.Join(root, "Big", "x.bin"), make([]byte, 65536)) + writeFile(t, filepath.Join(root, "Small", "y.bin"), make([]byte, 4096)) + + res := scanOrSkip(t, root, Options{TreeDepth: 2, TreeMinSize: 32 * 1024}) + + if findTreeChild(t, res.Tree, "Big").Size <= 0 { + t.Errorf("child Big missing or zero size") + } + for _, c := range res.Tree.Children { + if c.Name == "Small" { + t.Errorf("Tree.Children contains Small (size %d) but TreeMinSize should filter it", c.Size) + } + } +} diff --git a/builtins/internal/ntfsmft/matcher.go b/builtins/internal/ntfsmft/matcher.go new file mode 100644 index 00000000..2a13502c --- /dev/null +++ b/builtins/internal/ntfsmft/matcher.go @@ -0,0 +1,310 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "bytes" + "container/heap" + "fmt" + "path/filepath" + "regexp" + "sort" + "strings" + "unsafe" +) + +// FindQuery is one filename-matching find request. +// +// - Type "ext": Value is a comma-separated list of extensions. Leading +// dots and case are ignored (e.g. ".dmp,.etl,DMP"). +// - Type "glob": Value is a single filepath.Match pattern matched against +// the basename (no path separators). +// - Type "regex": Value is an RE2 expression matched against the basename. +// +// Limit caps this query's result block; 0 selects a sensible default (100). +// Label is opaque, carried into FindResultBlock for caller attribution. +type FindQuery struct { + Type string + Value string + Limit int + Label string +} + +// FindResultBlock pairs a FindQuery with the files it matched, sorted by +// size descending then basename ascending and trimmed to the query's Limit. +type FindResultBlock struct { + Query FindQuery + Matches []FileEntry +} + +// matchSlot is one per-query predicate + heap. Each FindQuery passed to +// newMatchSet becomes one slot. +type matchSlot struct { + query FindQuery + // Exactly one of the following is populated, per query.Type: + exts [][]byte // ext: pre-normalized lowercased extensions, no dot + glob string // glob: filepath.Match pattern + regex *regexp.Regexp // regex: compiled RE2 + heap fileHeap + cap int +} + +// matchSet evaluates per-file predicates against a list of independent +// FindQuery slots. Each slot retains its own top-Cap candidates via a +// min-heap. +// +// Hot-path cost when matchSet is nil: a single nil check per file. When +// multiple slots need the same input (extension or decoded basename), +// that work is done at most once per file via lazy locals in consider. +type matchSet struct { + slots []*matchSlot + // fast, when true, decodes ASCII filenames in-place into nameBuf and + // passes a non-allocating string view (unsafe.String) to filepath.Match + // and regexp.MatchString. Non-ASCII filenames fall back to the heap + // allocator. The string MUST NOT escape predicate evaluation — nameBuf + // is reused on the next file. + fast bool + nameBuf [512]byte + // minSize is the file-size floor: files strictly smaller are not + // considered for any query, matching the top-files floor semantics. + minSize int64 +} + +// newMatchSet builds a matcher from a list of FindQuery. Returns (nil, nil) +// when queries is empty. Returns an error if any query is malformed (empty +// value, unknown type, bad glob, bad regex). minSize excludes files smaller +// than the threshold from all queries (0 = no floor). +func newMatchSet(queries []FindQuery, fast bool, minSize int64) (*matchSet, error) { + if len(queries) == 0 { + return nil, nil + } + m := &matchSet{fast: fast, minSize: minSize} + for i, q := range queries { + if q.Value == "" { + return nil, fmt.Errorf("find[%d]: value must not be empty", i) + } + c := q.Limit + if c <= 0 { + c = 100 + } + slot := &matchSlot{query: q, cap: c, heap: make(fileHeap, 0, c)} + switch q.Type { + case "ext": + for _, e := range splitAndNormalizeExts(q.Value) { + slot.exts = append(slot.exts, []byte(e)) + } + if len(slot.exts) == 0 { + return nil, fmt.Errorf("find[%d]: ext value %q yielded no extensions", i, q.Value) + } + case "glob": + // Validate eagerly via probe; filepath.Match only reports + // ErrBadPattern when it encounters the bad character. + if _, err := filepath.Match(q.Value, "x"); err != nil { + return nil, fmt.Errorf("find[%d]: invalid glob %q: %w", i, q.Value, err) + } + slot.glob = q.Value + case "regex": + re, err := regexp.Compile(q.Value) + if err != nil { + return nil, fmt.Errorf("find[%d]: invalid regex %q: %w", i, q.Value, err) + } + slot.regex = re + default: + return nil, fmt.Errorf("find[%d]: unknown type %q (want \"ext\", \"glob\", or \"regex\")", i, q.Type) + } + m.slots = append(m.slots, slot) + } + return m, nil +} + +// splitAndNormalizeExts parses ".dmp,.etl,DMP" into ["dmp", "etl", "dmp"] +// (lowercased, leading dot stripped, blanks dropped). +func splitAndNormalizeExts(csv string) []string { + if csv == "" { + return nil + } + parts := strings.Split(csv, ",") + out := make([]string, 0, len(parts)) + for _, p := range parts { + p = strings.TrimSpace(p) + p = strings.TrimPrefix(p, ".") + p = strings.ToLower(p) + if p == "" { + continue + } + out = append(out, p) + } + return out +} + +// consider runs in the pass-2 hot path. Evaluates each slot's predicate +// against the file and pushes to the slot's heap on match. Per-file work +// (extension extraction, name decode) happens at most once, lazily. +func (m *matchSet) consider(idx uint64, e *mftEntry, sz int64) { + if m == nil { + return + } + // Size floor: skip files below --min before any predicate work, so + // --find surfaces only large matches (e.g. "find large .dmp files"). + if sz < m.minSize { + return + } + + // Lazy extension extraction (24-byte buffer covers all real-world + // extensions, e.g. ".crdownload", ".application", ".compositions"). + var extBuf [24]byte + var extN int + extEvaluated := false + getExt := func() ([]byte, bool) { + if !extEvaluated { + extEvaluated = true + extN = extractAsciiExtension(e.nameBytes, extBuf[:]) + } + if extN <= 0 { + return nil, false + } + return extBuf[:extN], true + } + + // Lazy name decode. name is the value handed to predicate evaluators + // (may be an unsafe view of nameBuf in fast+ASCII path). realName, if + // set, is a heap-allocated copy safe to retain past this call. + var name string + var realName string + nameEvaluated := false + getName := func() string { + if nameEvaluated { + return name + } + nameEvaluated = true + if m.fast { + n := utf16ToASCIIFast(e.nameBytes, m.nameBuf[:]) + if n >= 0 { + name = unsafe.String(&m.nameBuf[0], n) + return name + } + } + realName = decodeUTF16Name(e.nameBytes) + name = realName + return name + } + + for _, s := range m.slots { + matched := false + switch { + case len(s.exts) > 0: + ext, ok := getExt() + if !ok { + continue + } + for _, want := range s.exts { + if bytes.Equal(ext, want) { + matched = true + break + } + } + case s.glob != "": + if ok, _ := filepath.Match(s.glob, getName()); ok { + matched = true + } + case s.regex != nil: + if s.regex.MatchString(getName()) { + matched = true + } + } + if !matched { + continue + } + // push needs a heap-allocated basename. realName is empty in the + // fast+ASCII match path until we promote it here, and in the pure- + // ext match path until push lazily decodes (push avoids the decode + // if the heap rejects the candidate). + pushName := realName + if pushName == "" && nameEvaluated { + realName = decodeUTF16Name(e.nameBytes) + pushName = realName + } + s.push(idx, e, sz, pushName) + } +} + +// utf16ToASCIIFast copies a UTF-16LE filename into out as ASCII bytes. +// Returns the number of bytes written, or -1 if the name contains any +// non-ASCII code unit (high byte != 0). No allocations. +func utf16ToASCIIFast(nameUTF16, out []byte) int { + if len(nameUTF16)%2 != 0 { + return -1 + } + n := len(nameUTF16) / 2 + if n > len(out) { + return -1 + } + for i := 0; i < n; i++ { + hi := nameUTF16[i*2+1] + if hi != 0 { + return -1 + } + out[i] = nameUTF16[i*2] + } + return n +} + +// push inserts a matched candidate into the slot's heap. If the heap is +// at capacity, the smallest entry is evicted unless the new entry is also +// smaller — in which case nothing happens. The basename is decoded lazily +// here when the caller didn't already have one (extension match path), +// avoiding the decode for files we end up evicting. +func (s *matchSlot) push(idx uint64, e *mftEntry, sz int64, name string) { + if len(s.heap) >= s.cap && sz <= s.heap[0].size { + return + } + if name == "" { + name = decodeUTF16Name(e.nameBytes) + } + cand := fileCandidate{idx: idx, sequence: e.sequence, size: sz, basename: name} + if len(s.heap) < s.cap { + heap.Push(&s.heap, cand) + return + } + s.heap[0] = cand + heap.Fix(&s.heap, 0) +} + +// drained returns one block per slot, in input-query order. Each block is +// sorted by size desc, then basename asc. +func (m *matchSet) drained() [][]fileCandidate { + if m == nil { + return nil + } + out := make([][]fileCandidate, len(m.slots)) + for i, s := range m.slots { + blk := make([]fileCandidate, len(s.heap)) + copy(blk, s.heap) + sort.Slice(blk, func(a, b int) bool { + if blk[a].size != blk[b].size { + return blk[a].size > blk[b].size + } + return blk[a].basename < blk[b].basename + }) + out[i] = blk + } + return out +} + +// queries returns the original FindQuery slice in slot order. Used by +// callers to pair drained() output back to the input queries. +func (m *matchSet) queries() []FindQuery { + if m == nil { + return nil + } + out := make([]FindQuery, len(m.slots)) + for i, s := range m.slots { + out[i] = s.query + } + return out +} diff --git a/builtins/internal/ntfsmft/parser.go b/builtins/internal/ntfsmft/parser.go new file mode 100644 index 00000000..db992415 --- /dev/null +++ b/builtins/internal/ntfsmft/parser.go @@ -0,0 +1,440 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +// Package ntfsmft computes disk usage for a target directory on an NTFS volume +// by reading the raw $MFT. +// +// The volume I/O and scan orchestration are Windows-only (see du_windows.go); +// this file holds the pure $MFT record/attribute parser, which has no +// platform dependencies (stdlib only) so it can be unit-tested and fuzzed on +// any OS. +// +// Scan pipeline (see Scan in du_windows.go for section markers): +// +// - Setup: open \\.\:, resolve the target and its immediate children +// to MFT indices via the Windows API (CreateFile, GetFileInformationByHandle, +// FindFirstFile). Exclusion paths are resolved to indices before the MFT +// walks so out-of-scope subtrees short-circuit cheaply. +// +// - Pass 1 (modeAll, one full MFT stream): build dirParent (directory → +// parent idx), plus extSize and extParents per file base. Extension records +// are folded into this pass so their $DATA sizes and spillover $FILE_NAME +// parents are not rescanned. The bulk walk does not decode UTF-16 names. +// +// - Classify scope: TreeDepth <= 1 (fast path) precomputes dirBucket via +// walkUp from target and its immediate children; TreeDepth >= 2 (general +// path) retains dirParent and tree-dir anchor maps for per-file chain walks +// in pass 2. +// +// - Pass 2 (modeFileBaseOnly, or modeAll when TreeDepth >= 2): tally in-use +// file base records into per-child / subtree totals; optional top-N files, +// extension aggregation, and find predicates run inline in this callback. +// The general path opportunistically decodes names only for dirs at +// depth ≤ TreeDepth. +// +// - Post-scan: assemble the optional Result.Tree; resolve top-file paths via +// OpenFileByID (bounded, not part of the MFT stream). +// +// - Pipelined ReadFile (double-buffered) overlaps disk I/O with parsing. +// +// - parseMode header-only early exit skips the attribute walk on records a +// pass cannot use (see modeAll / modeFileBaseOnly below). +// +// - No per-file info map: pass 2 unions base + extension parents and adds +// directly into totals. No per-file slice allocation on the hot path. +// +// Requires Administrator privileges (\\.\C: open). +package ntfsmft + +import ( + "encoding/binary" + "errors" +) + +// ------------------------------------------------------------------------- +// MFT record / attribute constants +// ------------------------------------------------------------------------- + +const ( + mftSignature = 0x454C4946 // "FILE" little-endian + + attrStandardInfo = 0x10 + attrAttributeList = 0x20 + attrFileName = 0x30 + attrData = 0x80 + attrEndMarker = 0xFFFFFFFF + + flagInUse = 0x01 + flagDirectory = 0x02 + + nsPosix = 0x00 + nsWin32 = 0x01 + nsDOS = 0x02 + nsWin32AndDOS = 0x03 + + // Records 0–15 are NTFS metafiles ($MFT, $MFTMirr, $LogFile, $Volume, + // $AttrDef, root, $Bitmap, $Boot, $BadClus, $Secure, $UpCase, $Extend, + // reserved 12–15). Real user-actionable system files (pagefile.sys, + // hiberfil.sys, swapfile.sys) have idx >= 16. + maxMetafileMFTIndex = 15 + + // Root directory MFT index (always 5 on NTFS). + rootDirMFTIndex = 5 +) + +// errBadSignature indicates the record does not start with "FILE". +var errBadSignature = errors.New("bad MFT signature") + +// MFTIndex masks the lower 48 bits of an MFT file reference. The upper 16 are +// the sequence number; we don't need them for disk-usage tally because we +// always cross-reference by record index, not sequence-stamped reference. +func MFTIndex(ref uint64) uint64 { + return ref & 0x0000FFFFFFFFFFFF +} + +// ------------------------------------------------------------------------- +// Parsed MFT entry — caller-buffer reuse via parseInto +// ------------------------------------------------------------------------- + +// hardlinkParents are the parent MFT indices from the $FILE_NAME attributes of +// a single record. A file with N hardlinks contributes N entries (one per +// $FILE_NAME, except DOS-only 8.3 aliases which we drop). For a directory or +// a single-link file this typically has 1 entry. +// +// The slice's backing array is reused across records via parseInto. + +// mftEntry is the result of parsing a single MFT record. parseInto resets the +// fields on entry but preserves hardlinkParents' backing array, so the only +// allocations across many records are when hardlinkParents grows. +type mftEntry struct { + // hardlinkParents is the list of parent MFT indices from non-DOS + // $FILE_NAME attributes on this record. Used by the scan to attribute + // hard-linked files to multiple buckets. + hardlinkParents []uint64 + + // primaryParent is the parent MFT index of the highest-namespace-priority + // $FILE_NAME on this record. Falls back to hardlinkParents[0] if needed. + primaryParent uint64 + + // nameBytes is a slice into the record buffer of the raw UTF-16 little- + // endian name from the highest-priority $FILE_NAME. Valid ONLY for the + // duration of the streamPipelined callback that produced this entry; the + // underlying buffer is reused on the next chunk. Callers needing the name + // past the callback must copy it out. + nameBytes []byte + + // sequence is the MFT record sequence number (header offset 0x10). + // Combined with the record idx it forms the 64-bit NTFS file reference + // used by OpenFileByID for post-scan path resolution. + sequence uint16 + + // allocatedSize / dataSize come from $DATA. fnAllocSize / fnDataSize are + // the cached sizes from the highest-priority $FILE_NAME, used as a + // fallback when $DATA sizes are missing (typically when $DATA is in an + // extension record — covered by pass 2). + allocatedSize int64 + dataSize int64 + fnAllocSize int64 + fnDataSize int64 + + isInUse bool + isDir bool + isSparse bool + isCompressed bool +} + +// ------------------------------------------------------------------------- +// Parser modes +// ------------------------------------------------------------------------- + +// parseMode lets each pass skip records it cannot use after a 0x28-byte +// header read, before the attribute walk. Header-only early exit saves +// ~25–30% wall on the file-tally pass. +type parseMode uint8 + +const ( + // modeAll: parse every in-use record fully. The map-building pass uses + // this so it can see directory base records AND extension records (for + // dir-name spillover reconciliation, $DATA size accumulation per base, + // and cross-bucket hardlink parents from extension $FILE_NAMEs). + modeAll parseMode = iota + + // modeFileBaseOnly: skip records with baseRef != 0 OR isDir. The + // file-tally pass uses this. The parent-only $FILE_NAME walk still runs + // on bases so callers can attribute hardlinks across buckets. + modeFileBaseOnly +) + +// ------------------------------------------------------------------------- +// Top-level parse +// ------------------------------------------------------------------------- + +// parseInto parses one MFT record into the caller-provided *mftEntry, +// preserving hardlinkParents' backing array. +// +// Returns (baseRef, error). baseRef is 0 for base records and non-zero (the +// MFT index of the file the extension belongs to) for extension records. +// +// For mode != modeAll, the function may return early after the header read +// without populating any attribute-derived fields. Callers must check the +// pass-specific predicates again before use; e.g. pass 2's callback re-checks +// isDir even though modeFileBaseOnly already filtered it, because dir flag +// is a header bit set early. (In practice it's a single-bit re-check.) +func parseInto(record []byte, recordSize int, entry *mftEntry, mode parseMode) (uint64, error) { + // Reset fields, preserve hardlinks backing array. + hl := entry.hardlinkParents[:0] + *entry = mftEntry{hardlinkParents: hl} + + if len(record) < recordSize { + return 0, errBadSignature + } + if binary.LittleEndian.Uint32(record[0:4]) != mftSignature { + return 0, errBadSignature + } + if err := applyFixups(record, recordSize); err != nil { + return 0, err + } + + flags := binary.LittleEndian.Uint16(record[0x16:0x18]) + firstAttrOffset := binary.LittleEndian.Uint16(record[0x14:0x16]) + entry.sequence = binary.LittleEndian.Uint16(record[0x10:0x12]) + + // base_record_file_reference at offset 0x20. Non-zero = extension record. + var baseRef uint64 + if recordSize >= 0x28 { + baseRef = MFTIndex(binary.LittleEndian.Uint64(record[0x20:0x28])) + } + + entry.isInUse = flags&flagInUse != 0 + entry.isDir = flags&flagDirectory != 0 + if !entry.isInUse { + return baseRef, nil + } + + // Pass-mode early-exit: skip the attribute walk for records the file- + // tally pass cannot use. Most records are extensions or directories and + // get skipped here. + if mode == modeFileBaseOnly { + if baseRef != 0 || entry.isDir { + return baseRef, nil + } + } + + // Walk the attribute chain. + bestNS := -1 + offset := int(firstAttrOffset) + for offset+8 <= recordSize { + attrType := binary.LittleEndian.Uint32(record[offset : offset+4]) + if attrType == attrEndMarker || attrType == 0 { + break + } + attrLen := int(binary.LittleEndian.Uint32(record[offset+4 : offset+8])) + if attrLen < 16 || offset+attrLen > recordSize { + break + } + + switch attrType { + case attrFileName: + parseFileNameParents(record[offset:offset+attrLen], entry, &bestNS) + case attrData: + nonResident := record[offset+8] + if nonResident == 1 { + parseNonResidentData(record[offset:offset+attrLen], entry) + } else { + parseResidentData(record[offset:offset+attrLen], entry) + } + } + + offset += attrLen + } + + return baseRef, nil +} + +// ------------------------------------------------------------------------- +// Attribute parsers +// ------------------------------------------------------------------------- + +// parseFileNameParents extracts only what the scan needs from $FILE_NAME: +// parent MFT idx (always), and sizes from the highest-priority namespace +// (used as fallback when $DATA is missing). The UTF-16 name is NEVER +// decoded — see package-level doc. +// +// DOS-only ($FILE_NAME with namespace == nsDOS) is the 8.3 alias of an +// existing Win32 entry; we drop it to avoid double-counting parents. +func parseFileNameParents(attr []byte, entry *mftEntry, bestNS *int) { + contentOffset := int(binary.LittleEndian.Uint16(attr[0x14:0x16])) + contentLen := int(binary.LittleEndian.Uint32(attr[0x10:0x14])) + if contentOffset+contentLen > len(attr) || contentLen < 0x42 { + return + } + c := attr[contentOffset : contentOffset+contentLen] + + parentRef := MFTIndex(binary.LittleEndian.Uint64(c[0x00:0x08])) + allocSize := int64(binary.LittleEndian.Uint64(c[0x28:0x30])) + realSize := int64(binary.LittleEndian.Uint64(c[0x30:0x38])) + namespace := int(c[0x41]) + + if namespace == nsDOS { + return + } + + entry.hardlinkParents = append(entry.hardlinkParents, parentRef) + + pri := nsPriority(namespace) + if pri > *bestNS { + *bestNS = pri + entry.primaryParent = parentRef + entry.fnAllocSize = allocSize + entry.fnDataSize = realSize + // Capture the raw UTF-16 name bytes for callers that need basename + // or extension. Slice points into the record buffer — valid for the + // callback duration only. + nameLen := int(c[0x40]) + if 0x42+nameLen*2 <= len(c) { + entry.nameBytes = c[0x42 : 0x42+nameLen*2] + } + } +} + +func nsPriority(ns int) int { + switch ns { + case nsWin32AndDOS: + return 4 + case nsWin32: + return 3 + case nsPosix: + return 2 + default: + return 0 + } +} + +// parseResidentData: $DATA is small enough to live inside the MFT record. +// Allocated == data size (no separate cluster allocation). +// +// A single MFT record can hold multiple $DATA attributes when the file has +// alternate data streams (e.g. the unnamed main stream + a Zone.Identifier +// ADS on a downloaded file). Each is its own $DATA attribute. We accumulate +// across them so the reported size matches the file's true on-disk usage. +func parseResidentData(attr []byte, entry *mftEntry) { + if len(attr) < 0x18 { + return + } + contentLen := int64(binary.LittleEndian.Uint32(attr[0x10:0x14])) + entry.dataSize += contentLen + entry.allocatedSize += contentLen +} + +// parseNonResidentData: $DATA is in cluster runs on disk. AllocatedLength / +// FileSize are valid only when LowestVcn == 0 (per MS spec). Continuation +// fragments must be ignored; the base $DATA's sizes are authoritative. +// +// For sparse or compressed files, offset 0x40 ("Total allocated size") gives +// the actual on-disk allocation accounting for sparse holes / compression. +// Offset 0x28 is the VIRTUAL allocation including holes — useful for apparent +// size reporting but wrong for "size on disk" mode. +// +// Multiple $DATA attributes (alternate data streams) on the same record each +// contribute their own first-fragment sizes. We accumulate; sparse/compressed +// flags are sticky (any-stream). +func parseNonResidentData(attr []byte, entry *mftEntry) { + if len(attr) < 0x40 { + return + } + dataFlags := binary.LittleEndian.Uint16(attr[0x0C:0x0E]) + isCompressed := dataFlags&0x0001 != 0 + isSparse := dataFlags&0x8000 != 0 + + lowestVcn := binary.LittleEndian.Uint64(attr[0x10:0x18]) + if lowestVcn != 0 { + return // continuation run — sizes are invalid + } + + entry.dataSize += int64(binary.LittleEndian.Uint64(attr[0x30:0x38])) + if isSparse { + entry.isSparse = true + } + if isCompressed { + entry.isCompressed = true + } + + if (isSparse || isCompressed) && len(attr) >= 0x48 { + entry.allocatedSize += int64(binary.LittleEndian.Uint64(attr[0x40:0x48])) + } else { + entry.allocatedSize += int64(binary.LittleEndian.Uint64(attr[0x28:0x30])) + } +} + +// ------------------------------------------------------------------------- +// Multi-sector transfer protection (fixups) +// ------------------------------------------------------------------------- + +// errTornWrite is returned by applyFixups when a sector-end USN does not +// match the header USN, indicating the write that produced this record +// did not complete atomically. The record's content is in an +// indeterminate state and must not be parsed. +var errTornWrite = errors.New("torn write detected (USN mismatch)") + +// applyFixups validates the multi-sector transfer protection on an MFT +// record and restores the original sector-end bytes in place. +// +// NTFS writes records sector-by-sector. At write time it places a USN at +// the last 2 bytes of every 512-byte sector (overwriting the real +// content) and stashes the original bytes in the update sequence array +// at the start of the record. On read we must: +// +// 1. Validate that every sector-end still equals the USN. A mismatch +// means the write was torn (process / power interrupted mid-write) +// and the sector contents are unreliable. +// 2. Restore the original bytes from the USA back to the sector ends, +// so the parser sees the record's real content rather than the USN. +// +// Without step 2 the parser reads USN garbage at every 512-byte boundary; +// without step 1 we silently parse a partially-written record. Matches +// what the in-kernel NTFS driver does at the file API layer. +func applyFixups(record []byte, recordSize int) error { + fixupOffset := binary.LittleEndian.Uint16(record[4:6]) + fixupCount := binary.LittleEndian.Uint16(record[6:8]) + if fixupCount < 2 || int(fixupOffset)+int(fixupCount)*2 > recordSize { + return nil + } + + // First word of the USA is the USN; the remaining words are the saved + // original bytes for each sector. + usn0 := record[int(fixupOffset)] + usn1 := record[int(fixupOffset)+1] + + // Pass 1: USN validation. Walk every sector and confirm its trailing + // 2 bytes still equal the header USN. If any sector fails, the write + // did not complete atomically and we must reject the record before + // touching its content. + for i := uint16(1); i < fixupCount; i++ { + sectorEnd := int(i)*512 - 2 + if sectorEnd+2 > recordSize { + break + } + if record[sectorEnd] != usn0 || record[sectorEnd+1] != usn1 { + return errTornWrite + } + } + + // Pass 2: restore. Copy the saved bytes from the USA back to each + // sector end. Cannot be folded into pass 1 because step 1 must + // complete (verify all sectors are intact) before we mutate anything. + for i := uint16(1); i < fixupCount; i++ { + sectorEnd := int(i)*512 - 2 + if sectorEnd+2 > recordSize { + break + } + fvOff := int(fixupOffset) + int(i)*2 + if fvOff+2 > recordSize { + break + } + record[sectorEnd] = record[fvOff] + record[sectorEnd+1] = record[fvOff+1] + } + return nil +} diff --git a/builtins/internal/ntfsmft/parser_fuzz_test.go b/builtins/internal/ntfsmft/parser_fuzz_test.go new file mode 100644 index 00000000..720a5e10 --- /dev/null +++ b/builtins/internal/ntfsmft/parser_fuzz_test.go @@ -0,0 +1,133 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +// Fuzz tests for the pure $MFT record parser. These run on every platform +// (the parser has no OS dependencies — see parser.go), so they exercise the +// code that consumes untrusted on-disk binary data — raw MFT records read +// straight off the volume — and must never panic on hostile input. +// +// Seed corpus follows the same A/B/C structure as the ip route parser fuzzer +// (builtins/tests/ip/ip_route_fuzz_linux_test.go): +// +// A. Implementation edge cases: valid synthetic records (recordBuilder), +// empty/zero, signature-only, hostile attribute offsets/lengths. +// B. CVE / binary class: null bytes, binary magic, truncation, all-0xFF. +// C. Existing coverage: byte patterns from parser_test.go. +package ntfsmft + +import ( + "encoding/binary" + "testing" +) + +// intoRecord copies fuzz bytes into a fixed testRecordSize buffer, mirroring +// how the streaming layer always hands parseInto / applyFixups a full, +// geometry-validated record (recordSize is never attacker-controlled — it comes +// from FSCTL_GET_NTFS_VOLUME_DATA — so fuzzing content within a valid-size +// record is the realistic threat model). +func intoRecord(data []byte) []byte { + buf := make([]byte, testRecordSize) + copy(buf, data) + return buf +} + +// hostileOffsets builds a record with a valid signature but an out-of-range +// first-attribute offset and a giant attribute length — the classic malformed +// inputs an attribute walk must reject without reading out of bounds. +func hostileOffsets() []byte { + buf := make([]byte, testRecordSize) + binary.LittleEndian.PutUint32(buf[0:4], mftSignature) + binary.LittleEndian.PutUint16(buf[0x14:0x16], 0xFFFF) // first-attr offset past end + binary.LittleEndian.PutUint16(buf[0x16:0x18], flagInUse) + return buf +} + +func giantAttrLen() []byte { + buf := make([]byte, testRecordSize) + binary.LittleEndian.PutUint32(buf[0:4], mftSignature) + binary.LittleEndian.PutUint16(buf[0x14:0x16], 0x38) + binary.LittleEndian.PutUint16(buf[0x16:0x18], flagInUse) + binary.LittleEndian.PutUint32(buf[0x38:0x3C], attrFileName) + binary.LittleEndian.PutUint32(buf[0x3C:0x40], 0xFFFFFFFF) // attrLen wraps/overruns + return buf +} + +func fuzzParserSeeds(f *testing.F) { + // Source A: valid synthetic records covering each parse path. + valid := [][]byte{ + newBuilder(0, 0).bytes(), // deleted record + func() []byte { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 4096, 1000, nsWin32AndDOS) + return rb.bytes() + }(), + func() []byte { + rb := newBuilder(flagInUse|flagDirectory, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + return rb.bytes() + }(), + newBuilder(flagInUse, 12345).bytes(), // extension record + func() []byte { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(100, 0, 0, nsWin32AndDOS) + rb.appendFileName(200, 0, 0, nsWin32) + rb.appendNonResidentData(0x8000, 0, 1<<30, 1<<30, 4096) + rb.appendResidentData(154) + return rb.bytes() + }(), + } + for _, v := range valid { + f.Add(v) + } + + // Source A: hostile offsets / lengths. + f.Add(hostileOffsets()) + f.Add(giantAttrLen()) + + // Source B: CVE / binary class. + f.Add([]byte{}) // empty + f.Add(make([]byte, testRecordSize)) // all zero (no signature) + f.Add([]byte("FILE")) // signature only, truncated + f.Add([]byte("\x7fELF\x02\x01\x01\x00")) + f.Add([]byte("MZ\x90\x00\x03\x00")) + f.Add([]byte("PK\x03\x04")) + allFF := make([]byte, testRecordSize) + for i := range allFF { + allFF[i] = 0xFF + } + f.Add(allFF) +} + +// FuzzParseInto verifies parseInto never panics on arbitrary record bytes, in +// both parse modes. parseInto drives parseFileNameParents / parseResidentData / +// parseNonResidentData, so this one fuzzer covers the whole attribute walk. +func FuzzParseInto(f *testing.F) { + fuzzParserSeeds(f) + f.Fuzz(func(t *testing.T, data []byte) { + if len(data) > 1<<20 { // cap at 1 MiB + return + } + rec := intoRecord(data) + var entry mftEntry + // A panic here fails the fuzz test. Errors are expected and fine. + _, _ = parseInto(rec, testRecordSize, &entry, modeAll) + entry = mftEntry{} + _, _ = parseInto(rec, testRecordSize, &entry, modeFileBaseOnly) + }) +} + +// FuzzApplyFixups verifies the update-sequence-array (fixup) logic never panics +// or reads out of bounds on arbitrary record bytes, including hostile USA +// offsets and counts. +func FuzzApplyFixups(f *testing.F) { + fuzzParserSeeds(f) + f.Fuzz(func(t *testing.T, data []byte) { + if len(data) > 1<<20 { + return + } + rec := intoRecord(data) + _ = applyFixups(rec, testRecordSize) + }) +} diff --git a/builtins/internal/ntfsmft/parser_test.go b/builtins/internal/ntfsmft/parser_test.go new file mode 100644 index 00000000..49681907 --- /dev/null +++ b/builtins/internal/ntfsmft/parser_test.go @@ -0,0 +1,510 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package ntfsmft + +import ( + "encoding/binary" + "testing" +) + +const testRecordSize = 1024 + +// recordBuilder constructs a synthetic MFT record for parser tests. The +// builder produces a record with a valid "FILE" signature, a 3-entry fixup +// array at offset 0x30 (covering two 512-byte sectors plus signature slot), +// and attributes appended starting at offset 0x38. Callers fill in the flags, +// baseRef, and attribute payload via the helpers below. +type recordBuilder struct { + buf [testRecordSize]byte + attrStart int // grows as Append* is called +} + +func newBuilder(flags uint16, baseRef uint64) *recordBuilder { + rb := &recordBuilder{} + binary.LittleEndian.PutUint32(rb.buf[0:4], mftSignature) + binary.LittleEndian.PutUint16(rb.buf[4:6], 0x30) // fixup array offset + binary.LittleEndian.PutUint16(rb.buf[6:8], 3) // fixup count: 1 sig + 2 sectors + binary.LittleEndian.PutUint16(rb.buf[0x14:0x16], 0x38) // first attribute offset + binary.LittleEndian.PutUint16(rb.buf[0x16:0x18], flags) + binary.LittleEndian.PutUint64(rb.buf[0x20:0x28], baseRef) + + rb.attrStart = 0x38 + return rb +} + +func (rb *recordBuilder) bytes() []byte { return rb.buf[:] } + +func (rb *recordBuilder) appendAttr(attrType uint32, body []byte) { + off := rb.attrStart + attrLen := 16 + len(body) + binary.LittleEndian.PutUint32(rb.buf[off:off+4], attrType) + binary.LittleEndian.PutUint32(rb.buf[off+4:off+8], uint32(attrLen)) + copy(rb.buf[off+16:off+attrLen], body) + rb.attrStart += attrLen + binary.LittleEndian.PutUint32(rb.buf[rb.attrStart:rb.attrStart+4], attrEndMarker) +} + +func (rb *recordBuilder) appendFileName(parentRef uint64, alloc, real int64, ns byte) { + const contentLen = 0x42 + body := make([]byte, contentLen+8) + binary.LittleEndian.PutUint32(body[0x00:0x04], contentLen) + binary.LittleEndian.PutUint16(body[0x04:0x06], 0x18) + c := body[0x08:] + binary.LittleEndian.PutUint64(c[0x00:0x08], parentRef) + binary.LittleEndian.PutUint64(c[0x28:0x30], uint64(alloc)) + binary.LittleEndian.PutUint64(c[0x30:0x38], uint64(real)) + c[0x40] = 0 + c[0x41] = ns + + rb.appendAttr(attrFileName, body) +} + +func (rb *recordBuilder) appendResidentData(contentLen int) { + body := make([]byte, 8+contentLen) + binary.LittleEndian.PutUint32(body[0x00:0x04], uint32(contentLen)) + binary.LittleEndian.PutUint16(body[0x04:0x06], 0x18) + rb.appendAttr(attrData, body) +} + +func (rb *recordBuilder) appendNonResidentData(flags uint16, lowestVcn uint64, allocSize, dataSize, totalAlloc int64) { + bodyLen := 0x38 + body := make([]byte, bodyLen) + binary.LittleEndian.PutUint64(body[0x00:0x08], lowestVcn) + binary.LittleEndian.PutUint64(body[0x18:0x20], uint64(allocSize)) + binary.LittleEndian.PutUint64(body[0x20:0x28], uint64(dataSize)) + binary.LittleEndian.PutUint64(body[0x30:0x38], uint64(totalAlloc)) + + attrStart := rb.attrStart + rb.appendAttr(attrData, body) + rb.buf[attrStart+8] = 1 // nonResident + binary.LittleEndian.PutUint16(rb.buf[attrStart+0x0C:attrStart+0x0E], flags) +} + +// ------------------------------------------------------------------------- +// Tests +// ------------------------------------------------------------------------- + +func TestParse_BadSignature(t *testing.T) { + buf := make([]byte, testRecordSize) + var entry mftEntry + if _, err := parseInto(buf, testRecordSize, &entry, modeAll); err == nil { + t.Fatal("expected error on missing signature") + } +} + +func TestParse_DeletedRecord(t *testing.T) { + rb := newBuilder(0, 0) + var entry mftEntry + baseRef, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll) + if err != nil { + t.Fatalf("parse: %v", err) + } + if baseRef != 0 { + t.Errorf("baseRef = %d, want 0", baseRef) + } + if entry.isInUse { + t.Error("isInUse = true, want false") + } +} + +func TestParse_DirectoryFlag(t *testing.T) { + rb := newBuilder(flagInUse|flagDirectory, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if !entry.isDir { + t.Error("isDir = false, want true") + } + if entry.primaryParent != 5 { + t.Errorf("primaryParent = %d, want 5", entry.primaryParent) + } +} + +func TestParse_ExtensionRecord(t *testing.T) { + rb := newBuilder(flagInUse, 12345) + var entry mftEntry + baseRef, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll) + if err != nil { + t.Fatalf("parse: %v", err) + } + if baseRef != 12345 { + t.Errorf("baseRef = %d, want 12345", baseRef) + } +} + +func TestParse_DOSNamespaceSkipped(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsDOS) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if got := len(entry.hardlinkParents); got != 1 { + t.Errorf("hardlinkParents len = %d, want 1 (DOS-only skipped)", got) + } +} + +func TestParse_MultipleHardlinks(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(100, 0, 0, nsWin32AndDOS) + rb.appendFileName(200, 0, 0, nsWin32) + rb.appendFileName(300, 0, 0, nsPosix) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if got := len(entry.hardlinkParents); got != 3 { + t.Fatalf("hardlinkParents = %v, want 3 entries", entry.hardlinkParents) + } + for i, want := range []uint64{100, 200, 300} { + if entry.hardlinkParents[i] != want { + t.Errorf("hardlinkParents[%d] = %d, want %d", i, entry.hardlinkParents[i], want) + } + } + if entry.primaryParent != 100 { + t.Errorf("primaryParent = %d, want 100", entry.primaryParent) + } +} + +func TestParse_NamespacePriority(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(10, 0, 0, nsPosix) + rb.appendFileName(20, 0, 0, nsWin32) + rb.appendFileName(30, 0, 0, nsWin32AndDOS) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.primaryParent != 30 { + t.Errorf("primaryParent = %d, want 30 (Win32+DOS wins)", entry.primaryParent) + } +} + +func TestParse_ResidentData(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendResidentData(123) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.dataSize != 123 { + t.Errorf("dataSize = %d, want 123", entry.dataSize) + } + if entry.allocatedSize != 123 { + t.Errorf("allocatedSize = %d, want 123 (resident)", entry.allocatedSize) + } +} + +func TestParse_NonResidentData_Normal(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0, 0, 8192, 5000, 0) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.dataSize != 5000 { + t.Errorf("dataSize = %d, want 5000", entry.dataSize) + } + if entry.allocatedSize != 8192 { + t.Errorf("allocatedSize = %d, want 8192", entry.allocatedSize) + } + if entry.isSparse || entry.isCompressed { + t.Error("normal data marked sparse/compressed") + } +} + +func TestParse_NonResidentData_Sparse(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + const virtualAlloc = 1 << 30 + const physicalAlloc = 4096 + rb.appendNonResidentData(0x8000, 0, virtualAlloc, virtualAlloc, physicalAlloc) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if !entry.isSparse { + t.Error("isSparse = false, want true") + } + if entry.allocatedSize != physicalAlloc { + t.Errorf("allocatedSize = %d, want %d (physical from offset 0x40)", entry.allocatedSize, physicalAlloc) + } + if entry.dataSize != virtualAlloc { + t.Errorf("dataSize = %d, want %d", entry.dataSize, virtualAlloc) + } +} + +func TestParse_NonResidentData_Compressed(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0x0001, 0, 65536, 60000, 32768) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if !entry.isCompressed { + t.Error("isCompressed = false, want true") + } + if entry.allocatedSize != 32768 { + t.Errorf("allocatedSize = %d, want 32768 (compressed physical)", entry.allocatedSize) + } +} + +func TestParse_NonResidentData_Continuation(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0, 100, 999999, 999999, 999999) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.dataSize != 0 || entry.allocatedSize != 0 { + t.Errorf("continuation fragment overwrote sizes: data=%d alloc=%d", + entry.dataSize, entry.allocatedSize) + } +} + +func TestParse_MultipleDataStreams_Resident(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendResidentData(100) + rb.appendResidentData(200) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.dataSize != 300 { + t.Errorf("dataSize = %d, want 300 (sum of two streams)", entry.dataSize) + } + if entry.allocatedSize != 300 { + t.Errorf("allocatedSize = %d, want 300 (sum of two streams)", entry.allocatedSize) + } +} + +func TestParse_MultipleDataStreams_NonResident(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0, 0, 4096, 4000, 0) + rb.appendNonResidentData(0, 0, 8192, 7000, 0) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.allocatedSize != 12288 { + t.Errorf("allocatedSize = %d, want 12288 (4096+8192)", entry.allocatedSize) + } + if entry.dataSize != 11000 { + t.Errorf("dataSize = %d, want 11000 (4000+7000)", entry.dataSize) + } +} + +func TestParse_MultipleDataStreams_MixedResidence(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0, 0, 1<<30, 1<<30, 0) + rb.appendResidentData(154) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + want := int64(1<<30) + 154 + if entry.allocatedSize != want { + t.Errorf("allocatedSize = %d, want %d (1 GiB main + 154 B ADS)", + entry.allocatedSize, want) + } +} + +func TestParse_MultipleDataStreams_SparseFlagSticky(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + rb.appendNonResidentData(0x8000, 0, 1<<30, 1<<30, 4096) + rb.appendNonResidentData(0, 0, 4096, 4000, 0) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if !entry.isSparse { + t.Error("isSparse = false, want true (one stream is sparse)") + } + if entry.allocatedSize != 8192 { + t.Errorf("allocatedSize = %d, want 8192 (4096 sparse-physical + 4096 normal)", + entry.allocatedSize) + } +} + +func TestParse_FileNameSizesAsFallback(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(5, 4096, 1234, nsWin32AndDOS) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.fnAllocSize != 4096 { + t.Errorf("fnAllocSize = %d, want 4096", entry.fnAllocSize) + } + if entry.fnDataSize != 1234 { + t.Errorf("fnDataSize = %d, want 1234", entry.fnDataSize) + } +} + +func TestParse_ModeFileBaseOnly_SkipsExtension(t *testing.T) { + rb := newBuilder(flagInUse, 555) + rb.appendFileName(1, 0, 0, nsWin32AndDOS) + var entry mftEntry + baseRef, err := parseInto(rb.bytes(), testRecordSize, &entry, modeFileBaseOnly) + if err != nil { + t.Fatalf("parse: %v", err) + } + if baseRef != 555 { + t.Errorf("baseRef = %d, want 555", baseRef) + } + if len(entry.hardlinkParents) != 0 { + t.Error("extension record was parsed under modeFileBaseOnly") + } +} + +func TestParse_ModeFileBaseOnly_SkipsDirectory(t *testing.T) { + rb := newBuilder(flagInUse|flagDirectory, 0) + rb.appendFileName(5, 0, 0, nsWin32AndDOS) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeFileBaseOnly); err != nil { + t.Fatalf("parse: %v", err) + } + if !entry.isDir { + t.Error("isDir = false, want true") + } + if len(entry.hardlinkParents) != 0 || entry.primaryParent != 0 { + t.Error("directory was parsed under modeFileBaseOnly") + } +} + +func TestParse_ModeFileBaseOnly_ParsesFileBase(t *testing.T) { + rb := newBuilder(flagInUse, 0) + rb.appendFileName(99, 4096, 1000, nsWin32AndDOS) + rb.appendNonResidentData(0, 0, 4096, 1000, 0) + var entry mftEntry + if _, err := parseInto(rb.bytes(), testRecordSize, &entry, modeFileBaseOnly); err != nil { + t.Fatalf("parse: %v", err) + } + if entry.primaryParent != 99 { + t.Errorf("primaryParent = %d, want 99", entry.primaryParent) + } + if entry.allocatedSize != 4096 { + t.Errorf("allocatedSize = %d, want 4096", entry.allocatedSize) + } +} + +func TestParse_HardlinkBackingArrayReused(t *testing.T) { + rb1 := newBuilder(flagInUse, 0) + rb1.appendFileName(1, 0, 0, nsWin32AndDOS) + rb1.appendFileName(2, 0, 0, nsWin32) + rb1.appendFileName(3, 0, 0, nsPosix) + + var entry mftEntry + if _, err := parseInto(rb1.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse 1: %v", err) + } + if cap(entry.hardlinkParents) < 3 { + t.Fatalf("expected cap >= 3 after first parse, got %d", cap(entry.hardlinkParents)) + } + firstAddr := &entry.hardlinkParents[:cap(entry.hardlinkParents)][0] + + rb2 := newBuilder(flagInUse, 0) + rb2.appendFileName(99, 0, 0, nsWin32AndDOS) + if _, err := parseInto(rb2.bytes(), testRecordSize, &entry, modeAll); err != nil { + t.Fatalf("parse 2: %v", err) + } + if len(entry.hardlinkParents) != 1 || entry.hardlinkParents[0] != 99 { + t.Errorf("hardlinkParents = %v, want [99]", entry.hardlinkParents) + } + secondAddr := &entry.hardlinkParents[:cap(entry.hardlinkParents)][0] + if firstAddr != secondAddr { + t.Error("hardlinkParents backing array was reallocated; reuse broke") + } +} + +func TestApplyFixups_RestoresSectorEnds(t *testing.T) { + buf := make([]byte, testRecordSize) + binary.LittleEndian.PutUint32(buf[0:4], mftSignature) + binary.LittleEndian.PutUint16(buf[4:6], 0x30) + binary.LittleEndian.PutUint16(buf[6:8], 3) + + binary.LittleEndian.PutUint16(buf[0x30:0x32], 0xCAFE) + binary.LittleEndian.PutUint16(buf[0x32:0x34], 0xDEAD) + binary.LittleEndian.PutUint16(buf[0x34:0x36], 0xBEEF) + + binary.LittleEndian.PutUint16(buf[0x1FE:0x200], 0xCAFE) + binary.LittleEndian.PutUint16(buf[0x3FE:0x400], 0xCAFE) + + if err := applyFixups(buf, testRecordSize); err != nil { + t.Fatalf("applyFixups: %v", err) + } + + if got := binary.LittleEndian.Uint16(buf[0x1FE:0x200]); got != 0xDEAD { + t.Errorf("sector 1 end = 0x%X, want 0xDEAD", got) + } + if got := binary.LittleEndian.Uint16(buf[0x3FE:0x400]); got != 0xBEEF { + t.Errorf("sector 2 end = 0x%X, want 0xBEEF", got) + } +} + +func TestApplyFixups_DetectsTornWrite(t *testing.T) { + build := func() []byte { + buf := make([]byte, testRecordSize) + binary.LittleEndian.PutUint32(buf[0:4], mftSignature) + binary.LittleEndian.PutUint16(buf[4:6], 0x30) // USA offset + binary.LittleEndian.PutUint16(buf[6:8], 3) // USA size: 1 USN + 2 sectors + // USN + saved-original bytes for each sector. + binary.LittleEndian.PutUint16(buf[0x30:0x32], 0xCAFE) // USN + binary.LittleEndian.PutUint16(buf[0x32:0x34], 0xDEAD) // sector 1 original + binary.LittleEndian.PutUint16(buf[0x34:0x36], 0xBEEF) // sector 2 original + // Both sector ends carry the USN (intact write). + binary.LittleEndian.PutUint16(buf[0x1FE:0x200], 0xCAFE) + binary.LittleEndian.PutUint16(buf[0x3FE:0x400], 0xCAFE) + return buf + } + + // Sector 2 was torn (USN replaced with stale/random bytes). + torn := build() + binary.LittleEndian.PutUint16(torn[0x3FE:0x400], 0xBEEF) + if err := applyFixups(torn, testRecordSize); err == nil { + t.Error("torn sector 2 was accepted; expected errTornWrite") + } + + // Sector 1 was torn. + torn1 := build() + binary.LittleEndian.PutUint16(torn1[0x1FE:0x200], 0xDEAD) + if err := applyFixups(torn1, testRecordSize); err == nil { + t.Error("torn sector 1 was accepted; expected errTornWrite") + } + + // On detected torn-write, the record buffer must NOT be modified. + // Otherwise a caller that ignores the error would still see partially- + // restored data. + torn2 := build() + binary.LittleEndian.PutUint16(torn2[0x3FE:0x400], 0xBEEF) + before := make([]byte, len(torn2)) + copy(before, torn2) + _ = applyFixups(torn2, testRecordSize) + for i := range before { + if before[i] != torn2[i] { + t.Fatalf("buffer mutated at offset 0x%X on torn-write detection", i) + } + } +} + +func TestMFTIndex_MasksUpperBits(t *testing.T) { + const seqStamped = 0x0007_0000_DEAD_BEEF + const wantIdx = 0x0000_0000_DEAD_BEEF + if got := MFTIndex(seqStamped); got != wantIdx { + t.Errorf("MFTIndex(0x%X) = 0x%X, want 0x%X", uint64(seqStamped), got, uint64(wantIdx)) + } +} diff --git a/builtins/internal/ntfsmft/scan_bench_test.go b/builtins/internal/ntfsmft/scan_bench_test.go new file mode 100644 index 00000000..ac472675 --- /dev/null +++ b/builtins/internal/ntfsmft/scan_bench_test.go @@ -0,0 +1,51 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "context" + "os" + "testing" + + "golang.org/x/sys/windows" +) + +// BenchmarkScan scans a real NTFS volume (default C:\, override with +// RSHELL_NTFSDU_BENCH_TARGET) so CPU/memory profiles reflect a full $MFT walk. +// Run one iteration for profiling: +// +// go test -run '^$' -bench '^BenchmarkScan$' -benchtime=1x \ +// -cpuprofile cpu.prof -memprofile mem.prof ./builtins/internal/ntfsmft/ +// +// Skips when raw MFT access is unavailable (non-admin, container filesystem, or +// a non-NTFS volume), so it is inert in CI. +func BenchmarkScan(b *testing.B) { + target := `C:\` + if t := os.Getenv("RSHELL_NTFSDU_BENCH_TARGET"); t != "" { + target = t + } + + // Cheap pre-check: open the volume without walking the $MFT so the skip + // path costs no full scan and does not pollute the profile. + h, _, err := openVolume(target[:1]) + if err != nil { + if isRawMFTUnsupported(err) { + b.Skipf("raw MFT access unavailable: %v", err) + } + b.Fatalf("openVolume(%s): %v", target[:1], err) + } + windows.CloseHandle(h) + + opts := Options{TopFiles: 10, TopExtensions: 10, TreeDepth: 1} + b.ResetTimer() + for i := 0; i < b.N; i++ { + if _, err := Scan(context.Background(), target, opts); err != nil { + b.Fatalf("Scan: %v", err) + } + } +} diff --git a/builtins/internal/ntfsmft/topn.go b/builtins/internal/ntfsmft/topn.go new file mode 100644 index 00000000..2f03da10 --- /dev/null +++ b/builtins/internal/ntfsmft/topn.go @@ -0,0 +1,392 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsmft + +import ( + "container/heap" + "sort" + "time" + "unsafe" + + "golang.org/x/sys/windows" +) + +// ------------------------------------------------------------------------- +// Top-N file heap (min-heap on Size) +// ------------------------------------------------------------------------- + +// fileCandidate is a top-N file candidate captured during pass 2. The basename +// is copied from the per-record buffer at heap-push time so it survives past +// the streamPipelined callback that produced it. +type fileCandidate struct { + idx uint64 + sequence uint16 + size int64 + basename string // decoded once at heap insertion +} + +// fileHeap is a min-heap of fileCandidate by size. We keep the smallest at the +// top so a bigger candidate can pop it. +type fileHeap []fileCandidate + +func (h fileHeap) Len() int { return len(h) } +func (h fileHeap) Less(i, j int) bool { return h[i].size < h[j].size } +func (h fileHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *fileHeap) Push(x interface{}) { *h = append(*h, x.(fileCandidate)) } +func (h *fileHeap) Pop() interface{} { + old := *h + n := len(old) + x := old[n-1] + *h = old[:n-1] + return x +} + +// topFiles maintains the top-N files seen so far by size. After the scan, +// drained() returns the candidates in descending size order. +type topFiles struct { + cap int + minSize int64 // candidates < minSize are ignored; raised by heap as it fills + heap fileHeap // capacity = cap +} + +func newTopFiles(n int, minSize int64) *topFiles { + if n <= 0 { + return nil + } + return &topFiles{cap: n, minSize: minSize, heap: make(fileHeap, 0, n)} +} + +// consider runs in the pass-3 hot path. It must be cheap when the candidate +// won't qualify; that case is a single int comparison plus an early return. +// When a candidate qualifies we decode its basename from the (still-valid) +// nameBytes slice — that's the only allocation, and it's bounded by the +// total number of distinct heap insertions over the scan (~O(N · log(total)) +// in the random-input case; in practice closer to O(N) once threshold rises). +func (t *topFiles) consider(idx uint64, e *mftEntry, size int64) { + if t == nil || size < t.minSize { + return + } + if len(t.heap) < t.cap { + heap.Push(&t.heap, fileCandidate{ + idx: idx, + sequence: e.sequence, + size: size, + basename: decodeUTF16Name(e.nameBytes), + }) + if len(t.heap) == t.cap && t.heap[0].size > t.minSize { + t.minSize = t.heap[0].size + } + return + } + if size <= t.heap[0].size { + return + } + t.heap[0] = fileCandidate{ + idx: idx, + sequence: e.sequence, + size: size, + basename: decodeUTF16Name(e.nameBytes), + } + heap.Fix(&t.heap, 0) + t.minSize = t.heap[0].size +} + +// drained returns the candidates in descending size order (largest first). +func (t *topFiles) drained() []fileCandidate { + if t == nil { + return nil + } + out := make([]fileCandidate, len(t.heap)) + copy(out, t.heap) + sort.Slice(out, func(i, j int) bool { + if out[i].size != out[j].size { + return out[i].size > out[j].size + } + return out[i].basename < out[j].basename + }) + return out +} + +// ------------------------------------------------------------------------- +// Extension aggregator +// ------------------------------------------------------------------------- + +// extAggregator counts total bytes and file count per file extension. +type extAggregator struct { + bySize map[string]int64 + byCount map[string]int +} + +func newExtAggregator(enabled bool) *extAggregator { + if !enabled { + return nil + } + // Most volumes have hundreds of distinct extensions; pre-size to avoid + // rehashing during the scan. + return &extAggregator{ + bySize: make(map[string]int64, 512), + byCount: make(map[string]int, 512), + } +} + +// addFromName extracts a lowercased ASCII extension (≤24 chars) from the +// raw UTF-16 name and aggregates size. Files with no extension or non-ASCII +// extensions are bucketed under "" / non-ASCII tag (kept for completeness). +// +// Hot path: must not allocate. Uses a stack array + the compiler's +// `m[string(byteSlice)]` optimization to update the map in place. +func (a *extAggregator) addFromName(nameBytes []byte, size int64) { + if a == nil { + return + } + var buf [24]byte + n := extractAsciiExtension(nameBytes, buf[:]) + if n < 0 { + // Non-ASCII or no extension — track under "" so totals are honest. + a.bySize[""] += size + a.byCount[""]++ + return + } + // `m[string(buf[:n])]` is recognized by the Go compiler and does NOT + // allocate when the key already exists in the map. New keys allocate + // once each (bounded by # distinct extensions). + a.bySize[string(buf[:n])] += size + a.byCount[string(buf[:n])]++ +} + +// ExtensionEntry is one extension and its aggregated stats. +type ExtensionEntry struct { + Ext string // lowercased, no leading dot; "" = no extension or non-ASCII + Size int64 + Count int +} + +func (a *extAggregator) topN(n int, minSize int64) []ExtensionEntry { + if a == nil { + return nil + } + out := make([]ExtensionEntry, 0, len(a.bySize)) + for ext, size := range a.bySize { + if size < minSize { + continue + } + out = append(out, ExtensionEntry{Ext: ext, Size: size, Count: a.byCount[ext]}) + } + sort.Slice(out, func(i, j int) bool { + if out[i].Size != out[j].Size { + return out[i].Size > out[j].Size + } + return out[i].Ext < out[j].Ext + }) + if n > 0 && len(out) > n { + out = out[:n] + } + return out +} + +// ------------------------------------------------------------------------- +// Name helpers +// ------------------------------------------------------------------------- + +// decodeUTF16Name converts a UTF-16 little-endian byte slice (NOT null- +// terminated) to a Go string. Used only for top-N heap entries — bounded. +func decodeUTF16Name(b []byte) string { + if len(b) < 2 { + return "" + } + u := make([]uint16, len(b)/2) + for i := range u { + u[i] = uint16(b[i*2]) | uint16(b[i*2+1])<<8 + } + return windows.UTF16ToString(u) +} + +// extractAsciiExtension finds the lowercased ASCII extension in a raw UTF-16 +// name (no leading dot returned). Writes up to len(out) bytes; returns the +// number written, or -1 if there is no extension or it contains non-ASCII. +// +// Walks backward at most ~24 UTF-16 chars looking for '.'. Bounded work; no +// allocation. Empty extension (".") returns -1. +func extractAsciiExtension(nameUTF16, out []byte) int { + if len(nameUTF16) < 4 { + return -1 + } + // Search at most the last min(len, 48) bytes (24 UTF-16 chars) for a dot. + end := len(nameUTF16) + limit := end - 48 + if limit < 0 { + limit = 0 + } + dotAt := -1 + for i := end - 2; i >= limit; i -= 2 { + hi := nameUTF16[i+1] + lo := nameUTF16[i] + if hi != 0 { + return -1 // non-ASCII somewhere in the tail + } + if lo == '.' { + dotAt = i + break + } + } + if dotAt < 0 { + return -1 + } + tailBytes := end - (dotAt + 2) + tailLen := tailBytes / 2 + if tailLen == 0 { + return -1 + } + if tailLen > len(out) { + tailLen = len(out) + } + for j := 0; j < tailLen; j++ { + if nameUTF16[dotAt+2+j*2+1] != 0 { + return -1 + } + ch := nameUTF16[dotAt+2+j*2] + if ch >= 'A' && ch <= 'Z' { + ch += 32 + } + out[j] = ch + } + return tailLen +} + +// ------------------------------------------------------------------------- +// Path resolution via OpenFileByID +// ------------------------------------------------------------------------- + +// fileIDDescriptor mirrors Windows FILE_ID_DESCRIPTOR. The struct size is +// 24 bytes on x64: dwSize(4) + Type(4) + 16-byte union (FILE_ID_128 is the +// largest variant). For Type=FileIdType (0), we use the first 8 bytes of +// the union as the 64-bit NTFS file reference: low 48 bits = MFT idx, high +// 16 bits = sequence number. +type fileIDDescriptor struct { + Size uint32 + Type uint32 // 0 = FileIdType + FileID uint64 + _pad uint64 // pad to FILE_ID_128 union size +} + +// resolveCandidatePaths opens each candidate by FILE_ID and translates it to +// a Win32 path via GetFinalPathNameByHandle. On failure (file deleted between +// scan and resolution, permission, etc.) the basename is used as a fallback. +// +// Bounded by the size of the heap (≤ topN). ~2 syscalls per file; for default +// N=25 this is well under a millisecond. +func resolveCandidatePaths(volumeRoot string, candidates []fileCandidate) []FileEntry { + if len(candidates) == 0 { + return nil + } + rootW, err := windows.UTF16PtrFromString(volumeRoot) + if err != nil { + return fallbackPaths(candidates) + } + hRoot, err := windows.CreateFile( + rootW, + windows.GENERIC_READ, + windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE|windows.FILE_SHARE_DELETE, + nil, + windows.OPEN_EXISTING, + windows.FILE_FLAG_BACKUP_SEMANTICS, + 0, + ) + if err != nil { + return fallbackPaths(candidates) + } + defer windows.CloseHandle(hRoot) + + kernel32 := windows.NewLazySystemDLL("kernel32.dll") + openByID := kernel32.NewProc("OpenFileById") + getFinalPath := kernel32.NewProc("GetFinalPathNameByHandleW") + + out := make([]FileEntry, 0, len(candidates)) + // 32K wchars covers any Windows extended-length path (\\?\ + 32767). + const maxPathChars = 32768 + pathBuf := make([]uint16, maxPathChars) + for _, c := range candidates { + fid := fileIDDescriptor{ + Size: uint32(unsafe.Sizeof(fileIDDescriptor{})), + Type: 0, + FileID: (uint64(c.sequence) << 48) | (c.idx & 0x0000FFFFFFFFFFFF), + } + hr, _, _ := openByID.Call( + uintptr(hRoot), + uintptr(unsafe.Pointer(&fid)), + uintptr(windows.FILE_READ_ATTRIBUTES), + uintptr(windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE|windows.FILE_SHARE_DELETE), + 0, + uintptr(windows.FILE_FLAG_BACKUP_SEMANTICS|windows.FILE_FLAG_OPEN_REPARSE_POINT), + ) + h := windows.Handle(hr) + // INVALID_HANDLE_VALUE on x64 is ^uintptr(0). Treat any handle that + // is 0 or invalid as failure; otherwise the call succeeded regardless + // of GetLastError state. + if hr == 0 || h == windows.InvalidHandle { + out = append(out, FileEntry{Path: "?\\" + c.basename, Size: c.size}) + continue + } + // Read creation / last-write times from the handle we already hold + // (opened with FILE_READ_ATTRIBUTES, which is all this needs). This is + // one extra syscall per displayed file — bounded by topN + find limits, + // not the per-record hot path. Best-effort: on failure the times stay + // zero and the entry is still emitted with its path. + var created, modified time.Time + var info windows.ByHandleFileInformation + if windows.GetFileInformationByHandle(h, &info) == nil { + created = time.Unix(0, info.CreationTime.Nanoseconds()).UTC() + modified = time.Unix(0, info.LastWriteTime.Nanoseconds()).UTC() + } + n, _, _ := getFinalPath.Call( + uintptr(h), + uintptr(unsafe.Pointer(&pathBuf[0])), + uintptr(maxPathChars), + 0, // VOLUME_NAME_DOS | FILE_NAME_NORMALIZED + ) + windows.CloseHandle(h) + if n == 0 || n >= uintptr(maxPathChars) { + out = append(out, FileEntry{Path: "?\\" + c.basename, Size: c.size, Created: created, Modified: modified}) + continue + } + path := windows.UTF16ToString(pathBuf[:n]) + // Strip the \\?\ prefix that GetFinalPathNameByHandleW returns by + // default; users expect "C:\..." not "\\?\C:\...". + path = stripExtendedPathPrefix(path) + out = append(out, FileEntry{Path: path, Size: c.size, Created: created, Modified: modified}) + } + return out +} + +func fallbackPaths(candidates []fileCandidate) []FileEntry { + out := make([]FileEntry, len(candidates)) + for i, c := range candidates { + out[i] = FileEntry{Path: "?\\" + c.basename, Size: c.size} + } + return out +} + +func stripExtendedPathPrefix(p string) string { + const prefix = `\\?\` + if len(p) >= len(prefix) && p[:len(prefix)] == prefix { + return p[len(prefix):] + } + return p +} + +// FileEntry is one large-file entry in Result.TopFiles or a --find match. +type FileEntry struct { + Path string + Size int64 + // Created and Modified are the file's creation and last-data-modification + // times in UTC, read from the file's $STANDARD_INFORMATION via + // GetFileInformationByHandle during post-scan path resolution. Zero when + // the file could not be opened (deleted between scan and resolution, etc.). + Created time.Time + Modified time.Time +} diff --git a/builtins/ntfsdu/doc_notwindows.go b/builtins/ntfsdu/doc_notwindows.go new file mode 100644 index 00000000..94b2eda0 --- /dev/null +++ b/builtins/ntfsdu/doc_notwindows.go @@ -0,0 +1,10 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build !windows + +// Package ntfsdu implements the Windows-only ntfs-du builtin. +// ntfs-du has no non-Windows behavior. +package ntfsdu diff --git a/builtins/ntfsdu/ntfsdu.go b/builtins/ntfsdu/ntfsdu.go new file mode 100644 index 00000000..292fa754 --- /dev/null +++ b/builtins/ntfsdu/ntfsdu.go @@ -0,0 +1,268 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +// Package ntfsdu implements the ntfs-du builtin command. +// +// ntfs-du — whole-disk NTFS disk-usage analysis (Windows only) +// +// Usage: ntfs-du [OPTION]... [FOLDER] +// +// ntfs-du reports disk usage across an NTFS volume by reading the raw Master +// File Table ($MFT) directly from the volume device (\\.\:). Unlike du, +// which walks a directory tree through the AllowedPaths sandbox, ntfs-du's +// runtime is a function of the MFT size (roughly the number of records on the +// volume), *independent of the starting directory*. Prefer du for a small, +// bounded subtree (e.g. a logs folder); reach for ntfs-du to locate the +// largest files, folders, and extensions across an entire disk. +// +// Windows only, by construction. ntfs-du reads the raw NTFS $MFT through the +// Windows volume API and has no cross-platform implementation +// — the entire package is behind //go:build windows and is registered only on +// Windows (see interp/register_builtins_windows.go). On other platforms it is +// not a recognized command. +// +// Other requirements and constraints: +// +// - NTFS-formatted volumes only. Non-NTFS volumes (FAT, ReFS, network +// shares) fail to open or parse and return an error. +// - Administrator privileges. Opening the raw volume device requires +// GENERIC_READ on \\.\:, which is only granted to elevated +// processes. A non-elevated run fails with "access is denied". +// +// Security note: ntfs-du reads the raw volume device directly via the Windows +// API, deliberately bypassing the AllowedPaths sandbox — the same trade-off +// documented for ss, df, and ip route. Because it enumerates the entire MFT, +// it can surface file names and sizes across the whole volume regardless of the +// configured sandbox roots. See the Security Design Decisions section in +// AGENTS.md. +// +// Output: machine-readable only for now (a human-readable renderer is +// planned). --output json emits a single pretty-printed object carrying the +// scan target, deduplicated subtree total, a depth-limited folder tree (whose +// depth-1 nodes are the target's immediate children; omitted entirely at +// --max-depth 0), the largest files, the largest extensions, and any --find +// results. +// +// Accepted flags: +// +// --apparent-size Report logical (apparent) sizes instead of on-disk +// allocation. Default reports allocation, matching +// Explorer's "size on disk". +// --top-files N Report the N largest files (default 10; 0 disables). +// --top-ext N Report the N largest extensions by aggregated size +// (default 10; 0 disables). +// -d, --max-depth N Folder-tree depth from the target (default 1; capped +// at 16). Depth 1 lists the immediate children; 0 omits +// the tree entirely (totals and top files/extensions +// only). +// --min SIZE Size floor (e.g. 100M, 1G): hides folder-tree nodes +// smaller than SIZE, excludes smaller files from the +// top-files list and from --find results, and drops +// extensions whose aggregated total is below SIZE from +// the top-extensions list. Does not affect the subtree +// total. Suffixes K/M/G/T (base 1024) are accepted. +// Defaults to 100M, since the command is aimed at large +// space consumers; pass --min 0 to include everything. +// --exclude PATH Exclude an absolute path's subtree from all totals +// (repeatable). +// --find-ext CSV Report files matching these comma-separated +// extensions (e.g. .dmp,.etl). Repeatable. +// --find-glob PATTERN Report files whose basename matches this glob +// (filepath.Match syntax). Repeatable. +// --find-regex PATTERN Report files whose basename matches this RE2 regex. +// Repeatable. +// --find-limit N Cap each --find query's result block (default 100; +// max 1000). +// --output FORMAT Output format; currently only "json" (default), a +// single pretty-printed object. +// -h, --help Print usage to stdout and exit 0. +// +// With no FOLDER operand, ntfs-du scans the root of the drive containing the +// shell's current working directory (e.g. C:\). +// +// Exit codes: +// +// 0 Scan completed successfully. +// 1 An error occurred (not elevated, non-NTFS volume, invalid argument, etc.). +package ntfsdu + +import ( + "context" + "fmt" + "strconv" + "strings" + + "github.com/DataDog/rshell/builtins" +) + +// Cmd is the ntfs-du builtin command descriptor. +var Cmd = builtins.Command{ + Name: "ntfs-du", + Description: "quickly find large folders, files, and file extensions on an NTFS volume (Windows only, requires Administrator)", + MakeFlags: registerFlags, +} + +// maxTreeDepth caps the requested directory-tree depth. +const maxTreeDepth = 16 + +// maxFindLimit caps a single --find query's result block. Requests above this +// are rejected rather than silently truncated. +const maxFindLimit = 1000 + +// options holds the resolved flag values after pflag parsing. Fields are plain +// scalars/slices so this struct is platform-independent; the Windows-only run() +// translates it into ntfsmft.Options. +type options struct { + apparent bool + topFiles int + topExt int + maxDepth int + minSize int64 + exclude []string + findExt []string + findGlob []string + findRegex []string + findLimit int + target string // positional operand, or "" for the default drive root +} + +func registerFlags(fs *builtins.FlagSet) builtins.HandlerFunc { + // Preserve registration order so PrintDefaults emits flags in a stable + // shape rather than alphabetical. + fs.SortFlags = false + + apparent := fs.Bool("apparent-size", false, "report apparent (logical content) size instead of the default on-disk allocation; on-disk is the clusters actually used (\"size on disk\": reflects cluster rounding, sparse files, and compression)") + topFiles := fs.Int("top-files", 10, "report the N largest files (0 disables)") + topExt := fs.Int("top-ext", 10, "report the N largest extensions by size (0 disables)") + maxDepth := fs.IntP("max-depth", "d", 1, "folder tree depth from the target (0 = no tree, max 16)") + minSize := fs.String("min", "100M", "size floor for the folder tree, top-files, top-extensions, and --find results (e.g. 100M, 1G; 0 shows all)") + exclude := fs.StringArray("exclude", nil, "exclude an absolute path's subtree from totals (repeatable)") + findExt := fs.StringArray("find-ext", nil, "find files with these comma-separated extensions (repeatable)") + findGlob := fs.StringArray("find-glob", nil, "find files whose basename matches this glob (repeatable)") + findRegex := fs.StringArray("find-regex", nil, "find files whose basename matches this RE2 regex (repeatable)") + findLimit := fs.Int("find-limit", 100, "cap each --find query's results (max 1000)") + output := fs.String("output", "json", "output format (currently only \"json\")") + helpFlag := fs.BoolP("help", "h", false, "print usage and exit") + + return func(ctx context.Context, callCtx *builtins.CallContext, args []string) builtins.Result { + // Validate explicitly-set flag values BEFORE the --help short-circuit, + // matching head/tail (builtins/head/head.go) and the args-trim contract + // in builtins/builtins.go: pflag accepts these values syntactically (a + // bad --output, a negative --max-depth, etc. all parse), so an invalid + // value followed by --help must report the value rather than silently + // printing help. + sz, err := parseSize(*minSize) + if err != nil { + callCtx.Errf("ntfs-du: invalid --min value: %s\n", err) + return builtins.Result{Code: 1} + } + + if *maxDepth < 0 { + callCtx.Errf("ntfs-du: invalid --max-depth %d\n", *maxDepth) + return builtins.Result{Code: 1} + } + depth := min(*maxDepth, maxTreeDepth) + + if *findLimit < 0 { + callCtx.Errf("ntfs-du: invalid --find-limit %d\n", *findLimit) + return builtins.Result{Code: 1} + } + if *findLimit > maxFindLimit { + callCtx.Errf("ntfs-du: --find-limit %d exceeds maximum of %d\n", *findLimit, maxFindLimit) + return builtins.Result{Code: 1} + } + + if *topFiles < 0 || *topExt < 0 { + callCtx.Errf("ntfs-du: --top-files and --top-ext must be non-negative\n") + return builtins.Result{Code: 1} + } + + if *output != "json" { + callCtx.Errf("ntfs-du: invalid --output format %q (want \"json\")\n", *output) + return builtins.Result{Code: 1} + } + + if *helpFlag { + fs.SetOutput(callCtx.Stdout) + callCtx.Out("Usage: ntfs-du [OPTION]... [FOLDER]\n") + callCtx.Out("Quickly find large folders, files, and file extensions on an NTFS\n") + callCtx.Out("volume by reading the raw $MFT.\n") + callCtx.Out("Windows only; requires Administrator. With no FOLDER, scans the\n") + callCtx.Out("root of the drive containing the current working directory.\n\n") + fs.PrintDefaults() + return builtins.Result{} + } + + // Operand count is checked after the --help short-circuit so that --help + // wins over an operand-count error, matching GNU (`du a b --help` prints + // help rather than erroring on the extra operand). + if len(args) > 1 { + callCtx.Errf("ntfs-du: at most one folder operand may be given\n") + return builtins.Result{Code: 1} + } + + opts := options{ + apparent: *apparent, + topFiles: *topFiles, + topExt: *topExt, + maxDepth: depth, + minSize: sz, + exclude: *exclude, + findExt: *findExt, + findGlob: *findGlob, + findRegex: *findRegex, + findLimit: *findLimit, + } + if len(args) == 1 { + opts.target = args[0] + } + + return run(ctx, callCtx, opts) + } +} + +// parseSize accepts "" -> 0, or a number with an optional K/M/G/T suffix +// (binary multiples). An optional trailing "B", "b", "iB", or "ib" is allowed. +// Examples: "100", "1K", "100M", "5G", "2T". Negative values are rejected. +func parseSize(s string) (int64, error) { + if s == "" { + return 0, nil + } + s = strings.TrimSpace(s) + s = strings.TrimSuffix(s, "B") + s = strings.TrimSuffix(s, "b") + s = strings.TrimSuffix(s, "i") + if s == "" { + return 0, fmt.Errorf("empty size") + } + mult := int64(1) + switch s[len(s)-1] { + case 'K', 'k': + mult = 1024 + s = s[:len(s)-1] + case 'M', 'm': + mult = 1024 * 1024 + s = s[:len(s)-1] + case 'G', 'g': + mult = 1024 * 1024 * 1024 + s = s[:len(s)-1] + case 'T', 't': + mult = 1024 * 1024 * 1024 * 1024 + s = s[:len(s)-1] + } + n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64) + if err != nil { + return 0, fmt.Errorf("parse %q: %w", s, err) + } + if n < 0 { + return 0, fmt.Errorf("negative size") + } + if n > (1<<63-1)/mult { + return 0, fmt.Errorf("size overflow") + } + return n * mult, nil +} diff --git a/builtins/ntfsdu/ntfsdu_other_test.go b/builtins/ntfsdu/ntfsdu_other_test.go new file mode 100644 index 00000000..0c8a6793 --- /dev/null +++ b/builtins/ntfsdu/ntfsdu_other_test.go @@ -0,0 +1,30 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build !windows + +package ntfsdu_test + +import ( + "slices" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/DataDog/rshell/builtins" + "github.com/DataDog/rshell/interp" +) + +// On non-Windows platforms ntfs-du must not be registered, so it never appears +// in the builtin registry (and therefore not in `help`). Constructing a runner +// triggers builtin registration. +func TestNotRegisteredOffWindows(t *testing.T) { + r, err := interp.New() + require.NoError(t, err) + r.Close() + + require.False(t, slices.Contains(builtins.Names(), "ntfs-du"), + "ntfs-du must not be registered on non-Windows platforms") +} diff --git a/builtins/ntfsdu/ntfsdu_pentest_test.go b/builtins/ntfsdu/ntfsdu_pentest_test.go new file mode 100644 index 00000000..4db7e703 --- /dev/null +++ b/builtins/ntfsdu/ntfsdu_pentest_test.go @@ -0,0 +1,166 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +// Adversarial (pentest) tests for ntfs-du. ntfs-du's attack surface is much +// smaller than a POSIX file command: it reads no stdin and no file *content*, +// only its own volume device. So the classic pentest categories that do not +// apply (stdin/infinite sources, long-line handling, binary-content parsing, +// symlink-to-special-file) are intentionally omitted here. What remains is +// flag-value validation, target/exclude path handling, and flag injection — +// modeled on builtins/df/builtin_df_pentest_test.go and +// builtins/ss/builtin_ss_pentest_test.go. +// +// Two classes of case: +// - Rejected pre-scan: deterministic exit 1 + stderr, everywhere. +// - Reaches the scan with a guaranteed-nonexistent target: the target fails +// to resolve (getMFTIdxFromPath) before any volume read, so these exit 1 +// fast in every environment (no elevation, no real MFT walk). We assert +// "clean exit 1, no panic" — the point is that hostile arguments never +// crash or hang the command. +package ntfsdu_test + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +// nonexistentTarget never resolves, so Scan errors out at target resolution +// before opening the volume — keeping these tests fast and elevation-free. +// Single-quoted in scripts because rshell is a POSIX shell (backslashes escape). +const nonexistentTarget = `'C:\__rshell_ntfsdu_pentest_does_not_exist__\x'` + +func TestNtfsduPentestRejectedFlagShapes(t *testing.T) { + // ntfs-du has no write/execute flags; every unrecognized flag (including + // dangerous-looking ones) must be rejected, not silently ignored. Long + // flags report GNU "unrecognized option '--x'"; short flags report + // "invalid option -- 'x'" (see the wrapper in builtins/builtins.go). + longFlags := []string{"--delete", "--write", "--output-file", "--follow", "--exec", "--no-such-flag"} + shortFlags := []string{"-f", "-Z", "-w"} + for _, flag := range longFlags { + _, stderr, code := cmdRun(t, "ntfs-du "+flag+" "+nonexistentTarget) + assert.Equal(t, 1, code, "flag %q must be rejected", flag) + assert.Contains(t, stderr, "unrecognized option", "flag %q", flag) + assert.Contains(t, stderr, "Try 'ntfs-du --help'", "flag %q", flag) + } + for _, flag := range shortFlags { + _, stderr, code := cmdRun(t, "ntfs-du "+flag+" "+nonexistentTarget) + assert.Equal(t, 1, code, "flag %q must be rejected", flag) + assert.Contains(t, stderr, "invalid option", "flag %q", flag) + assert.Contains(t, stderr, "Try 'ntfs-du --help'", "flag %q", flag) + } +} + +func TestNtfsduPentestFlagViaWordExpansion(t *testing.T) { + // A rejected flag arriving via shell word expansion must still be rejected + // (the flag is not visible as a literal token). Mirrors + // TestDfPentestFlagViaExpansion / TestSSPentestFlagViaWordExpansion. + _, stderr, code := cmdRun(t, "for f in --bogus; do ntfs-du $f "+nonexistentTarget+"; done") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "unrecognized option") +} + +func TestNtfsduPentestIntegerEdgeValues(t *testing.T) { + // Negative / overflowing / out-of-range numeric flag values must be + // rejected with a clear message, never wrap around or clamp silently. + cases := []struct { + script string + wantErr string + }{ + {"ntfs-du --max-depth -1 " + nonexistentTarget, "invalid --max-depth"}, + {"ntfs-du --find-limit -1 " + nonexistentTarget, "invalid --find-limit"}, + {"ntfs-du --find-limit 1001 " + nonexistentTarget, "exceeds maximum"}, + {"ntfs-du --top-files -1 " + nonexistentTarget, "must be non-negative"}, + {"ntfs-du --top-ext -1 " + nonexistentTarget, "must be non-negative"}, + {"ntfs-du --min -5 " + nonexistentTarget, "invalid --min value"}, + {"ntfs-du --min 999999999999999T " + nonexistentTarget, "invalid --min value"}, + {"ntfs-du --min 10Q " + nonexistentTarget, "invalid --min value"}, + } + for _, c := range cases { + stdout, stderr, code := cmdRun(t, c.script) + assert.Equal(t, 1, code, "script: %s", c.script) + assert.Contains(t, stderr, c.wantErr, "script: %s", c.script) + assert.Empty(t, stdout, "no output on validation failure: %s", c.script) + } +} + +func TestNtfsduPentestTargetPathEdgeCases(t *testing.T) { + // Hostile / degenerate target operands must fail cleanly (exit 1), never + // panic. Every target below is an absolute, guaranteed-nonexistent path so + // resolution fails fast (no full-volume scan, no elevation needed) in every + // environment. Mirrors TestDfPentestFileOperandTraversal. + // + // Note: drive-relative ("C:") and empty ("") operands are deliberately NOT + // tested here — they resolve to the cwd / default drive root and would + // trigger a real scan; their handling is covered by the scan tests instead. + targets := []string{ + `'C:\Windows\System32\..\..\..\..\__rshell_ntfsdu_nope__'`, // traversal, clamps at root + `'C:\\\\__rshell_ntfsdu_nope__\\\\x'`, // redundant separators + `'C:\__rshell_ntfsdu_nope__\with space\and.dots.\trailing '`, // spaces/dots/trailing space + `'C:\__rshell_ntfsdu_nope__\日本語\ünïcödé'`, // unicode + `'C:\__rshell_ntfsdu_nope__\` + strings.Repeat("a", 300) + `'`, // long segment + } + for _, target := range targets { + _, _, code := cmdRun(t, "ntfs-du --max-depth 0 --top-files 0 --top-ext 0 "+target) + assert.Equal(t, 1, code, "target %q must fail cleanly", target) + } +} + +func TestNtfsduPentestEndOfFlagsSeparator(t *testing.T) { + // After --, a flag-like token is a positional operand (the target), not a + // flag. It becomes a nonexistent target and exits 1 — it must NOT be parsed + // as an unknown flag nor crash. Mirrors TestDfPentestEndOfFlagsSeparator. + _, stderr, code := cmdRun(t, `ntfs-du -- '--weird-target'`) + assert.Equal(t, 1, code) + assert.NotContains(t, stderr, "unrecognized option", "token after -- is an operand, not a flag") +} + +func TestNtfsduPentestManyExcludesAndFinds(t *testing.T) { + // Many repeated repeatable flags must not exhaust resources or crash; they + // validate, then the nonexistent target fails resolution. Mirrors + // TestDfPentestManyTypeFilters. + script := "ntfs-du --max-depth 0" + for i := 0; i < 128; i++ { + script += ` --exclude 'C:\nope' --find-ext .dmp` + } + script += " " + nonexistentTarget + _, _, code := cmdRun(t, script) + assert.Equal(t, 1, code) +} + +func TestNtfsduPentestFindRegexReDoS(t *testing.T) { + // A pathological regex must not hang: the matcher uses RE2 (linear time), + // and the nonexistent target fails fast regardless. Mirrors the intent of + // builtins/grep/grep_vuln_hunt_test.go. + for _, re := range []string{`(a+)+$`, `(a|a)*$`, `(.*a){20}`} { + _, _, code := cmdRun(t, "ntfs-du --find-regex '"+re+"' "+nonexistentTarget) + assert.Equal(t, 1, code, "regex %q", re) + } +} + +func TestNtfsduPentestAllFlagsAtOnce(t *testing.T) { + // Smoke test: every flag set at once must parse and then fail only on the + // nonexistent target — no crash. Mirrors TestDfPentestAllFlagsAtOnce / + // TestSSPentestAllFlagCombinations. + script := "ntfs-du --apparent-size --top-files 5 --top-ext 5 --max-depth 3 " + + "--min 1M --exclude 'C:\\nope' --find-ext .dmp --find-glob '*.log' " + + "--find-regex 'x.*' --find-limit 10 --output json " + nonexistentTarget + _, _, code := cmdRun(t, script) + assert.Equal(t, 1, code) +} + +func TestNtfsduPentestHelpDoesNotEmitScanData(t *testing.T) { + // --help is not a scan: it must print usage to stdout, touch no volume, and + // emit no JSON. Mirrors TestSSPentestHelpDoesNotLeakSensitiveInfo. + stdout, stderr, code := cmdRun(t, "ntfs-du --help") + assert.Equal(t, 0, code) + assert.Empty(t, stderr) + assert.Contains(t, stdout, "Usage: ntfs-du") + assert.NotContains(t, stdout, "{", "help must not emit a JSON document") + assert.NotContains(t, stdout, "sizeBytes") +} diff --git a/builtins/ntfsdu/ntfsdu_test.go b/builtins/ntfsdu/ntfsdu_test.go new file mode 100644 index 00000000..3e92f7a1 --- /dev/null +++ b/builtins/ntfsdu/ntfsdu_test.go @@ -0,0 +1,112 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +// The entire ntfsdu package is Windows-only (//go:build windows) and the +// command is registered only on Windows (see interp/register_builtins_windows.go), +// so all of its tests — these flag-parsing / help / error tests plus the +// parseSize and scan tests — run on Windows only. The sole non-Windows test is +// ntfsdu_other_test.go, which asserts the command is absent off Windows. + +package ntfsdu_test + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/DataDog/rshell/builtins/testutil" + "github.com/DataDog/rshell/interp" +) + +// cmdRun runs a script with the temp dir as the sole AllowedPaths root. The +// behavior asserted here (flag parsing, --help, argument validation) is +// platform-independent, but ntfs-du is only registered on Windows (see the file +// header), so these tests run there. +func cmdRun(t *testing.T, script string) (string, string, int) { + t.Helper() + dir := t.TempDir() + return testutil.RunScript(t, script, dir, interp.AllowedPaths([]string{dir})) +} + +func TestHelpToStdout(t *testing.T) { + stdout, stderr, code := cmdRun(t, "ntfs-du --help") + assert.Equal(t, 0, code) + assert.Empty(t, stderr) + for _, want := range []string{ + "Usage: ntfs-du", + "find large folders, files, and file extensions", + "--max-depth", + "--find-ext", + "--output", + } { + assert.Contains(t, stdout, want) + } +} + +func TestUnknownFlagRejected(t *testing.T) { + stdout, stderr, code := cmdRun(t, "ntfs-du --bogus") + assert.Equal(t, 1, code) + assert.Empty(t, stdout) + assert.Contains(t, stderr, "unrecognized option '--bogus'") + assert.Contains(t, stderr, "Try 'ntfs-du --help'") +} + +func TestInvalidMinRejected(t *testing.T) { + stdout, stderr, code := cmdRun(t, "ntfs-du --min 10Q C:\\") + assert.Equal(t, 1, code) + assert.Empty(t, stdout) + assert.Contains(t, stderr, "invalid --min value") +} + +func TestNegativeMaxDepthRejected(t *testing.T) { + _, stderr, code := cmdRun(t, "ntfs-du --max-depth -1 C:\\") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "invalid --max-depth") +} + +func TestFindLimitTooLargeRejected(t *testing.T) { + _, stderr, code := cmdRun(t, "ntfs-du --find-limit 5000 C:\\") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "exceeds maximum") +} + +func TestInvalidOutputRejected(t *testing.T) { + _, stderr, code := cmdRun(t, "ntfs-du --output yaml C:\\") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "invalid --output format") +} + +func TestTooManyOperandsRejected(t *testing.T) { + _, stderr, code := cmdRun(t, "ntfs-du a b") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "at most one folder operand") +} + +func TestInvalidValueBeforeHelpErrors(t *testing.T) { + // An invalid flag value ahead of --help must report the value, not print + // help (matches head/tail; see the validation-ordering comment in ntfsdu.go). + stdout, stderr, code := cmdRun(t, "ntfs-du --max-depth -1 --help") + assert.Equal(t, 1, code) + assert.Contains(t, stderr, "invalid --max-depth") + assert.NotContains(t, stdout, "Usage: ntfs-du", "help must not print when a flag value is invalid") +} + +func TestOperandCountLosesToHelp(t *testing.T) { + // Positional-operand errors, by contrast, yield to --help (GNU semantics). + stdout, stderr, code := cmdRun(t, "ntfs-du a b --help") + assert.Equal(t, 0, code) + assert.Empty(t, stderr) + assert.Contains(t, stdout, "Usage: ntfs-du") +} + +func TestJSONBadDoesNotLeakToStdout(t *testing.T) { + // A validation failure must not print a partial JSON document. + stdout, _, code := cmdRun(t, "ntfs-du --top-files -5 C:\\") + assert.Equal(t, 1, code) + assert.False(t, strings.Contains(stdout, "{"), "no JSON should be emitted on error") +} diff --git a/builtins/ntfsdu/ntfsdu_windows.go b/builtins/ntfsdu/ntfsdu_windows.go new file mode 100644 index 00000000..51adb4aa --- /dev/null +++ b/builtins/ntfsdu/ntfsdu_windows.go @@ -0,0 +1,245 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsdu + +import ( + "context" + "encoding/json" + "path/filepath" + "time" + + "github.com/DataDog/rshell/builtins" + "github.com/DataDog/rshell/builtins/internal/ntfsmft" +) + +const ( + modeAllocated = "allocated" + modeApparent = "apparent" +) + +// jsonTreeNode is one node in the depth-limited directory tree. The tree is +// emitted as a flat, pre-order list; Path is the full path to the node and +// Pruned marks a leaf at the requested depth that has undisplayed descendants. +type jsonTreeNode struct { + Path string `json:"path"` + Kind string `json:"kind"` + SizeBytes int64 `json:"sizeBytes"` + Pruned bool `json:"pruned"` + FileCount int `json:"fileCount"` + FolderCount int `json:"folderCount"` +} + +// jsonFileEntry is one file in topFiles / find matches. Created/Modified are +// RFC 3339 UTC timestamps, omitted when unavailable (file not openable at +// resolution time). +type jsonFileEntry struct { + Path string `json:"path"` + SizeBytes int64 `json:"sizeBytes"` + Created string `json:"created,omitempty"` + Modified string `json:"modified,omitempty"` +} + +// jsonExtEntry is one aggregated file extension. +type jsonExtEntry struct { + Ext string `json:"ext"` + SizeBytes int64 `json:"sizeBytes"` + FileCount int `json:"fileCount"` +} + +// jsonFindQuery echoes the find predicate back in the output. +type jsonFindQuery struct { + Type string `json:"type"` + Value string `json:"value"` + Limit int `json:"limit,omitempty"` + Label string `json:"label,omitempty"` +} + +// jsonFindBlock pairs a find query with its matches. +type jsonFindBlock struct { + Query jsonFindQuery `json:"query"` + Matches []jsonFileEntry `json:"matches"` +} + +// jsonOutput is the top-level JSON document ntfs-du emits. The folder breakdown +// is the depth-limited Tree, whose depth-1 nodes are the target's immediate +// children; Tree is omitted entirely at --max-depth 0. +type jsonOutput struct { + Target string `json:"target"` + Mode string `json:"mode"` + SubtreeBytes int64 `json:"subtreeBytes"` + Tree []jsonTreeNode `json:"tree,omitempty"` + TopFiles []jsonFileEntry `json:"topFiles"` + TopExt []jsonExtEntry `json:"topExt"` + FindResults []jsonFindBlock `json:"findResults"` +} + +// run performs the scan on Windows and writes the JSON report to stdout. +func run(ctx context.Context, callCtx *builtins.CallContext, opts options) builtins.Result { + target := opts.target + if target == "" { + target = driveRoot(callCtx.WorkDir()) + } + + finds := buildFinds(opts) + + mode := modeAllocated + if opts.apparent { + mode = modeApparent + } + + res, err := ntfsmft.Scan(ctx, target, ntfsmft.Options{ + ShowApparent: opts.apparent, + TopFiles: opts.topFiles, + TopExtensions: opts.topExt, + MinFileSize: opts.minSize, + Finds: finds, + Exclude: opts.exclude, + TreeDepth: opts.maxDepth, + TreeMinSize: opts.minSize, + }) + if err != nil { + callCtx.Errf("ntfs-du: %s\n", err) + return builtins.Result{Code: 1} + } + + out := buildOutput(res, mode, opts.maxDepth) + enc, err := json.MarshalIndent(out, "", " ") + if err != nil { + callCtx.Errf("ntfs-du: encoding output: %s\n", err) + return builtins.Result{Code: 1} + } + callCtx.Out(string(enc)) + callCtx.Out("\n") + return builtins.Result{} +} + +// driveRoot returns the volume root ("C:\") for a drive-letter path. Non +// drive-shaped inputs are returned unchanged so Scan surfaces a clear error. +func driveRoot(wd string) string { + if len(wd) >= 2 && wd[1] == ':' { + return wd[:2] + `\` + } + return wd +} + +// buildFinds translates the per-type find flags into ntfsmft.FindQuery values, +// preserving the order ext, glob, regex. +func buildFinds(opts options) []ntfsmft.FindQuery { + total := len(opts.findExt) + len(opts.findGlob) + len(opts.findRegex) + if total == 0 { + return nil + } + finds := make([]ntfsmft.FindQuery, 0, total) + for _, v := range opts.findExt { + finds = append(finds, ntfsmft.FindQuery{Type: "ext", Value: v, Limit: opts.findLimit}) + } + for _, v := range opts.findGlob { + finds = append(finds, ntfsmft.FindQuery{Type: "glob", Value: v, Limit: opts.findLimit}) + } + for _, v := range opts.findRegex { + finds = append(finds, ntfsmft.FindQuery{Type: "regex", Value: v, Limit: opts.findLimit}) + } + return finds +} + +// buildOutput maps an ntfsmft.Result into the JSON document. +// depth is the requested tree depth, used to mark pruned leaves. +func buildOutput(res *ntfsmft.Result, mode string, depth int) jsonOutput { + out := jsonOutput{ + Target: res.Target, + Mode: mode, + SubtreeBytes: res.Subtree, + TopFiles: make([]jsonFileEntry, 0, len(res.TopFiles)), + TopExt: make([]jsonExtEntry, 0, len(res.TopExtensions)), + FindResults: make([]jsonFindBlock, 0, len(res.FindResults)), + } + + // The engine only builds a tree at TreeDepth > 0; at --max-depth 0 it is + // nil, so Tree stays empty and (via omitempty) is dropped from the output. + if res.Tree != nil { + out.Tree = flattenTree(res.Tree, depth) + } + + for _, f := range res.TopFiles { + out.TopFiles = append(out.TopFiles, fileEntry(f)) + } + for _, e := range res.TopExtensions { + out.TopExt = append(out.TopExt, jsonExtEntry{Ext: e.Ext, SizeBytes: e.Size, FileCount: e.Count}) + } + for _, blk := range res.FindResults { + matches := make([]jsonFileEntry, 0, len(blk.Matches)) + for _, m := range blk.Matches { + matches = append(matches, fileEntry(m)) + } + out.FindResults = append(out.FindResults, jsonFindBlock{ + Query: jsonFindQuery{ + Type: blk.Query.Type, + Value: blk.Query.Value, + Limit: blk.Query.Limit, + Label: blk.Query.Label, + }, + Matches: matches, + }) + } + + return out +} + +// flattenTree walks an ntfsmft.TreeNode subtree in pre-order and returns a flat +// list of nodes. The root node carries its full path; descendants carry a path +// joined from the parent's path and the node basename. A node at the requested +// depth with no displayed children is marked Pruned. +func flattenTree(root *ntfsmft.TreeNode, depth int) []jsonTreeNode { + var nodes []jsonTreeNode + var walk func(n *ntfsmft.TreeNode, parentPath string) + walk = func(n *ntfsmft.TreeNode, parentPath string) { + path := n.Name + if n.Depth != 0 { + path = filepath.Join(parentPath, n.Name) + } + nodes = append(nodes, jsonTreeNode{ + Path: path, + Kind: dirKind(n.Reparse), + SizeBytes: n.Size, + Pruned: n.Depth == depth && len(n.Children) == 0, + FileCount: n.Files, + FolderCount: n.Dirs, + }) + for _, c := range n.Children { + walk(c, path) + } + } + walk(root, "") + return nodes +} + +// fileEntry maps an engine FileEntry to its JSON form, formatting the +// creation / modification times as RFC 3339 UTC (omitted when unavailable). +func fileEntry(f ntfsmft.FileEntry) jsonFileEntry { + return jsonFileEntry{ + Path: f.Path, + SizeBytes: f.Size, + Created: rfc3339(f.Created), + Modified: rfc3339(f.Modified), + } +} + +// rfc3339 formats t as RFC 3339 UTC, or "" if t is the zero value. +func rfc3339(t time.Time) string { + if t.IsZero() { + return "" + } + return t.UTC().Format(time.RFC3339) +} + +func dirKind(reparse bool) string { + if reparse { + return "reparse" + } + return "dir" +} diff --git a/builtins/ntfsdu/ntfsdu_windows_test.go b/builtins/ntfsdu/ntfsdu_windows_test.go new file mode 100644 index 00000000..aa1ff652 --- /dev/null +++ b/builtins/ntfsdu/ntfsdu_windows_test.go @@ -0,0 +1,139 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsdu_test + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/DataDog/rshell/builtins/testutil" + "github.com/DataDog/rshell/interp" +) + +// scanResult is a minimal projection of the JSON document for assertions. +type scanResult struct { + Target string `json:"target"` + Mode string `json:"mode"` + SubtreeBytes int64 `json:"subtreeBytes"` + Tree []struct { + Path string `json:"path"` + SizeBytes int64 `json:"sizeBytes"` + Pruned bool `json:"pruned"` + FileCount int `json:"fileCount"` + FolderCount int `json:"folderCount"` + } `json:"tree"` + TopFiles []struct { + Path string `json:"path"` + SizeBytes int64 `json:"sizeBytes"` + Created string `json:"created"` + Modified string `json:"modified"` + } `json:"topFiles"` +} + +// TestScanTempDirJSON opportunistically validates the full builtin→engine→JSON +// path against a temp directory on the current volume. A real scan needs a +// genuine NTFS volume opened with elevation; environments that can't provide +// that are skipped rather than failed: +// - containers (including Windows CI containers) expose C: as a filesystem +// layer, not a raw volume, so raw $MFT reads fail with ERROR_NOT_SUPPORTED / +// ERROR_INVALID_FUNCTION ("not supported" / "incorrect function"); +// - non-elevated processes are denied the volume handle ("access is denied"). +// +// Because CI cannot reliably exercise the scan, this test is best-effort only. +// Real validation of scan correctness belongs in a VM-based E2E test (see the +// Testing note in AGENTS.md). +func TestScanTempDirJSON(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(dir, "big.bin"), make([]byte, 64*1024), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(dir, "small.txt"), []byte("hello"), 0o644)) + + // Single-quote the path: rshell is a POSIX shell, so the backslashes in a + // Windows path would otherwise be consumed as escape characters. + stdout, stderr, code := testutil.RunScript(t, + "ntfs-du --output json --top-files 5 '"+dir+"'", dir, interp.AllowedPaths([]string{dir})) + + if code != 0 { + low := strings.ToLower(stderr) + if strings.Contains(low, "access is denied") || strings.Contains(low, "need admin") || + strings.Contains(low, "not supported") || strings.Contains(low, "incorrect function") { + t.Skipf("ntfs-du scan unavailable in this environment: %s", strings.TrimSpace(stderr)) + } + t.Fatalf("ntfs-du failed (code %d): %s", code, stderr) + } + + var res scanResult + require.NoError(t, json.Unmarshal([]byte(stdout), &res)) + assert.Equal(t, "allocated", res.Mode) + assert.NotEmpty(t, res.Target) + require.NotEmpty(t, res.Tree, "flattened tree should have at least the root node at default depth 1") + assert.NotEmpty(t, res.Tree[0].Path, "root tree node should carry the target path") + assert.GreaterOrEqual(t, res.SubtreeBytes, int64(64*1024), "subtree should include the 64 KiB file") + // The temp dir holds exactly two files and no subfolders; counts are not + // filtered by --min, so the root node reports them in full. + assert.Equal(t, 2, res.Tree[0].FileCount, "root fileCount") + assert.Equal(t, 0, res.Tree[0].FolderCount, "root folderCount") +} + +// top-files entries carry RFC 3339 created/modified timestamps, read from the +// file handle during post-scan path resolution. +func TestScanTopFilesTimestamps(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(dir, "big.bin"), make([]byte, 256*1024), 0o644)) + + stdout, stderr, code := testutil.RunScript(t, + "ntfs-du --output json --apparent-size --min 0 --max-depth 0 --top-files 5 '"+dir+"'", + dir, interp.AllowedPaths([]string{dir})) + if code != 0 { + low := strings.ToLower(stderr) + if strings.Contains(low, "access is denied") || strings.Contains(low, "need admin") || + strings.Contains(low, "not supported") || strings.Contains(low, "incorrect function") { + t.Skipf("ntfs-du scan unavailable in this environment: %s", strings.TrimSpace(stderr)) + } + t.Fatalf("ntfs-du failed (code %d): %s", code, stderr) + } + + var res scanResult + require.NoError(t, json.Unmarshal([]byte(stdout), &res)) + require.NotEmpty(t, res.TopFiles, "big.bin should appear in top-files at --min 0") + f := res.TopFiles[0] + assert.NotEmpty(t, f.Created, "top-file should carry a created timestamp") + assert.NotEmpty(t, f.Modified, "top-file should carry a modified timestamp") + _, err := time.Parse(time.RFC3339, f.Created) + assert.NoError(t, err, "created must be RFC 3339: %q", f.Created) + _, err = time.Parse(time.RFC3339, f.Modified) + assert.NoError(t, err, "modified must be RFC 3339: %q", f.Modified) +} + +// At --max-depth 0 the folder tree is omitted entirely (the output carries only +// totals and the top files/extensions). +func TestScanDepthZeroOmitsTree(t *testing.T) { + dir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(dir, "f.bin"), make([]byte, 4096), 0o644)) + + stdout, stderr, code := testutil.RunScript(t, + "ntfs-du --output json --min 0 --max-depth 0 '"+dir+"'", dir, interp.AllowedPaths([]string{dir})) + if code != 0 { + low := strings.ToLower(stderr) + if strings.Contains(low, "access is denied") || strings.Contains(low, "need admin") || + strings.Contains(low, "not supported") || strings.Contains(low, "incorrect function") { + t.Skipf("ntfs-du scan unavailable in this environment: %s", strings.TrimSpace(stderr)) + } + t.Fatalf("ntfs-du failed (code %d): %s", code, stderr) + } + assert.NotContains(t, stdout, `"tree"`, "tree key must be omitted at --max-depth 0") + var res scanResult + require.NoError(t, json.Unmarshal([]byte(stdout), &res)) + assert.Empty(t, res.Tree, "no tree at --max-depth 0") +} diff --git a/builtins/ntfsdu/parsesize_internal_test.go b/builtins/ntfsdu/parsesize_internal_test.go new file mode 100644 index 00000000..64e0bacf --- /dev/null +++ b/builtins/ntfsdu/parsesize_internal_test.go @@ -0,0 +1,58 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package ntfsdu + +import "testing" + +func TestParseSize(t *testing.T) { + cases := []struct { + in string + want int64 + wantErr bool + }{ + {"", 0, false}, + {"0", 0, false}, + {"100", 100, false}, + {"1K", 1024, false}, + {"1k", 1024, false}, + {"100M", 100 * 1024 * 1024, false}, + {"5G", 5 * 1024 * 1024 * 1024, false}, + {"2T", 2 * 1024 * 1024 * 1024 * 1024, false}, + {"10KB", 10 * 1024, false}, + {"10KiB", 10 * 1024, false}, + {" 4M ", 4 * 1024 * 1024, false}, + {"10Q", 0, true}, + {"-5", 0, true}, + {"abc", 0, true}, + {"B", 0, true}, + } + for _, c := range cases { + got, err := parseSize(c.in) + if c.wantErr { + if err == nil { + t.Errorf("parseSize(%q): expected error, got %d", c.in, got) + } + continue + } + if err != nil { + t.Errorf("parseSize(%q): unexpected error: %v", c.in, err) + continue + } + if got != c.want { + t.Errorf("parseSize(%q) = %d, want %d", c.in, got, c.want) + } + } +} + +func TestParseSizeOverflow(t *testing.T) { + // A value that overflows int64 when multiplied by the suffix must error, + // not wrap around to a negative or truncated size. + if _, err := parseSize("9999999999999T"); err == nil { + t.Errorf("parseSize: expected overflow error for very large T value") + } +} diff --git a/interp/register_builtins.go b/interp/register_builtins.go index 8d766ef5..8705cdaa 100644 --- a/interp/register_builtins.go +++ b/interp/register_builtins.go @@ -51,7 +51,7 @@ var registerOnce sync.Once func registerBuiltins() { registerOnce.Do(func() { - for _, cmd := range []builtins.Command{ + cmds := []builtins.Command{ breakcmd.Cmd, cat.Cmd, cd.Cmd, @@ -89,7 +89,12 @@ func registerBuiltins() { uniq.Cmd, wc.Cmd, xargs.Cmd, - } { + } + // Append platform-specific builtins (e.g. the Windows-only ntfs-du) so + // they are registered — and thus listed by `help` and runnable — only on + // platforms that support them. + cmds = append(cmds, platformBuiltins()...) + for _, cmd := range cmds { cmd.Register() } }) diff --git a/interp/register_builtins_other.go b/interp/register_builtins_other.go new file mode 100644 index 00000000..06a1a5f2 --- /dev/null +++ b/interp/register_builtins_other.go @@ -0,0 +1,16 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build !windows + +package interp + +import "github.com/DataDog/rshell/builtins" + +// platformBuiltins returns builtins that exist only on non-Windows platforms and thus +// listed by `help` and runnable only on non-Windows platforms. +func platformBuiltins() []builtins.Command { + return nil +} diff --git a/interp/register_builtins_windows.go b/interp/register_builtins_windows.go new file mode 100644 index 00000000..38de3da1 --- /dev/null +++ b/interp/register_builtins_windows.go @@ -0,0 +1,19 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +//go:build windows + +package interp + +import ( + "github.com/DataDog/rshell/builtins" + "github.com/DataDog/rshell/builtins/ntfsdu" +) + +// platformBuiltins returns builtins that exist only on Windows and thus +// listed by `help` and runnable only on Windows. +func platformBuiltins() []builtins.Command { + return []builtins.Command{ntfsdu.Cmd} +}