Harden openat2 validation and /dev/fd resolution

[abnormal749]

Summary

Harden openat2(2) emulation to better match Linux semantics:

• reject unsupported/nonzero open_how extension fields with E2BIG
• reject oversized open_how structures with E2BIG
• reject unknown/conflicting how.flags and invalid how.mode with EINVAL
• reject /dev/fd/N magic-link traversal under RESOLVE_NO_XDEV
• make RESOLVE_NO_SYMLINKS / RESOLVE_NO_MAGICLINKS also reject /dev/fd/N

Linux intentionally defines openat2(2) as a stricter API than openat(2):
unknown or conflicting how.flags must fail, invalid how.mode must fail, nonzero unsupported extension fields must fail with E2BIG, and RESOLVE_NO_XDEV must reject mount-point crossings during path resolution.

Reference: https://man7.org/linux/man-pages/man2/openat2.2.html

Motivation

Before this change, elfuse accepted some invalid open_how inputs that Linux rejects. It also allowed a /dev dirfd-relative fd/N path to reach host /dev/fd/N before RESOLVE_NO_XDEV enforcement, which could bypass the intended guest path-resolution constraints.

Tests

make elfuse
make -B build/test-syscall-fidelity
build/elfuse build/test-syscall-fidelity
make test-hello

  Result:

  test-syscall-fidelity: 39 passed, 0 failed, 0 skipped - PASS

---

Summary by cubic

Hardened openat2(2) emulation to match Linux: strict open_how ABI checks and early flag/resolve validation, plus blocking fd magic-link escapes via /proc/*/fd and /dev/fd. This avoids cross-mount and magic-link bypasses and rejects invalid inputs before path lookup.

• Bug Fixes
  • Normalize absolute, dirfd-, and cwd-relative paths within the guest root; treat /proc/self/fd/N, /proc/<self_pid>/fd/N, and /dev/fd/N as fd magic-link anchors.
  • Enforce resolve policy: RESOLVE_NO_SYMLINKS/RESOLVE_NO_MAGICLINKS return ELOOP for those anchors; RESOLVE_NO_XDEV returns EXDEV.
  • Validate open_how ABI: nonzero tails fail with E2BIG; sizes >4096 fail with E2BIG.
  • Validate flags/mode before lookup: reject unknown flags, mode bits outside 07777, mode without O_CREAT, RESOLVE_BENEATH|RESOLVE_IN_ROOT, O_DIRECTORY|O_CREAT, O_PATH misuse, and __O_TMPFILE usage with EINVAL.
  • Added flag constants (LINUX_O_DSYNC, LINUX_O_SYNC, LINUX___O_TMPFILE, LINUX_O_TMPFILE), expanded tests for /proc/<pid>/fd and /dev/fd variants, and stabilized cwd-sensitive tests using fchdir.

^{Written for commit 988364d. Summary will update on new commits.}

[jserv]

Solid direction: the validation rejections match Linux semantics for the cases they cover, the new tests are tight, and the 39-case fidelity suite stays green on this branch. A few gaps and one validation/translation inconsistency worth addressing before merge; details inline.

[cubic-dev-ai]

2 issues found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/syscall/syscall.c">

<violation number="1" location="src/syscall/syscall.c:1408">
P1: openat2_flags_valid() unconditionally rejects all O_TMPFILE usage instead of only invalid combinations, breaking legitimate O_TMPFILE calls</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: openat2_flags_valid() unconditionally rejects all O_TMPFILE usage instead of only invalid combinations, breaking legitimate O_TMPFILE calls

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/syscall/syscall.c, line 1408:

<comment>openat2_flags_valid() unconditionally rejects all O_TMPFILE usage instead of only invalid combinations, breaking legitimate O_TMPFILE calls</comment>

<file context>
@@ -1405,16 +1405,14 @@ static int openat2_flags_valid(uint64_t flags, uint64_t mode)
         return 0;
-    if ((flags & LINUX___O_TMPFILE) &&
-        (flags & LINUX_O_TMPFILE) != LINUX_O_TMPFILE)
+    if (flags & LINUX___O_TMPFILE)
         return 0;
-    if ((flags & LINUX_O_TMPFILE) == LINUX_O_TMPFILE &&
</file context>

[abnormal749]

commit 1: Fix openat2 magic-link path normalization

• centralize fd magic-link detection through path_openat2_is_fd_magiclink_anchor()
• broaden /dev/fd relative path detection
• handle absolute /dev/fd/N
• handle dirfd=/ + dev/fd/N
• handle AT_FDCWD with cwd / + dev/fd/N
• handle normalized /dev paths such as shm/../fd/N
• handle /proc/<pid>/fd/N where pid == getpid()
• keep the extra check limited to openat2 resolve-flag prechecks

I also consolidated the /dev/fd, /proc/self/fd, and /proc/<self_pid>/fd checks into path_openat2_is_fd_magiclink_anchor(). The intent is to treat them as the same openat2 policy concept: fd magic-link anchors. The helper only performs lexical normalization against the dirfd/cwd base and does not follow symlinks or change general proc/dev path handling. This keeps the extra work limited to resolve-flag prechecks while avoiding scattered special cases.

commit 2: Tighten openat2 flag validation

• reject RESOLVE_BENEATH | RESOLVE_IN_ROOT
• reject O_DIRECTORY | O_CREAT
• reject __O_TMPFILE until Linux O_TMPFILE unnamed-file semantics are actually implemented
• avoid accepting flags that translate_open_flags() cannot faithfully preserve

commit 3: Add fidelity tests for reviewed edge cases

• RESOLVE_BENEATH | RESOLVE_IN_ROOT -> EINVAL
• O_DIRECTORY | O_CREAT -> EINVAL
• O_TMPFILE | O_RDONLY -> EINVAL
• /proc/<self_pid>/fd/N absolute and dirfd=/proc relative forms under NO_MAGICLINKS / NO_SYMLINKS / NO_XDEV
• /dev/fd/N absolute, dirfd=/ + dev/fd/N, cwd / + dev/fd/N, and dirfd=/dev + shm/../fd/N under the same three resolve policies

One detail: I did not keep the literal dirfd=/tmp + ../dev/fd/N case because under macOS/elfuse open("/tmp") resolves to /private/tmp, so ../dev/fd becomes /private/dev/fd and correctly returns ENOENT. I used dirfd=/dev + shm/../fd/N to cover the same .. lexical normalization into /dev/fd/N without depending on macOS /tmp symlink behavior.

Thank you for the detailed review. I tried to keep the fixes narrowly scoped, separate the behavior changes from the tests, and make the validation path closer to Linux semantics while avoiding unsupported O_TMPFILE behavior for now.

[jserv]

等級 意見 修復 Commit

Please don't create tables like this. We're here to discuss and improve the software together, not to act as task trackers whose sole purpose is to clear action items and close pending items without genuine interaction.

[abnormal749]

Understood. I removed the checklist-style table and will keep the discussion focused on the reasoning and tradeoffs instead.

[jserv]

Squash the commits and revise the commit messages to comply with the guidelines at https://cbea.ms/git-commit/

[abnormal749]

I have pushed the cleaned branch with the openat2 work and squashed into one commit on top of the latest upstream/main, after the PTY ioctl changes.

The latest review comments are addressed as well: bool is used for boolean helpers, open_how now has a short comment explaining the Linux extensible ABI size behavior, and the self-pid /proc fd parser uses a named pid_start pointer instead of the raw path + 5 offset.

Local validation passes, including format/security/shellcheck/cppcheck, clang-tidy, scan-build, and the syscall fidelity test.

[jserv]

I have pushed the cleaned branch with the openat2 work and squashed into one commit on top of the latest upstream/main, after the PTY ioctl changes.

You don't need to summarize your changes, as we can always review them using Git. Instead, simply click "Resolve conversation" via GitHub web when the issue is addressed, or leave comments wherever you have concerns. Please avoid imitating the workflow or actions of AI agents.