Nix build system, static analysis, and README rewrite#2
Open
randomizedcoder wants to merge 35 commits intomainfrom
Open
Nix build system, static analysis, and README rewrite#2randomizedcoder wants to merge 35 commits intomainfrom
randomizedcoder wants to merge 35 commits intomainfrom
Conversation
The eBPF verifier doesn't like functions with six arguments so reduce the number of arguments for extract matadata abd handler ops functions from six to five. This is done by eliminating the hdr_offset argument. To compensate for the lodd of the argument we add a start pointer as ctrl->pkt.start. This is set to point to the first byte of the packet in XDP2_CTRL_SET_BASIC_PKT_DATA. The offset can then be computed as hdr - ctrl->pkt.start. xdp2_parse_hdr_offset is a utility function for this that takes a hdr and ctrl pointer as arguments. Eliminate the argument in calls to extract matadata abd handler ops functions. In xdp2_parse_tlvs, xdp2_parse_one_tlv, xdp2_parse_flag_fields, and xdp2_parse_array functions remove the offset argument. Also, remove an computation of the offset in parser functions.
Remove hdr_off argument from metadata extract functions. Call xdp2_parse_hdr_offset to get the header offset.
Remove hdr_off argument from metadata extract and handler functions in arrays.h, flag_fields,h, tlvs.h.
Remove hdr_off argument from metadata extract functions.
Remove hdr_off argument from metadata extract functions in handler functions from falcon, uet, sunh, superp, and sue.
Remove xdp2_bpf_*_tcpopt_* functions from bpf.h. This is obviously not the right palce for them.
Remove hdr_off argument from metadata extract and handler functions in templates. Call xdp2_parse_hdr_offset to get the header offset. Remove tracking offsets in flag-fields and TLVs. Use plain ctrl pointer in xdp template for parse TLV function..
Remove hdr_off argument from metadata extract functions and handler functions. Call xdp2_parse_hdr_offset to get the header offset. Call XDP2_CTRL_SET_BASIC_PKT_DATA with new arguments. Call xdp2_parse_hdr_offset to get the header offset.
Remove hdr_off argument from metadata extract functions. Call xdp2_parse_hdr_offset to get the header offset. Call XDP2_CTRL_SET_BASIC_PKT_DATA with new arguments.
Remove hdr_off argument from metadata extract functions and handler functions.
Safe, self-contained build system hygiene with no functional change. Boost.System is header-only since 1.69 — try linking without -lboost_system first, falling back for older versions. - src/configure.sh: backtick→$(), quote basename args, exit -1→exit 1, PLATFORMS=($(ls))→mapfile -t - src/configure.sh: Boost Wave/Thread/System/Filesystem tests try header-only link first - .gitignore: add install/ directory (Nix build output) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nix bypasses the cc-wrapper, so xdp2-compiler's ClangTool API misses system includes. extract_struct_constants() had no include path config at all, causing proto table extraction failures on Nix. Adds environment-variable-driven ClangToolConfig loaded from XDP2_C_INCLUDE_PATH, XDP2_GLIBC_INCLUDE_PATH, XDP2_LINUX_HEADERS_PATH. No-op on systems where these env vars are unset (Ubuntu/Fedora). - NEW src/tools/compiler/include/xdp2gen/clang-tool-config.h - NEW src/tools/compiler/src/clang-tool-config.cpp - NEW src/tools/compiler/include/xdp2gen/assert.h - src/tools/compiler/src/main.cpp: from_environment() + apply_config() for both create_clang_tool() and extract_struct_constants() - src/tools/compiler/Makefile: add clang-tool-config.o, drop -lboost_system (header-only since 1.69) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nix clang 18.1.8 handles C tentative definitions differently from Ubuntu clang 18.1.3: hasInit() returns true with void-type InitListExpr. getAs<RecordType>() returns nullptr → segfault on ->getDecl(). Also, XDP2_MAKE_PROTO_TABLE creates two declarations per group, but HandleTopLevelDecl used isSingleDecl() which missed the second one. - proto-tables.h: iterate all decls in HandleTopLevelDecl; null-check getAs<RecordType>() before ->getDecl() - graph_consumer.h: add "proto_def" to fields of interest - log_handler.h: make static members inline (ODR fix, multi-TU) - processing_utilities.h: debug logging in connect_vertices - template.cpp: import sys + debug logging for graph vertex info Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root parse function expects pointer to frame, but template passed frame by value. Causes segfault only in optimized (-O) builds where the compiler optimizes away the copy. - src/templates/xdp2/common_parser.template.c: frame → &frame Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
XDP2 headers now compile when __bpf__ is defined (clang -target bpf). Replaces libc functions with __builtin_memset/__builtin_memcpy, maps htons/ntohs to bpf_htons/bpf_ntohs via new bpf_compat.h, and provides minimal struct definitions to avoid heavy kernel header dependency chains that fail under the BPF backend. Core change pattern: #ifndef __KERNEL__ becomes #if !defined(__KERNEL__) && !defined(__bpf__) - NEW src/include/xdp2/bpf_compat.h: byte-order compat for BPF/kernel/userspace; IPPROTO_* via linux/in.h - arrays.h, parser_metadata.h, tlvs.h: guard stddef.h, sys/types.h - bitmap.h: XDP2_BITMAP_BITS_PER_WORD from __SIZEOF_LONG__ (literal 32/64) — __BITS_PER_LONG is computed, breaks token pasting - parser.h: BPF memset/memcpy builtins; guard siphash - parser_types.h: BPF stddef.h+stdint.h from clang resource dir; ssize_t as __s64, bool as _Bool - utility.h: split includes into 3 contexts - proto_defs/proto_*.h: arpa/inet.h → bpf_compat.h; BPF-specific minimal struct definitions for ARP, ICMP, ICMPv6 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows sample binaries to find libxdp2.so at runtime without
LD_LIBRARY_PATH, which is required for Nix-based test automation.
- samples/parser/{simple,offset,ports}_parser/Makefile: -Wl,-rpath
- samples/xdp/flow_tracker_combo/Makefile: -Wl,-rpath
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces monolithic ~1000-line flake.nix with modular imports from nix/.
This commit covers native x86_64 build and devshell only — tests,
cross-compilation, and MicroVMs added in subsequent commits.
derivation.nix has patches = [] — source fixes from commits 2-3
supersede the patch files, which are kept as documentation of the
Nix-specific issues encountered.
- NEW nix/llvm.nix, packages.nix, env-vars.nix, derivation.nix,
devshell.nix
- NEW nix/shell-functions/{ascii-art,build,clean,configure,
navigation,validation}.nix
- NEW nix/patches/01-nix-clang-system-includes.patch,
02-tentative-definition-null-check.patch (not applied, docs only)
- flake.nix: refactored; outputs: xdp2, xdp2-debug, devShell
- flake.lock: updated nixpkgs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds Nix-based test infrastructure that builds samples from source
(native mode) or uses pre-built binaries (cross-compilation mode),
then runs them against pcap test data with expected output validation.
XDP sample compilation uses unwrapped clang to avoid Nix cc-wrapper
hardening flags that are incompatible with BPF target.
Tests: simple-parser, offset-parser, ports-parser, flow-tracker-combo
xdp-build test currently SKIPPED (blocked on BPF stack/API fixes).
- NEW nix/xdp-samples.nix: BPF bytecode compilation
- NEW nix/samples/default.nix: pre-built sample binaries
- NEW nix/tests/{default,simple-parser,offset-parser,ports-parser,
flow-tracker-combo,xdp-build,simple-parser-debug}.nix
- flake.nix: add xdp-samples, tests, run-sample-tests, per-test
convenience aliases
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generates .deb package from nix build output for x86_64 distribution.
- NEW nix/packaging/{default,metadata,deb}.nix
- flake.nix: add deb-staging, deb-x86_64 outputs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QEMU-based MicroVMs for full-system testing on x86_64, aarch64, and
riscv64. Expect-based automation handles VM boot, login, command
execution, and shutdown — required for cross-arch test runners that
cannot use binfmt for kernel-level XDP testing.
- NEW nix/microvms/{default,mkVm,lib,constants}.nix
- NEW nix/microvms/scripts/{vm-expect,vm-verify-service,vm-debug}.exp
- flake.nix: add microvm input; add microvms.* outputs + legacy flat
aliases for lifecycle scripts
- flake.lock: add microvm + spectrum inputs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cross-compiles xdp2 + samples for riscv64-linux. xdp2-compiler runs natively on x86_64 to generate .p.c files, which are then compiled with the RISC-V GCC toolchain. Tests run via binfmt/QEMU user-mode or inside MicroVMs. - NEW nix/cross-tests.nix: reusable cross-compilation module (included for future DRY refactoring; flake.nix currently inlines this) - flake.nix: add pkgsCrossRiscv, xdp2-debug-riscv64, prebuiltSamplesRiscv64, testsRiscv64, run-riscv64-tests (guarded by system == "x86_64-linux") Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same cross-compilation pattern as RISC-V, targeting aarch64-linux. - flake.nix: add pkgsCrossAarch64, xdp2-debug-aarch64, prebuiltSamplesAarch64, testsAarch64, run-aarch64-tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convenience make targets wrapping nix build/run commands for the full
build/test/cross-compilation matrix. Includes binfmt prerequisite
checks for cross-arch targets.
- NEW Makefile (332 lines): build, build-debug, samples, test,
test-{simple,offset,ports,flow}, riscv64, riscv64-samples,
test-riscv64, test-riscv64-vm, aarch64, aarch64-samples,
test-aarch64, test-aarch64-vm, vm-{x86,aarch64,riscv64},
vm-test-all, deb, dev, shell, check, eval, clean, gc
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- documentation/nix/nix.md: update with cross-compilation, MicroVM integration testing, sample test infrastructure, Makefile targets, and Debian packaging sections. Fix patches description (now historical, not applied). Fix file listing for new nix/ modules. - documentation/cpp-style-guide.md: C++ style guide for xdp2-compiler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents verified test results across x86_64, RISC-V, and AArch64. All 38 parser/XDP sample tests pass on all architectures. xdp_build tests are SKIPPED pending BPF stack/API fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port static analysis framework from the reference Nix implementation, adapted for XDP2's C codebase and Make-based build system. 8 analysis tools at 3 levels: - quick: clang-tidy + cppcheck - standard: + flawfinder, clang-analyzer, gcc-warnings - deep: + gcc-analyzer, semgrep, sanitizers Compilation database generated by parsing `make V=1 VERBOSE=1` output with a custom Python script, since bear's LD_PRELOAD fails in the Nix sandbox and compiledb doesn't recognize Nix wrapper compiler paths. Python triage system aggregates, deduplicates, and prioritizes findings across all tools. Exemptions documented in EXEMPTIONS.md cover cppcheck tool limitations (macro parsing, void pointer arithmetic, container_of patterns) and high-volume style checks (narrowing conversions, reserved identifiers, assignment-in-if) that are intentional in C networking code. Results (analysis-quick): clang-tidy: 18,108 raw → 3,653 after triage cppcheck: 202 raw triage: 14 high-confidence findings Usage: nix build .#analysis-quick nix build .#analysis-standard nix build .#analysis-deep make analysis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cmd is allocated via malloc but the FD_SETSIZE error path returns without freeing it. Also fix inconsistent null pointer initialization (*oldcmd = 0 → NULL). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fast_nodes is allocated via calloc but two early return paths return false without freeing it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use %d for signed int (was %u, undefined behavior per C standard) and %zu for size_t (was %lu, wrong on 32-bit platforms). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two 'if (cursor != l) continue;' checks are dead code — both are inside blocks already guarded by cursor == l (TAB completion requires no guard since it works at any cursor position; '?' block has cursor == l in the outer if condition). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use 'memcmp(...) != 0' instead of implicit boolean conversion for clarity in the prefix comparison function. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add default: break; to two switch statements on metadata->addr_type in parselite_hash_length() and parselite_hash_metadata(). The code is functionally correct without them (fallthrough uses full struct size), but the default cases satisfy static analysis and document intent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Exempt 4 false positives from high-confidence findings: - clang-diagnostic-implicit-function-declaration (pcap.h, env issue) - bugprone-signed-char-misuse (parser.c, intentional sign extension) - clang-analyzer-core.UndefinedBinaryOperatorResult (cli.h, linker syms) - alpha.security.ArrayBoundV2 (parselite, bounded hash length) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_bitmap.c includes the generated bitmap_funcs.h, but the implicit rule for test_bitmap.o did not declare this dependency. With parallel make (-j), compilation could start before header generation completed. Add an explicit dependency of test_bitmap.o on bitmap_funcs.h. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete rewrite of README.md: add table of contents, parse graph introduction with flow_tracker_combo example, dual quick starts (Ubuntu + Nix), project layout tree, documentation index, Nix build system target table, and cross-architecture test status (38/38 x3). Removes 80+ lines of inline configure docs (now in getting-started.md). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
make buildsucceedsmake testpasses 38/38 on x86_64make analysis-quickruns cleanmake helplists all documented targets🤖 Generated with Claude Code