From 81d5f69889c338167ec5a9784c3bcfc138606c43 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 21 May 2026 17:13:51 -0700 Subject: [PATCH 01/10] draft and iteration to provide ref docs, debugging guide, deploying references, and updates to existing articles to reference the rest --- .../ServerGuides.docc/Documentation.md | 7 + .../Sources/ServerGuides.docc/building.md | 66 +++ .../debugging-a-service-using-a-backtrace.md | 204 +++++++++ .../deploying-static-binaries.md | 26 ++ .../Sources/ServerGuides.docc/packaging.md | 30 +- .../swift-backtrace-configuration.md | 406 ++++++++++++++++++ 6 files changed, 738 insertions(+), 1 deletion(-) create mode 100644 server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md create mode 100644 server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md diff --git a/server-guides/Sources/ServerGuides.docc/Documentation.md b/server-guides/Sources/ServerGuides.docc/Documentation.md index 2b997d7e2..877cf91f0 100644 --- a/server-guides/Sources/ServerGuides.docc/Documentation.md +++ b/server-guides/Sources/ServerGuides.docc/Documentation.md @@ -6,6 +6,13 @@ ## Topics +### Guides + - - - +- + +### Reference + +- diff --git a/server-guides/Sources/ServerGuides.docc/building.md b/server-guides/Sources/ServerGuides.docc/building.md index a4ebdd4a1..9897392c5 100644 --- a/server-guides/Sources/ServerGuides.docc/building.md +++ b/server-guides/Sources/ServerGuides.docc/building.md @@ -70,6 +70,72 @@ For code that frequently calls small functions across module boundaries, this ca However, results vary by project because optimizations are specific to your code. Always benchmark your specific workload with and without this flag before deploying to production. +### Preserve debug information for symbolication + +Release builds optimize for performance and don't embed DWARF debug information by default. +Without DWARF, a crashing binary's stack trace reports raw addresses instead of function names, files, and line numbers. +You have two ways to keep symbolication working: embed the debug information in the binary, or split it into a sidecar file you ship separately. + +#### Embed DWARF in a release build + +Pass `-g` to the Swift compiler when building for release: + +```bash +swift build -c release -Xswiftc -g +``` + +If your package also compiles C or C++ sources, forward `-g` to the C compiler as well: + +```bash +swift build -c release -Xswiftc -g -Xcc -g +``` + +Confirm the resulting binary contains DWARF sections: + +```bash +file .build/release/MyServer +# .build/release/MyServer: ELF 64-bit LSB pie executable, ..., +# with debug_info, not stripped +``` + +Binaries built this way are larger — often two to five times the size of a stripped release binary — but they need no companion file at symbolication time. + +#### Split debug information into a sidecar file + +For a smaller deployable artifact, separate the debug information into its own file and strip the original binary. +This is the layout Linux distributions use for `-debuginfo` and `-dbgsym` packages. + +```bash +cd .build/release + +# Copy debug sections into a sidecar file. +objcopy --only-keep-debug MyServer MyServer.debug + +# Remove debug information from the deployed binary. +objcopy --strip-debug --strip-unneeded MyServer + +# Record a link from the stripped binary to its sidecar. +objcopy --add-gnu-debuglink=MyServer.debug MyServer +``` + +You now have two artifacts: a small, stripped `MyServer` to ship in your container or distribution package, and a `MyServer.debug` to publish to a symbol server or ship in a companion debug-info package. +For how symbolicators consume the sidecar at crash time, see . + +#### Verify build IDs match + +A sidecar is only useful if it shares a build ID with the binary it describes. +The linker writes the build ID into a `.note.gnu.build-id` section that survives stripping, and symbolicators use it to pair the two files. + +Read the build ID from each file: + +```bash +readelf -n MyServer | grep 'Build ID' +readelf -n MyServer.debug | grep 'Build ID' +``` + +Both commands print the same hexadecimal value when the files match. +A mismatch means the binary and sidecar came from different builds; in that case the sidecar can't symbolicate the binary and you need to rebuild both together. + ## Review your build artifacts After compiling, locate your build artifacts. diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md new file mode 100644 index 000000000..a5938b05e --- /dev/null +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -0,0 +1,204 @@ +# Debugging a service using a backtrace + +Read a captured backtrace, map it back to your code, and recognize what the crash is telling you. + +## Overview + +When your Swift service crashes on Linux, the runtime spawns a helper process, +`swift-backtrace`, that prints a stack trace describing what was running and where it failed. +This article assumes you have such a trace from a crash and walks through how to use it: +persisting it from a container, reading its structure, mapping frames back to your source, +recognizing the common crash patterns, and reproducing the crash locally. + +For the configuration options the trace responds to, see . +To install the backtracer in a container image, +see . + +### Persist backtraces from a container + +When a container exits, its filesystem is gone, +so a crash that wrote to a file inside the container loses the trace +on the next restart. +For durable, addressable crash files, write to a mounted volume: + +```Dockerfile +ENV SWIFT_BACKTRACE=enable=yes,interactive=no,symbolicate=off,output-to=/var/crash-logs/ +``` + +Mount a volume at `/var/crash-logs` when you run the container. +With `docker run`, use `-v`; with Kubernetes, mount a `PersistentVolume` +or an `emptyDir`-backed volume at that path. + +When `output-to` resolves to a directory, the backtracer writes each crash +to a uniquely named file inside it, so a burst of crashes doesn't overwrite +earlier traces. +The `symbolicate=off` setting keeps crash handling fast in production; +see for how to resolve symbols afterward. + +For the format of the captured file (text or JSON) and other output options, +see . + +### Read the trace structure + +A captured trace has four parts: a header naming the signal, +a thread block per thread the backtracer captured, +a frame list inside each thread, and an optional images list. +Here's a representative trace, abbreviated: + +``` +*** Program crashed: Bad pointer dereference at 0x0000000000000010 *** + +Platform: Linux x86_64 + +Thread 0 "myservice" crashed: + +0 0x000055a9c4001234 MyService.handleRequest(_:) + 132 in myservice at Handler.swift:84 +1 0x000055a9c4002345 closure #1 in MyService.run() + 66 in myservice at Service.swift:42 +2 0x000055a9c4003456 MyService.run() + 280 in myservice at Service.swift:28 +3 0x00007f1234567890 swift_runJob + 48 in libswift_Concurrency.so +``` + +The header line names the **signal** and the **fault address**. +On Linux the common signals are `SIGSEGV` (bad memory access), +`SIGABRT` (abort, including Swift's runtime traps and `fatalError`), +`SIGBUS` (misaligned access), `SIGFPE` (arithmetic exception), and `SIGILL` (illegal instruction). + +The **thread line** identifies which thread crashed. +The backtracer captures only the crashed thread by default; +to see every thread, which helps when diagnosing deadlocks, +use `SWIFT_BACKTRACE=threads=all` — see . + +Each **frame line** has the form: + +``` +
+ in at : +``` + +The backtracer demangles the symbol by default. +The `at :` portion is present only when the binary carries DWARF debug information +and `symbolicate=full` (the default) is in effect. +Stripped production binaries with `symbolicate=off` produce frames with the address and image name only. + +By default the trace includes registers and a list of loaded images for the crashed thread. +You can tune or suppress both; see . + +### Map a frame to source + +Symbolication — turning addresses into function names, files, and lines — has a real cost. +The default, `symbolicate=full`, walks DWARF for every captured frame, +which can extend crash-handling time noticeably on a large binary. +The established pattern for production services is to capture lightweight traces at crash time +and resolve symbols offline. + +**At crash time, in production.** +Build production binaries with debug info stripped and run with `SWIFT_BACKTRACE=symbolicate=off`. +Each captured frame has its address and the image (binary or shared library) it belongs to — +enough to resolve to source later. +See for the full set of `symbolicate` values +and the trade-offs between `off`, `fast`, and `full`. + +**Post-mortem, on a developer or build machine.** +Keep the unstripped binary, or a separate DWARF debug-info package, from the same build that produced the trace. +With that artifact in hand: + +```bash +# Resolve an address to a file:line and (mangled) symbol name. +llvm-symbolizer --obj=path/to/unstripped/myservice 0x000055a9c4001234 + +# Demangle the resulting Swift symbol names if needed. +swift demangle "$1MyService13handleRequestyAA8ResponseVAA7RequestVF" +``` + +Time isn't critical at this stage, so spending CPU on full demangling and inline-frame resolution is fine. +Tools like `addr2line` work as well; pick whichever your build environment already has. + +#### Resolve symbols from a separate debug-info file + +If you ship debug information as a sidecar file rather than embedded in the binary (see ), place the sidecar where symbolicators look for it. +Symbolicators check two locations in order: + +1. The path recorded by `objcopy --add-gnu-debuglink`, resolved relative to the binary and then under `/usr/lib/debug//`. +2. The build-ID index: `/usr/lib/debug/.build-id//.debug`, where the hex digits come from the binary's build ID. + +The build-ID index is what `-debuginfo` and `-dbgsym` packages populate when installed, so a developer machine with the matching debug-info package installed resolves symbols transparently: + +``` +/usr/bin/MyServer # stripped, build ID ab12cd34... +/usr/lib/debug/.build-id/ab/12cd34....debug # sidecar +``` + +With the sidecar in place, the same `llvm-symbolizer` invocation resolves to file and line: + +```bash +llvm-symbolizer --obj=/usr/bin/MyServer 0x000055a9c4001234 +``` + +`llvm-symbolizer` reads the build ID from the stripped binary, finds the matching sidecar on the index path, and pulls source locations from its DWARF. + +If the build IDs don't match, the symbolicator falls back to address-only output. +Confirm the pairing with `readelf -n` against both files; see . + +### Recognize the crash pattern + +The signal and the message that the runtime prints just before the trace are usually enough +to identify the kind of bug. + +| Signal and message | Typical cause | +|---|---| +| `SIGABRT` + `Fatal error: Index out of range` | Out-of-bounds collection access | +| `SIGABRT` + `Fatal error: Unexpectedly found nil while unwrapping an Optional value` | Force-unwrap (`!`) of a `nil` optional | +| `SIGABRT` + `Fatal error: 'try!' expression unexpectedly raised an error` | `try!` on a call that threw | +| `SIGABRT` + `Could not cast value of type ...` | Force-cast (`as!`) failure | +| `SIGABRT` + `Fatal error: Arithmetic operation ... overflow` | Trapping integer arithmetic | +| `SIGABRT` + `precondition failed` or a custom message | `precondition()` or `fatalError()` call | +| `SIGSEGV` with frames in your unsafe-pointer or C-interop code | Memory-safety violation in that code | +| `SIGSEGV` with frames only in libc or runtime | Likely heap corruption from earlier code | +| `SIGSEGV` with a deeply uniform recursive frame pattern | Stack overflow | + +For runtime-trap crashes (the `SIGABRT` rows in the previous table), the relevant fix is almost always +in the topmost frame of *your* code in the trace — +the standard library trap is a few frames up from there. +For `SIGSEGV` in unsafe code or dependencies, the topmost frame names the call that touched bad memory, +but the actual cause can be earlier code that produced the bad pointer or freed memory still in use. + +### Reproduce locally + +Once you've identified the suspect call site, capture the input that triggered it — +request body, environment variables, mounted file, and configured client state — +and rerun the service locally. +Two paths are useful: + +- **Run under `lldb`.** + Launch the service with `lldb` and let it stop at the crash for live inspection of variables and threads. +- **Rerun with the interactive backtracer.** + Setting `SWIFT_BACKTRACE=interactive=yes` causes the backtracer to drop into a debugger-like prompt at crash time + rather than printing and exiting. + See for the option. + Interactive mode requires a TTY, so it's a developer-machine tool, not a production setting. + +### Diagnose missing backtraces + +If a crash produced no trace at all, or one that ends after a few frames, +the cause is almost always one of the following: + +- term The helper binary isn't in the runtime image: + Slim and distroless base images don't ship `swift-backtrace`. + Copy `swift-backtrace-static` into the image — see . + +- term The runtime can't find the helper: + The runtime searches a fixed set of locations relative to a Swift root + directory; it doesn't consult `PATH`. + In a statically linked binary that doesn't follow the toolchain layout, + set `SWIFT_BACKTRACE=swift-backtrace=` or `SWIFT_ROOT=`. + See for the search order. + +- term `/proc` isn't mounted: + The backtracer enumerates threads and locates loaded images through `/proc//`. + Containers running with a stripped-down filesystem don't always have it mounted; re-enable `/proc`. + +- term Frame pointers are missing: + The fast unwinder follows frame pointers, so a binary built with `-fomit-frame-pointer` + (or a C/C++ dependency built that way) produces traces that stop at the first frame without one. + Swift code emits frame pointers by default since Swift 5.10. + Force the precise unwinder with `SWIFT_BACKTRACE=unwind=precise` to use DWARF instead, + or rebuild affected dependencies with `-fno-omit-frame-pointer`. diff --git a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md index 58dd83235..f70e4893c 100644 --- a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md +++ b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md @@ -120,6 +120,32 @@ container build -t :latest . > If you need a shell for troubleshooting, > use a slim base image during development and switch to `scratch` for production. +### Include the backtracer for crash diagnostics + +A static binary in a `scratch` container has no toolchain layout for the runtime +to fall back on, so the location where it expects to find `swift-backtrace` doesn't exist. +To get backtraces from crashes, copy the static helper from a Swift container image +into your final image and explicitly pass its path to the runtime: + +```Dockerfile +FROM swift:6.2-noble AS toolchain + +FROM scratch +COPY --from=toolchain /usr/libexec/swift/linux/swift-backtrace-static /swift-backtrace +COPY .build/release/ / + +ENV SWIFT_BACKTRACE=enable=yes,interactive=no,symbolicate=off,swift-backtrace=/swift-backtrace + +EXPOSE 8080 +ENTRYPOINT ["/"] +``` + +The `swift-backtrace=/swift-backtrace` setting overrides the runtime's default search +and points it to the correct location. + +For the full set of `SWIFT_BACKTRACE` options, see . +To debug your service using a captured trace, see . + ### Build and publish with Swift Container Plugin The [Swift Container Plugin](https://github.com/apple/swift-container-plugin) diff --git a/server-guides/Sources/ServerGuides.docc/packaging.md b/server-guides/Sources/ServerGuides.docc/packaging.md index de223ff27..77e2f2e01 100644 --- a/server-guides/Sources/ServerGuides.docc/packaging.md +++ b/server-guides/Sources/ServerGuides.docc/packaging.md @@ -117,7 +117,7 @@ CMD ["/"] The `--static-swift-stdlib` flag links the Swift standard library into your executable, so the final image doesn't need the Swift runtime installed. -If your service uses `FoundationNetworking` or `FoundationXML`, use `swift:6.2-noble-slim` instead of `ubuntu:noble` for the runtime image. +If your service uses `FoundationNetworking` or `FoundationXML`, use `swift:6.2-noble-slim` instead of `ubuntu:noble` for a smaller runtime image. This image includes system libraries that those frameworks require: `libcurl` and `libxml2`. Build and run the image the same way: @@ -131,6 +131,34 @@ container build -t :latest . > Without it, `COPY . /workspace` sends everything to the build daemon, > which slows builds and can bloat image layers. +### Include the backtracer in the runtime image + +When a Swift service crashes, the runtime invokes a helper binary, `swift-backtrace`, +to capture a stack trace and write it to stderr. +Slim runtime images don't include this helper, +so crashes terminate silently without a trace. +The toolchain ships a statically linked backtracer in the builder image +that you can copy into the runtime stage as is: + +```Dockerfile +COPY --from=builder /usr/libexec/swift/linux/swift-backtrace-static \ + /usr/bin/swift-backtrace +``` + +For containers without a TTY, set `SWIFT_BACKTRACE` so the backtracer +runs non-interactively and writes machine-readable output: + +```Dockerfile +ENV SWIFT_BACKTRACE=enable=yes,interactive=no +``` + +Swift 5.10 and later emit frame pointers by default, +so typical builds don't need extra flags. +Linux requires frame pointers for complete backtraces. + +For the full set of `SWIFT_BACKTRACE` options, see . +To debug your service using a captured trace, see . + ### Cache build artifacts to speed up rebuilds The Dockerfiles above copy all source files into the image and build from scratch every time. diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md new file mode 100644 index 000000000..4bb3c5542 --- /dev/null +++ b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md @@ -0,0 +1,406 @@ +# Swift backtrace configuration + +Control how the Swift runtime captures and reports stack traces when your service crashes. + +## Overview + +When a Swift program receives a fatal signal on Linux — +`SIGSEGV`, `SIGABRT`, `SIGBUS`, `SIGFPE`, `SIGILL`, `SIGTRAP`, or `SIGQUIT` — +the runtime's signal handler suspends the program's threads and launches a separate +helper process, `swift-backtrace`, to gather and format a crash report. +The helper reads the crashed process's memory, walks each thread's stack, +symbolicates frames, and writes the result before the original process exits. + +This two-process design keeps the signal handler small and signal-safe, +and lets the backtracer use ordinary Swift code to format output and look up symbols. + +The runtime only installs its signal handler for a given signal if no handler +is already in place, and only sets up an alternate signal stack if one isn't +already configured. +A library that installs its own handler for one of the catchable signals — +even `SIG_IGN` — silently disables backtracing for that signal. + +You configure the backtracer through the `SWIFT_BACKTRACE` environment variable. +It accepts a comma-separated list of `key=value` pairs: + +```bash +SWIFT_BACKTRACE=enable=yes,interactive=no,format=json,output-to=/var/log/crashes/ +``` + +The defaults target interactive use at a terminal. +For server deployments, the most common adjustments are +`interactive=no` for containers without a TTY, +and either `format=json` or `output-to=` so traces land somewhere your log pipeline can ingest. + +### Enabling and presentation + +| Option | Values | Default | Notes | +|---|---|---|---| +| `enable` | `yes`, `no`, `tty` | `yes` | `tty` enables backtracing only when stderr is a terminal. | +| `interactive` | `yes`, `no`, `tty` | `tty` | Interactive mode drops into a debugger-like prompt; disable for non-TTY containers. | +| `color` | `yes`, `no`, `tty` | `tty` | ANSI color in output. | +| `demangle` | `yes`, `no` | `yes` | Demangle Swift symbols in frames. | +| `preset` | `friendly`, `medium`, `full`, `auto` | `auto` | Bundles `threads`, `registers`, and `images` defaults. `friendly` is concise; `full` shows everything. | + +### Captured content + +| Option | Values | Default | Notes | +|---|---|---|---| +| `threads` | `crashed`, `all`, `preset` | `preset` | `crashed` shows only the failing thread; `all` is useful for deadlocks. | +| `registers` | `none`, `crashed`, `all`, `preset` | `preset` | Whether to dump CPU registers alongside frames. | +| `images` | `none`, `mentioned`, `all`, `preset` | `preset` | Loaded shared-library list; `mentioned` includes only those referenced by frames. | +| `limit` | integer, `none` | `64` | Maximum frames per thread. Prevents runaway output on infinite recursion. | +| `top` | integer | `16` | When `limit` truncates, always keep this many frames from the top of stack. | +| `sanitize` | `yes`, `no`, `preset` | `preset` | Strip PII from paths in captured frames. Has no effect on Linux; the runtime parses the option but doesn't transform paths outside macOS. | + +When the captured stack has more than `limit` frames, the backtracer keeps +`top` frames from the top of the stack (the deepest, most recent calls) +and `limit - 1 - top` frames from the bottom (the entry into the program), +joined by a `...` marker. +For example, a 10-frame stack with `limit=5,top=2` shows frames 10, 9, then +`...`, then 2, 1 — letting you see both where execution started and where +the fault occurred when a runaway recursion blows past the limit. + +### Unwinding and symbolication + +| Option | Values | Default | Notes | +|---|---|---|---| +| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` consults DWARF unwind info. `auto` picks per-frame. | +| `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF; `fast` resolves only the outermost symbol; `off` reports raw addresses. | +| `cache` | `yes`, `no` | `yes` | Cache symbol lookups across frames. | + +Symbolication quality depends on what's in your binary. +Stripped binaries report addresses without symbol names. +To embed DWARF directly in a release binary, or to split debug info into a sidecar file the symbolicator can find by build ID, see . +To resolve addresses from a captured trace using either form, see . + +### Output + +| Option | Values | Default | Notes | +|---|---|---|---| +| `format` | `text`, `json` | `text` | JSON is structured for log aggregators. | +| `output-to` | `stderr`, `stdout`, file path, directory path | `stderr` | If the path resolves to an existing directory, the backtracer writes each crash to a uniquely named file inside it. Otherwise the backtracer treats the path as a filename. | + +A text-format trace looks like this (abbreviated; registers and image list omitted): + +``` +*** Signal 11: Backtracing from 0xaaaaaaaa1804... done *** + +*** Program crashed: Bad pointer dereference at 0x0000000000000004 *** + +Thread 0 crashed: + +0 0x0000aaaaaaaa1804 MyService.handleRequest(_:) + 180 in myservice at /work/Sources/myservice/Service.swift:7:21 +1 [ra] 0x0000aaaaaaaa11a8 closure #1 in MyService.run() + 15 in myservice at /work/Sources/myservice/Service.swift:13:17 +2 [ra] 0x0000aaaaaaaa117c MyService.run() + 27 in myservice at /work/Sources/myservice/Service.swift:15:9 +3 [async] 0x0000aaaaaaaa1218 static Main.main() in myservice at /work/Sources/myservice/Service.swift:22 +4 [async] [system] 0x0000aaaaaaaa1344 static Main.$main() in myservice at // +... +``` + +Bracketed labels denote frame attributes: `[ra]` for return-address frames, +`[async]` for async resumption points, `[system]` for compiler-generated or +runtime frames, and `[thunk]` for compiler-generated bridges. The same +attributes appear in JSON output as the frame's `kind` field and the +`system` and `thunk` Boolean markers. +For the same crash represented as JSON, see . + +JSON output produces one object per crash, with a top-level +`description` describing the failure, a `faultAddress`, `platform`, +`architecture`, and a `threads` array containing per-thread frame lists, +along with registers when configured. See +for the full schema. + +Pipe it to your log shipper or write to a mounted volume: + +```bash +SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ +``` + +### Advanced options + +| Option | Values | Default | Notes | +|---|---|---|---| +| `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtracer to finish. | +| `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. | +| `warnings` | `enabled`, `suppressed` | `enabled` | Diagnostic messages from the backtracer itself. | +| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace. Useful in CI environments where leaked file descriptors can cause resource contention. | + +The runtime locates `swift-backtrace` by deriving a Swift root directory +and searching a fixed set of subdirectories underneath it. +The root is whichever of these the runtime finds first: + +1. The value of the `SWIFT_ROOT` environment variable, if set. +2. A path computed from the location of `libswiftCore` (when the runtime is + dynamically linked). +3. A path computed from the location of the running executable (when the + runtime is statically linked into the program). + +Within the root, the runtime checks these locations in order, where +`` is the Swift platform name (for example, `linux`) and `` is the +CPU architecture (for example, `x86_64` or `aarch64`): + +``` +/libexec/swift/ +/libexec/swift// +/libexec/swift +/libexec/swift/ +/bin +/bin/ + +``` + +The runtime doesn't consult `PATH`. If none of the above contain the helper, +crashes go uncaught — set `SWIFT_BACKTRACE=swift-backtrace=` to point +at it explicitly, or place the binary at one of the search locations. + +### JSON crash log schema + +When `format=json`, the backtracer emits one JSON object per crash. +Addresses appear as hexadecimal strings (with `0x` prefix); raw byte data +such as captured memory or build IDs appears as un-prefixed hexadecimal +strings with no inter-byte whitespace. +The backtracer omits Boolean fields when `false`, and omits unknown or empty +values entirely to save space. + +#### Top-level fields + +The following fields are always present: + +| Field | Value | +|---|---| +| `timestamp` | ISO-8601 timestamp string. | +| `kind` | The string `crashReport`. | +| `description` | Textual description of the crash or runtime failure. | +| `faultAddress` | Fault address associated with the crash. | +| `platform` | Platform string; first token names the platform, followed by version info — for example, `"linux (Ubuntu 22.04.5 LTS)"`. | +| `architecture` | Processor architecture name. | +| `threads` | Array of thread records. | + +The following fields appear conditionally, depending on backtracer settings: + +| Field | Value | +|---|---| +| `omittedThreads` | Count of threads omitted when `threads=crashed`. Omitted if zero. | +| `capturedMemory` | Dictionary of captured memory contents, keyed by hex address strings. Absent when `sanitize` is enabled or no data was captured. | +| `omittedImages` | When `images=mentioned`, count of images whose details were omitted. | +| `images` | Array of image records (unless `images=none`). | +| `backtraceTime` | Time taken to generate the report, in seconds. | + +#### Thread records + +| Field | Value | +|---|---| +| `name` | Thread name; omitted if not set. | +| `crashed` | `true` for the crashing thread; omitted otherwise. | +| `registers` | Dictionary of register name → hex value string. The backtracer omits this on non-crashed threads when `registers=crashed`. | +| `frames` | Array of frame records. | + +#### Frame records + +Each frame has a `kind`: + +| Kind | Meaning | +|---|---| +| `programCounter` | Frame address is a directly captured program counter. | +| `returnAddress` | Frame address is a return address. | +| `asyncResumePoint` | Frame address is a resumption point in an `async` function. | +| `omittedFrames` | Frame-omission record (carries a `count` field). | +| `truncated` | Backtrace was truncated at this point. | + +Address-bearing frames also include `address` (hex string). +Symbolicated frames can add `inlined`, `runtimeFailure`, `thunk`, or +`system` Boolean markers, plus the following fields when symbol lookup succeeds: + +| Field | Value | +|---|---| +| `symbol` | Mangled symbol name. | +| `offset` | Offset from the symbol to the frame address. | +| `description` | Demangled, human-readable description (when `demangle=yes`). | +| `image` | Name of the image containing the symbol. | +| `sourceLocation` | `{ file, line, column }` dictionary, when `symbolicate=full` and DWARF info is available. | + +#### Image records + +| Field | Value | +|---|---| +| `name` | Image name. | +| `buildId` | Build ID as un-prefixed hex string. | +| `path` | Path to the image. | +| `baseAddress` | Base address of the image text (hex string). | +| `endOfText` | End of the image text (hex string). | + +#### Example payload + +The following report comes from a small service whose `MyService.run()` calls +into a synchronous closure that invokes `MyService.handleRequest`, which +dereferences a NULL-adjacent pointer. +The binary was built with debug information and run on Linux arm64 with +default backtracer settings (`format=json`, preset `auto`, `demangle=yes`, +`symbolicate=full`). +For length, the `registers` and `capturedMemory` objects show only a +representative subset of their entries; a real trace contains every +general-purpose register and many more captured memory snapshots. + +```json +{ + "timestamp": "2026-05-21T23:51:47.905791Z", + "kind": "crashReport", + "description": "Bad pointer dereference", + "faultAddress": "0x0000000000000004", + "platform": "Linux (Ubuntu 24.04.4 LTS)", + "architecture": "arm64", + "threads": [ + { + "crashed": true, + "registers": { + "x0": "0x0000fffff702f830", + "x9": "0x0000000000000004", + "fp": "0x0000fffff702e5f0", + "lr": "0x0000aaaaaaaa11a8", + "sp": "0x0000fffff702e5a0", + "pc": "0x0000aaaaaaaa1804" + }, + "frames": [ + { + "kind": "programCounter", + "address": "0x0000aaaaaaaa1804", + "symbol": "$s9myservice9MyServiceV13handleRequestyAA8ResponseVAA0E0VF", + "offset": 180, + "description": "MyService.handleRequest(_:) + 180", + "image": "myservice", + "sourceLocation": { + "file": "/work/Sources/myservice/Service.swift", + "line": 7, + "column": 21 + } + }, + { + "kind": "returnAddress", + "address": "0x0000aaaaaaaa11a8", + "symbol": "$s9myservice9MyServiceV3runyyYaKFyycfU_", + "offset": 15, + "description": "closure #1 in MyService.run() + 15", + "image": "myservice", + "sourceLocation": { + "file": "/work/Sources/myservice/Service.swift", + "line": 13, + "column": 17 + } + }, + { + "kind": "returnAddress", + "address": "0x0000aaaaaaaa117c", + "symbol": "$s9myservice9MyServiceV3runyyYaKFTY0_", + "offset": 27, + "description": "MyService.run() + 27", + "image": "myservice", + "sourceLocation": { + "file": "/work/Sources/myservice/Service.swift", + "line": 15, + "column": 9 + } + }, + { + "kind": "asyncResumePoint", + "address": "0x0000aaaaaaaa1218", + "symbol": "$s9myservice4MainV4mainyyYaKFZTQ1_", + "offset": 0, + "description": "static Main.main()", + "image": "myservice", + "sourceLocation": { + "file": "/work/Sources/myservice/Service.swift", + "line": 22, + "column": 0 + } + }, + { + "kind": "asyncResumePoint", + "address": "0x0000aaaaaaaa1344", + "system": true, + "symbol": "$s9myservice4MainV5$mainyyYaKFZTQ0_", + "offset": 0, + "description": "static Main.$main()", + "image": "myservice", + "sourceLocation": { + "file": "//", + "line": 0, + "column": 0 + } + }, + { + "kind": "asyncResumePoint", + "address": "0x0000aaaaaaaa15d8", + "thunk": true, + "symbol": "$sIetH_yts5Error_pIegHrzo_TRTQ0_", + "offset": 0, + "description": "thunk for @escaping @convention(thin) @async () -> ()", + "image": "myservice", + "sourceLocation": { + "file": "//", + "line": 0, + "column": 0 + } + }, + { + "kind": "asyncResumePoint", + "address": "0x0000fffff78a4a18", + "system": true, + "symbol": "_ZL23completeTaskWithClosurePN5swift12AsyncContextEPNS_10SwiftErrorE", + "offset": 0, + "description": "completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*)", + "image": "libswift_Concurrency.so" + } + ] + } + ], + "capturedMemory": { + "0x0000aaaaaaaa1804": "280100f9d1ffff97fd7b45a9ff830191", + "0x0000aaaaaaaa11a8": "fd7bc1a8c0035fd6ff8300d1fd7b01a9", + "0x0000fffff702e5f0": "00e602f7ffff0000a811aaaaaaaa0000", + "0x0000fffff702e5a0": "f81410f7ffff00005fe602f7ffff0000" + }, + "omittedImages": 12, + "images": [ + { + "name": "myservice", + "buildId": "06b835b6531c5086c472cc7c15b89c97ea973e71", + "path": "/work/.build/aarch64-unknown-linux-gnu/debug/myservice", + "baseAddress": "0x0000aaaaaaaa0000", + "endOfText": "0x0000aaaaaaaa70e8" + }, + { + "name": "libswift_Concurrency.so", + "buildId": "76c30ca7c36aab49fe44cfdf7917c95cb33dfa01", + "path": "/usr/lib/swift/linux/libswift_Concurrency.so", + "baseAddress": "0x0000fffff7840000", + "endOfText": "0x0000fffff78bdc48" + } + ], + "backtraceTime": 0.0019300830000000002 +} +``` + +A few things to notice in this trace: + +- The async boundary in this program sits between `Main.main()` and + `MyService.run()` — that's where the `asyncResumePoint` frames begin. + `MyService.run()` itself appears as a `returnAddress` because at the moment + of the crash it was running on a regular stack inside the synchronous + closure it dispatched. +- The frames marked `system: true` (`Main.$main()` and + `completeTaskWithClosure`) come from compiler-generated entry-point glue + and the Swift Concurrency runtime. They don't usually have a meaningful + source location, so the backtracer reports ``. +- The frame marked `thunk: true` is a compiler-generated bridge that adapts + one async calling convention to another. +- `omittedImages: 12` means twelve loaded shared libraries weren't included + because no captured frame referenced them — the default `images=mentioned` + keeps the report compact. Set `images=all` to include every loaded image. + +### See also + +The packaging guide shows the recommended copy location for container images; +see . +To diagnose backtraces that don't appear or that are missing information, +see . From 369a1738b039e74ff33bcac541fe3ec0903cb948 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 21 May 2026 17:39:35 -0700 Subject: [PATCH 02/10] editorial pass --- .../debugging-a-service-using-a-backtrace.md | 39 ++++++++++--------- .../deploying-static-binaries.md | 5 ++- .../Sources/ServerGuides.docc/packaging.md | 9 +++-- .../swift-backtrace-configuration.md | 37 ++++++++++-------- 4 files changed, 50 insertions(+), 40 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md index a5938b05e..b6f663ca7 100644 --- a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -1,14 +1,14 @@ # Debugging a service using a backtrace -Read a captured backtrace, map it back to your code, and recognize what the crash is telling you. +Interpret a Swift service's crash trace to find the failing code on Linux. ## Overview When your Swift service crashes on Linux, the runtime spawns a helper process, `swift-backtrace`, that prints a stack trace describing what was running and where it failed. -This article assumes you have such a trace from a crash and walks through how to use it: -persisting it from a container, reading its structure, mapping frames back to your source, -recognizing the common crash patterns, and reproducing the crash locally. +This article assumes you already have a trace from a crash. +It covers how to persist the trace from a container, read its structure, +map frames back to your source, recognize common crash patterns, and reproduce the crash locally. For the configuration options the trace responds to, see . To install the backtracer in a container image, @@ -17,7 +17,7 @@ see . ### Persist backtraces from a container When a container exits, its filesystem is gone, -so a crash that wrote to a file inside the container loses the trace +so a crash that writes to a file inside the container loses the trace on the next restart. For durable, addressable crash files, write to a mounted volume: @@ -41,7 +41,7 @@ see . ### Read the trace structure A captured trace has four parts: a header naming the signal, -a thread block per thread the backtracer captured, +a thread block for each thread the backtracer captures, a frame list inside each thread, and an optional images list. Here's a representative trace, abbreviated: @@ -75,8 +75,8 @@ Each **frame line** has the form: ``` The backtracer demangles the symbol by default. -The `at :` portion is present only when the binary carries DWARF debug information -and `symbolicate=full` (the default) is in effect. +The `at :` portion appears only when the binary carries DWARF debug information +and you run with `symbolicate=full` (the default). Stripped production binaries with `symbolicate=off` produce frames with the address and image name only. By default the trace includes registers and a list of loaded images for the crashed thread. @@ -114,13 +114,16 @@ Tools like `addr2line` work as well; pick whichever your build environment alrea #### Resolve symbols from a separate debug-info file -If you ship debug information as a sidecar file rather than embedded in the binary (see ), place the sidecar where symbolicators look for it. -Symbolicators check two locations in order: +You can ship debug information as a sidecar file rather than embedding it in the binary +(see ). +Place the sidecar where symbolicators look for it. +Symbolicators look first at the path recorded by `objcopy --add-gnu-debuglink` +(resolved relative to the binary, then under `/usr/lib/debug//`), +and then at the build-ID index at `/usr/lib/debug/.build-id//.debug`, +where the hex digits come from the binary's build ID. -1. The path recorded by `objcopy --add-gnu-debuglink`, resolved relative to the binary and then under `/usr/lib/debug//`. -2. The build-ID index: `/usr/lib/debug/.build-id//.debug`, where the hex digits come from the binary's build ID. - -The build-ID index is what `-debuginfo` and `-dbgsym` packages populate when installed, so a developer machine with the matching debug-info package installed resolves symbols transparently: +The `-debuginfo` and `-dbgsym` packages populate the build-ID index when you install them, +so a developer machine with the matching debug-info package installed resolves symbols transparently: ``` /usr/bin/MyServer # stripped, build ID ab12cd34... @@ -159,7 +162,7 @@ For runtime-trap crashes (the `SIGABRT` rows in the previous table), the relevan in the topmost frame of *your* code in the trace — the standard library trap is a few frames up from there. For `SIGSEGV` in unsafe code or dependencies, the topmost frame names the call that touched bad memory, -but the actual cause can be earlier code that produced the bad pointer or freed memory still in use. +but the actual cause is often earlier code that produces a bad pointer or frees memory still in use. ### Reproduce locally @@ -171,8 +174,8 @@ Two paths are useful: - **Run under `lldb`.** Launch the service with `lldb` and let it stop at the crash for live inspection of variables and threads. - **Rerun with the interactive backtracer.** - Setting `SWIFT_BACKTRACE=interactive=yes` causes the backtracer to drop into a debugger-like prompt at crash time - rather than printing and exiting. + Set `SWIFT_BACKTRACE=interactive=yes` so the backtracer drops into a debugger-like prompt at crash time + instead of printing and exiting. See for the option. Interactive mode requires a TTY, so it's a developer-machine tool, not a production setting. @@ -199,6 +202,6 @@ the cause is almost always one of the following: - term Frame pointers are missing: The fast unwinder follows frame pointers, so a binary built with `-fomit-frame-pointer` (or a C/C++ dependency built that way) produces traces that stop at the first frame without one. - Swift code emits frame pointers by default since Swift 5.10. + Current Swift toolchains preserve frame pointers in release builds on Linux server architectures. Force the precise unwinder with `SWIFT_BACKTRACE=unwind=precise` to use DWARF instead, or rebuild affected dependencies with `-fno-omit-frame-pointer`. diff --git a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md index f70e4893c..fa2e63672 100644 --- a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md +++ b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md @@ -122,8 +122,8 @@ container build -t :latest . ### Include the backtracer for crash diagnostics -A static binary in a `scratch` container has no toolchain layout for the runtime -to fall back on, so the location where it expects to find `swift-backtrace` doesn't exist. +A static binary in a `scratch` container has no toolchain layout for the runtime to fall back on. +As a result, the runtime's default path to `swift-backtrace` doesn't exist in the image. To get backtraces from crashes, copy the static helper from a Swift container image into your final image and explicitly pass its path to the runtime: @@ -142,6 +142,7 @@ ENTRYPOINT ["/"] The `swift-backtrace=/swift-backtrace` setting overrides the runtime's default search and points it to the correct location. +The `symbolicate=off` setting keeps crash handling fast; you resolve addresses against the original binary later. For the full set of `SWIFT_BACKTRACE` options, see . To debug your service using a captured trace, see . diff --git a/server-guides/Sources/ServerGuides.docc/packaging.md b/server-guides/Sources/ServerGuides.docc/packaging.md index 77e2f2e01..74f0dff77 100644 --- a/server-guides/Sources/ServerGuides.docc/packaging.md +++ b/server-guides/Sources/ServerGuides.docc/packaging.md @@ -145,6 +145,8 @@ COPY --from=builder /usr/libexec/swift/linux/swift-backtrace-static \ /usr/bin/swift-backtrace ``` +The runtime looks for a binary named `swift-backtrace`, so the COPY renames the static helper as it copies it into `/usr/bin`. + For containers without a TTY, set `SWIFT_BACKTRACE` so the backtracer runs non-interactively and writes machine-readable output: @@ -152,9 +154,10 @@ runs non-interactively and writes machine-readable output: ENV SWIFT_BACKTRACE=enable=yes,interactive=no ``` -Swift 5.10 and later emit frame pointers by default, +Current Swift toolchains preserve frame pointers in release builds on Linux server architectures, so typical builds don't need extra flags. -Linux requires frame pointers for complete backtraces. +The fast unwinder relies on frame pointers; without them, the runtime falls back to DWARF-based unwinding, +which is slower but still produces complete traces when `.eh_frame` information is present. For the full set of `SWIFT_BACKTRACE` options, see . To debug your service using a captured trace, see . @@ -221,7 +224,7 @@ before the second stage picks it up. ### Build for a different platform If your development machine and deployment target use different CPU architectures — -for example, building on Apple Silicon (arm64) and deploying to x86_64 cloud infrastructure — +for example, building on Apple silicon (arm64) and deploying to x86_64 cloud infrastructure — you need to specify the target platform when building the image. With `docker`, use the `--platform` flag: diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md index 4bb3c5542..7478cbf30 100644 --- a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md +++ b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md @@ -30,18 +30,21 @@ SWIFT_BACKTRACE=enable=yes,interactive=no,format=json,output-to=/var/log/crashes The defaults target interactive use at a terminal. For server deployments, the most common adjustments are `interactive=no` for containers without a TTY, -and either `format=json` or `output-to=` so traces land somewhere your log pipeline can ingest. +and either `format=json` or `output-to=` so traces are written to a location your log pipeline can ingest. ### Enabling and presentation | Option | Values | Default | Notes | |---|---|---|---| -| `enable` | `yes`, `no`, `tty` | `yes` | `tty` enables backtracing only when stderr is a terminal. | -| `interactive` | `yes`, `no`, `tty` | `tty` | Interactive mode drops into a debugger-like prompt; disable for non-TTY containers. | -| `color` | `yes`, `no`, `tty` | `tty` | ANSI color in output. | +| `enable` | `yes`, `no`, `tty` | `yes` | `tty` enables backtracing when stdout is a terminal. | +| `interactive` | `yes`, `no`, `tty` | `tty` | Drops into a debugger-like prompt; `tty` enables it when both stdout and stdin are terminals. Disable for non-TTY containers. | +| `color` | `yes`, `no`, `tty` | `tty` | ANSI color in output; `tty` enables it when the resolved output stream is a terminal (stderr or stdout, depending on `output-to`). | | `demangle` | `yes`, `no` | `yes` | Demangle Swift symbols in frames. | | `preset` | `friendly`, `medium`, `full`, `auto` | `auto` | Bundles `threads`, `registers`, and `images` defaults. `friendly` is concise; `full` shows everything. | +When `output-to` points at a file, the `tty` resolutions above are overridden: +`enable=tty` becomes `yes`, while `interactive=tty` and `color=tty` become `no`. + ### Captured content | Option | Values | Default | Notes | @@ -49,9 +52,9 @@ and either `format=json` or `output-to=` so traces land somewhere your log | `threads` | `crashed`, `all`, `preset` | `preset` | `crashed` shows only the failing thread; `all` is useful for deadlocks. | | `registers` | `none`, `crashed`, `all`, `preset` | `preset` | Whether to dump CPU registers alongside frames. | | `images` | `none`, `mentioned`, `all`, `preset` | `preset` | Loaded shared-library list; `mentioned` includes only those referenced by frames. | -| `limit` | integer, `none` | `64` | Maximum frames per thread. Prevents runaway output on infinite recursion. | +| `limit` | integer, `none` | `64` | Maximum frames per thread, to prevent runaway output on infinite recursion. | | `top` | integer | `16` | When `limit` truncates, always keep this many frames from the top of stack. | -| `sanitize` | `yes`, `no`, `preset` | `preset` | Strip PII from paths in captured frames. Has no effect on Linux; the runtime parses the option but doesn't transform paths outside macOS. | +| `sanitize` | `yes`, `no`, `preset` | `preset` | Strips PII from paths in captured frames. Has no effect on Linux; the runtime parses the option but doesn't transform paths outside macOS. | When the captured stack has more than `limit` frames, the backtracer keeps `top` frames from the top of the stack (the deepest, most recent calls) @@ -59,7 +62,7 @@ and `limit - 1 - top` frames from the bottom (the entry into the program), joined by a `...` marker. For example, a 10-frame stack with `limit=5,top=2` shows frames 10, 9, then `...`, then 2, 1 — letting you see both where execution started and where -the fault occurred when a runaway recursion blows past the limit. +the fault occurred when a runaway recursion exceeds the limit. ### Unwinding and symbolication @@ -67,7 +70,7 @@ the fault occurred when a runaway recursion blows past the limit. |---|---|---|---| | `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` consults DWARF unwind info. `auto` picks per-frame. | | `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF; `fast` resolves only the outermost symbol; `off` reports raw addresses. | -| `cache` | `yes`, `no` | `yes` | Cache symbol lookups across frames. | +| `cache` | `yes`, `no` | `yes` | Caches symbol lookups across frames. | Symbolication quality depends on what's in your binary. Stripped binaries report addresses without symbol names. @@ -111,7 +114,7 @@ JSON output produces one object per crash, with a top-level along with registers when configured. See for the full schema. -Pipe it to your log shipper or write to a mounted volume: +Pipe the JSON output to your log shipper or write it to a mounted volume: ```bash SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ @@ -124,7 +127,7 @@ SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ | `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtracer to finish. | | `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. | | `warnings` | `enabled`, `suppressed` | `enabled` | Diagnostic messages from the backtracer itself. | -| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace. Useful in CI environments where leaked file descriptors can cause resource contention. | +| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace, useful in CI environments where leaked file descriptors cause resource contention. | The runtime locates `swift-backtrace` by deriving a Swift root directory and searching a fixed set of subdirectories underneath it. @@ -151,7 +154,7 @@ CPU architecture (for example, `x86_64` or `aarch64`): ``` The runtime doesn't consult `PATH`. If none of the above contain the helper, -crashes go uncaught — set `SWIFT_BACKTRACE=swift-backtrace=` to point +the runtime can't capture crashes — set `SWIFT_BACKTRACE=swift-backtrace=` to point at it explicitly, or place the binary at one of the search locations. ### JSON crash log schema @@ -185,7 +188,7 @@ The following fields appear conditionally, depending on backtracer settings: | `capturedMemory` | Dictionary of captured memory contents, keyed by hex address strings. Absent when `sanitize` is enabled or no data was captured. | | `omittedImages` | When `images=mentioned`, count of images whose details were omitted. | | `images` | Array of image records (unless `images=none`). | -| `backtraceTime` | Time taken to generate the report, in seconds. | +| `backtraceTime` | Time to generate the report, in seconds. | #### Thread records @@ -206,7 +209,7 @@ Each frame has a `kind`: | `returnAddress` | Frame address is a return address. | | `asyncResumePoint` | Frame address is a resumption point in an `async` function. | | `omittedFrames` | Frame-omission record (carries a `count` field). | -| `truncated` | Backtrace was truncated at this point. | +| `truncated` | The backtrace is truncated at this point. | Address-bearing frames also include `address` (hex string). Symbolicated frames can add `inlined`, `runtimeFailure`, `thunk`, or @@ -238,7 +241,7 @@ dereferences a NULL-adjacent pointer. The binary was built with debug information and run on Linux arm64 with default backtracer settings (`format=json`, preset `auto`, `demangle=yes`, `symbolicate=full`). -For length, the `registers` and `capturedMemory` objects show only a +To keep this example concise, the `registers` and `capturedMemory` objects show only a representative subset of their entries; a real trace contains every general-purpose register and many more captured memory snapshots. @@ -390,12 +393,12 @@ A few things to notice in this trace: closure it dispatched. - The frames marked `system: true` (`Main.$main()` and `completeTaskWithClosure`) come from compiler-generated entry-point glue - and the Swift Concurrency runtime. They don't usually have a meaningful + and the Swift concurrency runtime. They don't usually have a meaningful source location, so the backtracer reports ``. - The frame marked `thunk: true` is a compiler-generated bridge that adapts one async calling convention to another. -- `omittedImages: 12` means twelve loaded shared libraries weren't included - because no captured frame referenced them — the default `images=mentioned` +- `omittedImages: 12` means 12 loaded shared libraries aren't included + because no captured frame references them — the default `images=mentioned` keeps the report compact. Set `images=all` to include every loaded image. ### See also From e72e297a28ecb7281ee057553def1443c1dd1551 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 21 May 2026 17:47:00 -0700 Subject: [PATCH 03/10] restructuring building doc to level out the article structure after adding the subsection --- .../Sources/ServerGuides.docc/building.md | 59 ++++++++++--------- 1 file changed, 30 insertions(+), 29 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/building.md b/server-guides/Sources/ServerGuides.docc/building.md index 9897392c5..31930064b 100644 --- a/server-guides/Sources/ServerGuides.docc/building.md +++ b/server-guides/Sources/ServerGuides.docc/building.md @@ -1,7 +1,9 @@ -# Building Swift Server Applications +# Building Swift server applications Assemble your server applications using Swift Package Manager. +## Overview + Use [Swift Package Manager](/documentation/package-manager/) to build server applications. It provides a cross-platform foundation for building Swift code. You can build using the command line or through an integrated development environment (IDE) such as Xcode or Visual Studio Code. @@ -70,13 +72,13 @@ For code that frequently calls small functions across module boundaries, this ca However, results vary by project because optimizations are specific to your code. Always benchmark your specific workload with and without this flag before deploying to production. -### Preserve debug information for symbolication +## Preserve debug information for symbolication Release builds optimize for performance and don't embed DWARF debug information by default. Without DWARF, a crashing binary's stack trace reports raw addresses instead of function names, files, and line numbers. You have two ways to keep symbolication working: embed the debug information in the binary, or split it into a sidecar file you ship separately. -#### Embed DWARF in a release build +### Embed DWARF in a release build Pass `-g` to the Swift compiler when building for release: @@ -100,7 +102,7 @@ file .build/release/MyServer Binaries built this way are larger — often two to five times the size of a stripped release binary — but they need no companion file at symbolication time. -#### Split debug information into a sidecar file +### Split debug information into a sidecar file For a smaller deployable artifact, separate the debug information into its own file and strip the original binary. This is the layout Linux distributions use for `-debuginfo` and `-dbgsym` packages. @@ -121,7 +123,7 @@ objcopy --add-gnu-debuglink=MyServer.debug MyServer You now have two artifacts: a small, stripped `MyServer` to ship in your container or distribution package, and a `MyServer.debug` to publish to a symbol server or ship in a companion debug-info package. For how symbolicators consume the sidecar at crash time, see . -#### Verify build IDs match +### Verify build IDs match A sidecar is only useful if it shares a build ID with the binary it describes. The linker writes the build ID into a `.note.gnu.build-id` section that survives stripping, and symbolicators use it to pair the two files. @@ -136,23 +138,7 @@ readelf -n MyServer.debug | grep 'Build ID' Both commands print the same hexadecimal value when the files match. A mismatch means the binary and sidecar came from different builds; in that case the sidecar can't symbolicate the binary and you need to rebuild both together. -## Review your build artifacts - -After compiling, locate your build artifacts. -Swift Package Manager places them in directories that vary by platform and architecture: - -```bash -# Show where debug build artifacts are located -swift build --show-bin-path - -# Show where release build artifacts are located -swift build --show-bin-path -c release -``` - -The build products are written to the scratch path, which defaults to `.build`, but the specific location can vary based on platform or Swift compiler. -Use the `--show-bin-path` flag in deployment scripts to locate the build product without hardcoding platform-specific paths. - -### Build services for other platforms +## Build for another platform Swift build artifacts are both platform- and architecture-specific. Artifacts you create on macOS run only on macOS; those you create on Linux run only on Linux. @@ -161,7 +147,7 @@ This creates a challenge when you work on macOS and deploy to Linux servers. You can use Xcode for development, but it can't produce Linux artifacts for deployment. Swift provides two main approaches for cross-platform building. -#### Build with Linux containers +### Build with Linux containers On macOS, you can use [Container](https://github.com/apple/container) or the Docker CLI to verify your project builds under Linux. Apple publishes official Swift Docker images to [Docker Hub](https://hub.docker.com/_/swift), which provide complete Linux build environments. @@ -210,7 +196,13 @@ However, Docker container builds can be slower than native builds, especially on For more information on packaging your application or service, read [Packaging Swift Server Applications](./packaging.md). -#### Choose static or dynamic linking for the standard library +### Build with a VS Code Dev Container + +Visual Studio Code supports the Dev Container feature that lets you open, build, and debug your project within a container running on your local machine. +The Dev Container builds your code in the Linux environment that the container provides. +For more information on using a Dev Container, read [Visual Studio Code Dev Containers](https://docs.swift.org/vscode/documentation/userdocs/remote-dev/). + +### Choose static or dynamic linking for the standard library By default, Swift build artifacts link the standard library dynamically. This keeps individual build artifacts smaller, and multiple programs can share a single copy of the Swift runtime. @@ -235,7 +227,7 @@ For deploying to VMs or bare metal where you don't control the system configurat > Note: This technique doesn't apply on Apple platforms. -#### Cross-compile with the Static Linux SDK +### Cross-compile with the Static Linux SDK If the performance overhead of Docker-based builds affects your workflow, Swift 5.9 and later provide Static Linux SDKs. Find the link to install the static Linux SDK on the [install page at Swift.org](https://www.swift.org/install/), and detailed instructions in the article [Getting Started with the Static Linux SDK](https://www.swift.org/documentation/articles/static-linux-getting-started.html). @@ -261,12 +253,21 @@ your code can't use `dlopen` or similar mechanisms to dynamically load libraries For most projects, this distinction doesn't matter. However, packages with complex C dependencies can behave differently when built natively on Linux versus cross-compiled. +## Locate your build artifacts -### Build with VS Code using a Dev Container +After compiling, locate your build artifacts. +Swift Package Manager places them in directories that vary by platform and architecture: -Visual Studio Code supports the Dev Container feature that lets you open, build, and debug your project within a container running on your local machine. -The Dev Container builds your code in the Linux environment that the container provides. -For more information on using a Dev Container, read [Visual Studio Code Dev Containers](https://docs.swift.org/vscode/documentation/userdocs/remote-dev/). +```bash +# Show where debug build artifacts are located +swift build --show-bin-path + +# Show where release build artifacts are located +swift build --show-bin-path -c release +``` + +The build products are written to the scratch path, which defaults to `.build`, but the specific location can vary based on platform or Swift compiler. +Use the `--show-bin-path` flag in deployment scripts to locate the build product without hardcoding platform-specific paths. ### Inspect a binary From 8b5f978742c0a99bd15ca66dc9e947320d3b8459 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 21 May 2026 18:03:43 -0700 Subject: [PATCH 04/10] editorial fixes --- .../Sources/ServerGuides.docc/building.md | 80 +++++++++---------- 1 file changed, 39 insertions(+), 41 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/building.md b/server-guides/Sources/ServerGuides.docc/building.md index 31930064b..8d07202c1 100644 --- a/server-guides/Sources/ServerGuides.docc/building.md +++ b/server-guides/Sources/ServerGuides.docc/building.md @@ -4,14 +4,14 @@ Assemble your server applications using Swift Package Manager. ## Overview -Use [Swift Package Manager](/documentation/package-manager/) to build server applications. +Use [Swift Package Manager](https://www.swift.org/documentation/package-manager/) to build server applications. It provides a cross-platform foundation for building Swift code. -You can build using the command line or through an integrated development environment (IDE) such as Xcode or Visual Studio Code. +You can build using the command line or through an integrated development environment such as Xcode or Visual Studio Code. ## Choose a build configuration Swift Package Manager supports two distinct build configurations, each optimized for different stages of your development workflow. -The configurations are `debug`, frequently used during development, and `release`, which you use when profiling or creating production artifacts. +The configurations are `debug`, which you use during active development, and `release`, which you use when profiling or creating production artifacts. ### Use debug builds during development @@ -23,7 +23,7 @@ swift build Debug builds include full debugging symbols and runtime safety checks, which are essential during active development. The compiler skips most optimizations to keep compilation times fast and preserve maximum debugging information. -This lets you quickly test changes and debug your app using lldb and breakpoints. +This lets you quickly test changes and debug your app using `lldb` and breakpoints. However, skipping optimizations can come at a significant cost to runtime performance. Debug builds typically run more slowly than their release counterparts. @@ -72,6 +72,31 @@ For code that frequently calls small functions across module boundaries, this ca However, results vary by project because optimizations are specific to your code. Always benchmark your specific workload with and without this flag before deploying to production. +### Choose static or dynamic linking for the standard library + +By default, Swift build artifacts link the standard library dynamically. +This keeps individual build artifacts smaller, and multiple programs can share a single copy of the Swift runtime. +However, dynamic linking requires the Swift runtime to be installed on your deployment target. + +For deployment scenarios where you want more self-contained build artifacts, statically link the Swift standard library: + +```bash +swift build -c release --static-swift-stdlib +``` + +The resulting build artifacts still dynamically link to `glibc`, but have fewer other dependencies on the target system. +These executables bundle the Swift runtime directly: + +| Aspect | Dynamic linking | Static linking | +|--------|----------------|----------------| +| Build artifact size | Smaller (runtime not included) | Larger (runtime included in binary) | +| Deployment complexity | Requires Swift runtime on target system | Self-contained, no runtime needed | +| Version management | Must match runtime version on system | Each artifact includes its own runtime version | + +For deploying to VMs or bare metal where you don't control the system configuration, static linking removes the dependency on a pre-installed Swift runtime. + +> Note: This technique doesn't apply on Apple platforms. + ## Preserve debug information for symbolication Release builds optimize for performance and don't embed DWARF debug information by default. @@ -100,7 +125,7 @@ file .build/release/MyServer # with debug_info, not stripped ``` -Binaries built this way are larger — often two to five times the size of a stripped release binary — but they need no companion file at symbolication time. +This configuration produces larger binaries — often two to five times the size of a stripped release binary — but they need no companion file at symbolication time. ### Split debug information into a sidecar file @@ -149,13 +174,13 @@ Swift provides two main approaches for cross-platform building. ### Build with Linux containers -On macOS, you can use [Container](https://github.com/apple/container) or the Docker CLI to verify your project builds under Linux. +On macOS, you can use [Apple's container tool](https://github.com/apple/container) or the Docker CLI to verify your project builds under Linux. Apple publishes official Swift Docker images to [Docker Hub](https://hub.docker.com/_/swift), which provide complete Linux build environments. To build your application using the latest Swift release image: ```bash -# Build with Container +# Build with container container run -c 2 -m 8g --rm -it \ -v "$PWD:/code" -w /code \ swift:latest swift build @@ -169,10 +194,10 @@ docker run --rm -it \ These commands mount your current directory as `/code` in the container, set it as the working directory, and run `swift build` inside the Linux environment. The `swift:latest` container image provides this environment and `swift build` produces Linux-compatible build artifacts. -If you're on Apple silicon and need to target x86_64 Linux servers, you need to specify the target platform with the `--platform` option: +If you're on Apple silicon and need to target `x86_64` Linux servers, you need to specify the target platform with the `--platform` option: ```bash -# Build with Container +# Build with container container run -c 2 -m 8g --rm -it \ -v "$PWD:/code" -w /code \ --platform linux/amd64 \ @@ -192,9 +217,9 @@ The `-e QEMU_CPU=max` environment variable enables the maximum set of CPU featur To build your code into a container, you typically use a container declaration — a Dockerfile or Containerfile — that specifies all the steps to assemble the container image holding your build artifacts. Container-based builds work well in CI/CD pipelines and for validating that your code builds cleanly on Linux. -However, Docker container builds can be slower than native builds, especially on Apple silicon where x86_64 containers run through emulation. +However, Docker container builds can be slower than native builds, especially on Apple silicon where `x86_64` containers run through emulation. -For more information on packaging your application or service, read [Packaging Swift Server Applications](./packaging.md). +For more information on packaging your application or service, see . ### Build with a VS Code Dev Container @@ -202,31 +227,6 @@ Visual Studio Code supports the Dev Container feature that lets you open, build, The Dev Container builds your code in the Linux environment that the container provides. For more information on using a Dev Container, read [Visual Studio Code Dev Containers](https://docs.swift.org/vscode/documentation/userdocs/remote-dev/). -### Choose static or dynamic linking for the standard library - -By default, Swift build artifacts link the standard library dynamically. -This keeps individual build artifacts smaller, and multiple programs can share a single copy of the Swift runtime. -However, dynamic linking requires the Swift runtime to be installed on your deployment target. - -For deployment scenarios where you want more self-contained build artifacts, statically link the Swift standard library: - -```bash -swift build -c release --static-swift-stdlib -``` - -The resulting build artifacts still dynamically link to glibc, but have fewer other dependencies on the target system. -These executables bundle the Swift runtime directly: - -| Aspect | Dynamic linking | Static linking | -|--------|----------------|----------------| -| Build artifact size | Smaller (runtime not included) | Larger (runtime included in binary) | -| Deployment complexity | Requires Swift runtime on target system | Self-contained, no runtime needed | -| Version management | Must match runtime version on system | Each artifact includes its own runtime version | - -For deploying to VMs or bare metal where you don't control the system configuration, static linking removes the dependency on a pre-installed Swift runtime. - -> Note: This technique doesn't apply on Apple platforms. - ### Cross-compile with the Static Linux SDK If the performance overhead of Docker-based builds affects your workflow, Swift 5.9 and later provide Static Linux SDKs. @@ -266,12 +266,10 @@ swift build --show-bin-path swift build --show-bin-path -c release ``` -The build products are written to the scratch path, which defaults to `.build`, but the specific location can vary based on platform or Swift compiler. +Swift Package Manager writes the build products to the scratch path, which defaults to `.build`, but the specific location can vary by platform or Swift compiler version. Use the `--show-bin-path` flag in deployment scripts to locate the build product without hardcoding platform-specific paths. -### Inspect a binary - -If you're uncertain what platform a binary was built for, use the `file` command to inspect it: +If you're uncertain which platform a binary targets, use the `file` command to inspect it: ``` file .build/debug/MyServer @@ -293,7 +291,7 @@ Output from a debug build on Linux, built inside a container on Apple silicon: with debug_info, not stripped ``` -Output from a debug build on macOS, built using the Container tool with x86_64 emulation: +Output from a debug build on macOS, built using the container tool with `x86_64` emulation: ``` .build/debug/MyServer: ELF 64-bit LSB pie executable, From bbf11acc28f91a7dec7a174cdd092e182607aba9 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Fri, 22 May 2026 17:27:14 -0700 Subject: [PATCH 05/10] editorial cleanup, and validation of the types of traps through experimentation on x86 and aarch64 --- .../debugging-a-service-using-a-backtrace.md | 77 ++++++++++--------- .../deploying-static-binaries.md | 4 +- .../swift-backtrace-configuration.md | 39 +++++----- 3 files changed, 62 insertions(+), 58 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md index b6f663ca7..30ae1a2c1 100644 --- a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -5,9 +5,8 @@ Interpret a Swift service's crash trace to find the failing code on Linux. ## Overview When your Swift service crashes on Linux, the runtime spawns a helper process, -`swift-backtrace`, that prints a stack trace describing what was running and where it failed. -This article assumes you already have a trace from a crash. -It covers how to persist the trace from a container, read its structure, +`swift-backtrace`, that prints a stack trace that describes what was running and where it failed. +This article covers how to persist a trace from a container, read its structure, map frames back to your source, recognize common crash patterns, and reproduce the crash locally. For the configuration options the trace responds to, see . @@ -17,20 +16,18 @@ see . ### Persist backtraces from a container When a container exits, its filesystem is gone, -so a crash that writes to a file inside the container loses the trace -on the next restart. -For durable, addressable crash files, write to a mounted volume: +so a crash that writes to a file inside the container loses the trace. +For durable, addressable crash files, write the trace to a volume that you mount when you run the container. For example, if you mounted a volume at `/var/crash-logs`, use the configuration: ```Dockerfile ENV SWIFT_BACKTRACE=enable=yes,interactive=no,symbolicate=off,output-to=/var/crash-logs/ ``` -Mount a volume at `/var/crash-logs` when you run the container. -With `docker run`, use `-v`; with Kubernetes, mount a `PersistentVolume` -or an `emptyDir`-backed volume at that path. +With `docker run` or Apple container, use the `-v` option to mount a volume; +with Kubernetes, mount a `PersistentVolume` or an `emptyDir`-backed volume at that path. When `output-to` resolves to a directory, the backtracer writes each crash -to a uniquely named file inside it, so a burst of crashes doesn't overwrite +to a uniquely named file inside it, so a burst of crashes won't overwrite earlier traces. The `symbolicate=off` setting keeps crash handling fast in production; see for how to resolve symbols afterward. @@ -60,13 +57,20 @@ Thread 0 "myservice" crashed: The header line names the **signal** and the **fault address**. On Linux the common signals are `SIGSEGV` (bad memory access), -`SIGABRT` (abort, including Swift's runtime traps and `fatalError`), -`SIGBUS` (misaligned access), `SIGFPE` (arithmetic exception), and `SIGILL` (illegal instruction). +`SIGILL` and `SIGTRAP` (Swift runtime traps — see the table below), +`SIGABRT` (a deliberate abort, raised by `abort()` and by force-cast (`as!`) failures), +`SIGBUS` (misaligned access or access past the end of a memory-mapped file), +and `SIGFPE` (arithmetic exception, including integer division by zero). + +Swift runtime traps (force-unwrap of `nil`, index-out-of-range, `try!` on a throwing call, +arithmetic overflow, `precondition`, `fatalError`) lower to a CPU trap instruction. +The kernel reports this as `SIGILL` on x86_64 (where the trap lowers to `ud2`) +and as `SIGTRAP` on aarch64 (where it lowers to `brk #1`). The **thread line** identifies which thread crashed. -The backtracer captures only the crashed thread by default; -to see every thread, which helps when diagnosing deadlocks, -use `SWIFT_BACKTRACE=threads=all` — see . +The backtracer captures only the crashed thread by default. +Use `SWIFT_BACKTRACE=threads=all` to see every thread which helps when diagnosing deadlocks. +For more information, see . Each **frame line** has the form: @@ -86,11 +90,12 @@ You can tune or suppress both; see for the full and the trade-offs between `off`, `fast`, and `full`. **Post-mortem, on a developer or build machine.** -Keep the unstripped binary, or a separate DWARF debug-info package, from the same build that produced the trace. + +Keep the unstripped binary, or a separate DWARF debug-info package, that came from the same build as the trace. With that artifact in hand: ```bash @@ -109,19 +115,15 @@ llvm-symbolizer --obj=path/to/unstripped/myservice 0x000055a9c4001234 swift demangle "$1MyService13handleRequestyAA8ResponseVAA7RequestVF" ``` -Time isn't critical at this stage, so spending CPU on full demangling and inline-frame resolution is fine. -Tools like `addr2line` work as well; pick whichever your build environment already has. +**With a separate debug-info file.** -#### Resolve symbols from a separate debug-info file - -You can ship debug information as a sidecar file rather than embedding it in the binary +Ship debug information as a sidecar file rather than embedding it in the binary (see ). -Place the sidecar where symbolicators look for it. -Symbolicators look first at the path recorded by `objcopy --add-gnu-debuglink` -(resolved relative to the binary, then under `/usr/lib/debug//`), -and then at the build-ID index at `/usr/lib/debug/.build-id//.debug`, +Place the sidecar where symbolicators look for it: first at the path that +`objcopy --add-gnu-debuglink` records (resolved relative to the binary, +then under `/usr/lib/debug//`), +then at the build-ID index at `/usr/lib/debug/.build-id//.debug`, where the hex digits come from the binary's build ID. - The `-debuginfo` and `-dbgsym` packages populate the build-ID index when you install them, so a developer machine with the matching debug-info package installed resolves symbols transparently: @@ -137,9 +139,11 @@ llvm-symbolizer --obj=/usr/bin/MyServer 0x000055a9c4001234 ``` `llvm-symbolizer` reads the build ID from the stripped binary, finds the matching sidecar on the index path, and pulls source locations from its DWARF. +If the build IDs don't match, the symbolicator falls back to address-only output; +confirm the pairing with `readelf -n` against both files (see ). -If the build IDs don't match, the symbolicator falls back to address-only output. -Confirm the pairing with `readelf -n` against both files; see . +Time isn't critical at this stage, so spending CPU on full demangling and inline-frame resolution is fine. +Tools like `addr2line` from Linux binutils work as well; pick whichever your build environment already has. ### Recognize the crash pattern @@ -148,17 +152,18 @@ to identify the kind of bug. | Signal and message | Typical cause | |---|---| -| `SIGABRT` + `Fatal error: Index out of range` | Out-of-bounds collection access | -| `SIGABRT` + `Fatal error: Unexpectedly found nil while unwrapping an Optional value` | Force-unwrap (`!`) of a `nil` optional | -| `SIGABRT` + `Fatal error: 'try!' expression unexpectedly raised an error` | `try!` on a call that threw | +| **trap** + `Fatal error: Index out of range` | Out-of-bounds collection access | +| **trap** + `Fatal error: Unexpectedly found nil while unwrapping an Optional value` | Force-unwrap (`!`) of a `nil` optional | +| **trap** + `Fatal error: 'try!' expression unexpectedly raised an error` | `try!` on a call that threw | +| **trap** + `Fatal error: Arithmetic operation ... overflow` | Trapping integer arithmetic | +| **trap** + `Precondition failed: ` or `Fatal error: ` | `precondition()` or `fatalError()` call | | `SIGABRT` + `Could not cast value of type ...` | Force-cast (`as!`) failure | -| `SIGABRT` + `Fatal error: Arithmetic operation ... overflow` | Trapping integer arithmetic | -| `SIGABRT` + `precondition failed` or a custom message | `precondition()` or `fatalError()` call | +| `SIGABRT` | Direct `abort()` call from Swift or a C dependency | | `SIGSEGV` with frames in your unsafe-pointer or C-interop code | Memory-safety violation in that code | | `SIGSEGV` with frames only in libc or runtime | Likely heap corruption from earlier code | | `SIGSEGV` with a deeply uniform recursive frame pattern | Stack overflow | -For runtime-trap crashes (the `SIGABRT` rows in the previous table), the relevant fix is almost always +For runtime-trap crashes (the **trap** rows in the previous table), the relevant fix is almost always in the topmost frame of *your* code in the trace — the standard library trap is a few frames up from there. For `SIGSEGV` in unsafe code or dependencies, the topmost frame names the call that touched bad memory, diff --git a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md index fa2e63672..8bb43fe0a 100644 --- a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md +++ b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md @@ -120,10 +120,10 @@ container build -t :latest . > If you need a shell for troubleshooting, > use a slim base image during development and switch to `scratch` for production. -### Include the backtracer for crash diagnostics +### Include swift-backtrace for crash diagnostics A static binary in a `scratch` container has no toolchain layout for the runtime to fall back on. -As a result, the runtime's default path to `swift-backtrace` doesn't exist in the image. +As a result, the runtime's default path to find `swift-backtrace` doesn't exist in the image. To get backtraces from crashes, copy the static helper from a Swift container image into your final image and explicitly pass its path to the runtime: diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md index 7478cbf30..b0ef6ca07 100644 --- a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md +++ b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md @@ -12,24 +12,23 @@ The helper reads the crashed process's memory, walks each thread's stack, symbolicates frames, and writes the result before the original process exits. This two-process design keeps the signal handler small and signal-safe, -and lets the backtracer use ordinary Swift code to format output and look up symbols. +and lets the backtracer use unconstrained Swift code to look up symbols and format output. -The runtime only installs its signal handler for a given signal if no handler -is already in place, and only sets up an alternate signal stack if one isn't +The runtime only installs its signal handler for a given signal if there isn't one +already in place, and only sets up an alternate signal stack if one isn't already configured. -A library that installs its own handler for one of the catchable signals — +Any library that installs its own handler for one of the catchable signals — even `SIG_IGN` — silently disables backtracing for that signal. -You configure the backtracer through the `SWIFT_BACKTRACE` environment variable. -It accepts a comma-separated list of `key=value` pairs: +You configure the backtracer through the `SWIFT_BACKTRACE` environment variable +with a comma-separated list of `key=value` pairs: ```bash SWIFT_BACKTRACE=enable=yes,interactive=no,format=json,output-to=/var/log/crashes/ ``` The defaults target interactive use at a terminal. -For server deployments, the most common adjustments are -`interactive=no` for containers without a TTY, +For server deployments, common adjustments are `interactive=no` for containers without a TTY, and either `format=json` or `output-to=` so traces are written to a location your log pipeline can ingest. ### Enabling and presentation @@ -49,7 +48,7 @@ When `output-to` points at a file, the `tty` resolutions above are overridden: | Option | Values | Default | Notes | |---|---|---|---| -| `threads` | `crashed`, `all`, `preset` | `preset` | `crashed` shows only the failing thread; `all` is useful for deadlocks. | +| `threads` | `crashed`, `all`, `preset` | `preset` | `crashed` shows only the failing thread; `all` is useful to debug deadlocks. | | `registers` | `none`, `crashed`, `all`, `preset` | `preset` | Whether to dump CPU registers alongside frames. | | `images` | `none`, `mentioned`, `all`, `preset` | `preset` | Loaded shared-library list; `mentioned` includes only those referenced by frames. | | `limit` | integer, `none` | `64` | Maximum frames per thread, to prevent runaway output on infinite recursion. | @@ -61,14 +60,14 @@ When the captured stack has more than `limit` frames, the backtracer keeps and `limit - 1 - top` frames from the bottom (the entry into the program), joined by a `...` marker. For example, a 10-frame stack with `limit=5,top=2` shows frames 10, 9, then -`...`, then 2, 1 — letting you see both where execution started and where +`...`, then 2, and 1. This lets you see both where execution started and where the fault occurred when a runaway recursion exceeds the limit. ### Unwinding and symbolication | Option | Values | Default | Notes | |---|---|---|---| -| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` consults DWARF unwind info. `auto` picks per-frame. | +| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` consults DWARF unwind information. `auto` picks per-frame. | | `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF; `fast` resolves only the outermost symbol; `off` reports raw addresses. | | `cache` | `yes`, `no` | `yes` | Caches symbol lookups across frames. | @@ -84,7 +83,7 @@ To resolve addresses from a captured trace using either form, see -for the full schema. +along with registers when configured. +See for the full schema. -Pipe the JSON output to your log shipper or write it to a mounted volume: +Pipe the JSON output to a process that ships off your logs or write it to a mounted volume: ```bash SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ @@ -124,7 +123,7 @@ SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ | Option | Values | Default | Notes | |---|---|---|---| -| `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtracer to finish. | +| `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtrace to finish. | | `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. | | `warnings` | `enabled`, `suppressed` | `enabled` | Diagnostic messages from the backtracer itself. | | `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace, useful in CI environments where leaked file descriptors cause resource contention. | @@ -184,7 +183,7 @@ The following fields appear conditionally, depending on backtracer settings: | Field | Value | |---|---| -| `omittedThreads` | Count of threads omitted when `threads=crashed`. Omitted if zero. | +| `omittedThreads` | Count of threads omitted when `threads=crashed`; omitted if zero. | | `capturedMemory` | Dictionary of captured memory contents, keyed by hex address strings. Absent when `sanitize` is enabled or no data was captured. | | `omittedImages` | When `images=mentioned`, count of images whose details were omitted. | | `images` | Array of image records (unless `images=none`). | @@ -238,7 +237,7 @@ Symbolicated frames can add `inlined`, `runtimeFailure`, `thunk`, or The following report comes from a small service whose `MyService.run()` calls into a synchronous closure that invokes `MyService.handleRequest`, which dereferences a NULL-adjacent pointer. -The binary was built with debug information and run on Linux arm64 with +The binary for this example was built with debug information and run on Linux arm64 with default backtracer settings (`format=json`, preset `auto`, `demangle=yes`, `symbolicate=full`). To keep this example concise, the `registers` and `capturedMemory` objects show only a @@ -384,7 +383,7 @@ general-purpose register and many more captured memory snapshots. } ``` -A few things to notice in this trace: +A few things to note in this trace: - The async boundary in this program sits between `Main.main()` and `MyService.run()` — that's where the `asyncResumePoint` frames begin. @@ -392,7 +391,7 @@ A few things to notice in this trace: of the crash it was running on a regular stack inside the synchronous closure it dispatched. - The frames marked `system: true` (`Main.$main()` and - `completeTaskWithClosure`) come from compiler-generated entry-point glue + `completeTaskWithClosure`) come from compiler-generated entry-point and the Swift concurrency runtime. They don't usually have a meaningful source location, so the backtracer reports ``. - The frame marked `thunk: true` is a compiler-generated bridge that adapts From fcfa2ef518faf85dc7f873590bece2373061b703 Mon Sep 17 00:00:00 2001 From: Joseph Heck Date: Wed, 27 May 2026 08:30:09 -0700 Subject: [PATCH 06/10] Update server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md Co-authored-by: Alastair Houghton --- .../Sources/ServerGuides.docc/swift-backtrace-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md index b0ef6ca07..ab68768c1 100644 --- a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md +++ b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md @@ -67,7 +67,7 @@ the fault occurred when a runaway recursion exceeds the limit. | Option | Values | Default | Notes | |---|---|---|---| -| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` consults DWARF unwind information. `auto` picks per-frame. | +| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` may consult debug information, if present. `auto` picks a default based on the platform. | | `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF; `fast` resolves only the outermost symbol; `off` reports raw addresses. | | `cache` | `yes`, `no` | `yes` | Caches symbol lookups across frames. | From ec3999d485422acc18405250a463a91be8590f1e Mon Sep 17 00:00:00 2001 From: Joseph Heck Date: Wed, 27 May 2026 08:30:40 -0700 Subject: [PATCH 07/10] Update server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md Co-authored-by: Alastair Houghton --- .../ServerGuides.docc/debugging-a-service-using-a-backtrace.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md index 30ae1a2c1..6ef8cc313 100644 --- a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -208,5 +208,4 @@ the cause is almost always one of the following: The fast unwinder follows frame pointers, so a binary built with `-fomit-frame-pointer` (or a C/C++ dependency built that way) produces traces that stop at the first frame without one. Current Swift toolchains preserve frame pointers in release builds on Linux server architectures. - Force the precise unwinder with `SWIFT_BACKTRACE=unwind=precise` to use DWARF instead, - or rebuild affected dependencies with `-fno-omit-frame-pointer`. + Rebuild affected dependencies with `-fno-omit-frame-pointer`. From c1db945aeb018146443bdfa8c30dfe28f64edae4 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Wed, 27 May 2026 11:04:16 -0700 Subject: [PATCH 08/10] applying feedback details from review --- .../Sources/ServerGuides.docc/building.md | 33 +++++-------------- .../debugging-a-service-using-a-backtrace.md | 5 ++- .../swift-backtrace-configuration.md | 13 ++++---- 3 files changed, 20 insertions(+), 31 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/building.md b/server-guides/Sources/ServerGuides.docc/building.md index 8d07202c1..6ba614f1d 100644 --- a/server-guides/Sources/ServerGuides.docc/building.md +++ b/server-guides/Sources/ServerGuides.docc/building.md @@ -103,30 +103,6 @@ Release builds optimize for performance and don't embed DWARF debug information Without DWARF, a crashing binary's stack trace reports raw addresses instead of function names, files, and line numbers. You have two ways to keep symbolication working: embed the debug information in the binary, or split it into a sidecar file you ship separately. -### Embed DWARF in a release build - -Pass `-g` to the Swift compiler when building for release: - -```bash -swift build -c release -Xswiftc -g -``` - -If your package also compiles C or C++ sources, forward `-g` to the C compiler as well: - -```bash -swift build -c release -Xswiftc -g -Xcc -g -``` - -Confirm the resulting binary contains DWARF sections: - -```bash -file .build/release/MyServer -# .build/release/MyServer: ELF 64-bit LSB pie executable, ..., -# with debug_info, not stripped -``` - -This configuration produces larger binaries — often two to five times the size of a stripped release binary — but they need no companion file at symbolication time. - ### Split debug information into a sidecar file For a smaller deployable artifact, separate the debug information into its own file and strip the original binary. @@ -145,6 +121,15 @@ objcopy --strip-debug --strip-unneeded MyServer objcopy --add-gnu-debuglink=MyServer.debug MyServer ``` +The Swift source repository has scripts to make debug sidecar files: + +- [make-debuglink](https://github.com/swiftlang/swift/blob/main/test/Backtracing/Inputs/make-debuglink) +- [make-minidebug](https://github.com/swiftlang/swift/blob/main/test/Backtracing/Inputs/make-minidebug) + +That you can use to create debug sidecars. +The Swift backtracer supports gzip, zstd, and lzma compressed data, +provided it can locate the relevant dynamic libraries at runtime. + You now have two artifacts: a small, stripped `MyServer` to ship in your container or distribution package, and a `MyServer.debug` to publish to a symbol server or ship in a companion debug-info package. For how symbolicators consume the sidecar at crash time, see . diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md index 6ef8cc313..0d02881cd 100644 --- a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -78,10 +78,13 @@ Each **frame line** has the form:
+ in at : ``` +> Note: In the plaintext format, Swift backtracing may also include the source code in the output if it's accessible when we're generating the crash report. + The backtracer demangles the symbol by default. The `at :` portion appears only when the binary carries DWARF debug information and you run with `symbolicate=full` (the default). -Stripped production binaries with `symbolicate=off` produce frames with the address and image name only. +Stripped production binaries may produce frames with the address and image name only. +Swift backtracing tryies to find the closest visible symbol. By default the trace includes registers and a list of loaded images for the crashed thread. You can tune or suppress both; see . diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md index ab68768c1..9ce81c4cd 100644 --- a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md +++ b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md @@ -68,11 +68,11 @@ the fault occurred when a runaway recursion exceeds the limit. | Option | Values | Default | Notes | |---|---|---|---| | `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` may consult debug information, if present. `auto` picks a default based on the platform. | -| `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF; `fast` resolves only the outermost symbol; `off` reports raw addresses. | -| `cache` | `yes`, `no` | `yes` | Caches symbol lookups across frames. | +| `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF and source locations using DWARF debug information, if it exists; `fast` only tries to resolve the source location; `off` reports raw addresses. | +| `cache` | `yes`, `no` | `yes` | Enable cache mechanisms in the symbol fetching. `cache` as no effect on Linux. | Symbolication quality depends on what's in your binary. -Stripped binaries report addresses without symbol names. +Stripped binaries typically report addresses without symbol names, and backtracing may display the name of a visible symbol preceeding the address we're looking up along with an offset. To embed DWARF directly in a release binary, or to split debug info into a sidecar file the symbolicator can find by build ID, see . To resolve addresses from a captured trace using either form, see . @@ -124,9 +124,9 @@ SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ | Option | Values | Default | Notes | |---|---|---|---| | `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtrace to finish. | -| `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. | +| `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. This option should be disabled in distro-supplied runtimes, since in that case the relevant helperis located at a fixed path. | | `warnings` | `enabled`, `suppressed` | `enabled` | Diagnostic messages from the backtracer itself. | -| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace, useful in CI environments where leaked file descriptors cause resource contention. | +| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace. Used to make sure that in container environments, centralised routing machinery detects the failure of the container early, before Swift-backtrace starts trying to generate a backtrace, so that the running service doesn't have additional requests routed to it that external systems would need to retry. | The runtime locates `swift-backtrace` by deriving a Swift root directory and searching a fixed set of subdirectories underneath it. @@ -159,6 +159,7 @@ at it explicitly, or place the binary at one of the search locations. ### JSON crash log schema When `format=json`, the backtracer emits one JSON object per crash. +The JSON object is presented as a single line — no line breaks in the JSON output — so that it's easier for log processing tooling to read the output. Addresses appear as hexadecimal strings (with `0x` prefix); raw byte data such as captured memory or build IDs appears as un-prefixed hexadecimal strings with no inter-byte whitespace. @@ -384,7 +385,7 @@ general-purpose register and many more captured memory snapshots. ``` A few things to note in this trace: - +- This example has been reformatted (pretty printed) to make it easier to read. Swift-backtrace outputs the JSON as a single line. - The async boundary in this program sits between `Main.main()` and `MyService.run()` — that's where the `asyncResumePoint` frames begin. `MyService.run()` itself appears as a `returnAddress` because at the moment From 771769b82e4377f8add81070f9e129677b83ab47 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 28 May 2026 08:45:48 -0700 Subject: [PATCH 09/10] switching to reference rst file in GitHub for reference content --- .../ServerGuides.docc/Documentation.md | 4 +- .../debugging-a-service-using-a-backtrace.md | 14 +- .../deploying-static-binaries.md | 2 +- .../Sources/ServerGuides.docc/packaging.md | 2 +- .../swift-backtrace-configuration.md | 409 ------------------ 5 files changed, 11 insertions(+), 420 deletions(-) delete mode 100644 server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md diff --git a/server-guides/Sources/ServerGuides.docc/Documentation.md b/server-guides/Sources/ServerGuides.docc/Documentation.md index 877cf91f0..c20108e42 100644 --- a/server-guides/Sources/ServerGuides.docc/Documentation.md +++ b/server-guides/Sources/ServerGuides.docc/Documentation.md @@ -13,6 +13,6 @@ - - -### Reference +## See Also -- +- [Backtracing reference (swiftlang/swift)](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst) diff --git a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md index 0d02881cd..62189c398 100644 --- a/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md +++ b/server-guides/Sources/ServerGuides.docc/debugging-a-service-using-a-backtrace.md @@ -9,7 +9,7 @@ When your Swift service crashes on Linux, the runtime spawns a helper process, This article covers how to persist a trace from a container, read its structure, map frames back to your source, recognize common crash patterns, and reproduce the crash locally. -For the configuration options the trace responds to, see . +For the configuration options the trace responds to, see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). To install the backtracer in a container image, see . @@ -33,7 +33,7 @@ The `symbolicate=off` setting keeps crash handling fast in production; see for how to resolve symbols afterward. For the format of the captured file (text or JSON) and other output options, -see . +see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). ### Read the trace structure @@ -70,7 +70,7 @@ and as `SIGTRAP` on aarch64 (where it lowers to `brk #1`). The **thread line** identifies which thread crashed. The backtracer captures only the crashed thread by default. Use `SWIFT_BACKTRACE=threads=all` to see every thread which helps when diagnosing deadlocks. -For more information, see . +For more information, see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). Each **frame line** has the form: @@ -87,7 +87,7 @@ Stripped production binaries may produce frames with the address and image name Swift backtracing tryies to find the closest visible symbol. By default the trace includes registers and a list of loaded images for the crashed thread. -You can tune or suppress both; see . +You can tune or suppress both; see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). ### Map a frame to source @@ -102,7 +102,7 @@ and resolve symbols offline. Build production binaries with debug info stripped and run with `SWIFT_BACKTRACE=symbolicate=off`. Each captured frame has its address and the image (binary or shared library) it belongs to — enough to resolve to source later. -See for the full set of `symbolicate` values +See the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing) for the full set of `symbolicate` values and the trade-offs between `off`, `fast`, and `full`. **Post-mortem, on a developer or build machine.** @@ -184,7 +184,7 @@ Two paths are useful: - **Rerun with the interactive backtracer.** Set `SWIFT_BACKTRACE=interactive=yes` so the backtracer drops into a debugger-like prompt at crash time instead of printing and exiting. - See for the option. + See the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing) for the option. Interactive mode requires a TTY, so it's a developer-machine tool, not a production setting. ### Diagnose missing backtraces @@ -201,7 +201,7 @@ the cause is almost always one of the following: directory; it doesn't consult `PATH`. In a statically linked binary that doesn't follow the toolchain layout, set `SWIFT_BACKTRACE=swift-backtrace=` or `SWIFT_ROOT=`. - See for the search order. + See the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing) for the search order. - term `/proc` isn't mounted: The backtracer enumerates threads and locates loaded images through `/proc//`. diff --git a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md index 8bb43fe0a..52c2a3a97 100644 --- a/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md +++ b/server-guides/Sources/ServerGuides.docc/deploying-static-binaries.md @@ -144,7 +144,7 @@ The `swift-backtrace=/swift-backtrace` setting overrides the runtime's default s and points it to the correct location. The `symbolicate=off` setting keeps crash handling fast; you resolve addresses against the original binary later. -For the full set of `SWIFT_BACKTRACE` options, see . +For the full set of `SWIFT_BACKTRACE` options, see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). To debug your service using a captured trace, see . ### Build and publish with Swift Container Plugin diff --git a/server-guides/Sources/ServerGuides.docc/packaging.md b/server-guides/Sources/ServerGuides.docc/packaging.md index 74f0dff77..2bfd67799 100644 --- a/server-guides/Sources/ServerGuides.docc/packaging.md +++ b/server-guides/Sources/ServerGuides.docc/packaging.md @@ -159,7 +159,7 @@ so typical builds don't need extra flags. The fast unwinder relies on frame pointers; without them, the runtime falls back to DWARF-based unwinding, which is slower but still produces complete traces when `.eh_frame` information is present. -For the full set of `SWIFT_BACKTRACE` options, see . +For the full set of `SWIFT_BACKTRACE` options, see the [Swift backtracing reference](https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing). To debug your service using a captured trace, see . ### Cache build artifacts to speed up rebuilds diff --git a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md b/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md deleted file mode 100644 index 9ce81c4cd..000000000 --- a/server-guides/Sources/ServerGuides.docc/swift-backtrace-configuration.md +++ /dev/null @@ -1,409 +0,0 @@ -# Swift backtrace configuration - -Control how the Swift runtime captures and reports stack traces when your service crashes. - -## Overview - -When a Swift program receives a fatal signal on Linux — -`SIGSEGV`, `SIGABRT`, `SIGBUS`, `SIGFPE`, `SIGILL`, `SIGTRAP`, or `SIGQUIT` — -the runtime's signal handler suspends the program's threads and launches a separate -helper process, `swift-backtrace`, to gather and format a crash report. -The helper reads the crashed process's memory, walks each thread's stack, -symbolicates frames, and writes the result before the original process exits. - -This two-process design keeps the signal handler small and signal-safe, -and lets the backtracer use unconstrained Swift code to look up symbols and format output. - -The runtime only installs its signal handler for a given signal if there isn't one -already in place, and only sets up an alternate signal stack if one isn't -already configured. -Any library that installs its own handler for one of the catchable signals — -even `SIG_IGN` — silently disables backtracing for that signal. - -You configure the backtracer through the `SWIFT_BACKTRACE` environment variable -with a comma-separated list of `key=value` pairs: - -```bash -SWIFT_BACKTRACE=enable=yes,interactive=no,format=json,output-to=/var/log/crashes/ -``` - -The defaults target interactive use at a terminal. -For server deployments, common adjustments are `interactive=no` for containers without a TTY, -and either `format=json` or `output-to=` so traces are written to a location your log pipeline can ingest. - -### Enabling and presentation - -| Option | Values | Default | Notes | -|---|---|---|---| -| `enable` | `yes`, `no`, `tty` | `yes` | `tty` enables backtracing when stdout is a terminal. | -| `interactive` | `yes`, `no`, `tty` | `tty` | Drops into a debugger-like prompt; `tty` enables it when both stdout and stdin are terminals. Disable for non-TTY containers. | -| `color` | `yes`, `no`, `tty` | `tty` | ANSI color in output; `tty` enables it when the resolved output stream is a terminal (stderr or stdout, depending on `output-to`). | -| `demangle` | `yes`, `no` | `yes` | Demangle Swift symbols in frames. | -| `preset` | `friendly`, `medium`, `full`, `auto` | `auto` | Bundles `threads`, `registers`, and `images` defaults. `friendly` is concise; `full` shows everything. | - -When `output-to` points at a file, the `tty` resolutions above are overridden: -`enable=tty` becomes `yes`, while `interactive=tty` and `color=tty` become `no`. - -### Captured content - -| Option | Values | Default | Notes | -|---|---|---|---| -| `threads` | `crashed`, `all`, `preset` | `preset` | `crashed` shows only the failing thread; `all` is useful to debug deadlocks. | -| `registers` | `none`, `crashed`, `all`, `preset` | `preset` | Whether to dump CPU registers alongside frames. | -| `images` | `none`, `mentioned`, `all`, `preset` | `preset` | Loaded shared-library list; `mentioned` includes only those referenced by frames. | -| `limit` | integer, `none` | `64` | Maximum frames per thread, to prevent runaway output on infinite recursion. | -| `top` | integer | `16` | When `limit` truncates, always keep this many frames from the top of stack. | -| `sanitize` | `yes`, `no`, `preset` | `preset` | Strips PII from paths in captured frames. Has no effect on Linux; the runtime parses the option but doesn't transform paths outside macOS. | - -When the captured stack has more than `limit` frames, the backtracer keeps -`top` frames from the top of the stack (the deepest, most recent calls) -and `limit - 1 - top` frames from the bottom (the entry into the program), -joined by a `...` marker. -For example, a 10-frame stack with `limit=5,top=2` shows frames 10, 9, then -`...`, then 2, and 1. This lets you see both where execution started and where -the fault occurred when a runaway recursion exceeds the limit. - -### Unwinding and symbolication - -| Option | Values | Default | Notes | -|---|---|---|---| -| `unwind` | `auto`, `fast`, `precise` | `auto` | `fast` uses frame pointers only; `precise` may consult debug information, if present. `auto` picks a default based on the platform. | -| `symbolicate` | `full`, `fast`, `off` | `full` | `full` resolves inlined frames using DWARF and source locations using DWARF debug information, if it exists; `fast` only tries to resolve the source location; `off` reports raw addresses. | -| `cache` | `yes`, `no` | `yes` | Enable cache mechanisms in the symbol fetching. `cache` as no effect on Linux. | - -Symbolication quality depends on what's in your binary. -Stripped binaries typically report addresses without symbol names, and backtracing may display the name of a visible symbol preceeding the address we're looking up along with an offset. -To embed DWARF directly in a release binary, or to split debug info into a sidecar file the symbolicator can find by build ID, see . -To resolve addresses from a captured trace using either form, see . - -### Output - -| Option | Values | Default | Notes | -|---|---|---|---| -| `format` | `text`, `json` | `text` | JSON is structured for log aggregators. | -| `output-to` | `stderr`, `stdout`, file path, directory path | `stderr` | If the path resolves to an existing directory, the backtracer writes each crash to a uniquely named file inside it. Otherwise the backtracer treats the path as a filename. | - -An abbreviated text-format trace looks something like this (the registers and image list omitted): - -``` -*** Signal 11: Backtracing from 0xaaaaaaaa1804... done *** - -*** Program crashed: Bad pointer dereference at 0x0000000000000004 *** - -Thread 0 crashed: - -0 0x0000aaaaaaaa1804 MyService.handleRequest(_:) + 180 in myservice at /work/Sources/myservice/Service.swift:7:21 -1 [ra] 0x0000aaaaaaaa11a8 closure #1 in MyService.run() + 15 in myservice at /work/Sources/myservice/Service.swift:13:17 -2 [ra] 0x0000aaaaaaaa117c MyService.run() + 27 in myservice at /work/Sources/myservice/Service.swift:15:9 -3 [async] 0x0000aaaaaaaa1218 static Main.main() in myservice at /work/Sources/myservice/Service.swift:22 -4 [async] [system] 0x0000aaaaaaaa1344 static Main.$main() in myservice at // -... -``` - -Bracketed labels denote frame attributes: `[ra]` for return-address frames, -`[async]` for async resumption points, `[system]` for compiler-generated or -runtime frames, and `[thunk]` for compiler-generated bridges. The same -attributes appear in JSON output as the frame's `kind` field and the -`system` and `thunk` Boolean markers. -For the same crash represented as JSON, see . - -JSON output produces one object per crash, with a top-level -`description` describing the failure, a `faultAddress`, `platform`, -`architecture`, and a `threads` array containing per-thread frame lists, -along with registers when configured. -See for the full schema. - -Pipe the JSON output to a process that ships off your logs or write it to a mounted volume: - -```bash -SWIFT_BACKTRACE=format=json,output-to=/var/crash-logs/ -``` - -### Advanced options - -| Option | Values | Default | Notes | -|---|---|---|---| -| `timeout` | duration (`30s`), `none` | `30s` | How long the runtime waits for the backtrace to finish. | -| `swift-backtrace` | path | auto-detected | Override the implicit search and use the given absolute path directly. Required for statically linked binaries in minimal containers, where the implicit search can't reliably find the helper. This option should be disabled in distro-supplied runtimes, since in that case the relevant helperis located at a fixed path. | -| `warnings` | `enabled`, `suppressed` | `enabled` | Diagnostic messages from the backtracer itself. | -| `close-fds` | `yes`, `no` | `no` | Close all open file descriptors in the crashing process before gathering the trace. Used to make sure that in container environments, centralised routing machinery detects the failure of the container early, before Swift-backtrace starts trying to generate a backtrace, so that the running service doesn't have additional requests routed to it that external systems would need to retry. | - -The runtime locates `swift-backtrace` by deriving a Swift root directory -and searching a fixed set of subdirectories underneath it. -The root is whichever of these the runtime finds first: - -1. The value of the `SWIFT_ROOT` environment variable, if set. -2. A path computed from the location of `libswiftCore` (when the runtime is - dynamically linked). -3. A path computed from the location of the running executable (when the - runtime is statically linked into the program). - -Within the root, the runtime checks these locations in order, where -`` is the Swift platform name (for example, `linux`) and `` is the -CPU architecture (for example, `x86_64` or `aarch64`): - -``` -/libexec/swift/ -/libexec/swift// -/libexec/swift -/libexec/swift/ -/bin -/bin/ - -``` - -The runtime doesn't consult `PATH`. If none of the above contain the helper, -the runtime can't capture crashes — set `SWIFT_BACKTRACE=swift-backtrace=` to point -at it explicitly, or place the binary at one of the search locations. - -### JSON crash log schema - -When `format=json`, the backtracer emits one JSON object per crash. -The JSON object is presented as a single line — no line breaks in the JSON output — so that it's easier for log processing tooling to read the output. -Addresses appear as hexadecimal strings (with `0x` prefix); raw byte data -such as captured memory or build IDs appears as un-prefixed hexadecimal -strings with no inter-byte whitespace. -The backtracer omits Boolean fields when `false`, and omits unknown or empty -values entirely to save space. - -#### Top-level fields - -The following fields are always present: - -| Field | Value | -|---|---| -| `timestamp` | ISO-8601 timestamp string. | -| `kind` | The string `crashReport`. | -| `description` | Textual description of the crash or runtime failure. | -| `faultAddress` | Fault address associated with the crash. | -| `platform` | Platform string; first token names the platform, followed by version info — for example, `"linux (Ubuntu 22.04.5 LTS)"`. | -| `architecture` | Processor architecture name. | -| `threads` | Array of thread records. | - -The following fields appear conditionally, depending on backtracer settings: - -| Field | Value | -|---|---| -| `omittedThreads` | Count of threads omitted when `threads=crashed`; omitted if zero. | -| `capturedMemory` | Dictionary of captured memory contents, keyed by hex address strings. Absent when `sanitize` is enabled or no data was captured. | -| `omittedImages` | When `images=mentioned`, count of images whose details were omitted. | -| `images` | Array of image records (unless `images=none`). | -| `backtraceTime` | Time to generate the report, in seconds. | - -#### Thread records - -| Field | Value | -|---|---| -| `name` | Thread name; omitted if not set. | -| `crashed` | `true` for the crashing thread; omitted otherwise. | -| `registers` | Dictionary of register name → hex value string. The backtracer omits this on non-crashed threads when `registers=crashed`. | -| `frames` | Array of frame records. | - -#### Frame records - -Each frame has a `kind`: - -| Kind | Meaning | -|---|---| -| `programCounter` | Frame address is a directly captured program counter. | -| `returnAddress` | Frame address is a return address. | -| `asyncResumePoint` | Frame address is a resumption point in an `async` function. | -| `omittedFrames` | Frame-omission record (carries a `count` field). | -| `truncated` | The backtrace is truncated at this point. | - -Address-bearing frames also include `address` (hex string). -Symbolicated frames can add `inlined`, `runtimeFailure`, `thunk`, or -`system` Boolean markers, plus the following fields when symbol lookup succeeds: - -| Field | Value | -|---|---| -| `symbol` | Mangled symbol name. | -| `offset` | Offset from the symbol to the frame address. | -| `description` | Demangled, human-readable description (when `demangle=yes`). | -| `image` | Name of the image containing the symbol. | -| `sourceLocation` | `{ file, line, column }` dictionary, when `symbolicate=full` and DWARF info is available. | - -#### Image records - -| Field | Value | -|---|---| -| `name` | Image name. | -| `buildId` | Build ID as un-prefixed hex string. | -| `path` | Path to the image. | -| `baseAddress` | Base address of the image text (hex string). | -| `endOfText` | End of the image text (hex string). | - -#### Example payload - -The following report comes from a small service whose `MyService.run()` calls -into a synchronous closure that invokes `MyService.handleRequest`, which -dereferences a NULL-adjacent pointer. -The binary for this example was built with debug information and run on Linux arm64 with -default backtracer settings (`format=json`, preset `auto`, `demangle=yes`, -`symbolicate=full`). -To keep this example concise, the `registers` and `capturedMemory` objects show only a -representative subset of their entries; a real trace contains every -general-purpose register and many more captured memory snapshots. - -```json -{ - "timestamp": "2026-05-21T23:51:47.905791Z", - "kind": "crashReport", - "description": "Bad pointer dereference", - "faultAddress": "0x0000000000000004", - "platform": "Linux (Ubuntu 24.04.4 LTS)", - "architecture": "arm64", - "threads": [ - { - "crashed": true, - "registers": { - "x0": "0x0000fffff702f830", - "x9": "0x0000000000000004", - "fp": "0x0000fffff702e5f0", - "lr": "0x0000aaaaaaaa11a8", - "sp": "0x0000fffff702e5a0", - "pc": "0x0000aaaaaaaa1804" - }, - "frames": [ - { - "kind": "programCounter", - "address": "0x0000aaaaaaaa1804", - "symbol": "$s9myservice9MyServiceV13handleRequestyAA8ResponseVAA0E0VF", - "offset": 180, - "description": "MyService.handleRequest(_:) + 180", - "image": "myservice", - "sourceLocation": { - "file": "/work/Sources/myservice/Service.swift", - "line": 7, - "column": 21 - } - }, - { - "kind": "returnAddress", - "address": "0x0000aaaaaaaa11a8", - "symbol": "$s9myservice9MyServiceV3runyyYaKFyycfU_", - "offset": 15, - "description": "closure #1 in MyService.run() + 15", - "image": "myservice", - "sourceLocation": { - "file": "/work/Sources/myservice/Service.swift", - "line": 13, - "column": 17 - } - }, - { - "kind": "returnAddress", - "address": "0x0000aaaaaaaa117c", - "symbol": "$s9myservice9MyServiceV3runyyYaKFTY0_", - "offset": 27, - "description": "MyService.run() + 27", - "image": "myservice", - "sourceLocation": { - "file": "/work/Sources/myservice/Service.swift", - "line": 15, - "column": 9 - } - }, - { - "kind": "asyncResumePoint", - "address": "0x0000aaaaaaaa1218", - "symbol": "$s9myservice4MainV4mainyyYaKFZTQ1_", - "offset": 0, - "description": "static Main.main()", - "image": "myservice", - "sourceLocation": { - "file": "/work/Sources/myservice/Service.swift", - "line": 22, - "column": 0 - } - }, - { - "kind": "asyncResumePoint", - "address": "0x0000aaaaaaaa1344", - "system": true, - "symbol": "$s9myservice4MainV5$mainyyYaKFZTQ0_", - "offset": 0, - "description": "static Main.$main()", - "image": "myservice", - "sourceLocation": { - "file": "//", - "line": 0, - "column": 0 - } - }, - { - "kind": "asyncResumePoint", - "address": "0x0000aaaaaaaa15d8", - "thunk": true, - "symbol": "$sIetH_yts5Error_pIegHrzo_TRTQ0_", - "offset": 0, - "description": "thunk for @escaping @convention(thin) @async () -> ()", - "image": "myservice", - "sourceLocation": { - "file": "//", - "line": 0, - "column": 0 - } - }, - { - "kind": "asyncResumePoint", - "address": "0x0000fffff78a4a18", - "system": true, - "symbol": "_ZL23completeTaskWithClosurePN5swift12AsyncContextEPNS_10SwiftErrorE", - "offset": 0, - "description": "completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*)", - "image": "libswift_Concurrency.so" - } - ] - } - ], - "capturedMemory": { - "0x0000aaaaaaaa1804": "280100f9d1ffff97fd7b45a9ff830191", - "0x0000aaaaaaaa11a8": "fd7bc1a8c0035fd6ff8300d1fd7b01a9", - "0x0000fffff702e5f0": "00e602f7ffff0000a811aaaaaaaa0000", - "0x0000fffff702e5a0": "f81410f7ffff00005fe602f7ffff0000" - }, - "omittedImages": 12, - "images": [ - { - "name": "myservice", - "buildId": "06b835b6531c5086c472cc7c15b89c97ea973e71", - "path": "/work/.build/aarch64-unknown-linux-gnu/debug/myservice", - "baseAddress": "0x0000aaaaaaaa0000", - "endOfText": "0x0000aaaaaaaa70e8" - }, - { - "name": "libswift_Concurrency.so", - "buildId": "76c30ca7c36aab49fe44cfdf7917c95cb33dfa01", - "path": "/usr/lib/swift/linux/libswift_Concurrency.so", - "baseAddress": "0x0000fffff7840000", - "endOfText": "0x0000fffff78bdc48" - } - ], - "backtraceTime": 0.0019300830000000002 -} -``` - -A few things to note in this trace: -- This example has been reformatted (pretty printed) to make it easier to read. Swift-backtrace outputs the JSON as a single line. -- The async boundary in this program sits between `Main.main()` and - `MyService.run()` — that's where the `asyncResumePoint` frames begin. - `MyService.run()` itself appears as a `returnAddress` because at the moment - of the crash it was running on a regular stack inside the synchronous - closure it dispatched. -- The frames marked `system: true` (`Main.$main()` and - `completeTaskWithClosure`) come from compiler-generated entry-point - and the Swift concurrency runtime. They don't usually have a meaningful - source location, so the backtracer reports ``. -- The frame marked `thunk: true` is a compiler-generated bridge that adapts - one async calling convention to another. -- `omittedImages: 12` means 12 loaded shared libraries aren't included - because no captured frame references them — the default `images=mentioned` - keeps the report compact. Set `images=all` to include every loaded image. - -### See also - -The packaging guide shows the recommended copy location for container images; -see . -To diagnose backtraces that don't appear or that are missing information, -see . From 9301c500e23580a69dcf4c17b316249e30e0d701 Mon Sep 17 00:00:00 2001 From: Joe Heck Date: Thu, 28 May 2026 09:28:56 -0700 Subject: [PATCH 10/10] stripped out the specific script references, but left it as a 'for more examples, see ...' reference in the section --- .../Sources/ServerGuides.docc/building.md | 23 +++++++------------ 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/server-guides/Sources/ServerGuides.docc/building.md b/server-guides/Sources/ServerGuides.docc/building.md index 6ba614f1d..b30dbb653 100644 --- a/server-guides/Sources/ServerGuides.docc/building.md +++ b/server-guides/Sources/ServerGuides.docc/building.md @@ -95,7 +95,7 @@ These executables bundle the Swift runtime directly: For deploying to VMs or bare metal where you don't control the system configuration, static linking removes the dependency on a pre-installed Swift runtime. -> Note: This technique doesn't apply on Apple platforms. +> Note: This technique doesn't apply to Apple platforms. ## Preserve debug information for symbolication @@ -121,18 +121,11 @@ objcopy --strip-debug --strip-unneeded MyServer objcopy --add-gnu-debuglink=MyServer.debug MyServer ``` -The Swift source repository has scripts to make debug sidecar files: - -- [make-debuglink](https://github.com/swiftlang/swift/blob/main/test/Backtracing/Inputs/make-debuglink) -- [make-minidebug](https://github.com/swiftlang/swift/blob/main/test/Backtracing/Inputs/make-minidebug) - -That you can use to create debug sidecars. -The Swift backtracer supports gzip, zstd, and lzma compressed data, -provided it can locate the relevant dynamic libraries at runtime. - You now have two artifacts: a small, stripped `MyServer` to ship in your container or distribution package, and a `MyServer.debug` to publish to a symbol server or ship in a companion debug-info package. For how symbolicators consume the sidecar at crash time, see . +For additional examples of this kind of process, review the [backtracing scripts](https://github.com/swiftlang/swift/blob/main/test/Backtracing/Inputs/) in the Swift GitHub repository. + ### Verify build IDs match A sidecar is only useful if it shares a build ID with the binary it describes. @@ -146,7 +139,7 @@ readelf -n MyServer.debug | grep 'Build ID' ``` Both commands print the same hexadecimal value when the files match. -A mismatch means the binary and sidecar came from different builds; in that case the sidecar can't symbolicate the binary and you need to rebuild both together. +A mismatch means the binary and sidecar came from different builds; in that case, the sidecar can't symbolicate the binary, and you need to rebuild both together. ## Build for another platform @@ -177,7 +170,7 @@ docker run --rm -it \ ``` These commands mount your current directory as `/code` in the container, set it as the working directory, and run `swift build` inside the Linux environment. -The `swift:latest` container image provides this environment and `swift build` produces Linux-compatible build artifacts. +The `swift:latest` container image provides this environment, and `swift build` produces Linux-compatible build artifacts. If you're on Apple silicon and need to target `x86_64` Linux servers, you need to specify the target platform with the `--platform` option: @@ -202,20 +195,20 @@ The `-e QEMU_CPU=max` environment variable enables the maximum set of CPU featur To build your code into a container, you typically use a container declaration — a Dockerfile or Containerfile — that specifies all the steps to assemble the container image holding your build artifacts. Container-based builds work well in CI/CD pipelines and for validating that your code builds cleanly on Linux. -However, Docker container builds can be slower than native builds, especially on Apple silicon where `x86_64` containers run through emulation. +However, Docker container builds can be slower than native builds, especially on Apple silicon, where `x86_64` containers run through emulation. For more information on packaging your application or service, see . ### Build with a VS Code Dev Container -Visual Studio Code supports the Dev Container feature that lets you open, build, and debug your project within a container running on your local machine. +Visual Studio Code supports the Dev Container feature, which lets you open, build, and debug your project within a container running on your local machine. The Dev Container builds your code in the Linux environment that the container provides. For more information on using a Dev Container, read [Visual Studio Code Dev Containers](https://docs.swift.org/vscode/documentation/userdocs/remote-dev/). ### Cross-compile with the Static Linux SDK If the performance overhead of Docker-based builds affects your workflow, Swift 5.9 and later provide Static Linux SDKs. -Find the link to install the static Linux SDK on the [install page at Swift.org](https://www.swift.org/install/), and detailed instructions in the article [Getting Started with the Static Linux SDK](https://www.swift.org/documentation/articles/static-linux-getting-started.html). +Find the link to install the static Linux SDK on the [install page at Swift.org](https://www.swift.org/install/); for detailed instructions, see [Getting Started with the Static Linux SDK](https://www.swift.org/documentation/articles/static-linux-getting-started.html). The SDK enables cross-compilation directly from macOS to Linux without using a container: ```bash