UBSan: null pointer passed to memcpy in NALList::serialize (nal.cpp:75) on every encoder startup

[xavier-redant]

Summary

Building with -fsanitize=address,undefined and running any encode triggers a UBSan diagnostic on the very first serialize() call:

source/encoder/nal.cpp:75:19: runtime error: null pointer passed as argument 2, which is declared to never be null

This fires on every encoder startup (via x265_encoder_headers), because the first serialize() call runs the buffer-grow path while m_buffer is still NULL.

Environment

• x265 version: 4.2+46-ef0259ef7 (current master at time of report)
• Build: Linux x86_64, GCC 16.1.1, 8-bit, cmake -DFSANITIZE="address,undefined"
• Reproduces with any input; no special content needed.

Exact command line

./x265 --input input.yuv --input-res 1280x720 --fps 30 --preset medium -o out.hevc

(input: any raw YUV; a synthetic clip generated with ffmpeg -f lavfi -i testsrc2=size=1280x720:rate=30:duration=1 -pix_fmt yuv420p -f rawvideo input.yuv is enough — the diagnostic fires while writing the stream headers, before the first frame is encoded.)

Analysis

NALList's constructor initializes m_buffer(NULL), m_occupancy(0), m_allocSize(0) (nal.cpp:30-39). On the first serialize() call, nextSize > m_allocSize is always true, so the grow path executes:

uint8_t *temp = X265_MALLOC(uint8_t, nextSize);
if (temp)
{
    memcpy(temp, m_buffer, m_occupancy);   // nal.cpp:75 — m_buffer == NULL, m_occupancy == 0

Passing a null pointer to memcpy is undefined behavior even when the size is zero (C11 §7.24.1p2; glibc declares both arguments __nonnull). It is harmless with current codegen, but UB of this kind licenses the optimizer to make surprising deductions (e.g. dropping later null checks on the same pointer), and it makes every -fsanitize=undefined run noisy from startup, which buries real findings.

Trivial fix:

if (m_occupancy)
    memcpy(temp, m_buffer, m_occupancy);

CLA signed and sent on 2026-07-17, with this patch attached to the CLA e-mail. Happy to open a PR as well if that is preferred.

Full UBSan stack trace

source/encoder/nal.cpp:75:19: runtime error: null pointer passed as argument 2, which is declared to never be null
    #0 x265::NALList::serialize(NalUnitType, x265::Bitstream const&, int, unsigned char) [clone .cold] (libx265.so.216+0x22852)
    #1 x265::Encoder::getStreamHeaders(x265::NALList&, x265::Entropy&, x265::Bitstream&) (libx265.so.216+0x5f7c37)
    #2 x265_encoder_headers.part.0 (libx265.so.216+0x6d3511)
    #3 x265::PassEncoder::threadMain() (x265+0x48f033)
    #4 x265::ThreadShim(void*) (libx265.so.216+0x10e98e7)
    #5 asan_thread_start(void*) (libasan.so.8+0x2b606)
    #6 start_thread (libc.so.6+0x72c18)
    #7 __GI___clone3 (libc.so.6+0xf65cb)

Related

The same ASan+UBSan run reports two further benign UB diagnostics I can file separately if useful: a shift-exponent-64 in common/quant.h:128 (Entropy::codeCoeffNxN path) and a negative shift exponent in common/cudata.h:302 (Analysis::compressInterCU_rd0_4 path). For what it's worth, this aligns with the UB/data-race cleanup already underway (#885, #916). ASan itself reports no memory errors and no leaks on this path, and the sanitized build's output is bit-identical to the regular build.