Skip to content

cmake: kafka: enable zstd compression using bundled library #11726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zshuang0316 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

cmake: kafka: enable zstd compression using bundled library #11726

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions cmake/kafka.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,51 @@ if (FLB_SYSTEM_MACOS)
FLB_OPTION(WITH_CURL Off)
endif()

# Enable zstd compression for librdkafka using the library already resolved
# by the top-level zstd detection block (LIBZSTD_LIBRARIES is set for all
# cases: bundled, system CMake config, and system pkg-config fallback).
if(LIBZSTD_LIBRARIES)
if(TARGET libzstd_static)
# Bundled: force-inject vars — librdkafka's find_package(ZSTD) cannot
# discover an in-tree target on its own.
set(ZSTD_FOUND TRUE CACHE BOOL "" FORCE)
set(ZSTD_INCLUDE_DIR "${FLB_PATH_ROOT_SOURCE}/${FLB_PATH_LIB_ZSTD}/lib" CACHE PATH "" FORCE)
set(ZSTD_LIBRARY_DEBUG libzstd_static CACHE STRING "" FORCE)
set(ZSTD_LIBRARY_RELEASE libzstd_static CACHE STRING "" FORCE)
set(ZSTD_LIBRARY libzstd_static CACHE STRING "" FORCE)
Comment on lines +87 to +91
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Badge Clear cached bundled ZSTD entries on mode switch

The bundled branch force-writes ZSTD_* cache entries, but the system branch never unsets or updates them. On reconfigure in the same build directory (e.g., switching from bundled to FLB_PREFER_SYSTEM_LIB_ZSTD=ON), stale libzstd_static cache values are reused by librdkafka, so configuration can continue to report/use bundled linkage while the feature summary says system. This cache contamination is introduced by the forced cache writes and should be cleaned up in the non-bundled path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Clear forced ZSTD cache when leaving bundled mode

Forcing ZSTD_LIBRARY* into the cache here leaks across reconfigures: if a build directory is first configured with bundled zstd and then reconfigured with -DFLB_PREFER_SYSTEM_LIB_ZSTD=On, the system branch never unsets these cache entries, so librdkafka's find_package(ZSTD) reuses libzstd_static from cache instead of the system library. In a reconfigure test, CMake still reported Found ZSTD: libzstd_static while the summary said ZSTD: system, and this can produce link failures (-lzstd_static) on hosts that only provide libzstd.

Useful? React with 👍 / 👎.

FLB_OPTION(WITH_ZSTD ON)
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Clear cached WITH_ZSTD before system autodetection

Setting WITH_ZSTD here uses FLB_OPTION, which writes an INTERNAL cache entry, but the system-zstd path does not unset it later. If a build directory is first configured with bundled zstd and then reconfigured to use system zstd where discovery fails, WITH_ZSTD remains ON while the code clears ZSTD_* cache entries, so librdkafka proceeds with zstd enabled but without a resolved library/include path (observed as Found ZSTD: with an empty value), which can break the Kafka build/link step.

Useful? React with 👍 / 👎.

set(FLB_KAFKA_ZSTD_SOURCE "bundled")
else()
# System: clear stale bundled-sentinel entries and WITH_ZSTD so librdkafka
# re-runs find_package(ZSTD) fresh and its option() sets its own default.
# Preserve explicit caller overrides such as -DZSTD_LIBRARY=/custom/path.
if(ZSTD_LIBRARY STREQUAL "libzstd_static")
unset(ZSTD_FOUND CACHE)
unset(ZSTD_INCLUDE_DIR CACHE)
Comment on lines +98 to +100
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass system zstd vars before entering librdkafka

When zstd is discovered through Fluent Bit's pkg-config fallback (CMakeLists.txt zstd block), this system path does not populate ZSTD_INCLUDE_DIR/ZSTD_LIBRARY or force WITH_ZSTD on unless a bundled sentinel is present. In that case, add_subdirectory(librdkafka) runs its own find_package(ZSTD) and may keep WITH_ZSTD=OFF, so Kafka zstd compression is silently disabled despite zstd having been found earlier. Please propagate the resolved system zstd include/library values (or explicitly enable WITH_ZSTD) in this branch.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

@zshuang0316 zshuang0316 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When zstd is found via pkg-config it is present in standard system paths that librdkafka's find_package(ZSTD) will also search. If the two diverge it indicates a non-standard prefix where the caller should supply CMAKE_PREFIX_PATH or -DZSTD_ROOT; silently forwarding pkg-config results into librdkafka's find variables would reintroduce the stale-cache issues this change was designed to avoid.

unset(ZSTD_LIBRARY CACHE)
unset(ZSTD_LIBRARY_DEBUG CACHE)
unset(ZSTD_LIBRARY_RELEASE CACHE)
unset(WITH_ZSTD CACHE)
endif()
set(FLB_KAFKA_ZSTD_SOURCE "system")
endif()
else()
# FLB didn't find zstd. Clear stale bundled entries so librdkafka's
# find_package/option() runs fresh and defaults to OFF.
# Preserve user-supplied custom paths (ZSTD_LIBRARY != sentinel).
if(ZSTD_LIBRARY STREQUAL "libzstd_static")
unset(ZSTD_FOUND CACHE)
unset(ZSTD_INCLUDE_DIR CACHE)
unset(ZSTD_LIBRARY CACHE)
unset(ZSTD_LIBRARY_DEBUG CACHE)
unset(ZSTD_LIBRARY_RELEASE CACHE)
unset(WITH_ZSTD CACHE)
elseif(NOT ZSTD_LIBRARY)
unset(WITH_ZSTD CACHE)
endif()
set(FLB_KAFKA_ZSTD_SOURCE "disabled")
endif()

include_directories(${FLB_PATH_ROOT_SOURCE}/${FLB_PATH_LIB_RDKAFKA}/src/)

add_subdirectory(${FLB_PATH_LIB_RDKAFKA} EXCLUDE_FROM_ALL)
Expand All @@ -88,4 +133,5 @@ message(STATUS "=== Kafka Feature Summary ===")
message(STATUS "SASL Auth: ${FLB_SASL_ENABLED}")
message(STATUS "OAuth Bearer: ${FLB_SASL_OAUTHBEARER_ENABLED}")
message(STATUS "MSK IAM: ${FLB_KAFKA_MSK_IAM_ENABLED}")
message(STATUS "ZSTD: ${FLB_KAFKA_ZSTD_SOURCE}")
message(STATUS "===============================")
Loading