Draft
Conversation
Validates the claim from WICKED_COOL_NEXT_STEPS.md that producing Arrow IPC on the Kafka wire enables a near-zero-copy consumer path. The librdkafka receive buffer is wrapped in place by Arrow C++ — no memcpy between librdkafka and DuckDB's arrow_scan. Includes: - Python producer (pyarrow + confluent-kafka) generating PostHog-shaped events as Arrow IPC record batches - C++ consumer (~250 LOC): librdkafka poll → Arrow IPC reader → DuckDB arrow_scan → INSERT into DuckLake - Standalone docker-compose stack (shifted ports to avoid parent collisions) - Smoke test results: 660K records ingested, zero crashes, zero data loss See experiments/arrow-payload/FINDINGS.md for detailed results.
Collaborator
Author
|
Not merging as this is just a PoC for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WICKED_COOL_NEXT_STEPS.md: a C++ consumer wraps librdkafka's receive buffer in place viaarrow::Buffer(ptr, len)— no memcpy between Kafka and DuckDB'sarrow_scantest-events-arrowtopicarrow_scan→ INSERT into DuckLakeSmoke-test results (Apple Silicon, 2026-04-12)
Test plan
cd experiments/arrow-payload && docker compose up --build— all services healthy, records flowingflushed batchwith record counts:9101shows parquet files underducklake/data/main/events_arrow/