Skip to content

fix(ducklake): improve COPY workload#890

Open
bnjjj wants to merge 10 commits into
mainfrom
bnjjj/fix_ducklake_default_config
Open

fix(ducklake): improve COPY workload#890
bnjjj wants to merge 10 commits into
mainfrom
bnjjj/fix_ducklake_default_config

Conversation

@bnjjj

@bnjjj bnjjj commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This PR adds several improvements to the COPY phase of ducklake:

  • Use arrow batch instead of sql literals
  • Do not enable data inlining when COPY
  • Disable maintenance while copying
  • Fix transaction conflict by removing the DELETEs from internal ETL markers tables and introducing an EPOCH table to know what we can delete in maintenances

Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested a review from a team as a code owner July 9, 2026 11:52
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@coveralls

coveralls commented Jul 10, 2026

Copy link
Copy Markdown

Coverage Status

Coverage is 73.221%bnjjj/fix_ducklake_default_config into main. No base build found for main.

bnjjj added 6 commits July 10, 2026 10:48
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj changed the title fix(ducklake): change default config fix(ducklake): improve COPY workload Jul 13, 2026
Comment thread crates/etl-destinations/src/ducklake/batches.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants