Skip to content

Pre-size LIST results during materialization#343

Open
skuirrels wants to merge 1 commit into
Giorgi:developfrom
skuirrels:feature/list-result-presizing
Open

Pre-size LIST results during materialization#343
skuirrels wants to merge 1 commit into
Giorgi:developfrom
skuirrels:feature/list-result-presizing

Conversation

@skuirrels

Copy link
Copy Markdown
Contributor

Problem

LIST values are materialized into an empty List<T> and grown as each item is added. That creates avoidable backing-array allocations and repeats reflective construction for every row.

Solution

Create normal List<T> results through a reader-local generic factory and pass DuckDB's known list length as the initial capacity. Custom collection targets keep the existing fallback.

Benchmark

100,000 rows with a 16-item List<int>, median of 7 runs after warmup:

Metric Before After Change
Elapsed 95.20 ms 68.58 ms -28.0%
Allocated 45.02 MiB 14.50 MiB -67.8%

Tests

  • 74 LIST, nested LIST, ARRAY, and type-reader tests passed
  • Full suite: 6,897 passed

@skuirrels
skuirrels marked this pull request as ready for review July 21, 2026 09:14
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.53%. Comparing base (fe2abc5) to head (ca3858a).

Files with missing lines Patch % Lines
....NET.Data/DataChunk/Reader/ListVectorDataReader.cs 69.23% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #343      +/-   ##
===========================================
- Coverage    87.59%   87.53%   -0.06%     
===========================================
  Files           77       77              
  Lines         3143     3154      +11     
  Branches       466      469       +3     
===========================================
+ Hits          2753     2761       +8     
- Misses         276      278       +2     
- Partials       114      115       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request optimizes LIST/ARRAY materialization in the DuckDB.NET data reader by pre-sizing List<T> instances using DuckDB’s known list length, reducing allocations and repeated reflection overhead during reads.

Changes:

  • Adds a regression test asserting List<int> results are created with initial capacity equal to the list length.
  • Introduces a cached, reader-local factory path for List<T> to construct new List<T>(capacity) during list materialization.
  • Preserves the existing reflection-based fallback for non-List<T> collection targets.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
DuckDB.NET.Test/DuckDBDataReaderListTests.cs Adds a test verifying list capacity is pre-sized during materialization.
DuckDB.NET.Data/DataChunk/Reader/ListVectorDataReader.cs Implements cached List<T> factory creation and uses DuckDB-provided length as initial capacity.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread DuckDB.NET.Data/DataChunk/Reader/ListVectorDataReader.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants