Skip to content

Reduce allocations when materializing MAP values#342

Draft
skuirrels wants to merge 1 commit into
Giorgi:developfrom
skuirrels:feature/map-materialization-performance
Draft

Reduce allocations when materializing MAP values#342
skuirrels wants to merge 1 commit into
Giorgi:developfrom
skuirrels:feature/map-materialization-performance

Conversation

@skuirrels

Copy link
Copy Markdown
Contributor

What changed

MAP values currently go through IDictionary, which boxes numeric keys and values and lets Dictionary grow as entries are added.

This keeps that fallback for custom or wider dictionary targets, but adds a typed path for normal Dictionary<TKey, TValue> reads:

  • cache one closed generic materializer per compatible dictionary type
  • pre-size the dictionary from DuckDB's MAP length
  • read keys and values through the generic vector readers
  • cache the selected materializer on the vector reader instead of resolving it per row
  • check compatibility before adding anything to the process-wide cache

Nullable values, provider-specific types, nested values, and derived/wider dictionaries keep the existing behavior.

Benchmark

The benchmark materializes 100,000 rows of Dictionary<int, int>, with 16 entries per row:

SELECT map(range(16)::INTEGER[], range(16)::INTEGER[]) AS values
FROM range(100000)

Results are the median of 7 measured runs after warmup, using a Release build on the same machine.

Metric Before After Change Winner
Median elapsed 94.35 ms 93.77 ms -0.6% No meaningful difference
Allocated 151.07 MiB 45.02 MiB -70.2% After

The allocation reduction is the useful result here. The elapsed-time difference is small enough to treat as noise.

Validation

  • MAP reader tests: 11 passed
  • cache probe: incompatible target remains uncached; compatible target is cached once
  • git diff --check passes
  • the full suite was also exercised; the MAP tests stay green, while repeated runs exposed an existing order-dependent UDF exception-store failure that passes when run alone

@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.59%. Comparing base (fe2abc5) to head (3774e96).

Files with missing lines Patch % Lines
...B.NET.Data/DataChunk/Reader/MapVectorDataReader.cs 97.05% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #342   +/-   ##
========================================
  Coverage    87.59%   87.59%           
========================================
  Files           77       77           
  Lines         3143     3176   +33     
  Branches       466      476   +10     
========================================
+ Hits          2753     2782   +29     
- Misses         276      277    +1     
- Partials       114      117    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant