Optimize mapped appends and prepare stable 1.5.5#8
Merged
Conversation
* Add Apache Arrow result streaming to DuckDBCommand Adds a managed Apache Arrow read surface built on DuckDB's current Arrow C Data Interface (duckdb_to_arrow_schema / duckdb_data_chunk_to_arrow), addressing Giorgi#26. - Bindings: duckdb_result_get_arrow_options, duckdb_to_arrow_schema, duckdb_data_chunk_to_arrow, error-data helpers, and a DuckDBArrowOptions safe handle. - Data: DuckDBArrowArrayStream (IArrowArrayStream) that builds the schema once and converts each data chunk into an Arrow RecordBatch via Apache.Arrow's C importers. - DuckDBCommand.ExecuteArrowStream() and ExecuteArrowBatchesAsync() public APIs. - Apache.Arrow dependency added to DuckDB.NET.Data only. - Tests covering schema, scalar values, nulls, multi-chunk streaming, and the stream API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Honor UseStreamingMode in Arrow APIs and fix result stream cleanup Pass UseStreamingMode through ExecuteArrowStream and fetch chunks via the streaming or materialized path accordingly. Dispose Arrow options and close the result if schema building fails in the stream constructor. Add tests for streaming mode, cancellation, and dispose semantics. * Refactor Arrow error handling to typed DuckDBErrorData handle - Add DuckDBErrorData SafeHandle wrapping duckdb_error_data - Move error-data P/Invokes to NativeMethods.ErrorData - Retype duckdb_to_arrow_schema/data_chunk_to_arrow returns to DuckDBErrorData - Add reusable ThrowOnError extension throwing DuckDBException - Make DuckDBArrowArrayStream internal --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the duckdb_error_data_error_type binding and pass the resolved error type to DuckDBException in ThrowOnError. Also drop unused usings in DuckDBArrowArrayStream, DuckDBCommand, and DuckDBDataReader.
Switch the appender from duckdb_appender_error (plain string) to duckdb_appender_error_data, which exposes both the message and the DuckDB error type. Route all appender error sites through the existing DuckDBErrorData.ThrowOnError extension (message prefix now optional), so appender exceptions carry ErrorType, consistent with the Arrow paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LWKN1SKSTFDyeqSrA4dfYT
NumericVectorDataReader.GetValue(offset, targetType) backs the non-generic read surface (GetValue(ordinal), this[ordinal], GetValues). It always ran the boxed value through Convert.ChangeType, even when targetType already matched the column's natural type — the common case, since those APIs read using the column's ClrType. Convert.ChangeType does not early-out on same-type for IConvertible values: it dispatches through ToXxx() and re-boxes the result, so every value was boxed twice. Add a value.GetType() == targetType fast path that returns the already boxed value directly. Benchmark (read 1,000,000 rows x 3 numeric columns via GetValue, .NET 10, M4 Pro): Before: 54.50 ms, 137.3 MB allocated After: 22.73 ms, 68.7 MB allocated -58% time and -50% allocations on the object-returning path (the remaining allocation is the unavoidable single box of the object API). The typed GetFieldValue<T> path is unaffected. Behavior is unchanged: ChangeType returned the same value for same-type, and BigInteger (not IConvertible) already used this same early-out inside ChangeType.
PreparedStatement.BindParameters ran a LINQ OfType<>().Any(...) and iterated the parameters with foreach over the non-generic collection (which boxed the List<T> struct enumerator) on every command execution. Replace both with index-based loops over the typed indexer. This removes ~128 bytes and 2-3 allocations per command execution, which matters in tight parameterized-query loops. Execution time is unchanged, as the native prepare/execute dominates.
Prepare consolidated fork preview release
Point fork README to preview packages
Use compact preview package badges
Removed project icon from README.
Updated description to reflect cutting-edge performance work.
skuirrels
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
List<T>, and typed enumerable collectionsv1.5.5Impact
The change reduces per-row reflection, interface dispatch, boxing, and temporary allocations across mapped appends, prepared commands, and nested collection writes while preserving the existing public API.
Representative LIST/ARRAY results for one million rows with 32 integers:
List<int>to LISTFor
ReadOnlyCollection<int>at 100,000 rows, runtime improved from 35.59 ms to 16.10 ms and allocations fell from 78,139.55 KB to 3,138.35 KB.Validation
net8.0andnet10.0net8.0net10.0Skuirrels.DuckDB.NET.Bindings.Full.1.5.5.nupkgcreated and validatedSkuirrels.DuckDB.NET.Data.Full.1.5.5.nupkgcreated and validatednet8.0andnet10.0Release
After merge, this change is intended to be published as the stable
v1.5.5release.