blog: Apache DataFusion Comet 0.17.0 release post#198
Merged
Conversation
Draft release announcement for Comet 0.17.0, focused on the JVM codegen dispatcher, expanded expression coverage, and the native-shuffle FFI round-trip removal, told through the Arrow-native framing. Stats and date are placeholders to finalize at release.
andygrove
changed the title
…oiding stage fallback
Contributor
|
Do we want a section that we're starting to discuss criteria for a 1.0.0 release, have a tracking issue for it apache/datafusion-comet#4082, and are looking for as much user feedback as possible? |
Fill in PR count (192) from the generated changelog and remove the pre-publish TODO. Expand the codegen dispatch coverage to reflect the expressions wired up since the draft (collection and higher-order functions, AES, mask, try_to_number, timezone conversions). Add the Arrow C Stream Interface input-path change and the native broadcast nested loop join. Correct Spark 4.1 to 4.1.2. Tighten editorializing prose throughout.
The post referenced a global spark.comet.expr.allowIncompatible, which does not exist. The flag is per-expression and lives under the spark.comet.expression prefix: spark.comet.expression.<name>.allowIncompatible.
Replace raw TPC-H runtimes with the combined improvement: the two FFI changes improve TPC-DS at 1TB by around 9%.
The 9% is the 0.16.0-to-0.17.0 improvement at 1TB; the FFI changes are the largest contributor but not the sole cause.
andygrove
changed the title
andygrove
requested review from
coderfender,
comphead,
kazuyukitanimura,
mbutrovich and
parthchandra
coderfender
approved these changes
Jun 21, 2026
martin-g
approved these changes
Jun 21, 2026
Contributor
|
Taking another pass this morning. |
mbutrovich
requested changes
Jun 22, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
Minor nits. Thanks for driving the blog post, @andygrove! 0.17.0 looks like a good one!
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft release announcement for Apache DataFusion Comet 0.17.0.
The post is structured as a standard PMC release announcement, with the focus on this cycle's headline work:
Incompatible-to-dispatch change).GroupsAccumulatorfor statistical aggregates.Opening as a draft because the following still need to be finalized at release time (flagged with a TODO comment in the post):
dev/changelog/0.17.0.md)Feedback welcome on scope: a few other 0.17.0 themes (Iceberg
RewriteDataFilesnative scan,NullTypeshuffle support, pluggable S3 credentials) were intentionally left out to keep the focus tight.