Datafusion codec - adding libs , data source plugin and data source aware plugin #37
Draft
bharath-techie wants to merge 3 commits intofeature/datafusionfrom
Draft
Datafusion codec - adding libs , data source plugin and data source aware plugin #37bharath-techie wants to merge 3 commits intofeature/datafusionfrom
bharath-techie wants to merge 3 commits intofeature/datafusionfrom
Conversation
Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
Signed-off-by: bharath-techie <bharath78910@gmail.com>
Signed-off-by: bharath-techie <bharath78910@gmail.com>
Bukhtawar
reviewed
Aug 15, 2025
| * Represents a stream of record batches from a DataFusion query execution. | ||
| * This interface provides access to query results in a streaming fashion. | ||
| */ | ||
| public interface RecordBatchStream extends AutoCloseable { |
Collaborator
There was a problem hiding this comment.
Maybe this can be generic typed RecordBatchStream<T>
Owner
Author
There was a problem hiding this comment.
Yes makes sense. I had a todo in CSV RBS to refactor a bit
Bukhtawar
reviewed
Aug 15, 2025
Comment on lines
+32
to
+55
| /** | ||
| * Create a new session context for query execution. | ||
| * | ||
| * @param globalRuntimeEnvId the global runtime environment ID | ||
| * @return a CompletableFuture containing the session context ID | ||
| */ | ||
| CompletableFuture<Long> createSessionContext(long globalRuntimeEnvId); | ||
|
|
||
| /** | ||
| * Execute a Substrait query plan. | ||
| * | ||
| * @param sessionContextId the session context ID | ||
| * @param substraitPlanBytes the serialized Substrait query plan | ||
| * @return a CompletableFuture containing the result stream | ||
| */ | ||
| CompletableFuture<RecordBatchStream> executeSubstraitQuery(long sessionContextId, byte[] substraitPlanBytes); | ||
|
|
||
| /** | ||
| * Close a session context and free associated resources. | ||
| * | ||
| * @param sessionContextId the session context ID to close | ||
| * @return a CompletableFuture that completes when the context is closed | ||
| */ | ||
| CompletableFuture<Void> closeSessionContext(long sessionContextId); |
Collaborator
There was a problem hiding this comment.
How is this specific to the data source? Isn't this more around query execution?
Bukhtawar
reviewed
Aug 15, 2025
Comment on lines
+4
to
+15
|
|
||
| use datafusion::prelude::*; | ||
| use datafusion::execution::context::SessionContext; | ||
| use std::collections::HashMap; | ||
| use std::sync::Arc; | ||
| use anyhow::Result; | ||
|
|
||
| /// Manages DataFusion session contexts | ||
| pub struct SessionContextManager { | ||
| contexts: HashMap<*mut SessionContext, Arc<SessionContext>>, | ||
| next_runtime_id: u64, | ||
| } |
Collaborator
There was a problem hiding this comment.
We want to avoid data fusion dependencies in dataformat-csv?
Bukhtawar
reviewed
Aug 15, 2025
| opensearchplugin { | ||
| name = 'dataformat-csv' | ||
| description = 'CSV data format plugin for OpenSearch DataFusion' | ||
| classname = 'org.opensearch.datafusion.csv.CsvDataFormatPlugin' |
Collaborator
There was a problem hiding this comment.
We want to decouple the package namespacing?
Owner
Author
There was a problem hiding this comment.
Yes , i'll work with mohit , this whole dataformat plugin will probably be moved/designed as something that can work for both query and indexing. I have todos in plugin class for the same.
Bukhtawar
reviewed
Aug 15, 2025
Comment on lines
+10
to
+13
| [dependencies] | ||
| # DataFusion dependencies | ||
| datafusion = "49.0.0" | ||
| datafusion-substrait = "49.0.0" |
Collaborator
There was a problem hiding this comment.
We should avoid this dependency?
bharath-techie
force-pushed
the
feature/datafusion
branch
from
8d0dfb6 to
3d06c82
Compare
bharath-techie
force-pushed
the
feature/datafusion
branch
from
0d2f104 to
4987cb8
Compare
bharath-techie
force-pushed
the
feature/datafusion
branch
2 times, most recently
from
31f431b to
53e2fa9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.