Wire NativeObjectStore through Store to DataFormatPlugin, IndexingEngine and ReaderManagers#7
Conversation
…ine and ReaderManagers Phase 1: Plumbing only — creates the wiring for a shard-scoped NativeStoreHandle to flow from the repository layer through to the data format write and read paths. No actual ObjectStore usage yet (createNativeStore returns EMPTY). Changes: - Store: add NativeStoreRepository field (repo-level, borrowed from Repository) - StoreFactory: 6/7-arg newStore become defaults, 8-arg with NativeStoreRepository is abstract - IndexService.createShard: resolve NativeStoreRepository from RepositoriesService - DataFormatPlugin: add createNativeStore(Store) default method - DataFormatRegistry: add createNativeStore(format, store), update getReaderManagers - DataFormatAwareEngine: orchestrate creation, pass handle to engine and readers, close on shutdown - IndexingEngineConfig/ReaderManagerConfig: carry NativeStoreHandle (shard-scoped) - ParquetDataFormatPlugin: override createNativeStore with Phase 2 TODOs
| // Plugin creates shard-scoped native store from the Store's repository reference | ||
| this.shardNativeStore = registry.createNativeStore(format, config().getStore()); | ||
|
|
There was a problem hiding this comment.
Instead of NativeStoreHandle I believe the createNativeStore should return NativeStoreRepository, the handle should be a lower level primitive that interacts with the FFM layer
There was a problem hiding this comment.
IMO, Consider passing the shard-scoped ObjectStore handle directly instead of the repository-level NativeStoreRepository. The plugin shouldn't need to know about repositories — it just needs an ObjectStore to read/write parquet files.
We can think this of how lucene works with FilterDirectory instead of RemoteDirectory. @Bukhtawar your thoughts ?
| // Plugin creates shard-scoped native store from the Store's repository reference | ||
| this.shardNativeStore = registry.createNativeStore(format, config().getStore()); | ||
|
|
There was a problem hiding this comment.
IMO, Consider passing the shard-scoped ObjectStore handle directly instead of the repository-level NativeStoreRepository. The plugin shouldn't need to know about repositories — it just needs an ObjectStore to read/write parquet files.
We can think this of how lucene works with FilterDirectory instead of RemoteDirectory. @Bukhtawar your thoughts ?
| OnClose onClose, | ||
| ShardPath shardPath, | ||
| IndexStorePlugin.DirectoryFactory directoryFactory, | ||
| NativeStoreRepository nativeStoreRepository |
There was a problem hiding this comment.
Store is a shard-level storage abstraction — it holds Directory for Lucene. For native formats, it should hold a NativeDirectory (shard-scoped object store) rather than NativeStoreRepository (repo-level handle). The repository is a cluster concept that shouldn't leak into the shard store directly
There was a problem hiding this comment.
Suggested pattern — mirror how Directory comes from DirectoryFactory:
NativeDirectoryFactory resolves the native pointer, creates the native object store returns NativeDirectory
Store holds NativeDirectory — doesn't know about repositories
Tomorrow the underlying pointer can be swapped (different backend) without changing Store
nishchay21
force-pushed
the
warm-composite-support
branch
8 times, most recently
from
9f3e9a5 to
f78e480
Compare
3 participants
DO NOT MERGE
Created PR only to get reviews on the NativeStore wiring.
PR Description
Summary
Wire a shard-scoped NativeStoreHandle (Rust ObjectStore pointer) through the storage and data format layers so that both the write path (IndexingEngine) and read path (ReaderManagers) can access it.
Phase 1 - plumbing only, no actual ObjectStore usage yet.
Phase 2 - actual usage of ObjectStore on native side for read and writes
Motivation
The parquet data format plugin currently uses direct file I/O for reads and writes. To support warm indices (where files may not be present locally), we need an ObjectStore abstraction
that can route I/O to local or remote storage. This PR creates the wiring so that a future Phase 2 can plug in the actual ObjectStore implementation via FFM/Rust.
Flow
DataFormatPlugin.createNativeStore(Store) — what it will do
Called once per shard at engine creation time. Creates a shard-scoped ObjectStore (Rust side) and returns a NativeStoreHandle pointer to it. Both the write path and read path share this handle.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.