feat(flagd): implement file sync for offline mode#98
feat(flagd): implement file sync for offline mode#98buildingisfun23 wants to merge 2 commits intoopen-feature:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements a file-based synchronization provider for flagd, enabling the loading and polling of feature flags from a local file. The changes include the core FileSync implementation, integration into the FlagdProvider, and comprehensive unit tests. Review feedback highlights a race condition in the initialization logic, the need to reset the shutdown flag during re-initialization, and opportunities to optimize file reading and parsing by using stream-based parsing and checking file modification times.
| lock.unlock(); | ||
| absl::Status status = ReadAndUpdateFlags(); | ||
| if (!status.ok()) { | ||
| return status; | ||
| } | ||
| lock.lock(); |
There was a problem hiding this comment.
There is a race condition in the Init method. The lifecycle_mutex_ is unlocked before calling ReadAndUpdateFlags() and re-locked before starting the polling thread. If multiple threads call Init concurrently, they could all pass the initial initialized_ check, perform redundant file reads, and attempt to spawn multiple polling threads, potentially leading to resource leaks or crashes when assigning to poll_thread_.
You should re-verify the initialized_ state after re-acquiring the lock at line 80.
lock.lock();
if (initialized_) {
return absl::OkStatus();
}There was a problem hiding this comment.
Fixed in commit 45075c5. After re-acquiring the lock, we now re-check initialized_ before spawning the poll thread, so concurrent callers that both pass the initial guard correctly short-circuit on the second check.
|
|
||
| try { | ||
| poll_thread_ = std::thread(&FileSync::PollForChanges, this); | ||
| } catch (const std::exception& e) { | ||
| return absl::InternalError( | ||
| absl::StrCat("Failed to spawn poll thread: ", e.what())); | ||
| } |
There was a problem hiding this comment.
The shutdown_requested_ flag is not reset to false during Init. If Shutdown() is called and then Init() is called again (e.g., in a re-initialization scenario), the newly spawned PollForChanges thread will immediately exit because shutdown_requested_ remains true from the previous shutdown. Additionally, ensure shutdown_requested_ is reset within the lock before spawning the thread, and return an absl::Status if thread creation fails.
shutdown_requested_ = false;
try {
poll_thread_ = std::thread(&FileSync::PollForChanges, this);
} catch (const std::exception& e) {
return absl::InternalError(e.what());
}References
- For functions returning absl::StatusOr, propagate errors by returning an appropriate absl::Status instead of throwing exceptions.
There was a problem hiding this comment.
Fixed in commit 45075c5. shutdown_requested_ is now reset to false inside the lock before spawning the thread (see the shutdown_requested_ = false line added prior to thread creation). Thread creation failure is also propagated as absl::InternalError.
| std::stringstream buffer; | ||
| buffer << file.rdbuf(); | ||
|
|
||
| Json parsed; | ||
| try { | ||
| parsed = Json::parse(buffer.str()); |
There was a problem hiding this comment.
Reading the entire file into a std::stringstream and then converting it to a string before parsing is inefficient. nlohmann::json::parse can directly consume the input stream, which avoids unnecessary memory allocations and copies. Additionally, ensure that any exceptions caught during parsing are propagated as an absl::Status rather than being thrown, as per repository standards.
Json parsed;
try {
parsed = Json::parse(file);
} catch (const std::exception& e) {
return absl::InternalError(e.what());
}References
- For functions returning absl::StatusOr, propagate errors by returning an appropriate absl::Status instead of throwing exceptions.
There was a problem hiding this comment.
Fixed in commit 45075c5. The std::stringstream intermediate is removed; ReadAndUpdateFlags now calls Json::parse(file) directly on the std::ifstream, avoiding the extra allocation and copy. Parse exceptions are still caught and returned as absl::InvalidArgumentError.
| absl::Status FileSync::ReadAndUpdateFlags() { | ||
| std::ifstream file(source_path_); | ||
| if (!file.is_open()) { | ||
| return absl::NotFoundError( | ||
| absl::StrCat("Unable to open flag source file: ", source_path_)); | ||
| } | ||
|
|
||
| std::stringstream buffer; | ||
| buffer << file.rdbuf(); | ||
|
|
||
| Json parsed; | ||
| try { | ||
| parsed = Json::parse(buffer.str()); | ||
| } catch (const std::exception& e) { | ||
| return absl::InvalidArgumentError( | ||
| absl::StrCat("Failed to parse flag source file: ", e.what())); | ||
| } | ||
|
|
||
| UpdateFlags(parsed); | ||
| return absl::OkStatus(); | ||
| } |
There was a problem hiding this comment.
The current implementation parses the JSON file every poll_interval_ms_ (default 5s) regardless of whether the file has changed. For better efficiency, consider checking the file's modification time (e.g., using std::filesystem::last_write_time) and only parsing the JSON if the file has been updated since the last read.
There was a problem hiding this comment.
Fixed in commit 45075c5. ReadAndUpdateFlags now calls std::filesystem::last_write_time at the top and returns early with OkStatus when the mtime matches the previously recorded last_write_time_ member. JSON parsing only happens when the file has actually changed.
Add FileSync provider that reads flag configuration from a local JSON file and polls for changes at a configurable interval, enabling offline flag evaluation without a gRPC connection to flagd. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: buildingisfun23 <buildingisfun23@users.noreply.github.com>
cfdeb3b to
ec990a9
Compare
- Re-check initialized_ after re-acquiring lock in Init() to fix race condition when concurrent callers both pass the initial guard - Reset shutdown_requested_ to false inside lock before spawning thread so re-initialization after Shutdown() works correctly - Parse JSON directly from ifstream instead of via stringstream to avoid unnecessary memory allocation and copy - Skip JSON parsing when file mtime is unchanged to avoid redundant work on every poll interval Signed-off-by: buildingisfun23 <buildingisfun23@users.noreply.github.com>
|
Hi @buildingisfun23 ! I'm super happy to see you started contributing to this repo. What is the status of this PR? Is it ready for review or is it still WIP? Do you need any help? Do you want me to assign you to related issue? |
Summary
FileSyncprovider that reads flag configuration from a local JSON file, resolving [flagd] Implement file sync #20offline_poll_interval_ms, default 5s)FlagdProviderso that whenoffline_flag_source_pathis set,FileSyncis used instead ofGrpcSync