Replace reindex API with bulk API and add multi-data-stream support#7
Open
AndersonQ wants to merge 1 commit intoelastic:masterfrom
Open
Replace reindex API with bulk API and add multi-data-stream support#7AndersonQ wants to merge 1 commit intoelastic:masterfrom
AndersonQ wants to merge 1 commit intoelastic:masterfrom
Conversation
Switch from the server-side _reindex API to scan + _bulk with "create" actions, mimicking how Elastic Agent sends data. This gives per-document feedback: created, duplicate (409 version conflict), or other errors. Failed and duplicate documents are saved to separate NDJSON files. Accept a list of data streams instead of a single one; the program now iterates over each data stream, scoping index names per data stream (e.g. tsdb-<data_stream>, tsdb-overwritten-<data_stream>). Also: - Read ES credentials from ELASTIC_PACKAGE_ELASTICSEARCH_* env vars with fallback to hardcoded defaults; mask passwords in output - Fall back to system/certifi CA bundle when elasticsearch_ca_path is not set - Replace exit(0) with sys.exit(1) for error paths - Clear time_series_fields between runs to avoid stale state - Update README with new configuration, examples, and output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When using the test kit some docs were failing because of how the reindex work and hiding the real issue. Thus I modified it to use the bulk as the Elastic Agent does. Also I made some small improvements:
Proposed commit message