feat(insights): OpenSearch AD and Elasticsearch ML anomaly pipeline#30
Open
kgrubb wants to merge 5 commits into
Open
feat(insights): OpenSearch AD and Elasticsearch ML anomaly pipeline#30kgrubb wants to merge 5 commits into
kgrubb wants to merge 5 commits into
Conversation
## Added - Idempotent OpenSearch AD discover/adopt/seed via transport and scoped anomaly result queries. - Elasticsearch ML job, datafeed, and `getRecords`-backed findings. - `elasticsearch_ml_anomaly` finding kind and enrichment when source IPs are present. ## Changed - Insight engine ensures the native pipeline after field mapping; polls OS or ES for anomalies and alerts on the existing path. - Heuristic egress polling skips only when native pipeline is healthy and native fetches report a healthy empty window. - OpenSearch AD/alerting searches accept string JSON bodies using shared `parseJsonOrNull` (shard counts and hits stay consistent). ## Fixed - Heuristics no longer stay suppressed when native anomaly setup is not actually ready.
…logs) ## Changed - Detect ML `resource_already_exists` via `meta.body.error.type` and `body.error.type` when present, with string fallback for older clients. - Normalize ML record `timestamp` from epoch ms or ISO strings for stable finding ids and windows. - Log throttled JSON parse warnings for OpenSearch shard counts when the search body is a malformed string (same as hits path). - Debug log when multiple egress-shaped AD detectors or ML jobs match so operators see deterministic single adoption. ## Fixed - Heuristic feature match comment clarifies JSON substring check is not a full aggregation AST.
## Changed - ML pipeline: extract create helper; derive jobIds without reassignment. - OpenSearch AD: extract egressDetectorIdsEnsure; single const detectorIds. - Insight engine: resolve native anomaly context via async IIFE; poll timer/inFlight held in one const object. - ML job tie-break: sort a copy of matches instead of mutating the array.
## Changed - Numeric ML record timestamps: assume ms from API; if value is below 1e12, treat as epoch seconds before converting to ISO. ## Fixed - OpenSearch AD relist failure path: assert three transport calls for empty relist; add case where relist returns hits that fail egress match.
…logs ## Added - verify: second AD list call requires Kaytoo egress detector name in JSON (confirms native seed/adopt against OpenSearch). - verify: optional log line when Anomaly appears (graded AD in console path). ## Changed - e2e README: document the new native AD assertion and optional Anomaly hint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added
postInsight).elasticsearch_ml_anomalyfinding kind; enrichment foropensearch_anomalyandelasticsearch_ml_anomalywhen contributing source IPs are present.Changed
healthyEmpty(so heuristics are not silenced when AD/ML is unavailable).Fixed
parseJsonOrNull, consistent with the rest of the codebase.