feat: elasticsearch 插件同步上游代码版本#1414
Merged
kongfei605 merged 6 commits intoflashcatcloud:mainfrom Mar 9, 2026
Merged
Conversation
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
该 PR 主要用于将 Elasticsearch input/collector 代码与上游 Prometheus Elasticsearch exporter 的实现对齐,并补充/更新对应的 fixture 与测试用例。
Changes:
- 将多个 collector(SLM / DataStream / Snapshots / ILM / ClusterSettings / HealthReport 等)改造成上游风格的
registerCollector + Update(ctx, ch)注册与采集模式,并新增通用 HTTP 工具函数 - 更新/新增大量 Elasticsearch 不同版本的 fixture(cluster info/health、node stats、mappings、indices、aliases、health_report 等)
- 调整部分测试用例以匹配新的 collector 行为,并新增 ILM、HealthReport、ClusterSettings 相关测试
Reviewed changes
Copilot reviewed 61 out of 80 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| inputs/elasticsearch/elasticsearch.go | 通过 filters 控制 exporter 内部启用的 collectors,并移除部分手动采集逻辑 |
| inputs/elasticsearch/collector/util.go | 新增通用 getURL 与 bool2Float 工具函数供各 collector 复用 |
| inputs/elasticsearch/collector/tasks.go | 调整 tasks 聚合结构体命名与返回类型 |
| inputs/elasticsearch/collector/tasks_test.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/snapshots.go | 重写 snapshots collector 为注册式 + Update(ctx) 模式并复用 getURL |
| inputs/elasticsearch/collector/snapshots_reponse.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/slm.go | 重写 SLM collector 为注册式 + Update(ctx) 模式,并内联 response 类型 |
| inputs/elasticsearch/collector/slm_response.go | 移除独立的 SLM response 定义文件(已内联到 slm.go) |
| inputs/elasticsearch/collector/shards.go | shards collector 获取 cluster name 的方式调整,并修改 cluster info 接收循环 |
| inputs/elasticsearch/collector/shards_test.go | shards collector 测试用例重写为读取 shards fixtures |
| inputs/elasticsearch/collector/nodes_response.go | 更新 nodes stats response 结构以适配新字段/减少字段 |
| inputs/elasticsearch/collector/indices_settings.go | 调整 indices settings collector 的结构与采集流程(拆分 fetch 方法等) |
| inputs/elasticsearch/collector/indices_settings_response.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/indices_mappings.go | 调整 indices mappings 采集流程与部分变量定义方式 |
| inputs/elasticsearch/collector/indices_mappings_response.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/indices_response.go | 调整 indices response 结构字段(指针字段/新增字段/移除字段) |
| inputs/elasticsearch/collector/ilm.go | 新增上游风格 ILM collector(Explain + Status)并注册 |
| inputs/elasticsearch/collector/ilm_test.go | 新增 ILM collector 的集成式测试(Explain + Status 两个 endpoint) |
| inputs/elasticsearch/collector/ilm_status.go | 删除旧的 ILM status collector(被新 ILM collector 替代) |
| inputs/elasticsearch/collector/ilm_indices.go | 删除旧的 ILM indices collector(被新 ILM collector 替代) |
| inputs/elasticsearch/collector/ilm_indices_test.go | 删除旧的 ILM indices 测试(被新 ILM 测试替代) |
| inputs/elasticsearch/collector/health_report.go | 新增 health-report collector 并注册,采集 _health_report 指标 |
| inputs/elasticsearch/collector/health_report_test.go | 新增 health-report collector 的测试用例 |
| inputs/elasticsearch/collector/data_stream.go | 重写 data-stream collector 为注册式 + Update(ctx) 模式 |
| inputs/elasticsearch/collector/data_stream_response.go | 移除独立的 data stream response 定义文件(已内联到 data_stream.go) |
| inputs/elasticsearch/collector/cluster_settings.go | cluster settings 改为注册式 + Update(ctx),并处理 watermark 字段多形态 |
| inputs/elasticsearch/collector/cluster_settings_test.go | 扩展 cluster settings 测试覆盖并新增 getValueInBytes/getValueAsRatio 单测 |
| inputs/elasticsearch/collector/cluster_settings_reponse.go | 删除旧的 cluster settings response 定义文件(已内联到 cluster_settings.go) |
| inputs/elasticsearch/collector/collector.go | 调整 import 排序/版权头一致性 |
| inputs/elasticsearch/collector/cluster_health_test.go | 更新 cluster health 测试期望(移除 active_shards_percent 指标) |
| inputs/elasticsearch/collector/cluster_health.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/cluster_health_response.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/cluster_info.go | 更新版权头信息(与上游一致) |
| inputs/elasticsearch/collector/categraf_utils.go | 新增 Categraf 侧工具函数(GetNodeID/GetClusterName/GetCatMaster/queryURL) |
| inputs/elasticsearch/fixtures/nodestats/6.8.8.json | nodestats fixture 由 pretty JSON 改为单行 JSON(内容等价) |
| inputs/elasticsearch/fixtures/nodestats/5.6.16.json | nodestats fixture 由 pretty JSON 改为单行 JSON(内容等价) |
| inputs/elasticsearch/fixtures/indices_settings/6.5.4.json | 新增 indices settings fixture |
| inputs/elasticsearch/fixtures/indices_mappings/counts.json | 修正 mappings fixture 的 JSON 结构与字段顺序(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/indices_mappings/7.8.0.json | 修正 mappings fixture 的 JSON 结构与字段顺序(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/indices/alias/7.17.3.json | 新增 alias fixtures(7.17.3) |
| inputs/elasticsearch/fixtures/indices/alias/5.4.2.json | 新增 alias fixtures(5.4.2) |
| inputs/elasticsearch/fixtures/indices/alias/2.4.5.json | 新增 alias fixtures(2.4.5) |
| inputs/elasticsearch/fixtures/indices/alias/1.7.6.json | 新增 alias fixtures(1.7.6) |
| inputs/elasticsearch/fixtures/indices/2.4.5.json | 新增 indices stats fixture(2.4.5) |
| inputs/elasticsearch/fixtures/indices/1.7.6.json | 新增 indices stats fixture(1.7.6) |
| inputs/elasticsearch/fixtures/ilm_indices/6.6.0.json | 调整 ILM indices fixture key 顺序(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/healthreport/8.7.0.json | 新增 _health_report fixture(8.7.0) |
| inputs/elasticsearch/fixtures/datastream/7.15.0.json | 调整 data stream fixture 字段顺序/补充 total_store_size_bytes |
| inputs/elasticsearch/fixtures/clusterinfo/7.13.1.json | 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/clusterinfo/5.4.2.json | 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/clusterinfo/2.4.5.json | 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出) |
| inputs/elasticsearch/fixtures/clusterhealth/5.4.2.json | 调整 cluster health fixture 字段顺序/active_shards_percent 类型(50.0) |
| inputs/elasticsearch/fixtures/clusterhealth/2.4.5.json | 调整 cluster health fixture 字段顺序/active_shards_percent 类型(50.0) |
| inputs/elasticsearch/fixtures/clusterhealth/1.7.6.json | 调整 cluster health fixture 字段顺序 |
Comments suppressed due to low confidence (1)
inputs/elasticsearch/collector/indices_mappings.go:1
- 之前当
len(indicesIncluded) > maxIndicesIncludeCount时,会先请求/_all/_mappings再对返回结果做本地过滤(仅保留indicesIncluded)。本次修改移除了 Collect 里的本地过滤逻辑,但fetchAndDecodeIndicesMappings()也没有补上过滤,从而导致“include 列表很大”时采集会把所有 index 的 mappings 都纳入指标输出,违背 include 语义且可能大幅放大输出规模。建议恢复本地过滤(在 fetch 或 Collect 任一处补回即可)。
// Copyright The Prometheus Authors
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+123
to
133
| type aggregatedTaskStats struct { | ||
| CountByAction map[string]int64 | ||
| } | ||
|
|
||
| func AggregateTasks(t tasksResponse) AggregatedTaskStats { | ||
| func AggregateTasks(t tasksResponse) aggregatedTaskStats { | ||
| actions := map[string]int64{} | ||
| for _, task := range t.Tasks { | ||
| actions[task.Action]++ | ||
| } | ||
| return AggregatedTaskStats{CountByAction: actions} | ||
| return aggregatedTaskStats{CountByAction: actions} | ||
| } |
There was a problem hiding this comment.
AggregateTasks 仍是导出函数,但返回类型改成了未导出的 aggregatedTaskStats。这会让包外调用者无法在类型层面使用/声明该返回值(属于对外 API 的破坏性变化)。建议要么将函数也改为未导出(aggregateTasks),要么恢复/保留导出返回类型(例如 AggregatedTaskStats)。
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.