Skip to content

feat: elasticsearch 插件同步上游代码版本#1414

Merged
kongfei605 merged 6 commits intoflashcatcloud:mainfrom
0x0034:feat/esFollowUpstreamVersion
Mar 9, 2026
Merged

feat: elasticsearch 插件同步上游代码版本#1414
kongfei605 merged 6 commits intoflashcatcloud:mainfrom
0x0034:feat/esFollowUpstreamVersion

Conversation

@0x0034
Copy link
Contributor

@0x0034 0x0034 commented Mar 4, 2026

No description provided.

Signed-off-by: ruochen <wanxialianwei@gmail.com>
@kongfei605 kongfei605 requested a review from Copilot March 5, 2026 01:59
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

该 PR 主要用于将 Elasticsearch input/collector 代码与上游 Prometheus Elasticsearch exporter 的实现对齐,并补充/更新对应的 fixture 与测试用例。

Changes:

  • 将多个 collector(SLM / DataStream / Snapshots / ILM / ClusterSettings / HealthReport 等)改造成上游风格的 registerCollector + Update(ctx, ch) 注册与采集模式,并新增通用 HTTP 工具函数
  • 更新/新增大量 Elasticsearch 不同版本的 fixture(cluster info/health、node stats、mappings、indices、aliases、health_report 等)
  • 调整部分测试用例以匹配新的 collector 行为,并新增 ILM、HealthReport、ClusterSettings 相关测试

Reviewed changes

Copilot reviewed 61 out of 80 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
inputs/elasticsearch/elasticsearch.go 通过 filters 控制 exporter 内部启用的 collectors,并移除部分手动采集逻辑
inputs/elasticsearch/collector/util.go 新增通用 getURLbool2Float 工具函数供各 collector 复用
inputs/elasticsearch/collector/tasks.go 调整 tasks 聚合结构体命名与返回类型
inputs/elasticsearch/collector/tasks_test.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/snapshots.go 重写 snapshots collector 为注册式 + Update(ctx) 模式并复用 getURL
inputs/elasticsearch/collector/snapshots_reponse.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/slm.go 重写 SLM collector 为注册式 + Update(ctx) 模式,并内联 response 类型
inputs/elasticsearch/collector/slm_response.go 移除独立的 SLM response 定义文件(已内联到 slm.go)
inputs/elasticsearch/collector/shards.go shards collector 获取 cluster name 的方式调整,并修改 cluster info 接收循环
inputs/elasticsearch/collector/shards_test.go shards collector 测试用例重写为读取 shards fixtures
inputs/elasticsearch/collector/nodes_response.go 更新 nodes stats response 结构以适配新字段/减少字段
inputs/elasticsearch/collector/indices_settings.go 调整 indices settings collector 的结构与采集流程(拆分 fetch 方法等)
inputs/elasticsearch/collector/indices_settings_response.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/indices_mappings.go 调整 indices mappings 采集流程与部分变量定义方式
inputs/elasticsearch/collector/indices_mappings_response.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/indices_response.go 调整 indices response 结构字段(指针字段/新增字段/移除字段)
inputs/elasticsearch/collector/ilm.go 新增上游风格 ILM collector(Explain + Status)并注册
inputs/elasticsearch/collector/ilm_test.go 新增 ILM collector 的集成式测试(Explain + Status 两个 endpoint)
inputs/elasticsearch/collector/ilm_status.go 删除旧的 ILM status collector(被新 ILM collector 替代)
inputs/elasticsearch/collector/ilm_indices.go 删除旧的 ILM indices collector(被新 ILM collector 替代)
inputs/elasticsearch/collector/ilm_indices_test.go 删除旧的 ILM indices 测试(被新 ILM 测试替代)
inputs/elasticsearch/collector/health_report.go 新增 health-report collector 并注册,采集 _health_report 指标
inputs/elasticsearch/collector/health_report_test.go 新增 health-report collector 的测试用例
inputs/elasticsearch/collector/data_stream.go 重写 data-stream collector 为注册式 + Update(ctx) 模式
inputs/elasticsearch/collector/data_stream_response.go 移除独立的 data stream response 定义文件(已内联到 data_stream.go)
inputs/elasticsearch/collector/cluster_settings.go cluster settings 改为注册式 + Update(ctx),并处理 watermark 字段多形态
inputs/elasticsearch/collector/cluster_settings_test.go 扩展 cluster settings 测试覆盖并新增 getValueInBytes/getValueAsRatio 单测
inputs/elasticsearch/collector/cluster_settings_reponse.go 删除旧的 cluster settings response 定义文件(已内联到 cluster_settings.go)
inputs/elasticsearch/collector/collector.go 调整 import 排序/版权头一致性
inputs/elasticsearch/collector/cluster_health_test.go 更新 cluster health 测试期望(移除 active_shards_percent 指标)
inputs/elasticsearch/collector/cluster_health.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/cluster_health_response.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/cluster_info.go 更新版权头信息(与上游一致)
inputs/elasticsearch/collector/categraf_utils.go 新增 Categraf 侧工具函数(GetNodeID/GetClusterName/GetCatMaster/queryURL)
inputs/elasticsearch/fixtures/nodestats/6.8.8.json nodestats fixture 由 pretty JSON 改为单行 JSON(内容等价)
inputs/elasticsearch/fixtures/nodestats/5.6.16.json nodestats fixture 由 pretty JSON 改为单行 JSON(内容等价)
inputs/elasticsearch/fixtures/indices_settings/6.5.4.json 新增 indices settings fixture
inputs/elasticsearch/fixtures/indices_mappings/counts.json 修正 mappings fixture 的 JSON 结构与字段顺序(更贴近 ES 输出)
inputs/elasticsearch/fixtures/indices_mappings/7.8.0.json 修正 mappings fixture 的 JSON 结构与字段顺序(更贴近 ES 输出)
inputs/elasticsearch/fixtures/indices/alias/7.17.3.json 新增 alias fixtures(7.17.3)
inputs/elasticsearch/fixtures/indices/alias/5.4.2.json 新增 alias fixtures(5.4.2)
inputs/elasticsearch/fixtures/indices/alias/2.4.5.json 新增 alias fixtures(2.4.5)
inputs/elasticsearch/fixtures/indices/alias/1.7.6.json 新增 alias fixtures(1.7.6)
inputs/elasticsearch/fixtures/indices/2.4.5.json 新增 indices stats fixture(2.4.5)
inputs/elasticsearch/fixtures/indices/1.7.6.json 新增 indices stats fixture(1.7.6)
inputs/elasticsearch/fixtures/ilm_indices/6.6.0.json 调整 ILM indices fixture key 顺序(更贴近 ES 输出)
inputs/elasticsearch/fixtures/healthreport/8.7.0.json 新增 _health_report fixture(8.7.0)
inputs/elasticsearch/fixtures/datastream/7.15.0.json 调整 data stream fixture 字段顺序/补充 total_store_size_bytes
inputs/elasticsearch/fixtures/clusterinfo/7.13.1.json 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出)
inputs/elasticsearch/fixtures/clusterinfo/5.4.2.json 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出)
inputs/elasticsearch/fixtures/clusterinfo/2.4.5.json 调整 cluster info fixture 字段顺序/结构(更贴近 ES 输出)
inputs/elasticsearch/fixtures/clusterhealth/5.4.2.json 调整 cluster health fixture 字段顺序/active_shards_percent 类型(50.0)
inputs/elasticsearch/fixtures/clusterhealth/2.4.5.json 调整 cluster health fixture 字段顺序/active_shards_percent 类型(50.0)
inputs/elasticsearch/fixtures/clusterhealth/1.7.6.json 调整 cluster health fixture 字段顺序
Comments suppressed due to low confidence (1)

inputs/elasticsearch/collector/indices_mappings.go:1

  • 之前当 len(indicesIncluded) > maxIndicesIncludeCount 时,会先请求 /_all/_mappings 再对返回结果做本地过滤(仅保留 indicesIncluded)。本次修改移除了 Collect 里的本地过滤逻辑,但 fetchAndDecodeIndicesMappings() 也没有补上过滤,从而导致“include 列表很大”时采集会把所有 index 的 mappings 都纳入指标输出,违背 include 语义且可能大幅放大输出规模。建议恢复本地过滤(在 fetch 或 Collect 任一处补回即可)。
// Copyright The Prometheus Authors

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +123 to 133
type aggregatedTaskStats struct {
CountByAction map[string]int64
}

func AggregateTasks(t tasksResponse) AggregatedTaskStats {
func AggregateTasks(t tasksResponse) aggregatedTaskStats {
actions := map[string]int64{}
for _, task := range t.Tasks {
actions[task.Action]++
}
return AggregatedTaskStats{CountByAction: actions}
return aggregatedTaskStats{CountByAction: actions}
}
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AggregateTasks 仍是导出函数,但返回类型改成了未导出的 aggregatedTaskStats。这会让包外调用者无法在类型层面使用/声明该返回值(属于对外 API 的破坏性变化)。建议要么将函数也改为未导出(aggregateTasks),要么恢复/保留导出返回类型(例如 AggregatedTaskStats)。

Copilot uses AI. Check for mistakes.
0x0034 added 5 commits March 5, 2026 16:30
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
Signed-off-by: ruochen <wanxialianwei@gmail.com>
@kongfei605 kongfei605 merged commit 1186788 into flashcatcloud:main Mar 9, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants