Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ on:
- main
workflow_dispatch:

permissions:
contents: read
checks: write
pull-requests: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
Expand Down
47 changes: 31 additions & 16 deletions INTERCEPTOR.md
Original file line number Diff line number Diff line change
Expand Up @@ -399,10 +399,14 @@ import com.netflix.conductor.client.http.ConductorClient;
import com.netflix.conductor.client.http.TaskClient;
import com.netflix.conductor.client.http.WorkflowClient;
import com.netflix.conductor.client.metrics.prometheus.PrometheusMetricsCollector;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

// 1. Create and start metrics
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector();
metricsCollector.startServer(); // port 9991, /metrics
// 1. Create a registry and a collector bound to it. The SDK does not start a
// web server; expose registry.scrape() from an endpoint you own (or, in
// Spring Boot, hand in the app's MeterRegistry and use Actuator).
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector(registry);

// 2. Create ConductorClient — withMetricsCollector installs the HTTP interceptor
// and enables automatic listener registration on downstream clients
Expand All @@ -427,12 +431,23 @@ For fine-grained control over which listeners are registered, create the `Conduc

### Custom Metrics Endpoint

The SDK no longer serves metrics itself. Expose the registry from an endpoint your application owns:

```java
// Start Prometheus server on custom port and endpoint
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector();
metricsCollector.startServer(8080, "/custom-metrics");
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector(registry);

var server = com.sun.net.httpserver.HttpServer.create(new java.net.InetSocketAddress(8080), 0);
server.createContext("/custom-metrics", exchange -> {
byte[] body = registry.scrape().getBytes();
exchange.sendResponseHeaders(200, body.length);
try (var os = exchange.getResponseBody()) { os.write(body); }
});
server.start();
```

In Spring Boot, prefer Actuator: hand the application's `MeterRegistry` to the collector (done automatically by `ConductorMetricsAutoConfiguration`) and configure the scrape path/port with `management.*` properties.

### Registering Custom Event Listeners

#### Approach 1: Using Builder's Listener API
Expand Down Expand Up @@ -908,8 +923,9 @@ public class CloudWatchMetricsCollector implements MetricsCollector {

```java
// Create multiple collectors
PrometheusMetricsCollector prometheus = new PrometheusMetricsCollector();
prometheus.startServer(9991, "/metrics");
PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector prometheus = new PrometheusMetricsCollector(prometheusRegistry);
// Expose prometheusRegistry.scrape() from an endpoint your application owns.

DatadogMetricsCollector datadog = new DatadogMetricsCollector(
System.getenv("DATADOG_API_KEY"),
Expand Down Expand Up @@ -1293,9 +1309,9 @@ public class ConductorMonitoringSetup {
new AnotherTaskWorker()
);

// 3. Setup Prometheus metrics
PrometheusMetricsCollector prometheus = new PrometheusMetricsCollector();
prometheus.startServer(9991, "/metrics");
// 3. Setup Prometheus metrics (expose the registry from an endpoint you own)
PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector prometheus = new PrometheusMetricsCollector(prometheusRegistry);

// 4. Setup custom monitoring
SLAMonitor slaMonitor = new SLAMonitor(
Expand Down Expand Up @@ -1443,12 +1459,11 @@ public class EventSystemTest {
**Problem**: Prometheus endpoint returns no metrics

**Solution**:
1. Verify server is started:
```java
metricsCollector.startServer(); // Don't forget this!
```
1. Verify the registry is exposed. The SDK does not start a server:
- Spring Boot: confirm `spring-boot-starter-actuator` + `micrometer-registry-prometheus` are present and the `prometheus` endpoint is exposed (`/actuator/prometheus` by default).
- Plain Java: confirm you are serving `registry.scrape()` from your own HTTP endpoint, and that the same `registry` was passed to `new PrometheusMetricsCollector(registry)`.

2. Check port availability:
2. Check port availability for whatever port your app exposes, e.g.:
```bash
lsof -i :9991
```
Expand Down
22 changes: 19 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ executor.initWorkers("com.mycompany.workers"); // Package to scan for @WorkerTa

## Monitoring Workers

Enable Prometheus metrics collection for monitoring workers:
Collect Micrometer/Prometheus metrics for monitoring workers. The SDK publishes into a `MeterRegistry` you provide; it does not start a metrics web server.

```groovy
// Using conductor-client-metrics module
Expand All @@ -307,11 +307,25 @@ dependencies {
}
```

**Spring Boot:** add Actuator and the Prometheus registry, and SDK metrics appear on `/actuator/prometheus` automatically (no wiring code needed):

```groovy
dependencies {
implementation 'org.conductoross:conductor-client-metrics:5.1.0'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-registry-prometheus'
}
```

**Plain Java:** create a registry, pass it to the collector, and expose it yourself:

```java
import com.netflix.conductor.client.metrics.prometheus.PrometheusMetricsCollector;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector();
metricsCollector.startServer(); // http://localhost:9991/metrics
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector(registry);

ConductorClient client = ConductorClient.builder()
.basePath("...")
Expand All @@ -324,6 +338,8 @@ WorkflowClient workflowClient = new WorkflowClient(client);
TaskRunnerConfigurer configurer = new TaskRunnerConfigurer.Builder(taskClient, workers)
.withThreadCount(10)
.build();

// Expose registry.scrape() from an HTTP endpoint your application owns.
```

When a `MetricsCollector` is attached to the `ConductorClient`, downstream clients (`TaskClient`, `WorkflowClient`, `TaskRunnerConfigurer`) automatically register themselves as event listeners.
Expand Down
71 changes: 62 additions & 9 deletions conductor-client-metrics/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# Conductor Client Metrics

The `conductor-client-metrics` module provides Prometheus metrics for Java SDK clients and workers. It helps operators monitor worker polling, task execution, task result updates, payload sizes, workflow starts, and HTTP client latency.
The `conductor-client-metrics` module records Micrometer metrics for Java SDK clients and workers. It helps operators monitor worker polling, task execution, task result updates, payload sizes, workflow starts, and HTTP client latency.

The SDK **should not start a metrics web server**. Metrics are published into a Micrometer `MeterRegistry` that you supply, and exposing them for scraping is the responsibility of the embedding application:

- **Spring Boot apps** hand in the application's `MeterRegistry` (the one backing `/actuator/prometheus`), so SDK metrics appear alongside the app's own metrics. This module ships a Spring Boot auto-configuration that does this automatically.
- **Plain Java apps** create a `PrometheusMeterRegistry` (or any other registry), pass it to the collector, and expose it however they like (e.g. serve `registry.scrape()` from an HTTP endpoint they own).

> **Deprecated:** the no-argument `new PrometheusMetricsCollector()` plus `startServer()` still spin up a minimal embedded scrape server for backward compatibility, but both are deprecated and will be removed in a future major release. Prefer supplying a `MeterRegistry`.

## Installation

Expand All @@ -14,13 +21,48 @@ dependencies {

## Usage

Create a `PrometheusMetricsCollector`, start the scrape server, and pass the collector to `ConductorClient.Builder`. All downstream clients and the task runner auto-register themselves as listeners.
### Spring Boot (recommended)

Add this module and Spring Boot Actuator with the Prometheus registry to a Spring Boot worker app:

```groovy
dependencies {
implementation 'org.conductoross:conductor-client-metrics:5.1.0'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-registry-prometheus'
}
```

That's it. When a `MeterRegistry` bean is present, `ConductorMetricsAutoConfiguration` creates a `MetricsCollector` bound to it, and the Conductor Spring auto-configuration wires that collector into the client and task runner. SDK metrics then show up on the app's existing scrape endpoint (by default `/actuator/prometheus`).

To serve them at a custom path/port, use the standard Actuator properties, e.g.:

```yaml
server:
port: 9991
management:
endpoints:
web:
base-path: /
exposure:
include: health,prometheus
path-mapping:
prometheus: metrics # scrape endpoint at /metrics
```

To provide your own collector (or a non-Prometheus registry), just declare a `MetricsCollector` bean; the auto-configuration backs off (`@ConditionalOnMissingBean`).

### Plain Java

Create a registry, pass it to the collector, and expose the registry yourself. Passing the collector to `ConductorClient.Builder` auto-registers all downstream clients and the task runner as listeners.

```java
import com.netflix.conductor.client.metrics.prometheus.PrometheusMetricsCollector;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector();
metricsCollector.startServer(); // http://localhost:9991/metrics
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector(registry);

ConductorClient client = ConductorClient.builder()
.basePath("http://conductor-server:8080/api")
Expand All @@ -34,9 +76,18 @@ TaskRunnerConfigurer configurer = new TaskRunnerConfigurer.Builder(taskClient, w
.withThreadCount(10)
.build();
configurer.init();

// Expose the metrics from an HTTP endpoint you own, e.g.:
// var server = com.sun.net.httpserver.HttpServer.create(new InetSocketAddress(9991), 0);
// server.createContext("/metrics", exchange -> {
// byte[] body = registry.scrape().getBytes();
// exchange.sendResponseHeaders(200, body.length);
// try (var os = exchange.getResponseBody()) { os.write(body); }
// });
// server.start();
```

`startServer()` also accepts `(port, endpoint)` for custom scrape configurations. The client builder accepts the usual timeouts, SSL, authentication, and other options alongside `withMetricsCollector` -- none of them change how metrics wiring works.
The client builder accepts the usual timeouts, SSL, authentication, and other options alongside `withMetricsCollector` -- none of them change how metrics wiring works.

### How Auto-Registration Works

Expand All @@ -57,9 +108,11 @@ For advanced use cases where you need fine-grained control over which listeners

```java
import com.netflix.conductor.client.metrics.prometheus.PrometheusMetricsCollector;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector();
metricsCollector.startServer(); // http://localhost:9991/metrics
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
PrometheusMetricsCollector metricsCollector = new PrometheusMetricsCollector(registry);

ConductorClient client = ConductorClient.builder()
.basePath("http://conductor-server:8080/api")
Expand Down Expand Up @@ -168,9 +221,9 @@ Micrometer publishes a `*_max` Gauge alongside every Timer and DistributionSumma

### Metrics Are Empty

- Verify that the collector is wired into the client. The simplest check: was `withMetricsCollector` called on `ConductorClient.Builder`?
- Verify that the collector is wired into the client. In Spring Boot, confirm a `MeterRegistry` bean exists (add `spring-boot-starter-actuator` + `micrometer-registry-prometheus`). In plain Java, confirm `withMetricsCollector` was called on `ConductorClient.Builder`.
- Verify workers have polled or executed tasks. Metrics are created lazily when the relevant event occurs.
- Confirm the scrape endpoint is reachable at the expected host and port.
- Confirm the scrape endpoint is reachable at the expected host and port (Actuator's `/actuator/prometheus` by default, or whatever your app exposes).

### Missing HTTP or Size Metrics

Expand Down
11 changes: 11 additions & 0 deletions conductor-client-metrics/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,20 @@ dependencies {
implementation project(":conductor-client")
implementation "com.squareup.okhttp3:okhttp:${versions.okHttp}"

// Optional Spring Boot auto-configuration. Present only at compile time so
// plain-Java consumers pull in no Spring dependency; the auto-config only
// activates when a Spring Boot app already has spring-boot-autoconfigure
// (and a Micrometer MeterRegistry bean) on its runtime classpath.
compileOnly 'org.springframework.boot:spring-boot-autoconfigure:3.5.13'

testImplementation 'org.mockito:mockito-core:5.12.0'
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.10.3'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.10.3'

// Exercises the Spring Boot auto-configuration in isolation.
testImplementation 'org.springframework.boot:spring-boot-autoconfigure:3.5.13'
testImplementation 'org.springframework.boot:spring-boot-test:3.5.13'
testImplementation 'org.assertj:assertj-core:3.25.3'
}

java {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/*
* Copyright 2026 Conductor Authors.
* <p>
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
* <p>
* http://www.apache.org/licenses/LICENSE-2.0
* <p>
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.netflix.conductor.client.metrics.prometheus;

import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.context.annotation.Bean;

import com.netflix.conductor.client.metrics.MetricsCollector;

import io.micrometer.core.instrument.MeterRegistry;

/**
* Spring Boot auto-configuration that exposes the SDK's {@link MetricsCollector}
* bound to the application's Micrometer {@link MeterRegistry}.
*
* <p>When a {@code MeterRegistry} bean exists (e.g. the one Spring Boot Actuator
* creates to back {@code /actuator/prometheus}), the SDK's metrics are published
* into it and therefore show up alongside the application's own metrics &mdash;
* no separate metrics web server is started by the SDK.
*
* <p>This class is only loaded in a Spring Boot application (it is referenced
* from {@code META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports}).
* Plain-Java consumers never trigger it and pull in no Spring dependency.
*/
@AutoConfiguration(
afterName = {
"org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration",
"org.springframework.boot.actuate.autoconfigure.metrics.export.prometheus.PrometheusMetricsExportAutoConfiguration"
})
@ConditionalOnClass(MeterRegistry.class)
public class ConductorMetricsAutoConfiguration {

@Bean
@ConditionalOnBean(MeterRegistry.class)
@ConditionalOnMissingBean(MetricsCollector.class)
public MetricsCollector conductorMetricsCollector(MeterRegistry meterRegistry) {
return new PrometheusMetricsCollector(meterRegistry);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
import com.netflix.conductor.client.metrics.ApiClientMetrics;

import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

/**
* Prometheus-backed implementation of {@link ApiClientMetrics} that emits the
Expand Down Expand Up @@ -47,9 +47,9 @@ public final class PrometheusApiClientMetrics implements ApiClientMetrics {
100, 1_000, 10_000, 100_000, 1_000_000, 10_000_000
};

private final PrometheusMeterRegistry registry;
private final MeterRegistry registry;

public PrometheusApiClientMetrics(PrometheusMeterRegistry registry) {
public PrometheusApiClientMetrics(MeterRegistry registry) {
this.registry = registry;
}

Expand Down
Loading
Loading