Kafka Collector Documentation

1. Metrics of Kafka Collector

The Kafka metrics supported by Datakit are listed in the official documentation:
https://docs.truewatch.com/integrations/kafka/#metric

These metrics are a subset of those provided by Apache Kafka. If you find any missing metrics, you can manually add them to the Datakit Kafka collector.

2. Explanation of Datakit.Kafka Configuration Blocks

In Datakit's kafka.conf file, you will see configuration blocks similar to the following:

# Add metrics to read
  [[inputs.kafka.metric]]
    name         = "kafka_controller"
    mbean        = "kafka.controller:name=*,type=*"
    #field_name   = "#1."
    field_prefix = "#1."
    tag_keys     = ["request"]

Explanation of fields in this configuration block:

• name: Indicates the metric set sent to the TrueWatch workspace for this configuration block. For example, kafka_controller in the above example means metrics collected by this block will be grouped under kafka_controller, which can be queried when creating charts or configuring alerts in the workspace.

• mbean: The query statement structure for specific metrics or metric sets provided in the official documentation. You can find required metrics in the official documentation:
  https://kafka.apache.org/documentation/#monitoring
  Copy the MBean recorded in the table and paste it into this configuration item. Datakit will automatically query and send the metric.

• field_name: Specifies the metric name for all data obtained by the current query. This field uses relative offsets to mark metric names, which is not intuitive, so this configuration is usually not used.

• field_prefix: A unified prefix for all metric data obtained by the current query. For example, when collecting controller metrics, you can set Controller_ as the prefix here, so all query results will carry this prefix to distinguish from other Kafka metrics.

• tag_keys: Represents attribute tags attached to data obtained by the current query. Different MBeans have different query conditions, and the keys of these conditions are filled into this list as tags. When querying in the TrueWatch workspace, you can filter or aggregate data by these tags. The available tags depend on the MBean statement you select—all items used as query conditions in the statement. For example:

  kafka.producer:type=producer-metrics,client-id=([-.\w]+)
  

  This statement queries all producer metrics and uses client-id as an aggregation condition to attach the client-id attribute to metrics. Configure:

  tag_keys     = ["client-id"]

  This will add client-id as a tag to all producer metrics sent to TrueWatch, allowing you to filter metrics by different client-id values when creating charts.

3. How to Manually Add Metrics to Query

Before adding metrics, you need to test the selected MBean using the following general test command:

curl "http://<jolokia_addr>:<jolokia_port>/jolokia/read/kafka.producer:type=producer-metrics,client-id=<your_client_id>"

Where:

• <jolokia_addr>:<jolokia_port> should be replaced with the actual Jolokia address and port in your environment.
• <your_client_id> should be replaced with the actual client ID, or use the wildcard *.

If the test command returns a response, the parameter value is valid. Then, copy a configuration block in Datakit's kafka.conf and fill in the MBean:

  [[inputs.kafka.metric]]
    name         = "kafka_producer"
    mbean        = "kafka.producer:type=producer-metrics,client-id=*"
    #field_name   = "#1."
    field_prefix = "producer_"
    tag_keys     = ["client-id"]

Save the configuration file, exit, restart Datakit, and check the collection results.

4. Metric Collection Configuration for Producer/Consumer/Connect

For Kafka, producer, consumer, and connect are separate JVM processes from the broker, so a single Kafka configuration cannot collect metrics for all objects. To collect these metrics, two conditions must be met:

I. The producer, consumer, and connect processes are started with the -javaagent parameter in KAFKA_OPTS to monitor these processes using Jolokia.

II. Separate kafka.conf files are copied for each object (e.g., named kafka-pro.conf, kafka-con.conf) for data collection.

The reason for separate kafka.conf files is that Jolokia used by these JVM processes typically has different addresses and ports. If all components are deployed in one environment, their Jolokia ports are usually different. If deployed in different environments, their Jolokia addresses are usually different. Therefore, you need to copy separate kafka.conf files, configure jolokia_urls in each to point to the correct agent, and modify [[inputs.kafka.metric]] blocks to configure individual MBeans for each metric to be collected.

5. General Troubleshooting for Missing Data

If you still cannot obtain the desired metrics after following the above steps, check if the metric is provided by the current Jolokia using the following Jolokia interface:

curl "http://<jolokia_addr>:<jolokia_port>/jolokia/search/*:*"

Or query a specific metric set, for example, to check how many producer-related metrics exist:

curl "http://<jolokia_addr>:<jolokia_port>/jolokia-agent/search/kafka.producer:*"

These queries will return a structure similar to:

{
  "request": {
    "pattern": "kafka.producer:*",
    "type": "search"
  },
  "value": [
    "kafka.producer:client-id=console-producer,type=producer-metrics",
    "kafka.producer:client-id=console-producer,topic=telkom-demo-topic,type=producer-topic-metrics"
  ],
  "status": 200,
  "timestamp": 1761234567
}

Each item in the value array is a full MBean name that can be directly used in read operations (as in the previous command for querying specific metrics).

To further obtain details of an MBean (e.g., available attributes, data types, or operation methods), use:

curl "http://<jolokia_addr>:<jolokia_port>/jolokia/list"

This interface returns not only MBean names but also their attribute lists (e.g., record-send-rate), data types (e.g., double), operation methods (e.g., JMX function calls), and hierarchical relationships between MBeans.

6. Summary of Steps

The general steps for collecting metrics from Kafka objects are:

1. Inject the Jolokia agent into Kafka processes. The broker, producer, consumer, and connect must be injected separately at startup.

2. Copy independent datakit.kafka.conf files for each process, using distinct names for differentiation.

3. In each independent kafka.conf, fill in the access address and port of the Jolokia Agent corresponding to the collection target in urls:

   urls = ["http://localhost:8080/jolokia"]

4. Prepare MBean configuration blocks for the Kafka metrics you need to collect. Each command occupies a separate block; multiple commands require multiple blocks:

   [[inputs.kafka.metric]]
     name         = "kafka_producer"
     mbean        = "kafka.producer:type=producer-metrics,client-id=*"
     #field_name   = "#1."
     field_prefix = "producer_"
     tag_keys     = ["client-id"]
   
   [[inputs.kafka.metric]]
     name         = "kafka_producer"
     mbean        = "kafka.producer:type=producer-topic-metrics,client-id=*"
     #field_name   = "#1."
     field_prefix = "producer_topic_"
     tag_keys     = ["client-id"]
   
   [[inputs.kafka.metric]]
     ...<OTHER METRICS>...

5. Save the configuration file, exit, and restart Datakit to enable collection.