Prometheus and Grafana

Prometheus, Grafana and the JMX Exporter

Prometheus 16,091 (https://github.com/prometheus/prometheus 66,345 ) (Apache 2.0, 3.15.0 of 25 September 2026) pulls metrics over HTTP; it cannot read JMX. The JMX exporter (https://github.com/prometheus/jmx_exporter 3,330 ) (Apache 2.0, 1.6.0) bridges the two as a Java agent inside the broker, turning MBeans into Prometheus series by the rules of its example kafka-kraft-3_0_0.yml, saved as jmx/kafka.yml:

monitoring/monitoring.yaml (excerpt)YAML

# monitoring.yaml: added to clients/kafka.yaml; paths are relative to clients/
services:
  kafka:
    environment:                       # Prometheus agent on :9404, remote JMX on :9999
      KAFKA_OPTS: >-
        -javaagent:/opt/jmx/jmx_prometheus_javaagent-1.6.0.jar=9404:/opt/jmx/kafka.yml
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: l2-kafka
    volumes: ["../monitoring/jmx:/opt/jmx:ro"]
    healthcheck:                       # tools inherit KAFKA_OPTS: clear it, or :9404 clashes
      test: KAFKA_OPTS= /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server :9092

Every Java process in the container inherits KAFKA_OPTS, including tools run with docker exec and the health check, and each tried to open port 9404 again: the first start stayed unhealthy (BindException: Address in use). Clear it for tools. Prometheus scrapes the agent and KMinion every 5 seconds; one scrape:

monitoring/scrape.shShell

# scrape.sh: what Prometheus reads from the agent on :9404
docker exec l2-prometheus wget -qO- http://l2-kafka:9404/metrics > metrics.txt
echo "$(grep -vc '^#' metrics.txt) series"
grep -E '^kafka_(server_replicamanager_underreplicated|controller_kafkacontroller_active|'\
'server_raft_metrics_current_state|server_brokertopicmetrics_messagesin_total)' metrics.txt
grep 'totaltimems{quantile="0.99",request="Produce"' metrics.txt

Output

6558 series
kafka_controller_kafkacontroller_activebrokercount 1.0
kafka_controller_kafkacontroller_activecontrollercount 1.0
kafka_server_brokertopicmetrics_messagesin_total 390737.0
kafka_server_brokertopicmetrics_messagesin_total{topic="booknest.order-events"} 390737.0
kafka_server_raft_metrics_current_state{state="leader"} 1.0
kafka_server_replicamanager_underreplicatedpartitions 0.0
kafka_network_requestmetrics_totaltimems{quantile="0.99",request="Produce"} 265.87999999999994

Grafana 2,264 (https://github.com/grafana/grafana 77,045 ) (AGPL-3.0, 13.2.3, not run here) draws dashboards from these series; Strimzi 326,081 and Confluent 17,245 publish ready-made Kafka 129 dashboards.

MENU