Monitore o Kafka no Kubernetes (Strimzi) com OpenTelemetry

Monitore seu cluster Kafka em execução no Kubernetes com o operador Strimzi, implantando o OpenTelemetry Collector. O coletor descobre automaticamente os pods do broker Kafka e coleta métricas abrangentes.

Instalação e configuração

Siga estas etapas para implantar e configurar o OpenTelemetry Collector no seu cluster Kubernetes para descobrir e monitorar automaticamente seus brokers Strimzi Kafka.

Antes de você começar

Certifique-se de ter:

Uma conta New Relic com uma
Cluster Kubernetes com acesso kubectl
Kafka implantado via operador Strimzi

Configurar o cluster Kafka para métricas JMX do Kafka

Configure seu cluster Strimzi Kafka para expor métricas JMX do Kafka via Prometheus JMX Exporter. Esta configuração será implantada como um ConfigMap e referenciada pelo seu cluster Kafka.

Criar ConfigMap de métricas JMX

Crie um ConfigMap com padrões do JMX Exporter que definem quais métricas do Kafka coletar. Salvar como kafka-jmx-metrics-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jmx-metrics
  namespace: newrelic
data:
  kafka-metrics-config.yml: |
    startDelaySeconds: 0
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true

    rules:
      # Cluster-level controller metrics
      - pattern: 'kafka.controller<type=KafkaController, name=GlobalTopicCount><>Value'
        name: kafka_cluster_topic_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=GlobalPartitionCount><>Value'
        name: kafka_cluster_partition_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=FencedBrokerCount><>Value'
        name: kafka_broker_fenced_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=PreferredReplicaImbalanceCount><>Value'
        name: kafka_partition_non_preferred_leader
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value'
        name: kafka_partition_offline
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value'
        name: kafka_controller_active_count
        type: GAUGE

      # Broker-level replica metrics
      - pattern: 'kafka.server<type=ReplicaManager, name=UnderMinIsrPartitionCount><>Value'
        name: kafka_partition_under_min_isr
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=LeaderCount><>Value'
        name: kafka_broker_leader_count
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=PartitionCount><>Value'
        name: kafka_partition_count
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
        name: kafka_partition_under_replicated
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=IsrShrinksPerSec><>Count'
        name: kafka_isr_operation_count
        type: COUNTER
        labels:
          operation: "shrink"

      - pattern: 'kafka.server<type=ReplicaManager, name=IsrExpandsPerSec><>Count'
        name: kafka_isr_operation_count
        type: COUNTER
        labels:
          operation: "expand"

      - pattern: 'kafka.server<type=ReplicaFetcherManager, name=MaxLag, clientId=Replica><>Value'
        name: kafka_max_lag
        type: GAUGE

      # Broker topic metrics (totals)
      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>Count'
        name: kafka_message_count
        type: COUNTER

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec><>Count'
        name: kafka_request_count
        type: COUNTER
        labels:
          type: "fetch"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec><>Count'
        name: kafka_request_count
        type: COUNTER
        labels:
          type: "produce"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec><>Count'
        name: kafka_request_failed
        type: COUNTER
        labels:
          type: "fetch"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec><>Count'
        name: kafka_request_failed
        type: COUNTER
        labels:
          type: "produce"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>Count'
        name: kafka_network_io
        type: COUNTER
        labels:
          direction: "in"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec><>Count'
        name: kafka_network_io
        type: COUNTER
        labels:
          direction: "out"

      # Per-topic metrics (only appear after traffic flows)
      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>Count'
        name: kafka_prod_msg_count
        type: COUNTER
        labels:
          topic: "$1"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=(.+)><>Count'
        name: kafka_topic_io
        type: COUNTER
        labels:
          topic: "$1"
          direction: "in"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=(.+)><>Count'
        name: kafka_topic_io
        type: COUNTER
        labels:
          topic: "$1"
          direction: "out"

      # Request metrics
      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>Count'
        name: kafka_request_time_total
        type: COUNTER
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>50thPercentile'
        name: kafka_request_time_50p
        type: GAUGE
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>99thPercentile'
        name: kafka_request_time_99p
        type: GAUGE
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>Mean'
        name: kafka_request_time_avg
        type: GAUGE
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestChannel, name=RequestQueueSize><>Value'
        name: kafka_request_queue
        type: GAUGE

      - pattern: 'kafka.server<type=DelayedOperationPurgatory, name=PurgatorySize, delayedOperation=(.+)><>Value'
        name: kafka_purgatory_size
        type: GAUGE
        labels:
          type: "$1"

      # Controller stats
      - pattern: 'kafka.controller<type=ControllerStats, name=LeaderElectionRateAndTimeMs><>Count'
        name: kafka_leader_election_rate
        type: COUNTER

      - pattern: 'kafka.controller<type=ControllerStats, name=UncleanLeaderElectionsPerSec><>Count'
        name: kafka_unclean_election_rate
        type: COUNTER

      # Log flush metrics
      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>Count'
        name: kafka_logs_flush_count
        type: COUNTER

      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>50thPercentile'
        name: kafka_logs_flush_time_50p
        type: GAUGE

      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>99thPercentile'
        name: kafka_logs_flush_time_99p
        type: GAUGE

      # JVM Garbage Collection
      - pattern: 'java.lang<name=(.+), type=GarbageCollector><>CollectionCount'
        name: jvm_gc_collections_count
        type: COUNTER
        labels:
          name: "$1"

      - pattern: 'java.lang<name=(.+), type=GarbageCollector><>CollectionTime'
        name: jvm_gc_collections_elapsed
        type: COUNTER
        labels:
          name: "$1"

      # JVM Memory
      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>committed'
        name: jvm_memory_heap_committed
        type: GAUGE

      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max'
        name: jvm_memory_heap_max
        type: GAUGE

      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used'
        name: jvm_memory_heap_used
        type: GAUGE

      # JVM Threading and System
      - pattern: 'java.lang<type=Threading><>ThreadCount'
        name: jvm_thread_count
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>SystemLoadAverage'
        name: jvm_system_cpu_load_1m
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>AvailableProcessors'
        name: jvm_cpu_count
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad'
        name: jvm_cpu_recent_utilization
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>SystemCpuLoad'
        name: jvm_system_cpu_utilization
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>OpenFileDescriptorCount'
        name: jvm_file_descriptor_count
        type: GAUGE

      - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount'
        name: jvm_class_count
        type: GAUGE

      # JVM Memory Pool
      - pattern: 'java.lang<type=MemoryPool, name=(.+)><Usage>used'
        name: jvm_memory_pool_used
        type: GAUGE
        labels:
          name: "$1"

      - pattern: 'java.lang<type=MemoryPool, name=(.+)><Usage>max'
        name: jvm_memory_pool_max
        type: GAUGE
        labels:
          name: "$1"

      - pattern: 'java.lang<type=MemoryPool, name=(.+)><CollectionUsage>used'
        name: jvm_memory_pool_used_after_last_gc
        type: GAUGE
        labels:
          name: "$1"

      # Broker uptime
      - pattern: 'java.lang<type=Runtime><>Uptime'
        name: kafka_broker_uptime
        type: GAUGE

Dica

Personalize métricas: Este ConfigMap inclui métricas abrangentes de broker, tópico, requisição, controlador e JVM do Kafka. Você pode adicionar ou modificar padrões consultando os exemplos do Prometheus JMX Exporter e a documentação do Kafka MBean. Consulte a documentação de regras do JMX Exporter para configurações adicionais.

Importante

Requisito de namespace: O ConfigMap de métricas JMX e o seu cluster Kafka devem estar no mesmo namespace. Neste guia, ambos são implantados no namespace newrelic.

Aplique o ConfigMap:

bash

$kubectl apply -f kafka-jmx-metrics-config.yaml

Atualize o cluster Kafka para usar o JMX Exporter

Atualize seu recurso Strimzi Kafka para referenciar o ConfigMap de métricas:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: newrelic
spec:
  kafka:
    version: X.X.X
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-jmx-metrics
          key: kafka-metrics-config.yml
    # ...rest of your Kafka configuration

Aplique as alterações. O Strimzi executará uma reinicialização contínua dos seus brokers Kafka:

bash

$kubectl apply -f kafka-cluster.yaml

Após a conclusão da reinicialização gradual, cada broker do Kafka exporá métricas do Prometheus na porta 9404.

Implantar o OpenTelemetry Collector

Implante o OpenTelemetry Collector para monitorar seu cluster Kafka. Escolha seu método de instalação preferido:

O método de instalação via Helm é a abordagem recomendada para implantar o OpenTelemetry Collector no Kubernetes.

Criar segredo de credenciais da New Relic

Crie um segredo do Kubernetes contendo sua chave de licença da New Relic e o endpoint OTLP. Escolha o endpoint para sua região New Relic:

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.nr-data.net:4317'

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://eu01-otlp.nr-data.net:4317'

Dica

Para outras configurações de endpoint, consulte Configure seu endpoint OTLP.

Crie o values.yaml com a configuração do coletor

Crie um arquivo values.yaml que contenha a configuração completa do OpenTelemetry Collector. Tanto o NRDOT quanto os coletores OpenTelemetry usam configuração idêntica e fornecem os mesmos recursos de monitoramento do Kafka. Escolha sua imagem de coletor preferida:

NRDOT é a distribuição do OpenTelemetry Collector suportada pela New Relic, oferecendo suporte completo da New Relic. Para mais informações, consulte o repositório do NRDOT Collector no GitHub.

Crie values.yaml com o seguinte conteúdo:

# Deployment mode
mode: deployment
replicaCount: 1

# Use NRDOT collector image
image:
  repository: newrelic/nrdot-collector
  tag: "latest"
  pullPolicy: Always

# Service account configuration
serviceAccount:
  create: true
  name: otel-collector

# RBAC for Kubernetes service discovery
clusterRole:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]

# Pod security context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

# Resource limits
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi
    cpu: 500m

# Load environment variables from secret
extraEnvsFrom:
  - secretRef:
      name: newrelic-otlp-secret

# Disable default ports
ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

# OpenTelemetry Collector Configuration
config:
  receivers:
    # Disable default receivers not available in NRDOT experimental
    jaeger: null
    zipkin: null

    kafkametrics/cluster:
      brokers:
        # TODO#1: Replace with your Kafka bootstrap service
        - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
      collection_interval: 30s
      protocol_version: 2.0.0
      scrapers:
        - brokers
        - topics
        - consumers
      metrics:
        kafka.topic.min_insync_replicas:
          enabled: true
        kafka.topic.replication_factor:
          enabled: true
        kafka.partition.replicas:
          enabled: false
        kafka.partition.oldest_offset:
          enabled: false
        kafka.partition.current_offset:
          enabled: false

    prometheus/kafka-jmx:
      config:
        scrape_configs:
          - job_name: 'kafka-jmx-metrics'
            scrape_interval: 30s
            kubernetes_sd_configs:
              - role: pod
                namespaces:
                  names:
                    # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                    - newrelic
            relabel_configs:
              # Filter for Kafka broker pods
              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
                action: keep
                # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
                regex: my-cluster-kafka

              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
                action: keep
                # TODO#4: Replace with your Strimzi Kafka cluster name
                regex: my-cluster

              # Extract broker ID from pod name
              - source_labels: [__meta_kubernetes_pod_name]
                target_label: broker.id
                regex: '.*-(\\d+)$'
                replacement: '$1'

              # Set scrape target to pod IP on port 9404
              - source_labels: [__meta_kubernetes_pod_ip]
                target_label: __address__
                replacement: '$1:9404'

  exporters:
    # New Relic OTLP exporter
    otlp/backend:
      endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
      tls:
        insecure: false
      sending_queue:
        num_consumers: 12
        queue_size: 5000
      retry_on_failure:
        enabled: true
      headers:
        api-key: ${NEW_RELIC_LICENSE_KEY}

  processors:
    # Batch processor for efficient export
    batch/export:
      send_batch_size: 1024
      timeout: 30s

    # Memory limiter to prevent OOM
    memory_limiter:
      limit_percentage: 80
      spike_limit_percentage: 30
      check_interval: 1s

    # Transform metric naming conventions
    transform/metric-naming:
      metric_statements:
        - context: metric
          statements:
            - replace_pattern(name, "_", ".")
            - replace_pattern(name, "\\.load\\.1", ".load_1")
            - replace_pattern(name, "\\.recent\\.util", ".recent_util")
            - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
            - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
            - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
            - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
            - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
            - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
            - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
            - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
        - context: datapoint
          statements:
            - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
            - delete_key(attributes, "gc") where attributes["gc"] != nil
            - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
            - delete_key(attributes, "pool") where attributes["pool"] != nil

    # Add cluster name to all metrics
    resource/cluster-name:
      attributes:
        - key: kafka.cluster.name
          # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
          value: my-cluster
          action: upsert

    # Remove broker.id for cluster-level metrics
    transform/remove_broker_id:
      metric_statements:
        - context: datapoint
          statements:
            - delete_key(attributes, "broker.id")

    # Filter out scrape overhead metrics
    filter/scrape-overhead:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^jmx_.*"
            - "^process_.*"
            - "^jvm_buffer_pool_.*"
            - "^jvm_threads_.*"
            - "^jvm_classes_.*"
            - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
            - "^jvm_compilation_.*"
            - "^jvm_(runtime|info).*"
            - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"

    # Include only cluster-level metrics for cluster pipeline
    filter/include_cluster_metrics:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Exclude cluster-level metrics from broker pipeline
    filter/exclude_cluster_metrics:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Remove unnecessary attributes
    transform/remove_attributes:
      metric_statements:
        - context: metric
          statements:
            - set(description, "") where description != ""
            - set(unit, "") where unit != ""
        - context: resource
          statements:
            - delete_key(attributes, "server.address")
            - delete_key(attributes, "server.port")
            - delete_key(attributes, "service.instance.id")
            - delete_key(attributes, "host.name")
            - delete_key(attributes, "k8s.pod.uid")
            - delete_key(attributes, "url.scheme")

    # Aggregate partition metrics to topic level
    metricstransform/topic-aggregation:
      transforms:
        - include: kafka.partition.replicas_in_sync
          action: insert
          new_name: kafka.partition.replicas_in_sync.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum
        - include: kafka.partition.replicas
          action: insert
          new_name: kafka.partition.replicas.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum

    # Filter out original partition replicas metric
    filter/exclude_partition_replicas_metric:
      metrics:
        exclude:
          match_type: strict
          metric_names:
            - kafka.partition.replicas_in_sync

    # Convert cumulative to delta metrics
    cumulativetodelta:

  service:
    pipelines:
      # Override default traces pipeline to only use receivers that exist in NRDOT
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [debug]

      # Broker-level metrics from Prometheus JMX scraping
      metrics/broker:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/exclude_cluster_metrics
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Prometheus JMX scraping
      metrics/cluster/prometheus:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/include_cluster_metrics
          - transform/remove_broker_id
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Kafka metrics receiver
      metrics/cluster/kafkametrics:
        receivers:
          - kafkametrics/cluster
        processors:
          - resource/cluster-name
          - transform/remove_attributes
          - metricstransform/topic-aggregation
          - filter/exclude_partition_replicas_metric
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

Personalize para seu cluster: Atualize os itens TODO no arquivo de configuração do helm acima:

TODO#1: Substitua pelo seu serviço de bootstrap do Kafka
TODO#2: Substitua pelo namespace onde seu cluster Kafka está implantado
TODO#3: Substitua pelo nome do seu cluster Strimzi Kafka seguido de -kafka
TODO#4: Substitua pelo nome do seu cluster Strimzi Kafka
TODO#5: Substitua pelo nome do seu cluster Kafka (isso será usado para identificar e filtrar suas métricas no New Relic)

Use o OpenTelemetry Collector da comunidade para obter máxima flexibilidade e uma implantação independente de fornecedor.

Crie values.yaml com o seguinte conteúdo:

# Deployment mode
mode: deployment
replicaCount: 1

# Use contrib image for kafkametrics receiver
image:
  repository: otel/opentelemetry-collector-contrib
  tag: "latest"
  pullPolicy: Always

# Service account configuration
serviceAccount:
  create: true
  name: otel-collector

# RBAC for Kubernetes service discovery
clusterRole:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]

# Pod security context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

# Resource limits
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi
    cpu: 500m

# Load environment variables from secret
extraEnvsFrom:
  - secretRef:
      name: newrelic-otlp-secret

# OpenTelemetry Collector Configuration
config:
  receivers:
    # Kafka metrics receiver for cluster-level metrics
    kafkametrics/cluster:
      brokers:
        # TODO#1: Replace with your Kafka bootstrap service
        - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
      collection_interval: 30s
      protocol_version: 2.0.0
      scrapers:
        - brokers
        - topics
        - consumers
      metrics:
        kafka.topic.min_insync_replicas:
          enabled: true
        kafka.topic.replication_factor:
          enabled: true
        kafka.partition.replicas:
          enabled: false
        kafka.partition.oldest_offset:
          enabled: false
        kafka.partition.current_offset:
          enabled: false

    # Prometheus receiver for JMX metrics from Kafka brokers
    prometheus/kafka-jmx:
      config:
        scrape_configs:
          - job_name: 'kafka-jmx-metrics'
            scrape_interval: 30s
            kubernetes_sd_configs:
              - role: pod
                namespaces:
                  names:
                    # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                    - newrelic
            relabel_configs:
              # Filter for Kafka broker pods
              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
                action: keep
                # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
                regex: my-cluster-kafka

              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
                action: keep
                # TODO#4: Replace with your Strimzi Kafka cluster name
                regex: my-cluster

              # Extract broker ID from pod name
              - source_labels: [__meta_kubernetes_pod_name]
                target_label: broker.id
                regex: '.*-(\\d+)$'
                replacement: '$1'

              # Set scrape target to pod IP on port 9404
              - source_labels: [__meta_kubernetes_pod_ip]
                target_label: __address__
                replacement: '$1:9404'

  exporters:
    # New Relic OTLP exporter
    otlp/backend:
      endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
      tls:
        insecure: false
      sending_queue:
        num_consumers: 12
        queue_size: 5000
      retry_on_failure:
        enabled: true
      headers:
        api-key: ${NEW_RELIC_LICENSE_KEY}

  processors:
    # Batch processor for efficient export
    batch/export:
      send_batch_size: 1024
      timeout: 30s

    # Memory limiter to prevent OOM
    memory_limiter:
      limit_percentage: 80
      spike_limit_percentage: 30
      check_interval: 1s

    # Transform metric naming conventions
    transform/metric-naming:
      metric_statements:
        - context: metric
          statements:
            - replace_pattern(name, "_", ".")
            - replace_pattern(name, "\\.load\\.1", ".load_1")
            - replace_pattern(name, "\\.recent\\.util", ".recent_util")
            - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
            - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
            - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
            - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
            - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
            - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
            - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
            - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
        - context: datapoint
          statements:
            - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
            - delete_key(attributes, "gc") where attributes["gc"] != nil
            - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
            - delete_key(attributes, "pool") where attributes["pool"] != nil

    # Add cluster name to all metrics
    resource/cluster-name:
      attributes:
        - key: kafka.cluster.name
          # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
          value: my-cluster
          action: upsert

    # Remove broker.id for cluster-level metrics
    transform/remove_broker_id:
      metric_statements:
        - context: datapoint
          statements:
            - delete_key(attributes, "broker.id")

    # Filter out scrape overhead metrics
    filter/scrape-overhead:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^jmx_.*"
            - "^process_.*"
            - "^jvm_buffer_pool_.*"
            - "^jvm_threads_.*"
            - "^jvm_classes_.*"
            - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
            - "^jvm_compilation_.*"
            - "^jvm_(runtime|info).*"
            - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"

    # Include only cluster-level metrics for cluster pipeline
    filter/include_cluster_metrics:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Exclude cluster-level metrics from broker pipeline
    filter/exclude_cluster_metrics:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Remove unnecessary attributes
    transform/remove_attributes:
      metric_statements:
        - context: metric
          statements:
            - set(description, "") where description != ""
            - set(unit, "") where unit != ""
        - context: resource
          statements:
            - delete_key(attributes, "server.address")
            - delete_key(attributes, "server.port")
            - delete_key(attributes, "service.instance.id")
            - delete_key(attributes, "host.name")
            - delete_key(attributes, "k8s.pod.uid")
            - delete_key(attributes, "url.scheme")

    # Aggregate partition metrics to topic level
    metricstransform/topic-aggregation:
      transforms:
        - include: kafka.partition.replicas_in_sync
          action: insert
          new_name: kafka.partition.replicas_in_sync.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum
        - include: kafka.partition.replicas
          action: insert
          new_name: kafka.partition.replicas.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum

    # Filter out original partition replicas metric
    filter/exclude_partition_replicas_metric:
      metrics:
        exclude:
          match_type: strict
          metric_names:
            - kafka.partition.replicas_in_sync

    # Convert cumulative to delta metrics
    cumulativetodelta:

  service:
    pipelines:
      # Override default pipelines to only use custom Kafka metrics pipelines
      traces: null
      logs: null
      metrics: null

      # Broker-level metrics from Prometheus JMX scraping
      metrics/broker:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/exclude_cluster_metrics
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Prometheus JMX scraping
      metrics/cluster/prometheus:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/include_cluster_metrics
          - transform/remove_broker_id
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Kafka metrics receiver
      metrics/cluster/kafkametrics:
        receivers:
          - kafkametrics/cluster
        processors:
          - resource/cluster-name
          - transform/remove_attributes
          - metricstransform/topic-aggregation
          - filter/exclude_partition_replicas_metric
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

Personalize para seu cluster: Atualize os itens TODO no arquivo de configuração do helm acima:

TODO#1: Substitua pelo seu serviço de bootstrap do Kafka
TODO#2: Substitua pelo namespace onde seu cluster Kafka está implantado
TODO#3: Substitua pelo nome do seu cluster Strimzi Kafka seguido de -kafka
TODO#4: Substitua pelo nome do seu cluster Strimzi Kafka
TODO#5: Substitua pelo nome do seu cluster Kafka (isso será usado para identificar e filtrar suas métricas no New Relic)

Para opções de configuração avançadas, consulte estas páginas de documentação do receptor:

Documentação do receiver Prometheus - Opções adicionais de configuração do receiver

Documentação do receptor de métricas do Kafka - Configuração de métricas do Kafka adicional

Instale o OpenTelemetry Collector com o Helm

Adicione o repositório Helm e instale o OpenTelemetry Collector usando o arquivo values.yaml:

bash

$helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$helm upgrade kafka-monitoring open-telemetry/opentelemetry-collector \
>  --install \
>  --namespace newrelic \
>  --create-namespace \
>  -f values.yaml

Verifique a implantação:

bash

$# Check pod status
$kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$
$# View logs to verify metrics collection
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --tail=50

Você deve ver logs indicando uma coleta bem-sucedida dos brokers Kafka na porta 9404.

O método de instalação via manifesto oferece controle direto sobre os recursos do Kubernetes sem usar o Helm.

Criar segredo de credenciais da New Relic

Crie um segredo do Kubernetes contendo sua chave de licença da New Relic e o endpoint OTLP. Escolha o endpoint para sua região New Relic:

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.nr-data.net:4317'

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://eu01-otlp.nr-data.net:4317'

Dica

Para outras configurações de endpoint, consulte Configure seu endpoint OTLP.

Criar arquivos de manifesto

Crie os arquivos de manifesto do Kubernetes para o coletor de sua preferência. Ambos os coletores usam configuração idêntica - apenas a imagem difere.

Escolha sua opção de coletor e crie os três arquivos necessários:

1. Crie collector-rbac.yaml - Configuração RBAC para acesso à API do Kubernetes:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
  labels:
    app: otel-collector
rules:
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
  labels:
    app: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: newrelic
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

2. Criar collector-configmap.yaml - Configuração do OpenTelemetry Collector:

---
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
namespace: newrelic
labels:
app: otel-collector
data:
otel-collector-config.yaml: |
receivers:
  kafkametrics/cluster:
    brokers:
      # TODO#1: Replace with your Kafka bootstrap service
      - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
    collection_interval: 30s
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers
    metrics:
      kafka.topic.min_insync_replicas:
        enabled: true
      kafka.topic.replication_factor:
        enabled: true
      kafka.partition.replicas:
        enabled: false
      kafka.partition.oldest_offset:
        enabled: false
      kafka.partition.current_offset:
        enabled: false

  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                  - kafka
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
              action: keep
              # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
              regex: my-cluster-kafka
            - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
              action: keep
              # TODO#4: Replace with your Strimzi Kafka cluster name
              regex: my-cluster
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: broker.id
              regex: '.*-(\\d+)$'
              replacement: '$1'
            - source_labels: [__meta_kubernetes_pod_ip]
              target_label: __address__
              replacement: '$1:9404'

exporters:
  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    tls:
      insecure: false
    sending_queue:
      num_consumers: 12
      queue_size: 5000
    retry_on_failure:
      enabled: true
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

processors:
  batch/export:
    send_batch_size: 1024
    timeout: 30s
  memory_limiter:
    limit_percentage: 80
    spike_limit_percentage: 30
    check_interval: 1s
  transform/metric-naming:
    metric_statements:
    - context: metric
      statements:
      - replace_pattern(name, "_", ".")
      - replace_pattern(name, "\\.load\\.1", ".load_1")
      - replace_pattern(name, "\\.recent\\.util", ".recent_util")
      - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
      - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
      - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
      - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
      - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
      - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
      - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
      - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
    - context: datapoint
      statements:
      - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
      - delete_key(attributes, "gc") where attributes["gc"] != nil
      - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
      - delete_key(attributes, "pool") where attributes["pool"] != nil
  resource/cluster-name:
    attributes:
    - key: kafka.cluster.name
      # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
      value: my-cluster
      action: upsert
  transform/remove_broker_id:
    metric_statements:
    - context: datapoint
      statements:
      - delete_key(attributes, "broker.id")
  filter/scrape-overhead:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
        - "^jmx_.*"
        - "^process_.*"
        - "^jvm_buffer_pool_.*"
        - "^jvm_threads_.*"
        - "^jvm_classes_.*"
        - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
        - "^jvm_compilation_.*"
        - "^jvm_(runtime|info).*"
        - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"
  filter/include_cluster_metrics:
    metrics:
      include:
        match_type: regexp
        metric_names:
        - "^kafka\\.partition\\.offline$"
        - "^kafka\\.(leader|unclean)\\.election\\.rate$"
        - "^kafka\\.partition\\.non_preferred_leader$"
        - "^kafka\\.broker\\.fenced\\.count$"
        - "^kafka\\.cluster\\.partition\\.count$"
        - "^kafka\\.cluster\\.topic\\.count$"
  filter/exclude_cluster_metrics:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
        - "^kafka\\.partition\\.offline$"
        - "^kafka\\.(leader|unclean)\\.election\\.rate$"
        - "^kafka\\.partition\\.non_preferred_leader$"
        - "^kafka\\.broker\\.fenced\\.count$"
        - "^kafka\\.cluster\\.partition\\.count$"
        - "^kafka\\.cluster\\.topic\\.count$"
  transform/remove_attributes:
    metric_statements:
    - context: metric
      statements:
      - set(description, "") where description != ""
      - set(unit, "") where unit != ""
    - context: resource
      statements:
      - delete_key(attributes, "server.address")
      - delete_key(attributes, "server.port")
      - delete_key(attributes, "service.instance.id")
      - delete_key(attributes, "host.name")
      - delete_key(attributes, "k8s.pod.uid")
      - delete_key(attributes, "url.scheme")
  metricstransform/topic-aggregation:
    transforms:
    - include: kafka.partition.replicas_in_sync
      action: insert
      new_name: kafka.partition.replicas_in_sync.total
      operations:
      - action: aggregate_labels
        label_set: [topic]
        aggregation_type: sum
    - include: kafka.partition.replicas
      action: insert
      new_name: kafka.partition.replicas.total
      operations:
      - action: aggregate_labels
        label_set: [topic]
        aggregation_type: sum
  filter/exclude_partition_replicas_metric:
    metrics:
      exclude:
        match_type: strict
        metric_names:
        - kafka.partition.replicas_in_sync
  cumulativetodelta:

service:
  pipelines:
    metrics/broker:
      receivers: [prometheus/kafka-jmx]
      processors:
        - resource/cluster-name
        - filter/scrape-overhead
        - transform/metric-naming
        - transform/remove_attributes
        - filter/exclude_cluster_metrics
        - memory_limiter
        - cumulativetodelta
        - batch/export
      exporters: [otlp/backend]
    metrics/cluster/prometheus:
      receivers: [prometheus/kafka-jmx]
      processors:
        - resource/cluster-name
        - filter/scrape-overhead
        - transform/metric-naming
        - transform/remove_attributes
        - filter/include_cluster_metrics
        - transform/remove_broker_id
        - memory_limiter
        - cumulativetodelta
        - batch/export
      exporters: [otlp/backend]
    metrics/cluster/kafkametrics:
      receivers: [kafkametrics/cluster]
      processors:
        - resource/cluster-name
        - transform/remove_attributes
        - metricstransform/topic-aggregation
        - filter/exclude_partition_replicas_metric
        - memory_limiter
        - cumulativetodelta
        - batch/export
      exporters: [otlp/backend]

3. Criar collector-deployment.yaml - Implantação do OpenTelemetry Collector:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: otel-collector
      containers:
      - name: otel-collector
        image: newrelic/nrdot-collector:latest
        command:
        - "/nrdot-collector"
        - "--config=/conf/otel-collector-config.yaml"
        env:
        - name: NEW_RELIC_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_LICENSE_KEY
        - name: NEW_RELIC_OTLP_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_OTLP_ENDPOINT
        - name: GOGC
          value: "80"
        resources:
          limits:
            cpu: "1000m"
            memory: "1Gi"
          requests:
            cpu: "200m"
            memory: "512Mi"
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
          items:
          - key: otel-collector-config.yaml
            path: otel-collector-config.yaml

Personalize antes de implantar: Atualize os itens TODO em collector-configmap.yaml:

TODO#1: Substitua pelo seu serviço de bootstrap do Kafka
TODO#2: Substitua pelo namespace onde seu cluster Kafka está implantado
TODO#3: Substitua pelo nome do seu cluster Strimzi Kafka seguido de -kafka
TODO#4: Substitua pelo nome do seu cluster Strimzi Kafka
TODO#5: Substitua pelo nome do seu cluster Kafka (isso será usado para identificar e filtrar suas métricas no New Relic)

Use o OpenTelemetry Collector da comunidade para uma implantação independente de fornecedor.

1. Criar collector-rbac.yaml - O mesmo que a opção NRDOT acima (a configuração do RBAC é idêntica)

2. Criar collector-configmap.yaml - O mesmo que a opção NRDOT acima (a configuração é idêntica)

3. Criar collector-deployment.yaml - Implantação do OpenTelemetry Collector (apenas a imagem difere):

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: otel-collector
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:latest
        command:
        - "/otelcol-contrib"
        - "--config=/conf/otel-collector-config.yaml"
        env:
        - name: NEW_RELIC_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_LICENSE_KEY
        - name: NEW_RELIC_OTLP_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_OTLP_ENDPOINT
        - name: GOGC
          value: "80"
        resources:
          limits:
            cpu: "1000m"
            memory: "1Gi"
          requests:
            cpu: "200m"
            memory: "512Mi"
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
          items:
          - key: otel-collector-config.yaml
            path: otel-collector-config.yaml

Personalize antes de implantar: Atualize os itens TODO em collector-configmap.yaml:

TODO#1: Substitua pelo seu serviço de bootstrap do Kafka
TODO#2: Substitua pelo namespace onde seu cluster Kafka está implantado
TODO#3: Substitua pelo nome do seu cluster Strimzi Kafka seguido de -kafka
TODO#4: Substitua pelo nome do seu cluster Strimzi Kafka
TODO#5: Substitua pelo nome do seu cluster Kafka (isso será usado para identificar e filtrar suas métricas no New Relic)

Para opções de configuração avançadas, consulte estas páginas de documentação do receptor:

Documentação do receiver Prometheus - Opções adicionais de configuração do receiver

Documentação do receptor de métricas do Kafka - Configuração de métricas do Kafka adicional

Implante os manifestos

Aplique os manifestos do Kubernetes para implantar o OpenTelemetry Collector:

bash

$# Create namespace if it doesn't exist
$kubectl create namespace newrelic --dry-run=client -o yaml | kubectl apply -f -
$
$# Apply RBAC configuration
$kubectl apply -f collector-rbac.yaml
$
$# Apply ConfigMap
$kubectl apply -f collector-configmap.yaml
$
$# Apply Deployment
$kubectl apply -f collector-deployment.yaml

Verifique a implantação:

bash

$# Check pod status
$kubectl get pods -n newrelic -l app=otel-collector
$
$# View logs to verify metrics collection
$kubectl logs -n newrelic -l app=otel-collector --tail=50

Você deve ver logs indicando uma coleta bem-sucedida dos brokers Kafka na porta 9404.

(Opcional) Instrumente aplicações produtoras ou consumidoras

Importante

Suporte a linguagens: Aplicações Java suportam instrumentação de cliente Kafka pronta para uso utilizando o OpenTelemetry Java Agent.

Para coletar telemetria no nível da aplicação de suas aplicações produtoras e consumidoras do Kafka, use o OpenTelemetry Java Agent.

Instrumente seu aplicativo Kafka

Use um init container para baixar o OpenTelemetry Java Agent em tempo de execução:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-producer-app
spec:
  template:
    spec:
      initContainers:
      - name: download-java-agent
        image: busybox:latest
        command:
        - sh
        - -c
        - |
          wget -O /otel-auto-instrumentation/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
        volumeMounts:
        - name: otel-auto-instrumentation
          mountPath: /otel-auto-instrumentation

      containers:
      - name: app
        image: your-kafka-app:latest
        env:
        - name: JAVA_TOOL_OPTIONS
          value: >-
            -javaagent:/otel-auto-instrumentation/opentelemetry-javaagent.jar
            -Dotel.service.name=order-process-service
            -Dotel.resource.attributes=kafka.cluster.name=my-cluster
            -Dotel.exporter.otlp.endpoint=http://localhost:4317
            -Dotel.exporter.otlp.protocol=grpc
            -Dotel.metrics.exporter=otlp
            -Dotel.traces.exporter=otlp
            -Dotel.logs.exporter=otlp
            -Dotel.instrumentation.kafka.experimental-span-attributes=true
            -Dotel.instrumentation.messaging.experimental.receive-telemetry.enabled=true
            -Dotel.instrumentation.kafka.producer-propagation.enabled=true
            -Dotel.instrumentation.kafka.enabled=true
        volumeMounts:
        - name: otel-auto-instrumentation
          mountPath: /otel-auto-instrumentation

      volumes:
      - name: otel-auto-instrumentation
        emptyDir: {}

Observações de configuração:

Substitua order-process-service por um nome exclusivo para sua aplicação produtora ou consumidora
Substitua my-cluster pelo mesmo nome do cluster usado na configuração do seu coletor
O endpoint http://localhost:4317 pressupõe que o coletor esteja em execução como um sidecar no mesmo pod ou acessível via localhost

Dica

A configuração acima envia telemetria para um OpenTelemetry Collector. Se você precisar enviar telemetria para o coletor, implante-o conforme descrito na Etapa 3 com esta configuração:

Adicione o receptor OTLP e os pipelines à configuração do seu coletor para receber telemetria de aplicações instrumentadas:

Adicione à seção receivers:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

  # ... existing receivers (prometheus/kafka-jmx, kafkametrics/cluster)

Adicione à seção de exportadores:

exporters:
  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

  # ... existing exporters

Adicione à seção service.pipelines:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/backend]

    metrics:
      receivers: [otlp]
      exporters: [otlp/backend]

    logs:
      receivers: [otlp]
      exporters: [otlp/backend]

    # ... existing pipelines (metrics/broker, metrics/cluster/prometheus, metrics/cluster/kafkametrics)

Isso permite que o coletor receba traces, métricas e logs de aplicativos de seus clientes Kafka instrumentados e os encaminhe para o New Relic juntamente com as métricas do broker.

O Java Agent fornece instrumentação Kafka pronta para uso com zero alterações de código, capturando:

Latências de solicitação
Métricas de throughput
Taxas de erro
Rastreamento distribuído

Para configuração avançada, consulte a documentação de instrumentação do Kafka.

Encontre seus dados

Após alguns minutos, suas métricas Kafka devem aparecer no New Relic. Consulte Encontre seus dados para obter instruções detalhadas sobre como explorar suas métricas Kafka em diferentes visualizações na interface do usuário do New Relic.

Você também pode consultar seus dados com NRQL:

FROM Metric SELECT * WHERE kafka.cluster.name = 'my-kafka-cluster'

Resolução de problemas

1. Ative os logs de depuração do coletor: Adicione logs detalhados para solucionar problemas de configuração

Para implantações do Helm, atualize seu values.yaml:

config:
  service:
    telemetry:
      logs:
        level: "debug"  # Enable detailed collector internal logs

Para implantações de manifesto, edite o ConfigMap do seu coletor:

bash

$kubectl edit configmap -n newrelic otel-collector-config

Adicione a seção de telemetria sob service::

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    # ... existing pipelines ...

2. Adicionar exportador de depuração: Visualize as métricas nos logs do coletor antes de enviar para a New Relic

Adicione à sua configuração:

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5        # Log first 5 metrics
    sampling_thereafter: 200   # Then log every 200th metric

  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    metrics/broker:
      receivers: [prometheus/kafka-jmx]
      processors: [resource/cluster-name, filter/scrape-overhead, transform/metric-naming, transform/remove_attributes, filter/exclude_cluster_metrics, memory_limiter, cumulativetodelta, batch/export]
      exporters: [debug, otlp/backend]  # Add debug exporter

Em seguida, reinicie o coletor e verifique os logs:

bash

$# Restart collector
$kubectl rollout restart deployment -n newrelic otel-collector
$
$# View logs with metric output
$kubectl logs -n newrelic -l app=otel-collector -f

Importante: Remova o exportador de depuração em produção para evitar estouro de log.

1. Verifique o status e os eventos do pod:

bash

$# Check pod status
$# For Helm:
$kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl get pods -n newrelic -l app=otel-collector
$
$# View detailed pod description
$# For Helm:
$kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl describe pod -n newrelic -l app=otel-collector
$
$# Check recent logs
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --previous --tail=50
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector --previous --tail=50

Problemas comuns e soluções:

2. Configuração inválida: Valide a sintaxe YAML do ConfigMap

bash

$# For manifest deployments, check ConfigMap
$kubectl get configmap -n newrelic otel-collector-config -o yaml
$
$# Validate YAML syntax
$kubectl get configmap -n newrelic otel-collector-config -o yaml | kubectl apply --dry-run=client -f -
$
$# For Helm deployments, check the values
$helm get values <release-name> -n newrelic

3. Permissões RBAC: Verifique se a ServiceAccount tem os ClusterRole bindings adequados

bash

$# Check ServiceAccount
$kubectl get serviceaccount -n newrelic otel-collector
$
$# Check ClusterRole and ClusterRoleBinding
$kubectl get clusterrole otel-collector -o yaml
$kubectl get clusterrolebinding otel-collector -o yaml

4. Restrições de recursos: Verifique se o pod foi OOMKilled ou está atingindo os limites de recursos

bash

$# Check resource usage
$# For Helm:
$kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl top pods -n newrelic -l app=otel-collector
$
$# Check for resource limits
$# For Helm:
$kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -A 5 "Limits\|Requests"
$# For Manifest:
$kubectl describe pod -n newrelic -l app=otel-collector | grep -A 5 "Limits\|Requests"

5. Segredo não encontrado: Verifique se o segredo das credenciais do New Relic existe

bash

$# Check if secret exists
$kubectl get secret -n newrelic newrelic-otlp-secret
$
$# Verify secret has required keys
$kubectl get secret -n newrelic newrelic-otlp-secret -o jsonpath='{.data}' | jq 'keys'

1. Verifique se o JMX Exporter está habilitado: Verifique se o seu recurso Strimzi Kafka possui o JMX Exporter configurado

bash

$# Check Kafka resource configuration for JMX Exporter
$kubectl get kafka -n kafka -o yaml | grep -A 10 jmxPrometheusExporter
$
$# Should show something like:
$# jmxPrometheusExporter:
$#   lowercaseOutputName: true

2. Verifique a porta do JMX Exporter: Verifique se o exporter está ouvindo na porta 9404

bash

$# Check if port 9404 is exposed on Kafka pods
$kubectl get pods -n kafka -l strimzi.io/name=<cluster-name>-kafka -o yaml | grep -A 3 "containerPort: 9404"
$
$# Test connectivity from collector pod
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- sh -c "curl -s http://<kafka-pod-ip>:9404/metrics" | head -10
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- sh -c "curl -s http://<kafka-pod-ip>:9404/metrics" | head -10

3. Verifique se o receptor do Prometheus pode coletar métricas:

bash

$# Check collector logs for Prometheus scraping
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -i "prometheus\|scrape"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep -i "prometheus\|scrape"
$
$# Look for successful scrape messages or errors
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "prometheus/kafka-jmx"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "prometheus/kafka-jmx"

4. Verifique a descoberta de serviços do Kubernetes: Garanta que os rótulos dos pods correspondam a relabel_configs

bash

$# Verify Kafka pod labels
$kubectl get pods -n kafka -l strimzi.io/name=<cluster-name>-kafka --show-labels
$
$# Should include labels like:
$# strimzi.io/cluster=<cluster-name>
$# strimzi.io/name=<cluster-name>-kafka

5. Teste a coleta manual: Verifique se as métricas estão disponíveis

bash

$# Get Kafka broker pod IP
$kubectl get pods -n kafka -o wide
$
$# Curl metrics endpoint
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- curl http://<kafka-pod-ip>:9404/metrics
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- curl http://<kafka-pod-ip>:9404/metrics

6. Verifique se há erros no receiver kafkametrics:

bash

$# Look for kafkametrics connection issues
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "kafkametrics"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "kafkametrics"
$
$# Common errors:
$# - Connection refused: Check broker address is correct
$# - Timeout: Check network policies allow access
$# - Authentication failed: Remove TLS configuration if using plaintext

1. Monitorar o uso de recursos:

bash

$# Check current memory usage
$# For Helm:
$kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl top pods -n newrelic -l app=otel-collector
$
$# Watch memory usage over time
$# For Helm:
$watch kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$watch kubectl top pods -n newrelic -l app=otel-collector

2. Reduza tópicos monitorados: Limite a coleta apenas a tópicos essenciais

# In your values.yaml (Helm) or ConfigMap (manifest), add topic filtering:
receivers:
  kafkametrics/cluster:
    brokers:
      - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
    collection_interval: 30s
    scrapers:
      - brokers
      - topics  # Consider removing if not needed
      - consumers  # Consider removing if not needed
    topic_match: "^(important-topic-1|important-topic-2)$"  # Filter specific topics

3. Aumente os intervalos de coleta: Reduza a frequência de coleta

receivers:
  kafkametrics/cluster:
    collection_interval: 60s  # Increase from 30s to 60s

  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 60s  # Increase from 30s to 60s

4. Otimize o processamento em lote: Ajuste as configurações do processador de lote

processors:
  batch/export:
    timeout: 60s  # Increase from 30s
    send_batch_size: 512  # Reduce from 1024

5. Ajuste o limitador de memória: Ajuste o processador de limitador de memória

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75  # Reduce from 80
    spike_limit_percentage: 20  # Reduce from 30

6. Atualize os limites de recursos: Para implantações do Helm, atualize o values.yaml

resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi  # Adjust as needed
    cpu: 500m

Para implantações de manifesto, atualize a implantação diretamente:

bash

$kubectl patch deployment -n newrelic otel-collector --patch '
$spec:
$  template:
$    spec:
$      containers:
$      - name: otel-collector
$        resources:
$          limits:
$            memory: "1Gi"
$          requests:
$            memory: "512Mi"
$'

7. Reinicie o coletor após as alterações:

bash

$# For Helm:
$kubectl rollout restart deployment -n newrelic kafka-monitoring-opentelemetry-collector
$# For Manifest:
$kubectl rollout restart deployment -n newrelic otel-collector

1. Verifique a conectividade de rede: Verifique se o coletor consegue alcançar os pods do broker Kafka

bash

$# Get Kafka broker pod IPs
$kubectl get pods -n kafka -l strimzi.io/kind=Kafka -o wide
$
$# Test connectivity from collector pod
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- curl -m 5 http://<kafka-pod-ip>:9404/metrics | head -20
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- curl -m 5 http://<kafka-pod-ip>:9404/metrics | head -20

2. Verifique se o JMX Exporter está em execução: Verifique se os brokers do Kafka têm o JMX Exporter habilitado

bash

$# Check Kafka pod for JMX Exporter container
$kubectl get pods -n kafka -o yaml | grep -A 5 "jmx-exporter"
$
$# Check if port 9404 is listening
$kubectl exec -n kafka <kafka-pod-name> -- netstat -tlnp | grep :9404
$
$# Or test from within the pod
$kubectl exec -n kafka <kafka-pod-name> -- curl -s localhost:9404/metrics | head

3. Verifique se relabel_configs corresponde aos rótulos dos pods: Garanta que o receiver do Prometheus possa descobrir os pods do Kafka

bash

$# Verify pod labels match the relabel_configs in your configuration
$kubectl get pods -n kafka -l strimzi.io/kind=Kafka --show-labels
$
$# Should show labels like:
$# strimzi.io/cluster=my-cluster
$# strimzi.io/name=my-cluster-kafka

4. Verifique a configuração do namespace: Verifique se o receiver do Prometheus está procurando no namespace correto

# In your configuration, verify namespace matches where Kafka is deployed
receivers:
  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  - kafka  # Must match your Kafka namespace

5. Verifique as permissões RBAC: Verifique se o ClusterRole permite a descoberta de pods

bash

$# Check ClusterRole has pod list/watch permissions
$kubectl get clusterrole otel-collector -o yaml | grep -A 3 "pods"
$
$# Should include:
$# - apiGroups: [""]
$#   resources: ["pods", "nodes"]
$#   verbs: ["get", "list", "watch"]

6. Aumente o tempo limite de scrape: Se o endpoint de métricas estiver lento para responder

receivers:
  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 30s
          scrape_timeout: 20s  # Increase from default 10s

7. Verifique os logs do coletor para detalhes da coleta:

bash

$# View Prometheus receiver logs
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "prometheus/kafka-jmx"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "prometheus/kafka-jmx"
$
$# Look for discovered targets
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -i "target\|scrape"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep -i "target\|scrape"

1. Verifique os processadores de filtro: Verifique se você não está filtrando métricas inadvertidamente

# Review your filter configurations:
processors:
  filter/scrape-overhead:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "^jmx_.*"  # These are excluded
          - "^process_.*"
          - "^jvm_.*"

2. Os nomes de métricas JMX do Kafka são transformados: Underscores são substituídos por pontos para seguir os padrões do OpenTelemetry

A configuração transforma automaticamente os nomes das métricas do JMX Exporter (definido no Passo 1) para se alinharem às convenções semânticas do OpenTelemetry. Por exemplo:

kafka_topic_io torna-se kafka.topic.io
kafka_broker_leader_count torna-se kafka.broker.leader.count

Ao pesquisar métricas no New Relic, use os nomes transformados com pontos em vez de sublinhados:

# This transformation is applied automatically:
processors:
  transform/metric-naming:
    metric_statements:
      - context: metric
        statements:
          - replace_pattern(name, "_", ".")

3. Ativar logs detalhados: Veja exatamente quais métricas estão sendo processadas

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 100  # Log first 100 metrics to see what's available

service:
  pipelines:
    metrics/broker:
      exporters: [debug, otlp/backend]

4. Consulte o New Relic para obter nomes de métricas: Verifique quais métricas estão sendo realmente recebidas

FROM Metric SELECT uniques(metricName)
WHERE kafka.cluster.name = 'my-cluster'
SINCE 1 hour ago

Próximos passos

Explore as métricas do Kafka - Visualize a referência completa de métricas
Criar dashboards personalizados - Crie visualizações para seus dados do Kafka
Configurar alertas - Monitore métricas críticas como lag do consumidor e partições sub-replicadas

Esta tradução de máquina é fornecida para sua comodidade.

Monitore o Kafka no Kubernetes (Strimzi) com OpenTelemetry

Instalação e configuração

Antes de você começar

Configurar o cluster Kafka para métricas JMX do Kafka

Dica

Importante

Implantar o OpenTelemetry Collector

Região dos EUA

região da UE

Dica

Usando o NRDOT Collector (recomendado)

Usando o OpenTelemetry Collector

Região dos EUA

região da UE

Dica

Usando o NRDOT Collector (recomendado)

Usando o OpenTelemetry Collector

(Opcional) Instrumente aplicações produtoras ou consumidoras

Importante

Instrumente seu aplicativo Kafka

Dica

Configuração do Coletor para telemetria de aplicação

Encontre seus dados

Resolução de problemas

Habilitar log de depuração

Pod do coletor não iniciando

Nenhuma métrica do Kafka coletada

Alto uso de memória

Erros de coleta do Prometheus

Algumas métricas estão ausentes

Próximos passos

Esta tradução de máquina é fornecida para sua comodidade.

Monitore o Kafka no Kubernetes (Strimzi) com OpenTelemetry

Instalação e configuração.css-21sua1{background:none;border:none;width:0;padding:0;}

Antes de você começar

Configurar o cluster Kafka para métricas JMX do Kafka

Dica

Importante

Implantar o OpenTelemetry Collector

região da UE

Dica

Usando o NRDOT Collector (recomendado)

Usando o OpenTelemetry Collector

Região dos EUA

região da UE

Dica

Usando o NRDOT Collector (recomendado)

Usando o OpenTelemetry Collector

(Opcional) Instrumente aplicações produtoras ou consumidoras

Importante

Instrumente seu aplicativo Kafka

Dica

Configuração do Coletor para telemetria de aplicação

Encontre seus dados

Resolução de problemas

Habilitar log de depuração

Pod do coletor não iniciando

Nenhuma métrica do Kafka coletada

Alto uso de memória

Erros de coleta do Prometheus

Algumas métricas estão ausentes

Próximos passos

Instalação e configuração