Monitor self-hosted Elasticsearch with OpenTelemetry

Monitor your self-hosted Elasticsearch cluster by installing the OpenTelemetry Collector directly on servers or virtual machines. New Relic provides flexible deployment options to match your infrastructure setup and monitoring requirements.

You can choose between three collector options:

NRDOT: New Relic Distribution of OpenTelemetry
OTel Collector Contrib: Standard OpenTelemetry Collector with community-contributed components
Prometheus Receiver: For environments already running a Prometheus Elasticsearch exporter

Installation options

Choose the collector distribution that matches your needs:

Before you begin

Before configuring Elasticsearch monitoring with NRDOT, ensure you have:

NRDOT collector installed:

You must have NRDOT collector installed and running as a systemd service on your host
If not installed, follow the official NRDOT installation guide to install the collector using the package manager (DEB/RPM) which sets up the systemd service automatically

Required access privileges:

Your New Relic
Root or sudo privileges on the host machine
Elasticsearch cluster admin privileges with monitor or manage cluster privilege (see Elasticsearch security privileges documentation for details)

System requirements:

Elasticsearch version 7.16 or higher - This integration requires a modern Elasticsearch cluster
Network connectivity - Outbound HTTPS (port 443) to New Relic's OTLP ingest endpoint

Configuration values ready:

Elasticsearch endpoint - Your Elasticsearch cluster URL (e.g., http://localhost:9200)
Cluster name - A unique identifier for your cluster in New Relic

Configure Elasticsearch monitoring

Once the NRDOT collector is installed, replace the collector's default configuration file with the Elasticsearch monitoring configuration. This will enable Elasticsearch metrics collection. Host metrics and logs are optional and can be added separately.

The configuration file is located at: /etc/nrdot-collector/config.yaml

Tip

Backup your default configuration: Before modifying the configuration file, create a backup copy to preserve the default settings:

bash

$sudo cp /etc/nrdot-collector/config.yaml /etc/nrdot-collector/config.yaml.backup

To configure the collector:

Open the configuration file with a text editor using root or sudo privileges:
bash
```
$sudo nano /etc/nrdot-collector/config.yaml
```
Delete all existing content and replace it with the following configuration for Elasticsearch monitoring:

Important

Replace the endpoint value with your Elasticsearch cluster endpoint and update elasticsearch.cluster.name in the processor block with a unique name to identify your cluster in New Relic.

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  attributes/cluster_state_aggregate:
    include:
      match_type: strict
      metric_names:
        - elasticsearch.cluster.state_update.time
    actions:
      - key: type
        action: delete
      - key: state
        action: delete
  filter/critical_operations:
    metrics:
      datapoint:
        # Filters to keep only: query, index, get, merge
        # Affects only 4 metrics: *.operations.completed and *.operations.time
        # All other metrics pass through unchanged
        - 'attributes["operation"] == "query" or attributes["operation"] == "index" or attributes["operation"] == "get" or attributes["operation"] == "merge" or attributes["operation"] == nil'
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, filter/critical_operations, attributes/cluster_state_aggregate, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Optional) For secured Elasticsearch with authentication and SSL, modify the receiver configuration:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Optional) To collect host metrics, add the hostmetrics receiver:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

And add to the service pipelines:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Optional) To collect Elasticsearch logs, add the filelog receiver. Ensure the user running the collector service (nrdot-collector) has read access to your Elasticsearch log files:

If running Elasticsearch on Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

If running Elasticsearch in Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

And add to the service pipelines:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Optional) To add custom metadata tags to your metrics, use the resource/static_override processor:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Save the configuration file.
Set the environment variables:
Create a systemd override directory:
bash
```
$sudo mkdir -p /etc/systemd/system/nrdot-collector.service.d
```
Create the environment configuration file:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/nrdot-collector.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Update the configuration with your values:
- Replace https://otlp.nr-data.net:4318 with your region's endpoint
- Replace YOUR_LICENSE_KEY_HERE with your actual New Relic license key
- Replace 100 with your desired memory limit in MiB for the collector (default: 100 MiB). Adjust based on your environment's needs

Restart the NRDOT collector to apply changes:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart nrdot-collector.service

Verify data collection

Verify that the NRDOT collector is running and collecting data without errors:

Check the collector service status:

bash

$sudo systemctl status nrdot-collector.service

Monitor the collector logs for any errors:
bash
```
$sudo journalctl -u nrdot-collector.service -f
```
Look for successful connections to Elasticsearch and New Relic. If you see errors, refer to the troubleshooting guide.

Before you begin

Before configuring the OTel Collector Contrib, ensure you have:

Required access privileges:

Your New Relic
Root or sudo privileges on the host machine
Elasticsearch cluster admin privileges with monitor or manage cluster privilege (see Elasticsearch security privileges documentation for details)

System requirements:

Elasticsearch version 7.16 or higher - This integration requires a modern Elasticsearch cluster
Network connectivity - Outbound HTTPS (port 443) to New Relic's OTLP ingest endpoint

Configuration values ready:

Elasticsearch endpoint - Your Elasticsearch cluster URL (e.g., http://localhost:9200)
Cluster name - A unique identifier for your cluster in New Relic

Important

You must have OpenTelemetry Collector Contrib installed on your host before proceeding. We recommend installing via official packages (.deb or .rpm) to ensure the systemd service unit is created correctly.

Configure Elasticsearch monitoring

Once the OTel Collector Contrib is installed, replace the collector's default configuration file with the Elasticsearch monitoring configuration. This will enable Elasticsearch metrics collection. Host metrics and logs are optional and can be added separately.

The configuration file is located at: /etc/otelcol-contrib/config.yaml

Tip

Backup your default configuration: Before modifying the configuration file, create a backup copy to preserve the default settings:

bash

$sudo cp /etc/otelcol-contrib/config.yaml /etc/otelcol-contrib/config.yaml.backup

To configure the collector:

Open the configuration file with a text editor using root or sudo privileges:
bash
```
$sudo nano /etc/otelcol-contrib/config.yaml
```
Delete all existing content and replace it with the following configuration for Elasticsearch monitoring:

Important

Replace the endpoint value with your Elasticsearch cluster endpoint and update elasticsearch.cluster.name in the processor block with a unique name to identify your cluster in New Relic.

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 10s
    send_batch_size: 1024
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Optional) For secured Elasticsearch with authentication and SSL, modify the receiver configuration:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Optional) To collect host metrics, add the hostmetrics receiver:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

And add to the service pipelines:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Optional) To collect Elasticsearch logs, add the filelog receiver. Ensure the user running the collector service (otelcol-contrib) has read access to your Elasticsearch log files:

If running Elasticsearch on Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

If running Elasticsearch in Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

And add to the service pipelines:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Optional) To add custom metadata tags to your metrics, use the resource/static_override processor:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Save the configuration file.
Set the environment variables:
Create a systemd override directory:
bash
```
$sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d
```
Create the environment configuration file:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Update the configuration with your values:
- Replace https://otlp.nr-data.net:4318 with your region's endpoint
- Replace YOUR_LICENSE_KEY_HERE with your actual New Relic license key
- Replace 100 with your desired memory limit in MiB for the collector (default: 100 MiB). Adjust based on your environment's needs

Restart the OTel Collector Contrib to apply changes:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart otelcol-contrib.service

Verify data collection

Verify that the OTel Collector Contrib is running and collecting data without errors:

Check the collector service status:

bash

$sudo systemctl status otelcol-contrib.service

Monitor the collector logs for any errors:
bash
```
$sudo journalctl -u otelcol-contrib.service -f
```
Look for successful connections to Elasticsearch and New Relic. If you see errors, refer to the troubleshooting guide.

Use this approach if you already have a Prometheus Elasticsearch exporter running in your environment, or if you're migrating from a Prometheus-based monitoring stack.

Tip

Recommended: If you don't already have a Prometheus exporter running, use the NRDOT Collector or OTel Collector Contrib tabs instead. They connect directly to the Elasticsearch API without needing an additional exporter component.

Install and configure the Elasticsearch exporter

The Prometheus Elasticsearch exporter exposes Elasticsearch metrics in Prometheus format on port 9114.

Download the latest release:

bash

$wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.8.0/elasticsearch_exporter-1.8.0.linux-amd64.tar.gz
$tar xzf elasticsearch_exporter-*.tar.gz
$cd elasticsearch_exporter-*

Start the exporter (point it to your Elasticsearch instance):
bash
```
$./elasticsearch_exporter --es.uri=http://localhost:9200
```
Tip
For secured Elasticsearch, use: --es.uri=https://username:password@localhost:9200 --es.ssl-skip-verify
Verify the exporter is running:
bash
```
$curl http://localhost:9114/metrics | grep elasticsearch_cluster_health
```
You should see Prometheus-formatted metrics like elasticsearch_cluster_health_status, elasticsearch_cluster_health_number_of_nodes, etc.

Configure the collector

This configuration works with both NRDOT and OTel Collector Contrib. Place it in the appropriate config location for your collector:

NRDOT: /etc/nrdot-collector/config.yaml
OTel Collector Contrib: /etc/otelcol-contrib/config.yaml

If you don't have a collector installed yet, follow the installation steps in the NRDOT Collector or OTel Collector Contrib tabs above first, then return here to apply this configuration.

This configuration scrapes metrics from the Elasticsearch exporter and translates them to OpenTelemetry-compatible metric names that power the New Relic Elasticsearch dashboard.

receivers:
  prometheus/elasticsearch:
    config:
      scrape_configs:
        - job_name: 'elasticsearch'
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:9114']  # <-- elasticsearch_exporter address

processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}

  metricstransform/prom_to_otel:
    transforms:
      - include: ^elasticsearch_cluster_health_number_of_data_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.data_nodes
      - include: ^elasticsearch_cluster_health_status$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.health
        operations:
          - action: update_label
            label: color
            new_label: status
      - include: ^elasticsearch_cluster_health_number_of_in_flight_fetch$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.in_flight_fetch
      - include: ^elasticsearch_cluster_health_number_of_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.nodes
      - include: ^elasticsearch_cluster_health_number_of_pending_tasks$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.pending_tasks
      - include: ^elasticsearch_cluster_health_active_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_cluster_health_active_primary_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active_primary
      - include: ^elasticsearch_cluster_health_initializing_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: initializing
      - include: ^elasticsearch_cluster_health_relocating_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: relocating
      - include: ^elasticsearch_cluster_health_unassigned_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: unassigned
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.index.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
          - action: add_label
            new_label: aggregation
            new_value: primary_shards
          - action: add_label
            new_label: elasticsearch.index.name
            new_value: _all
      - include: ^elasticsearch_indices_docs_deleted$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: deleted
      - include: ^elasticsearch_indices_fielddata_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_indices_fielddata_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_transport_rx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: received
      - include: ^elasticsearch_transport_tx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: sent
      - include: ^elasticsearch_filesystem_data_available_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.available
      - include: ^elasticsearch_filesystem_data_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.total
      - include: ^elasticsearch_process_open_files_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.open_files
      - include: ^elasticsearch_indices_indexing_index_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: index
      - include: ^elasticsearch_indices_search_query_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: query
      - include: ^elasticsearch_indices_get_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: get
      - include: ^elasticsearch_indices_merges_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: merge
      - include: ^elasticsearch_indices_indexing_index_time_seconds_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: index
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_indices_search_query_time_seconds$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: query
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_thread_pool_active_count$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.operations.current
        operations:
          - action: update_label
            label: type
            new_label: operation
      - include: ^elasticsearch_os_cpu_percent$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.usage
      - include: ^elasticsearch_os_load1$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.1m
      - include: ^elasticsearch_os_load5$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.5m
      - include: ^elasticsearch_os_load15$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.15m
      - include: ^elasticsearch_jvm_gc_collection_seconds_count$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.count
      - include: ^elasticsearch_jvm_gc_collection_seconds_sum$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.elapsed
        operations:
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_jvm_memory_max_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.max
      - include: ^elasticsearch_jvm_memory_used_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.used
      - include: ^elasticsearch_thread_pool_threads_count$
        match_type: regexp
        action: update
        new_name: jvm.threads.count
        operations:
          - action: aggregate_labels
            label_set: [name, cluster]
            aggregation_type: sum
      - include: ^elasticsearch_indices_segments_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.index.segments.count

  filter/allowed_metrics:
    metrics:
      include:
        match_type: strict
        metric_names:
          - elasticsearch.cluster.data_nodes
          - elasticsearch.cluster.health
          - elasticsearch.cluster.in_flight_fetch
          - elasticsearch.cluster.nodes
          - elasticsearch.cluster.pending_tasks
          - elasticsearch.cluster.shards
          - elasticsearch.index.documents
          - elasticsearch.index.segments.count
          - elasticsearch.node.cache.evictions
          - elasticsearch.node.cache.memory.usage
          - elasticsearch.node.cluster.io
          - elasticsearch.node.documents
          - elasticsearch.node.fs.disk.available
          - elasticsearch.node.fs.disk.total
          - elasticsearch.node.open_files
          - elasticsearch.node.operations.completed
          - elasticsearch.node.operations.current
          - elasticsearch.node.operations.time
          - elasticsearch.os.cpu.usage
          - elasticsearch.os.cpu.load_avg.1m
          - elasticsearch.os.cpu.load_avg.5m
          - elasticsearch.os.cpu.load_avg.15m
          - jvm.gc.collections.count
          - jvm.gc.collections.elapsed
          - jvm.memory.heap.max
          - jvm.memory.heap.used
          - jvm.threads.count

  transform/rename_node_name:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["elasticsearch.node.name"], attributes["name"]) where attributes["name"] != nil
          - delete_key(attributes, "name") where attributes["elasticsearch.node.name"] != nil

  groupbyattrs/node:
    keys:
      - elasticsearch.node.name
      - elasticsearch.cluster.name

  cumulativetodelta: {}

  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert

  transform/scope_override:
    metric_statements:
      - context: scope
        statements:
          - set(name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver")
          - set(version, "")

  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true

  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096

  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete

  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")

exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics/elasticsearch:
      receivers: [prometheus/elasticsearch]
      processors:
        - memory_limiter
        - metricstransform/prom_to_otel
        - filter/allowed_metrics
        - transform/rename_node_name
        - groupbyattrs/node
        - transform/scope_override
        - resourcedetection
        - resource/cluster_name_override
        - attributes/cardinality_reduction
        - cumulativetodelta
        - transform/metadata_nullify
        - batch
      exporters: [otlphttp]

Replace the following values in the configuration:

<elasticsearch-cluster-name>: Your Elasticsearch cluster name for identification in New Relic.
localhost:9114: The address of your elasticsearch_exporter if running on a different host or port.

Set up environment variables

Create a systemd override to inject the required environment variables. Replace <collector-service> with your collector service name (nrdot-collector or otelcol-contrib):

bash

$sudo mkdir -p /etc/systemd/system/<collector-service>.service.d

Create the file /etc/systemd/system/<collector-service>.service.d/environment.conf:

[Service]
Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
Environment="NEWRELIC_LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY"
Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"

Replace YOUR_NEWRELIC_LICENSE_KEY with your .

Tip

For EU accounts, use NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318

Start the collector

Replace <collector-service> with your collector service name (nrdot-collector or otelcol-contrib):

bash

$sudo systemctl daemon-reload
$sudo systemctl enable <collector-service>
$sudo systemctl restart <collector-service>

Check the collector status:

bash

$sudo systemctl status <collector-service>
$sudo journalctl -u <collector-service> -f

Verify data in New Relic

After a few minutes, verify that data is flowing to New Relic:

FROM Metric SELECT count(*)
WHERE metricName LIKE 'elasticsearch.%'
AND elasticsearch.cluster.name = '<elasticsearch-cluster-name>'
SINCE 10 minutes ago

Tip

Correlate APM with Elasticsearch: To connect your APM application and Elasticsearch cluster, include the resource attribute es.cluster.name="your-cluster-name" in your APM metrics. This enables cross-service visibility and faster troubleshooting within New Relic.

View your Elasticsearch data

Once the collector is running and sending data, you can explore your Elasticsearch metrics, create custom queries, and set up monitoring dashboards in New Relic.

For detailed information on accessing your data, writing NRQL queries, and configuring alerts, see Find and query Elasticsearch data.

Troubleshooting

If you encounter issues during installation or don't see data in New Relic, see our comprehensive troubleshooting guide for step-by-step solutions to common problems.

Monitor self-hosted Elasticsearch with OpenTelemetry

Installation options .css-21sua1{background:none;border:none;width:0;padding:0;}

Before you begin

Configure Elasticsearch monitoring

Tip

Important

Verify data collection

Before you begin

Important

Configure Elasticsearch monitoring

Tip

Important

Verify data collection

Tip

Install and configure the Elasticsearch exporter

Tip

Configure the collector

Set up environment variables

Tip

Start the collector

Verify data in New Relic

Tip

View your Elasticsearch data

Troubleshooting

Installation options