AMD GPU integration

Our AMD GPU integration uses our infrastructure agent with the Flex integration, which lets us access AMD's SMI utility.

After you set up our AMD GPU integration, you see a dashboard for your GPU metrics.

When you install, you'll get a pre-built dashboard containing crucial GPU metrics:

GPU utilization
Memory usage and availability
Active compute processes
Clock frequencies
Temperature monitoring
Power consumption
Performance states
Dynamic and static information about each supported device

Install the infrastructure agent

To capture data with New Relic, install our infrastructure agent. Our infrastructure agent collects and ingests data so you can keep track of your GPU performance.

You can install the infrastructure agent two different ways:

Our guided install is a CLI tool that inspects your system and installs the infrastructure agent alongside the application monitoring agent that best works for your system. To learn more about how our guided install works, refer to Guided install overview.
If you'd rather install our infrastructure agent manually, follow the tutorial for manual installation for Linux, Windows.

Install AMD SMI

Before configuring the Flex integration, install AMD's SMI utility:

Ubuntu/Debian:

bash

$# Add AMD GPU repository
$wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -
$echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list
$
$# Update and install
$sudo apt update
$sudo apt install amd-smi

RHEL/CentOS/Fedora:

bash

$# Add AMD GPU repository
$sudo tee /etc/yum.repos.d/rocm.repo <<EOF
$[ROCm]
$name=ROCm
$baseurl=https://repo.radeon.com/rocm/yum/rpm
$enabled=1
$gpgcheck=1
$gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
$EOF
$
$# Install AMD SMI
$sudo yum install amd-smi

Important

After installation, ensure the amd-smi binary is accessible in your system PATH and your user has proper permissions to access GPU devices.

Configure Flex integration for AMD GPUs

Flex comes bundled with the New Relic infrastructure agent and can be integrated with AMD SMI, a command line utility to monitor AMD GPU devices.

Follow these steps to configure Flex:

Create a file named amd-smi-gpu-monitoring.yml in this path:
bash
```
$sudo touch /etc/newrelic-infra/integrations.d/amd-smi-gpu-monitoring.yml
```
You may also download from the git repository.
Update the amd-smi-gpu-monitoring.yml file with the integration config:
Copy the file content (.yml) from the AMD SMI GPU monitoring configuration and paste it into your configuration file.

Confirm GPU metrics are being ingested

The Flex configuration will be automatically detected and executed by the infrastructure agent, there's no need to restart the agent. You can confirm metrics are being ingested by running this NRQL query:

SELECT * FROM AMDGpuSample

Monitor your application

You can create custom dashboards to monitor your AMD GPU metrics. Here are some example NRQL queries to get you started:

GPU Utilization

SELECT average(gfx_activity_percent) FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

Memory Usage

SELECT average(vram_used_mb), average(vram_total_mb), average(vram_free_mb) FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

Temperature Monitoring

SELECT average(temp_edge_c), average(temp_hotspot_c), average(temp_mem_c) 
FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

Power Consumption

SELECT average(socket_power_w), max(max_power_w) FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

Clock Frequencies

SELECT average(gfx_0_clk_mhz), average(mem_0_clk_mhz), average(fclk_0_clk_mhz) FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

GPU Activity by Type

SELECT average(gfx_activity_percent), average(umc_activity_percent), average(mm_activity_percent) 
FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

ECC Error Monitoring

SELECT sum(ecc_total_correctable), sum(ecc_total_uncorrectable) FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

PCIe Performance

SELECT average(pcie_bandwidth_mb_s), average(pcie_current_bandwidth_sent), average(pcie_current_bandwidth_received) 
FROM AMDGpuSample TIMESERIES AUTO FACET gpu_id

To create a comprehensive dashboard:

Go to one.newrelic.com and click on Dashboards.
Click Create a dashboard.
Add charts using the NRQL queries provided above.
Customize your dashboard layout and save it.

For docs on using and editing dashboards, refer to our dashboard docs.

Troubleshooting

Common issues and solutions:

AMD SMI not found:

Ensure AMD SMI is properly installed and accessible in your PATH
Verify AMD SMI with: amd-smi version

Permission denied errors:

Ensure your user has proper permissions to access GPU devices
You may need to add your user to the render group:
bash
```
$sudo usermod -a -G render $USER
```
Log out and log back in for changes to take effect

No GPU detected:

Verify AMD GPU is detected by the system: lspci | grep -i amd
Check if amdgpu driver is loaded: lsmod | grep amdgpu
Ensure your AMD GPU is supported by AMD SMI

Missing metrics:

Some older AMD GPUs may not support all sensors
Check which metrics are available: amd-smi metric --help
Verify GPU compatibility: amd-smi list

What's next?

You can adapt the Flex configuration to include or exclude information available from the AMD SMI utility. Additional AMD SMI options include:

amd-smi list - List available GPU devices
amd-smi static - Static GPU information
amd-smi metric - Real-time GPU metrics
amd-smi process - Process information using GPU
amd-smi monitor - Continuous monitoring mode

To learn more about building NRQL queries and generating dashboards, refer to these docs:

Introduction to the query builder to create basic and advanced queries.
Introduction to dashboards to customize your dashboard and carry out different actions.
Manage your dashboard to adjust your display mode, or to add more content to your dashboard.

Install the infrastructure agent.css-21sua1{background:none;border:none;width:0;padding:0;}