Chainguard Container for dcgm-exporter

Chainguard Containers are regularly-updated, secure-by-default container images.

Download this Container Image

For those with access, this container image is available on cgr.dev:

docker pull cgr.dev/ORGANIZATION/dcgm-exporter:latest

Be sure to replace the ORGANIZATION placeholder with the name used for your organization's private repository within the Chainguard Registry.

Compatibility Notes

The Chainguard dcgm-exporter-fips image is designed to be a drop-in replacement for the upstream NVIDIA/dcgm-exporter image, with an important difference: The upstream image contains an entrypoint script which adds cap_sys_admin=+ep to the dcgm-exporter binary at runtime when --cap-add SYS_ADMIN is passed to the container. This Entrypoint script needs packages like bash and libcap-utils which increases the attack surface area of the image. The Chainguard image excludes these extra packages as well as the entrypoint script while ensuring no changes in the Image behaviour.

This image contains 2 different tags:

• Why is SYS_ADMIN capability required? The requirement for SYS_ADMIN privileges comes from DCGM, while the dcgm-exporter binary is simply a client. The specific use cases for how and where DCGM (nv-hostengine) and dcgm-exporter (binary) run and communicate can vary. It is required for following cases:
  • DCP metrics or DCGM Profiling metrics require root privileges unless configured otherwise in the kernel parameters.
  • MIG-enabled systems
• All components are minimal, distroless images with low-to-zero CVEs

Usage

DCGM-Exporter is a tool based on the Go APIs to NVIDIA DCGM that allows users to gather GPU metrics and understand workload behavior or monitor GPUs in clusters. DCGM Exporter is written in Go and exposes GPU metrics at an HTTP endpoint (/metrics) for monitoring solutions such as Prometheus.

To test the functionality of NVIDIA DCGM Exporter Image, it requires an environment with connected GPUs. If you have connected GPUs, here's one way to use this image:

Using Docker

Run Image

Install Docker Engine and configure it with your credentials to pull image

Run the image:

docker run -d --rm \
   --gpus all \
   --net host \
   --cap-add SYS_ADMIN \
   cgr.dev/chainguard/dcgm-exporter:latest \
   -f /etc/dcgm-exporter/dcp-metrics-included.csv

Retreive the metrics

$ curl localhost:9400/metrics

Output should be something like this

# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge
...
DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 139
DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 405
DCGM_FI_DEV_MEMORY_TEMP{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 9223372036854775794
...

Helm Installation

Step 1: Add and Update Helm Repository Add the NVIDIA DCGM Exporter repository and update it to ensure you have access to the latest charts.

$ helm repo add gpu-helm-charts \
   https://nvidia.github.io/dcgm-exporter/helm-charts

$ helm repo update

Step 2: Install NVIDIA DCGM Exporter

Install NVIDIA DCGM Exporter using Helm with the specified version, namespace, and optional configuration settings.

$ helm install \
  --generate-name \
  gpu-helm-charts/dcgm-exporter \
  --set image.repository=cgr.dev/chainguard/dcgm-exporter \
  --set image.tag=latest

Step 3: Verify Installation

$ kubectl get pods -A

NAMESPACE     NAME                                                              READY   STATUS      RESTARTS   AGE
default       dcgm-exporter-2-1603213075-w27mx                                  1/1     Running     0          2m18s
kube-system   calico-kube-controllers-8f59968d4-g28x8                           1/1     Running     1          43m
kube-system   calico-node-zfnfk                                                 1/1     Running     1          43m
kube-system   coredns-f9fd979d6-p7djj                                           1/1     Running     1          43m
kube-system   coredns-f9fd979d6-qhhgq                                           1/1     Running     1          43m
kube-system   etcd-ip-172-31-92-253                                             1/1     Running     1          43m
kube-system   kube-apiserver-ip-172-31-92-253                                   1/1     Running     2          43m
kube-system   kube-controller-manager-ip-172-31-92-253                          1/1     Running     1          43m
kube-system   kube-proxy-mh528                                                  1/1     Running     1          43m
kube-system   kube-scheduler-ip-172-31-92-253                                   1/1     Running     1          43m
kube-system   nvidia-device-plugin-1603211071-7hlk6                             1/1     Running     0          35m
prometheus    alertmanager-kube-prometheus-stack-1603-alertmanager-0            2/2     Running     0          33m
prometheus    kube-prometheus-stack-1603-operator-6b95bcdc79-wmbkn              2/2     Running     0          33m
prometheus    kube-prometheus-stack-1603211794-grafana-67ff56c449-tlmxc         2/2     Running     0          33m
prometheus    kube-prometheus-stack-1603211794-kube-state-metrics-877df67c49f   1/1     Running     0          33m
prometheus    kube-prometheus-stack-1603211794-prometheus-node-exporter-b5fl9   1/1     Running     0          33m
prometheus    prometheus-kube-prometheus-stack-1603-prometheus-0                3/3     Running     1          33m

For more information and setting it up with prometheus stack, refer to the official documentation:

• Helm Installation Guide

What are Chainguard Containers?

Chainguard's free tier of Starter container images are built with Wolfi, our minimal Linux undistro.

All other Chainguard Containers are built with Chainguard OS, Chainguard's minimal Linux operating system designed to produce container images that meet the requirements of a more secure software supply chain.

The main features of Chainguard Containers include:

• Minimal design, without unnecessary software bloat
• Daily builds to ensure container images are up-to-date with available security patches
• High quality build-time SBOMs attesting to the provenance of all artifacts within the image
• Verifiable signatures provided by Sigstore
• Reproducible builds with Cosign and apko (read more about reproducibility)

For cases where you need container images with shells and package managers to build or debug, most Chainguard Containers come paired with a development, or -dev, variant.

In all other cases, including Chainguard Containers tagged as :latest or with a specific version number, the container images include only an open-source application and its runtime dependencies. These minimal container images typically do not contain a shell or package manager.

Although the -dev container image variants have similar security features as their more minimal versions, they include additional software that is typically not necessary in production environments. We recommend using multi-stage builds to copy artifacts from the -dev variant into a more minimal production image.

Need additional packages?

To improve security, Chainguard Containers include only essential dependencies. Need more packages? Chainguard customers can use Custom Assembly to add packages, either through the Console, chainctl, or API.

To use Custom Assembly in the Chainguard Console: navigate to the image you'd like to customize in your Organization's list of images, and click on the Customize image button at the top of the page.

Learn More

Refer to our Chainguard Containers documentation on Chainguard Academy. Chainguard also offers VMs and Libraries — contact us for access.

Trademarks

This software listing is packaged by Chainguard. The trademarks set forth in this offering are owned by their respective companies, and use of them does not imply any affiliation, sponsorship, or endorsement by such companies.