For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/pl/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/pl/llms-full.txt, and this page is available as Markdown at https://docs.ovhcloud.com/pl/guides/public-cloud/containers-orchestration/managed-kubernetes/deploy-gpu-application.md.
Przejdź do treści

Deploying a GPU application on OVHcloud Managed Kubernetes Service

Pokaż jako Markdown

Find out how to deploy a GPU application on OVHcloud Managed Kubernetes

Objective

In this tutorial we will show you how to deploy a GPU application on an OVHcloud Managed Kubernetes cluster.

GPUs provide compute power to drive AI/ML & Deep Learning tasks with intensive calculations such as image recognition, natural language processing (NLP), as well as other compute-intensive tasks such as video transcoding and image processing.

Using GPUs with Kubernetes allows you to extend the scalability of Kubernetes to AI/ML applications.

Before you begin

This tutorial presupposes that you already have a working OVHcloud Managed Kubernetes cluster, and some basic knowledge of how to operate it. If you want to know more on those topics, please look at the OVHcloud Managed Kubernetes Service Quickstart.

You also need to have Helm installed on your workstation and your cluster. Please refer to the How to install Helm on OVHcloud Managed Kubernetes Service tutorial.

Prerequisites

To get the best and most cost-effective benefit from GPUs on OVHcloud Managed Kubernetes, and to take advantage of cluster autoscaling, we recommend that you create separate GPU node pools in your Kubernetes clusters.

GPU node pools group nodes of the same GPU flavor and can be configured with autoscaling to handle variable workloads efficiently.

The GPU operator that you will install should run on Kubernetes nodes that are equipped with GPUs.

Create a node pool that uses a GPU flavor. The worker node flavors list shows the GPU flavors and their regions, and the Public Cloud prices page shows the GPU model of each flavor. The driver installed later depends on the architecture:

  • Current architectures: A100-180, H100-*, L4-*, L40S-* and RTX5000-* (Ampere, Hopper, Ada and Turing).
  • Tesla V100 and V100S: T1-* and T2-* (Volta).

If you already have an OVHcloud Kubernetes cluster with a GPU node pool, you can skip the following Autoscaling GPU Node Pool creation step.

Warning

GPU Virtual machines are extremely powerful and significantly more expensive than general purpose nodes. If you are usually using OVHcloud Public Cloud at small scale and/or are a new user, you may need to raise your Public Cloud quota to enable access to those type of machines for your project. Also note that these machines are only available in some of our Public Cloud regions.

Also note that while worker nodes (be them added manually or through cluster autoscaler) are created within a few minutes, our GPU worker nodes ready status can take up to a bit more than one hour.


OVHcloud Console Access

  • Direct link:
  • Navigation path: Public Cloud > Select your project > Containers > Managed Kubernetes Service

Instructions

Autoscaling GPU Node Pool creation

Click on the name of your cluster, then on the Node pools tab.

We will create our special GPU Node Pool. Click on Add a node pool.

On the Create a node pool page, enter a Node pool name for your GPU node pool (for example, gpu-node-pool).

On the flavor selection step, click the GPU tab. For a current GPU, choose a flavor such as H100, L4, L40S, A100-180 or RTX5000. For a Tesla V100 or V100S, choose a T1 or T2 flavor, for example T1-45.

Set the Number of nodes and, if desired, enable Autoscaling and define the minimum and maximum pool size (for example, 3 minimum and 10 maximum).

Select Hourly as the billing mode for this GPU node pool, then click Create Node pool.

The node pool creation is now in progress. It should be available within a few minutes in the OVHcloud Console.

kubectl get node --show-labels | grep "node.k8s.ovh/type=gpu"

Once the node pool status shows Ready in the OVHcloud Console, you can confirm the GPU nodes are correctly labelled using the command above.

NVIDIA GPU Operator

The NVIDIA GPU Operator uses the Operator Framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPU. These components include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin for GPUs, the NVIDIA Container Runtime, automatic node labelling, DCGM based monitoring and others.

NVIDIA GPU Kubernetes Operator

If you are interested about the operator, feel free to read the NVIDIA GPU operator official documentation.

Installing the NVIDIA GPU Operator Helm chart

For this tutorial we are using the NVIDIA GPU Operator Helm chart found on NVIDIA repository. The chart is fully configurable. The commands below set only the driver options required for the GPU architecture of your node pool.

Add the NVIDIA Helm repository:

Info

The Nvidia Helm chart has moved. If you already added a repo with the name nvidia, you can remove it: helm repo remove nvidia.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

This will add the NVIDIA repository and update all of your repositories:

$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
"nvidia" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nvidia" chart repository
[...]
Update Complete. ⎈Happy Helming!⎈

Install the GPU Operator in the gpu-operator namespace. Pick the tab that matches the GPU flavor of the node pool. By default, the operator compiles the driver modules for the node kernel.

L4, L40S, H100, A100
RTX 5000
Tesla V100 and V100S (T1, T2)

On the NGC driver catalog, select the latest Ubuntu 22.04 tag of the form <version>-ubuntu22.04. Pass that version to driver.version (e.g. 595.91.07).

helm install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.version=<version>

To skip the compilation, use a precompiled driver image instead. NVIDIA publishes these images for some Ubuntu 22.04 kernels only. Read the kernel of each GPU node:

kubectl get nodes -l node.k8s.ovh/type=gpu -o custom-columns=NODE:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion

On the NGC driver catalog, look for a tag of the form <branch>-<kernel>-ubuntu22.04 for each of these kernels (e.g. 595-5.15.0-187-generic-ubuntu22.04), and pass the branch to driver.version (e.g. 595). If one of the kernels has no tag, keep the command above: the GPUs of that node stay unavailable otherwise. In an autoscaled node pool, new nodes can start on a newer kernel that has no tag yet.

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.usePrecompiled=true \
  --set driver.version=<branch>
Info

If the cluster mixes Tesla V100 nodes with other GPU flavors, use the Tesla V100 command: that driver branch also supports the current architectures. If one of the flavors is RTX 5000, also create the ConfigMap and add the driver.kernelModuleType and driver.kernelModuleConfig.name options from the RTX 5000 tab.

If the cluster mixes RTX 5000 nodes with L4, L40S, H100 or A100 nodes, use the RTX 5000 command.

Compiling the modules takes a few minutes. The nvidia-driver-daemonset pod stays not ready until the build finishes.

You should have a GPU operator installed and running:

$ kubectl get pod -n gpu-operator
NAME                                                          READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-8xzzw                                   1/1     Running     0          22m
gpu-feature-discovery-kxtlh                                   1/1     Running     0          22m
gpu-feature-discovery-wdvr7                                   1/1     Running     0          22m
gpu-operator-689dbf694b-clz7f                                 1/1     Running     0          23m
gpu-operator-node-feature-discovery-master-7db9bfdd5b-9w2hj   1/1     Running     0          23m
gpu-operator-node-feature-discovery-worker-2wpmm              1/1     Running     0          23m
gpu-operator-node-feature-discovery-worker-4bsn7              1/1     Running     0          23m
gpu-operator-node-feature-discovery-worker-9klx5              1/1     Running     0          23m
gpu-operator-node-feature-discovery-worker-gn62n              1/1     Running     0          23m
gpu-operator-node-feature-discovery-worker-hdzpx              1/1     Running     0          23m
nvidia-container-toolkit-daemonset-hvx6x                      1/1     Running     0          22m
nvidia-container-toolkit-daemonset-lhmxn                      1/1     Running     0          22m
nvidia-container-toolkit-daemonset-tjrb2                      1/1     Running     0          22m
nvidia-cuda-validator-fcfwn                                   0/1     Completed   0          18m
nvidia-cuda-validator-mdbml                                   0/1     Completed   0          18m
nvidia-cuda-validator-sv979                                   0/1     Completed   0          17m
nvidia-dcgm-exporter-fvn8h                                    1/1     Running     0          22m
nvidia-dcgm-exporter-mt5qh                                    1/1     Running     0          22m
nvidia-dcgm-exporter-n65kl                                    1/1     Running     0          22m
nvidia-device-plugin-daemonset-hwc95                          1/1     Running     0          22m
nvidia-device-plugin-daemonset-wr5td                          1/1     Running     0          22m
nvidia-device-plugin-daemonset-zzzkm                          1/1     Running     0          22m
nvidia-device-plugin-validator-4k5wd                          0/1     Completed   0          17m
nvidia-device-plugin-validator-rjkzd                          0/1     Completed   0          17m
nvidia-device-plugin-validator-swdrr                          0/1     Completed   0          17m
nvidia-driver-daemonset-2jsmv                                 1/1     Running     0          22m
nvidia-driver-daemonset-5zq44                                 1/1     Running     0          22m
nvidia-driver-daemonset-v6qgx                                 1/1     Running     0          22m
nvidia-operator-validator-kk6nd                               1/1     Running     0          22m
nvidia-operator-validator-m9p9k                               1/1     Running     0          22m
nvidia-operator-validator-s6czx                               1/1     Running     0          22m

Verify GPU Operator Install

When the GPU operator is UP and running, it will add nvidia.com/gpu labels on GPU Nodes.

You can check new labels with the command:

kubectl get nodes --show-labels | grep "nvidia.com/gpu"

New labels should appear only on your GPU Nodes:

GPU Nodes NVIDIA labels

Running Sample GPU Applications

You can now proceed to running a GPU application/workload on your cluster.

To configure Pods to consume GPUs, you need to use a resource limit in your YAML manifest file. You have to specify a resource limit in a Pod specification using the following key-value pair:

Key: nvidia.com/gpu
Value: Number of GPUs to consume

Create a vector.yaml YAML manifest file with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1"
    resources:
      limits:
         nvidia.com/gpu: 1

Apply it:

kubectl apply -f vector.yaml -n default

And watch the Pod startup:

kubectl get pod -n default -w

You should have results like this:

$ kubectl apply -f vector.yaml -n default
pod/cuda-vectoradd created

$ kubectl get pod -n default -w
NAME             READY   STATUS              RESTARTS   AGE
cuda-vectoradd   0/1     ContainerCreating   0          4s
cuda-vectoradd   0/1     Completed   0          35s

When the cuda-vectoradd has started, run and completed its task, watch the logs with the following command:

kubectl logs cuda-vectoradd -n default
$ kubectl logs cuda-vectoradd -n default
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Our first GPU workload is just started up and has done its task in our OVHcloud Managed Kubernetes cluster.

Troubleshooting

If the nvidia-driver-daemonset pod stays not ready after the build, run nvidia-smi in it:

kubectl exec -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-driver-ctr -- nvidia-smi

If it reports No devices were found, read the kernel log:

kubectl exec -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-driver-ctr -- sh -c 'dmesg | grep -m 20 NVRM'

Lines such as Cannot initialize GSP firmware RM, kgspWaitForGfwBootOk, RmFetchGspRmImages: No firmware image found or RmInitAdapter failed mean that the GPU firmware did not start. This happens on RTX 5000 nodes installed without the options of the RTX 5000 tab, or with a driver of the 615 branch or later. Uninstall the operator, then install it with the command from the RTX 5000 tab.

Go further

To learn more about using your Kubernetes cluster the practical way, we invite you to look at our OVHcloud Managed Kubernetes documentation.

For every GPU Operator option, such as driver configuration, upgrades and monitoring, read the NVIDIA GPU Operator documentation.

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

Join our community of users.

Czy ta strona była pomocna?