Configuring multi-attach persistent volumes with OVHcloud Cloud Disk Array

Objective

OVHcloud Managed Kubernetes natively integrates Block Storage as persistent volumes. This technology may however not be suited to some legacy or non cloud-native applications, often requiring to share this persistent data accross different pods on multiple worker nodes (ReadWriteMany or RWX). If you would need to do this for some of your workloads, one solution is to use CephFS volumes.
OVHcloud Cloud Disk Array is a managed solution that lets you easily configure a Ceph cluster and multiple CephFS volumes. In this tutorial we are going to see how to configure your OVHcloud Managed Kubernetes cluster to use OVHcloud Cloud Disk Array as a CephFS provider for Kubernetes Persistent Volumes.

Requirements

This tutorial assumes that you already have a working OVHcloud Managed Kubernetes cluster, and some basic knowledge of how to operate it. If you want to know more on those topics, please look at the deploying a Hello World application documentation.

It also assumes you have an OVHcloud Cloud Disk Array already available. If you don't, you can order one in the OVHcloud Control Panel.

You also need to have Helm installed on your workstation, please refer to the How to install Helm on OVHcloud Managed Kubernetes Service tutorial.

OVHcloud Control Panel Access

• Direct link: All my Public Cloud projects
• Navigation path: Public Cloud > Select your project > Gateway

Instructions

To configure OVHcloud Cloud Disk Array, you need to use the OVHcloud API. If you have never used it, you can find the basics here: First steps with the OVHcloud API.

Step 1 - Creating a partition and granting your Managed Kubernetes Service access to it

• List you available Cloud Disk Array cluster:

The response returns the list of your Cloud Disk Array service names. Note the serviceName value you want to use for the following steps.

• Create a file system on your Cloud Disk Array:

Fill in serviceName with your Cloud Disk Array service name, and choose a name for fsName (e.g. fs-default). A successful call returns an empty response with HTTP 200.

• Create a user for the CephFS CSI that will be used by your Kubernetes cluster:

Set userName to a value such as ceph-csi. Note the user name for the following steps.

• Add permissions on fs-default for the Ceph CSI user:

Use the userName you created in the previous step and set the pool name to match your filesystem (e.g. fs-default). Grant read and write permissions.

Step 2 - Allow Kubernetes Nodes' IPs and/or Public Cloud Gateway IP to the Cloud Disk Array service

Your cluster is installed with Public Network or a private network without using an OVHcloud Internet Gateway or a custom one as your default route

Once the partition is created, we need to allow our Kubernetes nodes to access our newly created partition.

Get our Kubernetes nodes IP:

kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'

$ kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'
51.77.204.175 51.77.205.79

Your cluster is installed with Private Network and a default route via your Private Network (OVHcloud Internet Gateway/OpenStack Router or a custom one)

Because your nodes are configured to be routed by the private network gateway, you need to add the gateway IP address to the ACLs.

By using Public Cloud Gateway through our Managed Kubernetes Service, Public IPs on nodes are only for management purposes: MKS Known Limits

You can get your OVHcloud Internet Gateway's Public IP by navigating through the OVHcloud Control Panel:

Public Cloud > Select your tenant > Network / Gateway > Public IP

You can also use the following API endpoint to retrieve your OVHcloud Internet Gateway's Public IP:

If you want to use your Kubernetes cluster to retrieve your Gateway Public's IP, run this command:

kubectl run get-gateway-ip --image=ubuntu:latest -i --tty --rm

This will create a temporary pod and open a console.

You may have to wait a little for the pod to be created. Once the shell appears, you can run this command:

apt update && apt upgrade -y && apt install -y curl && curl ifconfig.ovh

This command will output the Public IP of the Gateway of your kubernetes cluster.

• Add the list of nodes IP or the Gateway IP to allow access to the Cloud Disk Array cluster:

Repeat this call for each node IP (or your gateway IP) you retrieved above. Each call adds one IP address to the allowlist.

• Get the key for the CephFS CSI user:

The response includes a key field. Copy this value — you will use it as <your_ceph_csi_user_key> in the configuration steps below.

• Get the Ceph monitors IP:

The response includes a monitors array listing the IP addresses of your Ceph monitor nodes. Note these values — you will use them as <your_ceph_monitor_ip_1>, <your_ceph_monitor_ip_2>, and <your_ceph_monitor_ip_3> in the configuration steps below.

• Create a Ubuntu pod on your Kubernetes cluster

kubectl run -it ubuntu --image=ubuntu

• Install the Ceph client and vim inside the Ubuntu pod:

apt-get update
apt-get install ceph-common vim

• Configure the Ceph client with the monitors IP and the key:

vim ceph.conf

[global]
mon_host = <your_ceph_monitor_ip_1>,<your_ceph_monitor_ip_2>,<your_ceph_monitor_ip_3>
keyring = /root/ceph.client.ceph-csi.keyring

vim /root/ceph.client.ceph-csi.keyring

[client.ceph-csi]
key = <your_ceph_csi_user_key>

• Test the Ceph client configuration:

ceph --conf ceph.conf --id ceph-csi fs ls

ceph --conf ceph.conf --id ceph-csi fs ls
name: fs-default, metadata pool: cephfs.fs-default.meta, data pools: [cephfs.fs-default.data ]

• Add a subvolumegroup to fs-default:

ceph --conf ceph.conf --id ceph-csi fs subvolumegroup create fs-default csi

• Exit the Ubuntu pod:

exit

• Destroy the Ubuntu pod:

kubectl delete pod ubuntu

Step 2 - Configuring Kubernetes to use our newly created CephFS partition

• Install the Ceph CSI helmchart repository:

helm repo add ceph-csi https://ceph.github.io/csi-charts

• Create a configuration file for the Ceph CSI helmchart:

vim values.yaml

csiConfig:
  - clusterID: "abcd123456789" # You can change this, but it needs to have at least one letter character
    monitors:
      - "<your_ceph_monitor_ip_1>:6789"
      - "<your_ceph_monitor_ip_2>:6789"
      - "<your_ceph_monitor_ip_3>:6789"
storageClass:
  create: true
  name: "cephfs"
  clusterID: "abcd123456789" # This should be the ID you chose above
  fsName: "fs-default"
secret:
  create: true
  adminID: "ceph-csi"
  adminKey: "<your_ceph_csi_user_key>"

• Install the Ceph CSI on the Managed Kubernetes cluster:

helm install --namespace "ceph-csi-cephfs" "ceph-csi-cephfs" ceph-csi/ceph-csi-cephfs --create-namespace -f ./values.yaml

Step 3 - Create and use a CephFS persistent volume

Let’s create a cephfs-persistent-volume-claim.yaml file:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: cephfs
  resources:
    requests:
      storage: 1Gi

And apply this to create the persistent volume claim:

kubectl apply -f cephfs-persistent-volume-claim.yaml

Let's now create a DaemonSet, which will create pods on all available nodes in order to use our CephFS volume simultaneously on multiple nodes. Let’s create a cephfs-nginx-daemonset.yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: cephfs-nginx
    namespace: default
spec:
    selector:
      matchLabels:
        name: nginx
    template:
        metadata:
          labels:
            name: nginx
        spec:
          volumes:
          - name: cephfs-volume
            persistentVolumeClaim:
              claimName: cephfs-pvc
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
              name: "http-server"
            volumeMounts:
            - mountPath: "/usr/share/nginx/html"
              name: cephfs-volume

And apply this to create the Nginx pods:

kubectl apply -f cephfs-nginx-pods.yaml

Let’s enter inside the first Nginx container to create a file on the NFS persistent volume:

FIRST_POD=$(kubectl get pod -l name=nginx --no-headers=true -o custom-columns=:metadata.name | head -1)
kubectl exec -it $FIRST_POD -n default -- bash

Create a new index.html file:

echo "CEPHFS volume!" > /usr/share/nginx/html/index.html

And exit the Nginx container:

exit

Generate the URL to open in your browser:

$ URL=$(echo "http://localhost:8001/api/v1/namespaces/default/pods/http:$FIRST_POD:/proxy/")
echo $URL

You can open the displayed URL to access the Nginx Service.

Use the following command to validate that the filesystem is shared with the second pod (given that you have more than one node deployed).

$ SECOND_POD=$(kubectl get pod -l name=nginx --no-headers=true -o custom-columns=:metadata.name | head -2 | tail -1)
URL2=$(echo "http://localhost:8001/api/v1/namespaces/default/pods/http:$SECOND_POD:/proxy/")
echo $URL2

Let’s try to access our new web page:

kubectl proxy

Open both URLs generated by the commands above to see if the data is shared with all the pods connected to the Ceph volume.

As you can see the data is correctly shared between the two Nginx pods running on two different Kubernetes nodes. Congratulations, you have successfully set up a multi-attach persistent volume with OVHcloud Cloud Disk Array!

Go further

To learn more about using your Kubernetes cluster the practical way, we invite you to look at our OVHcloud Managed Kubernetes documentation.

• For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

• Join our community of users.