Managing and rebuilding software RAID on servers using legacy boot (BIOS) mode
Objective
Redundant Array of Independent Disks (RAID) is a technology that mitigates data loss on a server by replicating data across two or more disks.
The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space.
This guide explains how to manage and rebuild a software RAID in the event of a disk replacement on your server in legacy boot mode (BIOS).
Before we begin, please note that this guide focuses on Dedicated servers that use legacy boot (BIOS) mode. If your server uses the UEFI mode (newer motherboards), refer to this guide Managing and rebuilding software RAID on servers in UEFI boot mode.
To check whether a server runs on legacy BIOS or UEFI mode, run the following command:
[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
Requirements
- A Dedicated server with a software RAID configuration
- Administrative (sudo) access to the server via SSH
- Understanding of RAID and partitions
Instructions
When you purchase a new server, you may feel the need to perform a series of tests and actions. One such test could be to simulate a disk failure to understand the RAID rebuild process and prepare yourself in case it ever happens.
Content overview
In a command line session, type the following code to determine the current RAID status:
[user@server_ip ~]# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p2[1] nvme0n1p20]
931954688 blocks super 1.2 [2/2] [UU]
bitmap: 2/7 pages [8KB], 65536KB chunk
md4 : active raid1 nvme0n1p4[0] nvme1n1p4[1]
1020767232 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
This command shows us that we have two software RAID devices currently set up, with md4 being the largest one. The md4 RAID device consists of two partitions, which are known as nvme0n1p4 and nvme1n1p4.
The [UU] means that all the disks are working normally. A _ would indicate a failed disk.
If you have a server with SATA disks, you would get the following results:
[user@server_ip ~]# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[1] sdb2[0]
931954688 blocks super 1.2 [2/2] [UU]
bitmap: 2/7 pages [8KB], 65536KB chunk
md4 : active raid1 sda4[0] sdb4[1]
1020767232 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information using fdisk -l:
[user@server_ip ~]# sudo fdisk -l
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: HGST HUS724020AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F
Device Start End Sectors Size Type
/dev/sdb1 2048 4095 2048 1M BIOS boot
/dev/sdb2 4096 1864177663 1864173568 888.9G Linux RAID
/dev/sdb3 1864177664 1865226239 1048576 512M Linux filesystem
/dev/sdb4 1865226240 3907024895 2041798656 973.6G Linux RAID
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: HGST HUS724020AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2E1DCCBA-8808-4D2B-BA33-9FEC3B96ADA8
Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 1864177663 1864173568 888.9G Linux RAID
/dev/sda3 1864177664 1865226239 1048576 512M Linux filesystem
/dev/sda4 1865226240 3907024895 2041798656 973.6G Linux RAID
/dev/sda5 3907025072 3907029134 4063 2M Linux filesystem
Disk /dev/md4: 973.5 GiB, 1045265645568 bytes, 2041534464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/md2: 888.8 GiB, 954321600512 bytes, 1863909376 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
The fdisk -l command also allows you to identify your partition type. This is important information when rebuilding your RAID after a disk failure.
For GPT partitions, line 6 will display: Disklabel type: gpt. This information can only be seen when the server is in normal mode.
Still going by the results of fdisk -l, we can see that /dev/md2 consists of 888.8GB and /dev/md4 contains 973.5GB.
Alternatively, the lsblk command offers a different view of the partitions:
[user@server_ip ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 888.9G 0 part
│ └─md2 9:2 0 888.8G 0 raid1 /
├─sda3 8:3 0 512M 0 part [SWAP]
├─sda4 8:4 0 973.6G 0 part
│ └─md4 9:4 0 973.5G 0 raid1 /home
└─sda5 8:5 0 2M 0 part
sdb 8:16 0 1.8T 0 disk
├─sdb1 8:17 0 1M 0 part
├─sdb2 8:18 0 888.9G 0 part
│ └─md2 9:2 0 888.8G 0 raid1 /
├─sdb3 8:19 0 512M 0 part [SWAP]
└─sdb4 8:20 0 973.6G 0 part
└─md4 9:4 0 973.5G 0 raid1 /home
Note the devices, partitions, and mount points, as this is important, especially after replacing a disk. This will allow you to verify that the partitions are correctly mounted on their respective mount points on the new disk.
In our example, we have:
- Partitions part of md2 (
/): sda2 and sdb2.
- Partitions part of md4 (
/home): sda4 and sdb4.
- Swap partitions: sda3 and sdb3.
- BIOS boot partitions: sda1 and sdb1.
The sda5 partition is a config drive, i.e. a read-only volume that provides the server with its initial configuration data. It is only read once during initial boot and can be removed afterwards.
Simulating a disk failure
We now have all the necessary information to simulate a disk failure. In this example, we will fail the disk sda.
The preferred way to do this is via the OVHcloud rescue mode environment.
First reboot the server in rescue mode and log in with the provided credentials.
To remove a disk from the RAID, the first step is to mark it as Failed and remove the partitions from their respective RAID arrays.
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[1] sdb2[0]
931954688 blocks super 1.2 [2/2] [UU]
bitmap: 2/7 pages [8KB], 65536KB chunk
md4 : active raid1 sda4[0] sdb4[1]
1020767232 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
From the above output, sda consists of two partitions in RAID which are sda2 and sda4.
Removing the failed disk
First we mark the partitions sda2 and sda4 as Failed.
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # mdadm --manage /dev/md2 --fail /dev/sda2
# mdadm: set /dev/sda2 faulty in /dev/md2
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # mdadm --manage /dev/md4 --fail /dev/sda4
# mdadm: set /dev/sda4 faulty in /dev/md4
We have now simulated a failure of the RAID, when we run the cat /proc/mdstat command, we have the following output:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[1](F) sdb2[0]
931954688 blocks super 1.2 [2/2] [_U]
bitmap: 2/7 pages [8KB], 65536KB chunk
md4 : active raid1 sda4[0](F) sdb4[1]
1020767232 blocks super 1.2 [2/2] [_U]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty.
Next, we remove these partitions from the RAID arrays.
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # sudo mdadm --manage /dev/md2 --remove /dev/sda2
# mdadm: hot removed /dev/sda2 from /dev/md2
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # sudo mdadm --manage /dev/md4 --remove /dev/sda4
# mdadm: hot removed /dev/sda4 from /dev/md4
To simulate a clean disk, run the following command. Replace sda with your own values:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ #
shred -s10M -n1 /dev/sda1
shred -s10M -n1 /dev/sda2
shred -s10M -n1 /dev/sda3
shred -s10M -n1 /dev/sda4
shred -s10M -n1 /dev/sda
The disk now appears as a new, empty drive:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
sdb 8:16 0 1.8T 0 disk
├─sdb1 8:17 0 1M 0 part
├─sdb2 8:18 0 888.9G 0 part
│ └─md2 9:2 0 888.8G 0 raid1 /
├─sdb3 8:19 0 512M 0 part [SWAP]
└─sdb4 8:20 0 973.6G 0 part
└─md4 9:4 0 973.5G 0 raid1 /home
If we run the following command, we see that our disk has been successfully "wiped":
parted /dev/sda
GNU Parted 3.5
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: /dev/sda: unrecognised disk label
Model: HGST HUS724020AL (SATA)
Disk /dev/sda: 1.8T
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
Our RAID status should now look like this:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdb2[0]
931954688 blocks super 1.2 [1/2] [_U]
bitmap: 2/7 pages [8KB], 65536KB chunk
md4 : active raid1 sdb4[1]
1020767232 blocks super 1.2 [1/2] [_U]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk sda and we can proceed with the disk replacement.
For more information on how to prepare and request for a disk replacement, consult this guide.
If you run the following command, you can have more details on the RAID array(s):
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # mdadm --detail /dev/md4
/dev/md4:
Version : 1.2
Creation Time : Tue Jan 24 15:35:02 2023
Raid Level : raid1
Array Size : 1020767232 (973.48 GiB 1045.27 GB)
Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jan 24 16:28:03 2023
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : md4
UUID : 7b5c1d80:0a7ab4c2:e769b5e5:9c6eaa0f
Events : 21
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 20 1 active sync /dev/sdb4
Rebuilding the RAID
Info
This process may vary depending on the operating system installed on your server. We recommend that you consult the official documentation for your operating system to obtain the appropriate commands.
Warning
For most servers in software RAID, after a disk replacement, the server is able to boot in normal mode (on the healthy disk) to rebuild the RAID. However, if the server is not able to boot in normal mode, it will be rebooted in rescue mode to proceed with the RAID rebuild.
Rebuilding the RAID in normal mode
In our example, we have replaced the disk sda.
Once the replacement is done, the next step is to copy the partition table from the healthy disk (in this example, sdb) to the new one (sda).
sudo sgdisk -R /dev/sdX /dev/sdX
The command should be in this format: sgdisk -R /dev/newdisk /dev/healthydisk.
[user@server_ip ~]# sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX
The command should be in this format: sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk.
Once this is done, the next step is to randomise the GUID of the new disk to prevent GUID conflicts with other disks:
If you receive the following message:
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Run partprobe. If you still cannot see the newly-created partitions (e.g. with lsblk), you need to reboot the server before continuing.
Next, we add the partitions to the RAID:
[user@server_ip ~]# sudo mdadm --add /dev/md2 /dev/sda2
# mdadm: added /dev/sda2
[user@server_ip ~]# sudo mdadm --add /dev/md4 /dev/sda4
# mdadm: re-added /dev/sda4
Use the following command to monitor the RAID rebuild:
[user@server_ip ~]# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[0] sdb2[1]
931954688 blocks super 1.2 [2/2] [UU]
bitmap: 4/4 pages [16KB], 65536KB chunk
md4 : active raid1 sda4[0](F) sdb4[1]
1020767232 blocks super 1.2 [2/1] [UU]
[============>........] recovery = 64.8% (822969856/1020767232) finish=7.2min speed=401664K/sec
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
Lastly, we add a label and mount the [SWAP] partition (if applicable).
To add a label to the SWAP partition:
[user@server_ip ~]# sudo mkswap /dev/sda4 -L swap-sda4
Next, retrieve the UUIDs of both SWAP partitions:
[user@server_ip ~]# sudo blkid -s UUID /dev/sda4
/dev/sda4: UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15"
[user@server_ip ~]# sudo blkid -s UUID /dev/sdb4
/dev/sdb4: UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a"
We replace the old UUID of the SWAP partition (sda4) with the new one in the /etc/fstab file.
Example:
[user@server_ip ~]# sudo nano /etc/fstab
UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1
UUID=f925a033-0087-40ec-817e-44efab0351ac /boot ext4 defaults 0 0
LABEL=BIOS /boot vfat defaults 0 1
UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0
UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0
Based on the above results, the old UUID is b7b5dd38-9b51-4282-8f2d-26c65e8d58ec and should be replaced with the new one b3c9e03a-52f5-4683-81b6-cc10091fcd15.
Make sure you replace the correct UUID.
Next, we verify that everything is properly mounted with the following command:
[user@server_ip ~]# sudo mount -av
/ : successfully mounted
/home : successfully mounted
swap : ignored
swap : ignored
We run the following command to enable the SWAP partition:
[user@server_ip ~]# sudo swapon -av
Then reload the system with the following command:
[user@server_ip ~]# sudo systemctl daemon-reload
The RAID rebuild is complete.
Rebuilding the RAID in rescue mode
If your server is unable to reboot in normal mode after a disk replacement, it will be rebooted in rescue mode by our datacentre team.
In this example, we have replaced the disk sdb.
Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sda) to the new one (sdb).
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # sgdisk -R /dev/sdX /dev/sdX
The command should be in this format: sgdisk -R /dev/newdisk /dev/healthydisk
Example:
sudo sgdisk -R /dev/sdb /dev/sda
sudo sfdisk -d /dev/sda | sfdisk /dev/sdb
The command should be in this format: sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk
Example:
sudo sfdisk -d /dev/sda | sfdisk /dev/sdb
Once this is done, the next step is to randomise the GUID of the new disk to prevent GUID conflicts with other disks:
If you receive the following message:
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
You can simply run the partprobe command.
We can now rebuild the RAID array by adding the new partitions (sdb2 and sdb4) back:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # sudo mdadm --add /dev/md2 /dev/sdb2
# mdadm: added /dev/sdb2
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # sudo mdadm --add /dev/md4 /dev/sdb4
# mdadm: re-added /dev/sdb4
Use the cat /proc/mdstat command to monitor the RAID rebuild:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[0] sdb2[1]
931954688 blocks super 1.2 [2/2] [UU]
bitmap: 4/4 pages [16KB], 65536KB chunk
md4 : active raid1 sda4[0](F) sdb4[1]
1020767232 blocks super 1.2 [2/1] [UU]
[============>........] recovery = 64.8% (822969856/1020767232) finish=7.2min speed=401664K/sec
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
Lastly, we add a label and mount the [SWAP] partition (if applicable).
Once the RAID rebuild is complete, we mount the partition containing the root of our operating system on /mnt. In our example, that partition is md2.
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # mount /dev/md2 /mnt
We add the label to our SWAP partition with the command:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # mkswap /dev/sdb4 -L swap-sdb4
mkswap: /dev/sdb4: warning: wiping old swap signature.
Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
LABEL=swap-sdb4, UUID=b3c9e03a-52f5-4683-81b6-cc10091fcd
Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ #
mount --types proc /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys
mount --rbind /dev /mnt/dev
mount --make-rslave /mnt/dev
mount --bind /run /mnt/run
mount --make-slave /mnt/run
Next, we access the chroot environment:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # chroot /mnt
We retrieve the UUIDs of both swap partitions:
root@rescue12-customer-eu:/# blkid -s UUID /dev/sda4
root@rescue12-customer-eu:/# blkid -s UUID /dev/sdb4
Example:
blkid /dev/sda4
/dev/sda4: UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15"
blkid /dev/sdb4
/dev/sdb4: UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a"
Next, we replace the old UUID of the swap partition (sdb4) with the new one in /etc/fstab:
root@rescue12-customer-eu:/# nano /etc/fstab
Example:
UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1
UUID=f925a033-0087-40ec-817e-44efab0351ac /home ext4 defaults 0 0
UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0
UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0
In our example above, the UUID to replace is d6af33cf-fc15-4060-a43c-cb3b5537f58a with the new one b3c9e03a-52f5-4683-81b6-cc10091fcd15.
Make sure you replace the proper UUID.
Next, we make sure everything is properly mounted:
root@rescue12-customer-eu:/# mount -av
/ : successfully mounted
/home : successfully mounted
swap : ignored
swap : ignored
Enable the SWAP partition with the following command:
root@rescue12-customer-eu:/# swapon -av
swapon: /dev/sda4: found signature [pagesize=4096, signature=swap]
swapon: /dev/sda4: pagesize=4096, swapsize=536870912, devsize=536870912
swapon /dev/sda4
swapon: /dev/sdb4: found signature [pagesize=4096, signature=swap]
swapon: /dev/sdb4: pagesize=4096, swapsize=536870912, devsize=536870912
swapon /dev/sdb4
We exit the chroot environment with exit and reload the system:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # systemctl daemon-reload
We unmount all the disks:
root@rescue12-customer-eu (nsxxxxx.ip-xx-xx-xx.eu) ~ # umount -R /mnt
The RAID rebuild is complete. Reboot the server in normal mode.
Go further
Hot Swap - Software RAID
OVHcloud API and Storage
Managing hardware RAID
Hot Swap - Hardware RAID
For specialised services (SEO, development, etc.), contact OVHcloud partners.
If you would like assistance using and configuring your OVHcloud solutions, please refer to our support offers.
If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts to assist with your specific use case.
Join our community of users.
