md is a Linux kernel facility with user space supporting tools and files.
md was originally "mirror disk". It was renamed "multiple devices" when functionality was extended to more than mirrored disks. Functionality now includes non-mirrored RAID and multipathing. At the time of writing, Blue Light had only used md to create RAID 1 devices.
md assembles multiple devices into a single virtual device.
The multiple devices are variously called "underlying devices" and "RAID devices" in the documentation. On this WIKI page they are always "underlying devices".
Originally md device files were /dev/md[[:digit:]]+. Now /dev/md/[[:digit:]]+ is preferred.
Create the block devices to be used as underlying devices. Typically these are HDD partitions but any block devices can be used. In the following example, /dev/sdb1 /dev/sdc1 are used.
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
In case only one underlying device is available (for example during system build or recovery) the same command can be used with the missing underlying devce name replaced by "missing". For example:
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing
If the md device provides the file system including /boot or is an LVM PV (physical volume) then /etc/mdadm/mdadm.conf must be re-populated as shown below and the initramfs re-generated by:
update-initramfs -u
Typically this is done when an HDD has failed.
In case the replacement HDD is the same size and has the same partitions as a working HDD, the partitions can be created and GRUB installed in a single command. Assuming sda is the working HDD, partitioned with an MBR and sdb the replacement:
dd if=/dev/sda of=/dev/sdb bs=512 count=1 |
The procedure continues at "All partition table types" below.
Create the new underlying device by partitioning.
Regardless of how the underlying block device was created, it can now be added to the md device:
mdadm --manage --add /dev/md/<md device index> /dev/<whatever>
For example:
mdadm --manage --add /dev/md/0 /dev/sdb1
This is intended to:
For example:
# mdadm --manage --fail /dev/md1 /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
# mdadm --manage --remove /dev/md1 /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md1
# mdadm --zero-superblock /dev/sdb2
mdadm: Unrecognised md component device - /dev/sdb2
[probably failed because sdb was bad]
# echo "1" > /sys/block/sdb/device/delete
Removing /dev/md1 as an example ...
Find which block devices the md device is built on
mdadm --detail /dev/md1
Stop the device:
mdadm --stop /dev/md1
For each block device the device is built on (here /dev/sda2):
mdadm --zero-superblock /dev/sda2
TODO: is this next step necessary or even possible (it was not in PTDC-361)
Finally remove the device:
mdadm --remove /dev/md1
When a RAID 1 md device is used for /boot, GRUB should be installed to both underlying devices, not to the md device. This is done using
dpkg-reconfigure pc-grub
GRUB2 can be configured to include md support so can assemble the /boot md before loading initrd (or the kernel directly) but Blue Light does not do this.
Some linux system integrated an mdadm tool called mdadm-startall. You can install it if not already done.
It doesn't require any argument and will assemble all the md devices found.
mdadm-startall |
At least Debian 7 Wheezy has /etc/cron.d/mdadm which runs
/usr/share/mdadm/checkarray --cron --all --idle --quiet
on the 14th of each month at 02:19.
The command only checks the md arrays, it does not fix them. For each array /dev/md/<n>, it results in messages in syslog like:
RebuildStart event detected on md device ...
Rebuild(20|40|60|80) event detected on md device ...
RebuildFinished event detected on md device ...
Despite "Rebuild" it is actually a check.
For small arrays the "Rebuild(20|40|60|80)
..." progress indicators are not generated. We have seen some of those messages missing.
Sometimes mismatches are reported. According to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=518834#38, that is a problem for RAID 5 but not for RAID 1. Presumably it is a problem for RAID 6 too.
/etc/mdadm/mdadm.conf is generated when md arrays are set up changes may require updating it by:
/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf
When initrd is responsible for assembling the md devices, it needs to be rebuilt when the md devices (not their underlying devices) change, for example when an md is removed and a new one created with the same name.
If this has not been done and the md devices were required to boot, a busybox prompt is presented. mdadm --assemble --scan
was not effective. The md device had to be created by running mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2
When the system has been booted, the damage can be fixed by:
mdadm --detail --scan
).update-initramfs -u
After fixing the missing md devices as above, the VGs and LVs can be enabled by running the busybox command lvm
and then using its help to find the commands to do so (cut down versions of the GNU Linux equivalents).
Populate /etc/mdadm/mdadm.conf as described above.
A non-zero number of mismatches is OK on RAID 1.