Recently I was working on a Linux Bare Metal Recovery releated requirement. Users should be allowed to use a customed Linux live CD to start a temporary Linux operating system that won’t make any changes to the storage, and then they can use the build-in tools to perform system recovery, e.g. using sfdisk to restore the volume/partition data and using GRUB toolchains to fix the startup items. Thus, we need to find a way to custom our a Linux recovery live CD firstly.

Due to I’m a newbie in such area, I googled about the Linux live CD and found there’s already some out-of-the-box images like Slax. However, it does not provide the feature customization kit so that does not meet our expectation. I searched linux live kit project from GitHub wishing to found a available solution. The projects are mainly some sorts of shell scripts to pack specified Linux distro into a ISO file.

I found the linux-live project by Tomas-M can pack any Linux system whose kernel supports squashfs , aufs and overlayfs into an live CD ISO image. All I need to do is just clone the project, make some configuration and run a script to produce the target ISO, which is quite simple and flexible. Here’s the steps in detail:

1. clone the project to a directory that won’t be packed latter, for example /opt.

   $ cd /opt && git clone https://github.com/Tomas-M/linux-live.git

2. edit the linux-live/config file to filter which directories need to be packed:

   # Kernel file, will be copied to your Live Kit # Your kernel must support aufs and squashfs. Debian Jessie's kernel is ready # out of the box. VMLINUZ=/vmlinuz # List of directories for root filesystem # No subdirectories are allowed, no slashes, # so You can't use /var/tmp here for example # Exclude directories like proc sys tmp MKMOD="bin etc home lib lib64 libx32 root sbin srv usr var"

   use find /boot | grep vmlinuz to find where the vmlinuz image is and create a /vmlinuz symlink to that path:

   $ find /boot | grep vmlinuz /boot/vmlinuz-0-rescue-2348d36f53074ace8b9850051a3d34b5 /boot/vmlinuz-5.10.0-182.0.0.95.oe2203sp3.x86_64 $ ln -s /boot/vmlinuz-5.10.0-182.0.0.95.oe2203sp3.x86_64 /vmlinuz

I have been wondering how operating system boots since the first time I installed my ArchLinux: What happened when I pressed the power on button? Why my computer knows where the system is installed? Why I need to run grub2-install and grub2-mkconfig from a rescue live CD when my Archlinux failed to boot after an update?

It just so happens that I’m working on a requirement of Bare Metal Recovery these days. Not only do we have to restore volumes sector by sector, but also “starts up recovery” is required. We need to mount origin volumes to a Linux live CD media and run commands like grub2-install and grub2-mkconfig to fix the start up items of the origin system, otherwise the system won’t boot after BMR performed. The two commands works on most cases, however, sometimes grub2-install will fail, sometimes grub2-mkconfig will succeed but origin system still can’t boot normally after restart and only got a GRUB shell prompted instead. These strange problems on specified distro force me to delve into the booting procedure of Linux, to figure out what is GRUB, and what did these commands actually do to the disk.

I will make a brief summary in this article of what I have learned these days about the os booting process, including BIOS/UEFI booting, GRUB bootloader, etc. I hope this blog might help for those who are also a newbie to such realm.

BIOS booting & UEFI booting

I seached from wikipedia and learned that modern computers usually have two ways for booting an operating system: BIOS (Basic Input/Output System) and UEFI (Unified Extensive Firmware Interface). They are both firmware interfaces that are responsible for initializing hardware components during the booting process of a computer. However, they differ in several aspects:

When we talk about BMR (Bare Metal Recovery), we are talking about a solution to recovery the operating system, the configurations and the documents. Distinguished from fileset backup and restore, BMR is usually used as the last resort for disaster recovery. This blog shows an feasible solution for BMR on Linux system.

I implemented a volume backup utility mentioned in the article before: VolumeBackup. This utility can perform full backup and forever increment backup to generate a volume copy with a *.img format, which can be used to perform block level recovery from. Actually, some free software providing mature BMR solution, e.g., CloneZilla, peforms BMR via cloning full partition or even full disk. Block-level backup is much faster than file-level backup, thus proven to be an effective way for BMR.

How to backup

To recover a Linux system fully, we need to backup it fully firstly. What we need to backup contains:

1. System state: This part contains system startup items like grub configurations, partition table, network configuration and so on.

The /boot folder stores EFI files, grub configuration and other files that are required for system startup, so it’s must be backuped. This directory neither contains too much files nor takes too much space (usually smaller than 500MB) so we can backup/recover it just at file-level. boot partition is usually initialized as a small standlone partition with FAT32 filesystem and must be the first partition on a disk with GPT partition table.

The root partition mentioned in the article refers to the volume that mount to /. It’s similiar to C:\ on Windows system where we call it System Drive and it’s must be backuped for BMR. Root partition is the place where Linux System and it’s applications and services installed, it may be large so it’s recommened to backup at block-level. If your root partition use a filesystem supporting snapshot creation like btrfs, you can create a snapshot and using VolumeBackup utility to clone it to a image format copy from the snapshot. If your root volume is a LVM logical volume, you can create the snapshot using LVM tools and perform the volume backup as well. If your root partition does not support consistency backup, you can shutdown the system and insert a new media to boot from, and backup the root volume from the new media.

Disk infomation and partitions infomation are also required to backup, they will be needed to recreate the partitions when doing BMR for the reason the partition infomation may be corrupted as well. If you are using LVM to manage your volumes, you alse need to backup your PV,VG,LV infomation.

Although volume-based backup/restore has significant limitations: using volume backup in scenarios with low file system utilization will result in low copy storage space utilization. In scenarios where the backed-up files are generally large, file-level backup can achieve good performance and efficiently use storage space. However, volume backup also has advantages in specific scenarios: it is common in the field of forensic examination of host hard disks based on volume backup. Backup of the entire volume can achieve high backup/restore speed, and the system volume backup can directly restore the operating system. This article starts by introducing the basic concepts of volumes, summarizing several solutions for volume backup, recovery, and subsequent utilization of copy data on the Windows/Linux platforms. Based on the technical solutions mentioned in this article, the author has implemented a set of volume backup tools, achieving full backup and permanent incremental backup of volumes under Windows/Linux, as well as recovery and timely mounting of volume copies. The source code can be found at: https://github.com/XUranus/VolumeBackup, and this article can be considered as documentation for this project.

Basic Concepts of Volumes, Partitions, and Disks

Before describing the logic of volume backup/restore, let’s first introduce what a volume is because volume (Volume), partition (Partition), and hard disk (Hard disk) are often confused concepts. The hard disk is a physical concept, with commonly used hard disks such as Hard Disk Drive (HDD) and Solid State Disk (SSD), but this article does not go into detail about hardware. Volumes and partitions are both storage areas for data, similar but not the same: a volume is an accessible storage area with a single file system, and a partition is a part of a hard disk partitioned out. This means that a partition is often a specific concept, existing in a continuous specific area on a specific disk. A partition may not have a file system (typically an uninitialized RAW partition is also considered a partition). A volume, on the other hand, is an abstract concept that must be strongly associated with a single file system, and a volume may exist on one or more disks. Since a volume is a logical concept, it exhibits differences for Linux/Windows operating systems, and physical partitions are the basis for forming logical volumes, so to understand volumes deeply, we need to start with partitions.