File-system integrity! Boring words, right? You’ve got work to finish. Your system froze in the middle of doing updates. When was my last backup? About as boring as a heart attack when your system won’t boot because your / (root) partition is corrupt.
All of those thoughts raced through my head when I started getting file system corruption on my home workstation. Ever see one of these errors?
Use dmesg to find file-system warnings.
ext4_iget: bogus i_node remounting as read-only
How did I find that message? Using dmesg. You can grep dmesg output if you want:
$ dmesg | egrep -i '(ext|mount)'
Or you can use the mount command to find things mounted read-only:
$ mount | grep "ro,"
If you’re caught in this predicament and cannot finish your boot sequence because you can’t find fsck on your hard drive–because you’re stuck in initrd…I hope you have a USB stick with Ubuntu MATE on it! (initrd is the boot ramdisk environment that loads the kernel.) My advice is go make a Mate live-CD right now…very useful!
When you’re booted into your live-CD environment, your desktop has not begun all the media discovery tasks that allow fsck to actually get work done. In my case, my /boot, and volume-groups were all on top of mdraid partitions. You will download fdisk, mdraid, and lvm2 to get started:
$ sudo apt install mdadm fdisk
This installs and then we can scan our partitions using partprobe or kpartx. (I really appreciate that the MATE live cd allows me to install new packages.)
$ sudo partprobe $ sudo fdisk -l /dev/sda
Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 1953791 1951744 953M fd Linux raid autodetect /dev/sda2 1953792 3907583 1953792 954M fd Linux raid autodetect /dev/sda3 3909630 250068991 246159362 117.4G 5 Extended /dev/sda5 3909632 101564415 97654784 46.6G fd Linux raid autodetect /dev/sda6 101566464 140625919 39059456 18.6G 83 Linux /dev/sda7 140627968 141125631 497664 243M 83 Linux /dev/sda8 141127680 250068991 108941312 52G 83 Linux
Because mirror partitions must be modified (sda1, sdb1) at the same time, you should NOT fsck them directly. You need to assemble the raid devices first:
$ sudo mdadm –assemble –scan
Then check which are assembled in /proc/mdstat:
$ cat /proc/mdstat md1 : active raid1 sdb2 sda2 976320 blocks super 1.2 [2/2] [UU] md2 : active raid1 sdb5 sda5 48794624 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdb1 sda1 975296 blocks super 1.2 [2/2] [UU]
My md0 device was mounted on /boot. Let’s run fsck right on it:
$ sudo fsck.ext4 -vfy /dev/md0
The other items are all managed by LVM. We need to do a vgscan to activate them:
$ sudo vgscan
That should detect your logical volumes and assemble them, but you may need to activate them at some point with
sudo vgchange -ay $your_vgname
My volume name is vg_cholla_root. Those logical volumes are accessible through the Device Mapper, so look in /dev/mapper/. Let’s fsck our root volume:
$ sudo fsck -vfy /dev/mapper/lv_cholla_root
That should not take very long if you’re on an SSD, especially a mirrored SSD. You’ll log out of your live-CD environment, remove your usb stick, reboot the machine and your original system should come backup up.
To Skip that Tedious Step
You don’t really want to do that twice. This actually kept happening to me (hence part 1). Before your system wedges again, you want to edit /etc/grub.conf and add some on-boot fsck options:
GRUB_CMDLINE_LINUX_DEFAULT="fsck.mode=force fsck.repair=yes" GRUB_CMDLINE_LINUX="fsck.mode=force fsck.repair=yes"
To be super cautious, also make sure both of your boot sectors (/dev/sda, /dev/sdb) have the grub boot loader installed:
$ sudo grub-install /dev/sda $ sudo grub-install /dev/sdb
Then run sudo update-grub2. Your every boot-up will force a file system check. On my system it takes about four seconds. I have my /home directory on a separate (ZFS) mount point, and that doesn’t get checked at boot.
Next episode, I’ll cover some of the additional steps I’ve taken to protect my workstation file-systems from being brutalized by the vagaries of my staggering hardware. Stay tuned to Freedom Penguin!