How to move Linux Home to ZFS

file manager
  • ( Linux Skill Level: Intermediate to Advanced )
  • Linux OS: Ubuntu 14.04 64-Bit // VM Environment: Virtualbox for Linux v5.1

Recently, I had an issue with a P2V VM (physical PC converted to virtual machine) where /home was running out of space and needed to be expanded. My /home in this case was a completely separate virtual disk (sdb) that was 10GB, and I needed to restore a multiple-GB Thunderbird email archive.

I was originally going to expand the virtual drive and partition with Virtualbox tools and gparted, but I decided to do things the ZFS way so A) It wouldn’t be as much of an issue in the future (easier to add more/larger disks if needed); and B) I could take advantage of ZFS filesystem compression to save disk space as part of the deal; AND C) Get free filesystem snapshots. NOTE: For best results, the VM should be 64-bit and have at least 2GB of RAM if you intend to use ZFS. And as always, more RAM is better for performance – at least 4GB RAM on the VM host is a good start.

You can refer to my previous series of articles if you would like to learn more about ZFS on Linux and you can also read up on datasets here *(1) before proceeding.

*(1) https://pthree.org/2012/12/17/zfs-administration-part-x-creating-filesystems/

https://blogs.oracle.com/orasysat/entry/so_what_makes_zfs_so

Goal: Move /home partition that is running low on space, to a separate ZFS single-drive pool with compression – Without having to reboot

First of all, in my case I had /home on a separate virtual drive and needed to either expand the disk or add a new one. So while the VM was still powered on, I added a new SATA drive (sdc) that was 20GB in size to my Virtualbox VM:

Controller SATA

zfs-home

This is a nice feature of working with VMs – and with Ubuntu Linux 14.04, you don’t need to issue any separate commands to re-read the SCSI bus, it should auto-detect the new drive right away!

If you are using another Linux distro (that is probably RPM-based) and need to rescan the SCSI bus to detect new disks without rebooting, refer to these: *(2)

*(2) https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/rescan-scsi-bus.html

https://geekpeek.net/rescan-scsi-bus-on-linux-system/

Before going ahead, you need to have ZFS on Linux up and running. Switch from your non-admin user to root and issue:

sudo su -
zpool status
no pools available

That should be what you get if you haven’t defined any ZFS pools yet; if you get an error, try:

modprobe zfs

…and then redo the above zpool command. If you still don’t have a working ‘ zpool status ‘ then you can follow my previous article on how to get ZFS installed on Ubuntu Linux 14.04.

We need to partition the new disk; issue ‘ # fdisk -l ‘ as root to list all disks and then issue:

parted -s /dev/sdc mklabel gpt
fdisk -l /dev/sdc
Disk /dev/sdc: 21.5 GB, 21474836480 bytes
256 heads, 63 sectors/track, 2600 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 41943039 20971519+ ee GPT

You may need to ‘ # apt-get install parted ‘ if you get a command not found error.

We need to find the virtual HD’s long-form name to define the ZFS pool:

ls -l /dev/disk/by-id
lrwxrwxrwx 1 root root 9 Apr 14 10:34 ata-VBOX_HARDDISK_VB7957d4c1-b4a9da8a -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 14 10:34 ata-VBOX_HARDDISK_VB7957d4c1-b4a9da8a-part1 -> ../../sda1
lrwxrwxrwx 1 root root 9 Apr 14 10:42 ata-VBOX_HARDDISK_VB8a5a6cd1-37a97479 -> ../../sdc
lrwxrwxrwx 1 root root 9 Apr 14 10:34 ata-VBOX_HARDDISK_VBb8b78e91-6b4c1ca2 -> ../../sdb
lrwxrwxrwx 1 root root 10 Apr 14 10:34 ata-VBOX_HARDDISK_VBb8b78e91-6b4c1ca2-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Apr 14 10:34 ata-VBOX_HARDDISK_VBb8b78e91-6b4c1ca2-part2 -> ../../sdb2

I already knew that disk “sda” is my linux root and “sdb” is /home, so I issued:

zp=zhome
zpool create -o ashift=12 -o autoexpand=on -o autoreplace=on \
-O atime=off -O compression=lz4 \
$zp \
ata-VBOX_HARDDISK_VB8a5a6cd1-37a97479
zpool status |awk 'NF>0'
pool: zhome
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zhome ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8a5a6cd1-37a97479 ONLINE 0 0 0
errors: No known data errors

This is a bit of a PROTIP on how to create a ZFS pool that some of the other articles won’t show you. The lower-case “-o” options set the ZFS pool properties to use 4K sector sizes regardless of physical sector size, and if you add another disk (or set of disks) it will auto-expand and use the extra space.

For a VM, you can probably go with the default ‘ ashift=9 ‘ for 512 sector sizes, but using ‘ ashift=12 ‘ everywhere translates into real-world usage and is good practice. It will also save you a lot of hassle when it comes time to either replace a failing disk, or want to expand the pool size in-place by using larger disks.

The upper-case “-O” options disable file/directory access-time updates for every READ on the entire pool (equivalent to “noatime” in /etc/fstab) and all future datasets, and fast+efficient LZ4 compression is enabled at the entire-pool level. If you need to, you can override these defaults at the dataset level later. therwise, they will be inherited. In my experience, setting “noatime” saves quite a bit of wear and tear on disks!

Now to create our new /home dataset on the single-disk pool (we don’t really need disk Mirroring since this is a VM; if doing this on a physical box, I would definitely recommend allocating at least 1 mirror disk of the same make/model/capacity to guard against disk failure):

zfs create -o sharesmb=off $zp/home
df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 24122780 12098696 10775676 53% /
/dev/sdb1 10190136 5771220 4297676 58% /home
zhome 20188288 0 20188288 1% /zhome
zhome/home 20188288 0 20188288 23% /zhome/home

Since this is Ubuntu, if you haven’t defined a root password yet you should do so with this before proceeding: (NOTE – these instructions are for home/personal use and I assume you own the PC you’re doing this on – DON’T do this on a shared/production system!!)

sudo passwd root

At this point, since I was logged into X windows and we’re moving /home entirely, I needed to LOGOUT of my X window manager all the way before proceeding, and then verify that nothing was holding on to any files in /home. So in Virtualbox, I hit right-Ctrl+F1 and switched to text console TTY1. If doing this on a physical box, you would hit Ctrl+Alt+F1.

( On tty1, login as root with the root password, and issue: )

lsof |grep home

This will list any Open Files in home – you will need to logout of all non-root users and/or issue kill commands to make them let go of open files. You can find more info on how to do that here: *(3)

*(3) https://unix.stackexchange.com/questions/18043/how-do-i-kill-all-a-users-processes-using-their-uid

http://www.tecmint.com/how-to-kill-a-process-in-linux/

Once the above ” lsof ” command returned nothing, I migrated my /home to ZFS with Midnight Commander. If you’re familiar with Norton Commander or other 2-pane text-based file managers, then you already know that MC is a Godsend – I use it especially to delete entire directory trees safely without using ‘ rm -f ‘. Verify that MC is installed with:

apt-get update; apt-get install mc## you only need to do this ONCE

mc /home /$zp/home## copy everything over, using the Insert key to mark files/directories and the F5 key to copy

file manager
( Note the right pane in the screenshot should read ” /zhome/home ” at the top )

You can learn more about Midnight Commander here: *(4)

*(4) https://en.wikipedia.org/wiki/Midnight_Commander

–If you would rather do the file copying programmatically, without using MC:

cd /home
time tar cpf - * |(cd /$zp/home; tar xpf - )
df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 24122780 12098712 10775660 53% /
/dev/sdb1 10190136 5771220 4297676 58% /home
zhome 15614336 128 15614208 1% /zhome
zhome/home 20188288 4574080 15614208 23% /zhome/home

You can see the benefits of ZFS data compression right away there, in the Used column.

Now once all the files are copied over, we don’t need the old /home anymore:

cd /; mv home home—old ## this should work if your /home is just a directory in "/"
/bin/mv: cannot move ‘home’ to ‘home--old’: Device or resource busy

# pstree -A
|-lightdm-+-Xorg
| |-lightdm-+-fluxbox-+-gnome-terminal-+-bash---su---bash---screen
| | | | |-bash
| | | | |-gnome-pty-helpe
| | | | `-3*[{gnome-terminal}]
| | | `-ssh-agent
| | `-{lightdm}
| `-2*[{lightdm}]

Ah, the X window manager is still running.

service lightdm stop

And then I remembered /home is on a separate drive/partition:

umount /home

I then modified my /etc/fstab to no longer mount the original home (I still have a swap partition on sdb, but I can switch that to a swapfile and remove the disk from the VM later):

# (EXAMPLE fstab line)
LABEL=home /home ext4 defaults,noauto,noatime,errors=remount-ro 0 1

Now to set things up permanently and move /home to ZFS:

zfs set mountpoint=/home $zp/home

zfs mount $zp/home## this proved to be extraneous, but I’m including it for completeness; you don’t have to enter it in.

df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 24122780 12098712 10775660 53% /
zhome 20188288 128 15614208 1% /zhome
zhome/home 20188288 4574080 15614208 23% /home

For comparison – original ext4 home before ZFS:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 10190136 5771220 4297676 58% /home

Home after migrating the original data to the compressed ZFS pool/dataset, before Thunderbird restore:

 zhome/home 20188288 4574080 15614208 23% /home

Home pool AFTER the thunderbird restore, still coming in well under my original 10GB limit:

 zhome/home 20187776 8912256 11275520 45% /home

Since my needs for this VM are light, I didn’t bother creating separate ZFS datasets for my individual users first, but you can certainly do that before copying data over. It will give you more fine-grained control over snapshots, and you can do per-dataset options like file sharing over Samba or NFS.

Five Cool things you can do with a ZFS-based /home now:

1) Replace existing 20GB disk in-place with 30GB (or arbitrarily larger size) and remove the old 20GB disk from the pool in TWO STEPS:

New HD to VM

Add the new HD to the VM

ls -l /dev/disk/by-id ## to find the new device long-form name; or use ' fdisk -l '
time zpool replace $zp \
ata-VBOX_HARDDISK_VB8a5a6cd1-37a97479 \
ata-VBOX_HARDDISK_VB8afce7ed-7f6e6a7c
real 0m16.749s
zpool status
pool: zhome
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Apr 14 15:30:35 2017
286M scanned out of 8.56G at 17.9M/s, 0h7m to go
286M resilvered, 3.27% done
config:
NAME STATE READ WRITE CKSUM
zhome ONLINE 0 0 0
replacing-0 ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8a5a6cd1-37a97479 ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8afce7ed-7f6e6a7c ONLINE 0 0 0 (resilvering)
errors: No known data errors

Before:

Filesystem 1K-blocks Used Available Use% Mounted on
zhome/home 20187520 8975488 11212032 45% /home

After: (Once resilvering is complete; you can keep checking every minute or so with ‘ zpool status ‘)

pool: zhome
state: ONLINE
scan: resilvered 8.56G in 0h8m with 0 errors on Fri Apr 14 15:38:41 2017
config:
NAME STATE READ WRITE CKSUM
zhome ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8afce7ed-7f6e6a7c ONLINE 0 0 0
errors: No known data errors
df
Filesystem 1K-blocks Used Available Use% Mounted on
zhome 21370112 0 21370112 0% /zhome
zhome/home 30345600 8975488 21370112 30% /home
zfs list
NAME USED AVAIL REFER MOUNTPOINT
zhome 8.56G 20.4G 96K /zhome
zhome/home 8.56G 20.4G 8.56G /home

Note that this increase in space has been done in real-time, with no “downtime”, and the new free space is available immediately without rebooting or having to resize disk partitions. 🙂

2) Add an additional 30GB disk and ‘ zpool add ‘ the extra space (NOT a mirror disk) to double the space pretty much instantly. NOTE you probably wouldn’t want to run this way on a physical box because it provides no Redundancy if any disk fails, but it’s OK in a VM (especially since I’m using mirrored ZFS disks for /home on the “host” side for Virtualbox):

( Example )

parted -s /dev/sde mklabel gpt ## on the NEW disk
zpool add -o ashift=12 $zp /dev/sde

You would of course want to use the “long-form” disk name in /dev/disk/by-id or /dev/disk/by-path, and MAKE SURE you’ve got the right disk – or risk data loss!

( As an example, I took a Virtualbox snapshot of the VM before doing this – creating an empty 1GB file and adding it to the pool for extra free space ):

cd /; time (dd if=/dev/zero of=zdisk2 bs=1M count=1024;sync)

1073741824 bytes (1.1 GB) copied, 38.5534 s, 27.9 MB/s
real 0m53.252s

/ # zpool add -o ashift=12 $zp /zdisk2
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses disk and new vdev is file
zpool add -f -o ashift=12 zhome /zdisk2
zpool status
pool: zhome
state: ONLINE
scan: resilvered 8.56G in 0h8m with 0 errors on Fri Apr 14 15:38:41 2017
config:
NAME STATE READ WRITE CKSUM
zhome ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8afce7ed-7f6e6a7c ONLINE 0 0 0
/zdisk2 ONLINE 0 0 0
errors: No known data errors
zfs list
NAME USED AVAIL REFER MOUNTPOINT
zhome 8.56G 21.3G 96K /zhome
zhome/home 8.56G 21.3G 8.56G /home

You can see the free space has increased. I reverted to the “before” VM snapshot because well, that configuration’s just not stable. 😉 It might not even survive a reboot; I’m just showing what’s possible.

Use the flexibility and capabilities of ZFS with solemn responsibility, and know the difference between ‘ zpool add ‘ and ‘ zpool attach ‘! If you issue the wrong command, you may have to backup your pool and recreate it to get it configured the way you intended!

3) Virtually Instant snapshot + rollback (restore) of ZFS datasets

ZFS Administration, Part XII- Snapshots and Clones

4) Set disk Quotas on a per-dataset basis to avoid overfilling the pool / runaway user data

http://stackoverflow.com/questions/18143081/how-to-set-default-user-quota-in-zfs-filesystem

This comes in handy if, say, you want to set aside some space for burning Blu-Ray backups:

zfs create -o compression=off -o atime=off \
-o mountpoint=/mnt/bluraytemp25 \
-o quota=23.5G $zp/bluraytemp; chown yournonrootuser /mnt/bluraytemp25
cd /mnt/bluraytemp25 && truncate -s 23652352K bdiscimage.udf
## Quickly create the UDF image file of size ~23GB
mkudffs --vid="YOURLABEL20170414" bdiscimage.udf
## Format the file with a descriptive label, as UDF
mkdir /mnt/bluray-ondisk ## Only once!
mount -t udf -o loop /mnt/bluraytemp25/bdiscimage.udf /mnt/bluray-ondisk -onoatime
 df
/dev/loop0 24413000 1500 24411500 1% /mnt/bluray-ondisk

Now you can copy files over to /mnt/bluray-ondisk and not have to worry about over-burning.

MAKE SURE TO UMOUNT BLURAY-ONDISK BEFORE BURNING THE IMAGE TO DISC!!

5) Have separate uncompressed / Samba-shared datasets without having to set up a full Active Directory environment:

zfs create -o compression=off -o sharesmb=on $zp/nocomprshare; chown youruser /$zp/ nocomprshare

I hope this article has been useful in showing the possibilities of migrating something like /home to ZFS. The capabilities of the filesystem and possibilities for easier disk management and expansion are pretty much incredible, but I highly recommend doing some research into it. You do need to know what you’re doing to leverage the potential in the right way.

Old Vista Laptop Into A Linux ZFS File Server Part 3

samba-share-discover

In the previous article, I showed you how to install Lubuntu 14.04 64-bit and install the important bits of Samba and the ZFS filesystem. In this article, I will give you the interesting details on how to get your Probox-connected disks up and running as a ZFS RAID10, starting with (1) disk and growing to a full 4-disk RAID10 in real-time. Please note: Follow these steps at your own risk. I take no responsibility for data loss! You will need to be careful and make sure you are using the right disks when entering administration commands.

It’s a common misconception that you can’t “grow” a ZFS RAIDZ. In fact, if you already have a RAIDZ, RAIDZ2, etc you can expand the free space – but only if you essentially duplicate the existing configuration, or replace the disks one at a time with bigger ones. There is more information here:
(See link’s Step 3): http://www.unixarena.com/2013/07/zfs-how-to-extend-zpool-and-re-layout.html
and here: http://alp-notes.blogspot.com/2011/09/adding-vdev-to-raidz-pool.html

Now, getting back to our project. First, you need to verify that the disks are detected by the OS. You can do this by entering (as root):

fdisk -l /dev/sd?

Depending on how many disks you have online in the Probox, you should see anything from sdb to sde:

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
256 heads, 63 sectors/track, 121126 cylinders, total 1953525168 sectors
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1953525167 976762583+ ee GPT
.
.
.
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
256 heads, 63 sectors/track, 121126 cylinders, total 1953525168 sectors
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Device Boot Start End Blocks Id System
/dev/sde1 1 1953525167 976762583+ ee GPT

If you get warnings like the following, you can ignore them:

Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.

virtualbox-storage

This is an example of how to setup your Virtualbox storage. Use a SATA controller for the Linux root (I used 120GB to duplicate my laptop’s SSD) and a SCSI controller for the ZFS disks to duplicate the Probox. The ZFS disks should all be the same size, whether 1GB for testing or 1TB.

To get a short list of the disks on your system, you can do this:

ls -l /dev/disk/by-id |egrep -v 'part|wwn'
lrwxrwxrwx 1 root root 9 Mar 31 10:47 ata-TSSTcorp_CDDVDW_SH-S222A -> ../../sr0
lrwxrwxrwx 1 root root 9 Mar 31 10:47 ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 -> ../../sdb
lrwxrwxrwx 1 root root 9 Mar 31 10:47 ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3 -> ../../sdd
lrwxrwxrwx 1 root root 9 Mar 31 10:47 ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J4 -> ../../sde
lrwxrwxrwx 1 root root 9 Mar 31 10:47 ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J6 -> ../../sdc
lrwxrwxrwx 1 root root 9 Mar 31 10:47 usb-Samsung_Flash_Drive_FIT_03557150-0:0 -> ../../sda

If you’re wanting to test this process in Virtualbox first, use ” /dev/disk/by-path ” instead.

This will display the long-form device name and the symlink to the short device name. The ‘egrep -v‘ part leaves out anything in the list that matches partitions and WWN device names. At any rate, we need to concentrate on the ata-WDC names in this case, because those will be our ZFS pool drives.

To start with, we should always run a full burn-in test of all physical drives that will be allocated to the ZFS pool (this can take several hours) – skip this at your own peril:

time badblocks -f -c 10240 -n /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 \
2>~/disk1-badblocks.txt &

This will start a non-destructive Read/Write “badblocks” disk-checker job in the background that redirects any errors to the file ” /root/disk1-badblocks.txt “.

Then you can hit the up-arrow and start another job at the same time, substituting the ata-WDC drive names and the disk numbers as needed:

dsk=2; time badblocks -f -c 10240 -n /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3 \ 2>~/disk$dsk-badblocks.txt &

And so on. Just make sure you change the “ata-” filenames, and you can check all 4 disks at the same time.

If you would rather do this programmatically:

for d in `ls -1 /dev/disk/by-id/ata-WDC_WD10* |egrep -v 'part|wwn'`; do echo "time badblocks -f -c 10240 -n -s -v $d 2>>~/disk-errors.txt &"; done

Which will result in something like the following:

time badblocks -f -c 10240 -n -s -v /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 2>>~/disk-errors.txt &
time badblocks -f -c 10240 -n -s -v /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3 2>>~/disk-errors.txt &
time badblocks -f -c 10240 -n -s -v /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J4 2>>~/disk-errors.txt &
time badblocks -f -c 10240 -n -s -v /dev/disk/by-id/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J6 2>>~/disk-errors.txt &

You can then copy/paste those resulting commands in the root terminal to run all 4 jobs at once, although they will all be logging errors to the same log file.

To get an idea of what is going on and see when the “badblocks” jobs are finished, you can open up another non-root Terminal (or File\Tab) and issue:

sudo apt-get install sysstat## You will only need to do this ONCE

iostat -k 5## This will update every 5 seconds until ^C (Control+C) is pressed

Linux 4.4.0-57-generic (hostname) 03/31/2017 _x86_64_ (1 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
1.10 0.00 2.20 47.19 0.00 49.50

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdd 108.00 37478.40 38912.00 187392 194560
sde 0.00 0.00 0.00 0 0
sdb 77.60 38912.00 38707.20 194560 193536

Once all the badblocks jobs are done (everything in the “kB_wrtn” column should go back to 0), it’s a good idea to check the /root/disk*.txt files to see if anything was found:

cat /root/disk*.txt

Of course, if anything really bad was found, you might want to delve in deeper and possibly replace the disk. If everything turns out OK, we can start by creating our initial ZFS pool using the first disk.

(Please note, the following commands assume you are using the BASH shell to enter commands)

export zp=zredpool1## Define our ZFS pool name – call yours whatever you like

export dpath=/dev/disk/by-id

PROTIP: If using Virtualbox, you would want to use /dev/disk/by-path

declare -a zfsdisk ## Declare a Bash array

zfsdisk[1]=$dpath/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1# Use whatever disk is appropriate to your system for the 1st disk in your pool

zfsdisk[2]=$dpath/ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3# Use whatever disk is appropriate to your system for the 2nd disk, if applicable

…and so on, for disks 3 and 4.

STEP 1: Initialize the disk(s) with GPT partition table

zpool labelclear ${zfsdisk[1]}; parted -s ${zfsdisk[1]} mklabel gpt

Repeat the below command, changing the number for each pooldisk – 2, 3, 4 as applicable:

dsk=2; zpool labelclear ${zfsdisk[$dsk]}; parted -s ${zfsdisk[$dsk]} mklabel gpt

Once those are done, all of your Probox disks should be initialized with a new GPT label and ready for use with a ZFS filesystem, even if they had previously been used in a pool. Check with this:

parted -l
Model: ATA WDC WD10EFRX-0 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

If for some reason the disks aren’t properly partitioned, you can ‘ apt-get install gparted ‘ as root and use ‘ sudo gparted ‘ as your non-root user to partition them manually.

Now for the EASY part – Create your initial ZFS pool using only 1 disk:

STEP 2: Create the Zpool

time zpool create -f -o ashift=12 -O compression=off -O atime=off $zp \
${zfsdisk[1]}

zpool set autoexpand=on $zp# Set some necessary options for expansion

 zpool set autoreplace=on $zp
df /$zp

Filesystem 1K-blocks Used Available Use% Mounted on zredpool1 942669440 128 942669440 1% /zredpool1

And you should have a 1-disk ZFS pool. 🙂 You can immediately start copying data to the pool if you want. BUT – we have no Redundancy yet!!

zpool status |awk 'NF>0'## this will skip blank lines in output

pool: zredpool1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zredpool1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
errors: No known data errors

Now if you already have the 2nd disk online and ready to go, we can add another drive on the fly to our 1-disk pool to make it RAID1 (mirrored):

STEP 3: Add a Mirror Disk to the 1-Disk Pool

time zpool attach $zp \
${zfsdisk[1]} ${zfsdisk[2]}

If you’re browsing this webpage on the same computer you’re creating your ZFS pool on, copy and paste this into your root terminal after entering the above command, especially if you already have “jumped the gun” a bit and copied some files to your new ZFS pool:

function waitforresilver () {
printf `date +%H:%M:%S`' ...waiting for resilver to complete...'
waitresilver=1
while [ $waitresilver -gt 0 ];do
waitresilver=`zpool status -v $zp |grep -c resilvering`
sleep 5
done
echo 'Syncing to be sure...'; time sync;
date
}
waitforresilver

Otherwise, just keep issuing ‘ zpool status ‘ every minute or so and/or monitoring things with ‘iostat’ in your non-root terminal until the resilver is done. You’ll get your prompt back when the ZFS disk-mirroring process is completed.

zpool status |awk 'NF>0'
pool: zredpool1
state: ONLINE
scan: resilvered 276K in 0h0m with 0 errors on Sat Apr 1 11:48:58 2017
config:
NAME STATE READ WRITE CKSUM
zredpool1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3 ONLINE 0 0 0
errors: No known data errors

At this point, we have a basic RAID1 (mirrored) ZFS pool so if a hard drive fails, it can be swapped out with a drive of the same make/model/capacity and there should be no loss of data. The basic capacity/free space available to the pool is the same as with a single disk, which you can verify with…

 ' df '.

Filesystem 1K-blocks Used Available Use% Mounted on
zredpool1 942669440 128 942669440 1% /zredpool1

You can safely skip this next part and go straight to STEP 4, but if you want to experiment by simulating a disk failure (and you have at least 3 disks ready), remove the 2nd disk from the Probox enclosure(!).

Now enter this: # touch /$zp/testfile; zpool status:

pool: zredpool1
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 276K in 0h0m with 0 errors on Sat Apr 1 12:40:03 2017
config:
NAME STATE READ WRITE CKSUM
zredpool1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J6 UNAVAIL 0 0 0
errors: No known data errors

To replace the “failed” disk #2 with disk #3, enter this:

time zpool replace $zp ${zfsdisk[2]} ${zfsdisk[3]}; zpool status -v
pool: zredpool1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Apr 1 13:30:20 2017
448M scanned out of 1.18G at 149M/s, 0h0m to go
444M resilvered, 37.03% done
config:
NAME STATE READ WRITE CKSUM
zredpool1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J6 UNAVAIL 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J4 ONLINE 0 0 0 (resilvering)
errors: No known data errors

Once the resilver is done, issue another ‘ zpool status ‘:

pool: zredpool1
state: ONLINE
scan: resilvered 1.18G in 0h0m with 0 errors on Sat Apr 1 13:30:30 2017
config:
NAME STATE READ WRITE CKSUM
zredpool1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J4 ONLINE 0 0 0
errors: No known data errors

And you will see that the original failed drive #2 that ended with “WCC4J6” is now gone from the pool and has been completely replaced with drive #3. Everything is back in order and the pool continues functioning with no downtime, and ONLY the “allocated data” has been copied from the “good” drive – ZFS RAID10 is MUCH FASTER with resilvering than a traditional RAID!

Now if you did this step, replace disk #2 in the Probox enclosure, then issue:
‘ zpool destroy zredpool1 ‘ and redo steps 1-3. Then go on to STEP 4.

If you already have all 4 disks ready to go in the Probox, you can now enter this:

STEP 4: Expand the ZFS RAID1 to zRAID10 using all (4) disks:

time zpool add -o ashift=12 $zp \
mirror ${zfsdisk[3]} ${zfsdisk[4]}
waitforresilver

And you’ll have a full zRAID10 ZFS pool, ready to receive – and share – files!

zpool status |awk 'NF>0'; df
pool: zredpool1
state: ONLINE
scan: resilvered 276K in 0h0m with 0 errors on Sat Apr 1 12:29:00 2017
config:
NAME STATE READ WRITE CKSUM
zredpool1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J6 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J4 ONLINE 0 0 0
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J3 ONLINE 0 0 0
errors: No known data errors

Filesystem 1K-blocks Used Available Use% Mounted on
zredpool1 1885339392 128 1885339264 1% /zredpool1

Wait, what’s that?! Take a look at the available space and compare it to what we started with at the single-disk and RAID1 stages:

Filesystem 1K-blocks Used Available Use% Mounted on
zredpool1 942669440 128 942669440 1% /zredpool1

By adding another set of 2 disks, we have expanded the ZFS RAID10 pool’s free space to about twice the size – minus some overhead. Expanding on this concept – if you bought another Probox and another 4 disks of the same make/model/size, you could continue dynamically increasing the available free space of your pool simply by adding more sets of 2 disks.

Or, you could use larger drives in the 2nd enclosure (2TB, 4TB, etc) and ‘ zpool replace ‘ them one at a time to migrate the entire pool from Probox 1 to Probox 2. You don’t have to wait for a drive to fail before issuing a ‘ zpool replace ‘ command, as long as the “necessary options” mentioned above are set.

If you run into any problems and want to start over, you can issue:

zpool destroy $zp; zpool status
no pools available

And go back to STEP 1. WARNING: NEVER do ‘zpool destroy’ on a “live” ZFS pool; ZFS assumes you know what you’re doing and will NOT ask for confirmation! Destroying a ZFS pool (intentionally or not) pretty much guarantees data loss if it’s not backed up!

Now we can get to the “productive” part of running ZFS – sharing files with Windows.

STEP 5: Creating and Sharing ZFS Datasets with Samba

First we need to define a Samba user. You should only need to do this once, and you need to use an existing ID on the system – so, since I installed Lubuntu earlier with a non-root user named ‘user’:

myuser=user## change this to suit yourself

smbpasswd -a $myuser
New SMB password:
Retype new SMB password:

You will need to enter a password at the prompt and confirm it. The password can be the same as your non-root user’s Linux login or different.

Now we can define some “datasets” on our ZFS pool. ZFS Datasets are a bit like regular directories, and a bit like disk partitions; but you have more control over them in regards to filesystem-level compression, disk quotas, and snapshots.

myds=sambasharecompr
zfs create -o compression=lz4 -o atime=off -o sharesmb=on $zp/$myds ; chown -v \
myuser /$zp/$myds
myds=notsharedcompr
zfs create -o compression=lz4 -o atime=off -o sharesmb=off $zp/$myds ; chown -v \
myuser /$zp/$myds
myds=notsharednotcompr
zfs create -o compression=off -o atime=off -o sharesmb=off $zp/$myds ; chown -v \
myuser /$zp/$myds
df
Filesystem 1K-blocks Used Available Use% Mounted on

zredpool1 942669184 128 942669056 1% /zredpool1
zredpool1/sambasharecompr 942669184 128 942669056 1% /zredpool1/sambasharecompr
zredpool1/notsharedcompr 942669184 128 942669056 1% /zredpool1/notsharedcompr
zredpool1/notsharednotcompr 942669184 128 942669056 1% /zredpool1/notsharednotcompr
ls -al /zredpool1/

total 6
drwxr-xr-x 5 root root 5 Apr 1 15:09 .
drwxr-xr-x 29 root root 4096 Apr 1 14:29 ..
drwxr-xr-x 2 user root 2 Apr 1 15:09 notsharedcompr
drwxr-xr-x 2 user root 2 Apr 1 15:09 notsharednotcompr
drwxr-xr-x 2 user root 2 Apr 1 15:08 sambasharecompr

At this point we should have (3) datasets on the pool; one is Samba shared and has fast and efficient Compression turned on (similar to NTFS filesystem compression or “zip folder” compression, but better.) The second is not Samba shared and also has compression; and the third is not shared or compressed. All three have “atime” turned off for speed and to avoid disk writes every time they’re read from, but again you can change this per-dataset.

You can find more information on ZFS datasets here:

https://www.nas4free.org/wiki/zfs/dataset

ZFS Administration, Part X- Creating Filesystems


Now if everything has gone well, you can go to a Windows box on the same network and issue:

‘ start \\hostname ‘ # where “hostname” is the name of your Lubuntu install

samba-share-discover

Click on the share name displayed, and it should ask you for a username and password; use the same credentials that we set a password for above. You should immediately be able to start copying files to the Samba share, and control file/directory access on the ZFS server side using standard Linux file permissions (rwx), users and groups (user:root). Since file compression is set on this share, almost anything copied to it will use less space (except for already-compressed files like MP3s, movies, or executable binary programs – the ones that end in “.exe”.)

Yes, once you have your pool up and running, standing up a new Samba share is literally that easy with ZFS. 🙂

To conclude, let’s take a snapshot of our pool for safety:

zfs snapshot -r $zp@firstsnapshot
zfs list -r -t snapshot

More info on snapshots: https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/

Old Vista Laptop Into A Linux ZFS File Server Part 2

zfslinuxweb

In the previous Linux ZFS File Server article I put forth a list of parts that allowed me to utilize an old Vista laptop as a Linux+ZFS fileserver. In this article, I will detail how to put all the pieces together, from installing the Linux OS to connecting all the hard drives.

First, we need to connect all the hardware. The eSATA card needs to be plugged into the slot, the USB3 Ethernet adapter needs to go in an available USB2 slot and connected with a CAT5 or better (CAT5e, CAT6) Ethernet cord to your existing router.

For the OS install part, we can leave the Probox unpowered and unconnected to the eSATA card.

If you’re planning on doing 2 separate ZFS pools, then you can (for example) have 2x1TB WD RED NAS drives and 2x2TB Seagate NAS drives in the same enclosure. This configuration can also come in handy if you want to practice growing a non-RAIDZ ZFS pool in-place – replacing 1TB drives with 2TB (or 4TB) one at a time, in order to give your existing pool more available free space.

Obtaining and installing the OS

Before starting on this, you will need to have downloaded and burned to DVD (it’s ~762MB, so won’t burn to a standard CDR) ” lubuntu-14.04.5-desktop-amd64.iso “:

  • At the initial boot screen, you will want to select “ Install Lubuntu “. You can choose to download updates while installing if you’ve already connected the Ethernet, but you don’t need to install Fluendo.
  • You could just as well choose to use Xubuntu, but I prefer Lubuntu for really low hardware requirements and resource usage. As long as it’s 64-bit, either will work for ZFS.

When it comes to disk partitioning, choose “Something else”:

partitionsetup

partitioning

I am using Virtualbox here to recreate the laptop hard disk setup and provide convenient screenshots. If you’re ditching Vista and reusing the hard drive as Linux only, you may want to use a partition scheme that goes something like this:

  • SDA2 as the root filesystem to install Lubuntu 14.04 – at least 15GB;
  • SDA3 as reserved space for another OS install;
  • SDA4 as Extended partition;
  • SDA5 as shared swap; (you might as well go ahead and make it 2GB to match installed RAM)
  • SDA6 as shared /home;
  • SDA7 as /mnt/extra.

It’s up to you whether you want to reserve space for another dual-boot Linux OS or just leave it as unused space for a future upgrade but I highly recommend doing this, as it will save you time later. (You could also install Centos or Fedora if you’re more familiar with them.) ~20GB each should be plenty for a 64-bit Linux root filesystem and /home unless you have really specific requirements.

The install will ask you what non-root username you want to use to login; you can use pretty much whatever is most useful to you, but use a non-weak password if you’re concerned about security. This example is setup for convenience and will automatically start the GUI desktop after booting:

usersetup

After the install completes and you’ve finished rebooting, open up a Terminal:

terminal

At the Terminal, type in ‘ sudo su – ‘ to become root.

Before proceeding with the ZFS install, we should bring the rest of the system up to date, so issue:

apt-get update && apt-get upgrade
  • As of this writing, from a base install of the latest Lubuntu 14.04.5 ISO, it will need to download 112MB of updates – Firefox being one of them, which is why we haven’t started the browser yet.
  • The update process will probably put a box up on the screen saying “ A new version of Ubuntu is available. Would you like to upgrade? ” IMPORTANT: YOU SHOULD CLICK ” DON’T UPGRADE “!! and then click OK on the next box. Otherwise it will attempt to upgrade Ubuntu 14.04 to 16.04(!) and at that point, you’re beyond the scope of this article. 😉

Once the updates are completed, reboot and then switch to virtual desktop #2 (by clicking on the 2nd “gray box” at the bottom left), fire up the Web browser and check out the ZFS on Linux Ubuntu instructions:

zfslinuxweb

Reopen a Terminal on virtual desktop #1 and become root again. Follow the instructions on the ZFS on Linux webpage to add the ZFS repository and some needed software to your system:

add-apt-repository ppa:zfs-native/stable
apt-get update && apt-get install ubuntu-zfs samba smbclient smartmontools mc screen openssh-server build-essential

As of this writing, this will download ~50MB more to install the packages and dependencies.

After it’s all done installing and configuring, you can issue the following:

modprobe zfs
zpool status

And you should get a response of:

 no pools available
  • Finally, you can now plug in the Probox’s power supply, connect it with the included eSATA cable, and put at least (1) hard drive in it. Then, power it on with the switch.
  • If you’re starting with 1 drive and working up to a full 4-drive RAID10, all four drives need to be the same manufacturer/model/capacity — e.g. Western Digital 1TB RED NAS drives, in my case.

PROTIP: Make sure the Probox is on the first setting – eSATA – not USB3.

After about 30 seconds (to allow things to settle), you can check to see if the new drive(s) were detected by issuing this command:

dmesg |egrep 'sd |scsi'which should show something similar to this:

[ 4.508395] scsi 9:0:0:0: Direct-Access ATA WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[ 4.520897] sd 9:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 4.521114] sd 9:0:0:0: Attached scsi generic sg2 type 0
[ 4.521483] scsi 11:0:0:0: Direct-Access ATA WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[ 4.521922] sd 11:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 4.521925] sd 11:0:0:0: [sdc] 4096-byte physical blocks
[ 4.522026] sd 11:0:0:0: [sdc] Write Protect is off
[ 4.522030] sd 11:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 4.522075] sd 11:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 4.522443] sd 11:0:0:0: Attached scsi generic sg3 type 0
[ 4.523423] scsi 12:0:0:0: Direct-Access ATA WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[ 4.524043] sd 12:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 4.524047] sd 12:0:0:0: [sdd] 4096-byte physical blocks
[ 4.524148] sd 12:0:0:0: [sdd] Write Protect is off
[ 4.524152] sd 12:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 4.524196] sd 12:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 4.524768] sd 12:0:0:0: Attached scsi generic sg4 type 0

You will need to have at least an “sdb” drive (sda should be your Linux install) before proceeding.

Once you have the drive(s) detected by the OS, we can get to the good parts – configuring your ZFS pool and sharing the data set(s) over Samba! 🙂

Next Article: Part 3 – Creating and Growing Your ZFS Pool

Old Vista Laptop Into A Linux ZFS File Server

probox-includes

You might be surprised at how much potential for usefulness still remains in older equipment.

My wife’s old laptop originally booted with Windows Vista, which (apart from being a generally substandard OS – Microsoft employees ran into so many problems with it, they reportedly didn’t even use it internally*) is no longer supported*(2). Mainstream support for Vista ended back in 2012, and as of April 11 2017, Vista will be officially dead.*(3)

*REF: http://www.nytimes.com/2008/03/09/business/09digi.html

http://www.computerworld.com/article/2537262/technology-law-regulation/microsoft-execs-struggled-with-vista-too–say-internal-e-mails.html

*(2)REF: https://www.extremetech.com/computing/245853-windows-vista-support-ends-next-month-upsetting-almost-no-one

*(3) REF: https://support.microsoft.com/en-us/help/13853/windows-lifecycle-fact-sheet

The laptop itself is a Dell Inspiron 1525, with a single-core 64-bit Celeron CPU that was bought around 2008. It has an HDMI video-out port, 2GB RAM, a DVD burner, and originally came with a 160GB laptop SATA drive.

More info on the laptop specs here: https://en.wikipedia.org/wiki/Dell_Inspiron_1525

The main limitations of this laptop are the 2.13GHz single CPU core and the 100Mbit Ethernet port. But it still has quite a bit of life left in it, thanks to Linux and a few accessories.

inspiron-1525usb3-gig-etheresata-2port-adapter

With a few bits of hardware I had lying around or bought for the purpose, I was able to increase the laptop’s network speed significantly and enable extra disk storage by using these two items:

1) (~$14.75) Plugable USB 3.0 to Gigabit Ethernet Network Adapter (ASIX AX88179 chipset)
2) (~$58.00) 2 Port SATA 6 Gbps ExpressCard eSATA Controller Card (ECESAT32)

My ZFS 4-disk pool file transfer speed went from ~12MB/sec over my Gigabit home network to ~30MB/sec sustained (transferring files with tar over netcat) with this one simple change!

I also replaced the laptop’s aging 160GB SATA hard drive with a 120GB SSD drive that I had leftover from a previous project, which made boot speed and the system in general MUCH faster.

3) SSD laptop drive: KINGSTON SSDNOW SH103S3120G
Nearest equivalent: (~$50.99) Kingston Digital 120GB SSDNow UV400 SATA 3 2.5″ SUV400S37/120G

To make a fairly spacious and redundant ZFS fileserver, I decided to use my existing 4-bay Probox:
(~$99.99) Mediasonic ProBox HF2-SU3S2 4 Bay 3.5” SATA HDD Enclosure – USB 3.0 & eSATA

proboxprobox-rearview

probox-includes

Some nice things about the Probox: It includes all necessary cables, supports up to (4) 5.25” SATA hard drives (NAS type drives recommended – I used Western Digital 1-2TB Red drives for lowest budget, although they are slower; I can also recommend Seagate 2TB NAS drives for budget and speed; both models are under $100/each) and it has a selectable eSATA and USB3 cable connection. USB3 on Linux has limitations — if disks are swapped out, drive assignments can change; also you lose SMART disk status and testing ability — so I set the Probox to use eSATA.

So if you had one of these laptops (or similar) and wanted to closely replicate my project (before filling the Probox with drives), your cash layout as of this writing would be about $173 plus tax, without the SSD drive. Now you can buy a basic Dell Inspiron 3000 laptop for around $130-150, or a 4-bay diskless QNAP or Synology NAS for around $280-$300 – BUT that’s not the point.

If you’re at all like me, you probably have some stuff lying around the house and would like to get further use out of your investment without relegating it to the trash or giving it away. This article is for you. Using a pre-existing piece of equipment and making it into something useful also gives me something to do in my free time while searching for a new Linux-related job, as well as rewarding me with a sense of accomplishment when I get it working as desired. And I will also mention that the above equipment is Optional. If your existing hard drive is still working fine after almost 9 years and you don’t need the best network speed, you can still go forward with what you have. In addition, you should be able to adapt these instructions to something other than a random Dell laptop.

The nice thing about having the extra equipment, however, is that it can be used for other projects as well. (I have several of the USB3 Gigabit Ethernet adapters and have also recommended them to friends.) Although the USB3 Ethernet adapter is limited to USB2 speeds for this project by using an eSATA card instead of a 2-port USB3 card, it’s still far better than the stock 100Mbit Ethernet.

The ZFS filesystem allows you a certain amount of flexibility, by the way. You don’t need to create a single ~1.8TB zRAID10 pool using all four drives – with the Probox, you could have (2) separate pools with 2 drives each if you wanted – but each pool’s size would be limited to the drive size in that case. Stepping up to 4TB NAS drives would make sense – however, they still cost ~$130/each at this time, and this is a “limited budget” project.

But getting on with the details… A critical step before installing the SSD was first making a copy of the Dell utility partition so I could still access the BIOS. I was able to do this by booting System Rescue CD*(4) and copying the 160GB drive’s first partition with DD to a USB thumbdrive I had lying about.

*(4) http://distrowatch.com/table.php?distribution=systemrescue

Command used in SystemrescueCD to backup the Dell partition:

dd if=/dev/sda1 of=/mnt/thumb/sda1-dell-utility-partition.dd bs=1M

Switching to a low-cost SSD had a number of benefits (besides speed) – less power usage, fewer moving parts, and less heat.

One of the good things about the “ZFS way” is that it simplifies things like Logical Volume Management (LVM) and RAID into a more manageable methodology, while providing end-to-end data protection – plus easy snapshots. ZFS also makes file-sharing your data with Windows machines dead simple by using Samba.

If budget is a real issue (say, you get paid once every couple of weeks), you can literally start off with (1) ZFS hard drive, do a burn-in test to make sure the drive doesn’t fail early, and start putting data on it; then add a mirror drive later – when you can afford it – for RAID1 redundancy. ZFS will copy the drive’s data in real-time and you can still access your files, albeit with something of a performance penalty. Once it’s done “resilvering” though, your read speeds should improve because ZFS can now pull the data from both spinning platters.

Now say it’s two weeks later, and you can afford to buy your 3rd drive (Important note: All drives in a given ZFS pool should be of the same manufacture, model, and capacity) – you get it shipped to your house, put it through a burn-in test, and then you can let it sit for a couple of weeks until you buy the 4th and final drive. Pop those into the Probox enclosure and you can expand your ZFS pool on-the-fly to a 4-drive RAID10. (Yes, I have done this. Yes, it actually works.)

Moving on… After replacing the laptop’s hard drive with the SSD, I again booted System Rescue CD and recreated the Dell utility partition with ‘ fdisk ‘ (you need to use fdisk to change the Partition type to “de”) and then used ‘ gparted ‘ for the rest of the layout. Note: run ‘ startx ‘ before doing gparted.

SSD Drive layout (fdisk -l):

Disk /dev/sda: 120 GB, 120031511040 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 6 48163 de Dell Utility
/dev/sda2 * 6 1572 12578895 83 Linux
/dev/sda3 1572 3139 12586927 83 Linux
/dev/sda4 3139 14594 92012287 5 Extended
/dev/sda5 3139 3330 1534207 82 Linux swap
/dev/sda6 3330 3591 2096482 83 Linux
/dev/sda7 3592 14594 88373565 83 Linux

## Put the Dell “de” partition data back on the SSD
# dd if=/mnt/thumb/sda1-dell-utility-partition.dd of=/dev/sda1 bs=1M

blkid details:

/dev/sda1: SEC_TYPE="msdos" LABEL="DellUtility" UUID="3030-3030" TYPE="vfat"
/dev/sda2: UUID="93fce241-xxx-cb6f1b01aee3" TYPE="ext4"
/dev/sda3: LABEL="rootantiX-16" TYPE="ext4"
/dev/sda5: LABEL="ssdswap" TYPE="swap"
/dev/sda6: LABEL="ssdhome" TYPE="ext4"
/dev/sda7: LABEL="ssdextra" TYPE="ext4"

After copying the Dell utility partition data back to the SSD, I downloaded and installed
” lubuntu-14.04.5-desktop-amd64.iso ” (*5) to /dev/sda2 and used a separate partition for /home.

For the curious, the laptop dual-boots with AntiX MX-16 on sda3 and is using the same swap and /home (sda6). The extra space on sda7 is used as /mnt/extra; it’s for storing ISOs, music, movies and backups. That’s another nice thing about using a general-purpose laptop for a project like this — the primary use that it’s been put to is actually as a DVD player.

(*5) http://distrowatch.com/?newsid=08395
http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/

If you are new to Linux and ZFS, I highly recommend starting with Ubuntu 14.04 LTS instead of the newer 16.04. In my experience, 14.04 is very stable and not very prone to unexpected behavior. The Xubuntu or Lubuntu flavors are fairly light on resources and leave quite a bit of room for ZFS caching if you have at least 2GB of RAM.

An important note: DON’T try to use this setup (laptop + Probox) in a “production” environment(!)
It’s OK for home use/light usage, demos, testing, and learning more – but get a real server going if you expect reliability.

If you need a 4-port SATA3 PCIe card and don’t want to delve into SAS, I can recommend this one since it has port multiplier, will work with the Probox, and delivers much better disk speed:

(~$50) Vantec 4-Channel 6-Port SATA 6Gb/s PCIe Card (UGT-ST644R)

(~$7) ESATA TO SATA Cable male to male M/M Shielded 6Gbps [sic]
^^ Use something like this cable if you want to connect the PCIe card’s Internal port to the Probox

Some PROTIPs / Best Practices for ZFS on Linux:

1) Don’t enable de-duplication on ZFS. Use compression=lz4 instead.

2) DO: Use the equivalent of zRAID10 (mirrored pairs), not raidz/raidz2/raidz3. You will thank me when it’s time to replace a failed drive! Rebuild times for a RAIDZ can be significantly longer than you might expect. In addition, it’s my considered opinion that any drive of 1TB or over should have at least 1 mirror. With ZFS, you can even do triple-mirroring if you like: (1 drive +2 mirrors)

REF: http://louwrentius.com/zfs-resilver-performance-of-various-raid-schemas.html

3) DO: USE ZFS SNAPSHOTS. Seriously. (Caveat: you will need to delete snapshots occasionally to free up disk space if you move files/directories around or delete them entirely.)

zfs snapshot -r zfspool/dataset@description-currentdate

4) DO: Burn-in Read/Write test all of your new drives before allocating them to a ZFS pool.
This is one of the most important things to remember!

time badblocks -f -c 10240 -n -s -v /dev/sdX

## ^^ ONLY run this command on new or “blank” drives – I take no responsibility if you run it by mistake on your boot drive! (Also note this can take hours to finish, especially for larger drives)

5) DO: Use GPT partitions on your ZFS drives, not “msdos” type partitions.
Also, allocate the entire drive to ZFS – don’t use partitions like you might with LVM.

parted -s /dev/sdX mklabel gpt

## NOTE data loss if used incorrectly! Only use to initialize a new drive as this will overwrite / re-partition the entire drive!

6) DO: Use /dev/disk/by-id when creating your ZFS pools, not /dev/sda naming.

7) DO: Always create your pools using ‘ ashift=12 ‘ for spinning hard drives, regardless if they have 512 sector sizes or 4K sectors. This will save you massive amounts of hassle.

8) DO: Use ‘ atime=off ‘ everywhere, unless you really need it. (Equivalent of “noatime” in fstab)

9) DON’T: make an unbalanced RAID10 (3 drives == 2x mirrored, 1x non-mirrored) mirror it ASAP!

10) DON’T: If using a RAID-capable card to connect your drives, don’t use it in RAID mode – use “JBOD” mode or “just a bunch of disks” to configure your ZFS pool.

Guideline: More RAM is almost always better. But if you’re in a “low budget” situation, you can achieve “usable” ZFS experiences with as little as 2GB RAM if you are willing to sacrifice some conveniences (like GUI and speed.)

Next Upcoming Article – Part 2: Putting it All Together

P2V: Debian Testing PXE server and Vmware Workstation 11

ASUS Spresso Open

MARCH 2017 – so I’ve had an old PXE an ASUS Spresso (S-presso) with a Pentium-4 sitting around for the last few years, and got a wild hair to do a P2V (Physical-to-Virtual) installation on it. First, I had to make sure it still works right after attempting an OS upgrade of Debian Testing (in a VM) to modern standards. Last time this box was booted was November 2013 and it was provisioned with “Dreamlinux 5″ back in January of 2012.

Thankfully, the SATA drive and CMOS battery have survived with apparently no ill effects, since the box has been moved around through a couple of house changes with no special storage arrangements – it’s basically been “unpowered” sitting in a corner.

ASUS Spresso

ASUS Spresso Open

ASUS Spresso back

 

( NOTE these are stock images. My unit has a PCI slot on the right that has a Gig Ethernet card in it and an unused AGP video card slot on the left. The motherboard only supports 100Mbit Ethernet. )

A few notes about PXE:

REFERENCE – https://en.wikipedia.org/wiki/Preboot_Execution_Environment

PXE HOWTO – https://debian-administration.org/article/478/Setting_up_a_server_for_PXE_network_booting

Tools used:

– FSArchiver

– Vmware Workstation 11

– SystemRescueCd

– KNOPPIX

– GParted

STEP 1: BACKUP the physical PC

This 32-bit box is so old, I had to install FSArchiver (my preferred bare-metal backup program) manually. Incidentally, it also weighs a ton with a DVD drive and a standard SATA hard drive, so removing the DVD and converting to an SSD (or laptop drive) to save weight and would be recommended if I wanted to haul it around.

(Commands entered on Physical box, bash shell, as root)

apt update && apt install fsarchiver -y

apt update && apt install fsarchiver -y
ddir=/mnt/extra
outfile=spresso-p2v-backup-20170302.fsarchive.fsa
rootdev=/dev/sda3
time fsarchiver -o -A -z 1 -j 2 savefs \
$ddir/$outfile \
$rootdev
cd $ddir
fsarchiver archinfo $outfile 2> flist--$outfile.txt

I then copied “spresso-p2v-backup-20170302.fsarchive.fsa” to my ZFS file server (method omitted), since we will need it later for the restore.

Also, since all of my Squid cache and ISO files for PXE are on /mnt/extra on the Spresso, I made a tar backup of those to my ZFS server.

cd /mnt/extra
time tar czpf /mnt/bkpspace/spresso-mnt-extra-bkp-20170302.tgz *

STEP 2: Setup the P2V VM in Vmware Workstation 11 and restore the backup

I tried as much as possible for the VM settings to duplicate the physical box, but in this case I already had a P2V VM of my daily workstation (XUbuntu 14.04 LTS) that I wasn’t using, so adding a 3rd virtual drive to the existing VM and reusing the existing GRUB saved me some steps. Running

update-grub ‘ on the Ubuntu side after restoring enabled me to boot my restored environment.

VM details:

RAM: 1.5GB

Processors: 1

SCSI1: 23.5GB (this is the Ubuntu drive)

SCSI2: 20GB ( this is /home and swap )

SCSI3: This was added “as new” and then Expanded from 10GB to 60GB (after running out of space)

Net1: Bridged (gets DHCP address from my LAN)

Net2: Started out as Host-only and ended up as LAN Segment – this enabled the “client” bare VM to boot over the network.

Details for the Spresso PXE P2V VM

I used a SystemRescueCd ISO to boot the VM, issued ‘ startx ‘ and used ‘ fdisk ‘ to make 2 partitions that fairly closely matched my physical layout.

PROTIP: In hindsight, I should have made the root partition around 15-20GB because of all the package upgrades that needed to be downloaded. (I ran out of space on root once during the upgrade and had to issue an ‘ apt-get clean ‘ to free up space.)

sdc3: 10GB ext4 (restored root) – Advice: Make yours bigger.
sdc4: 50GB ext4 (/mnt/extra)

PROTIP: As root, run ‘ tune2fs -m1 /dev/sdc3 ‘ — this will also reduce the Reserved space for ext4 and give you some extra usable free space.

Now, since I had plenty of space to work with, I copied my FSArchiver backup file from my ZFS server to the 50GB VM partition.

(Still in SystemRescueCd, in the VM)

mkdir /mnt/tmp
mount /dev/sdc4 /mnt/tmp
cd /mnt/tmp
ifconfig # Make note of my VM's IP address: 192.168.1.95
nc -l -p 32100 |tar xpvf -
##Netcat - Listen on port 32100 and untar whatever is received

(Now on my ZFS server)

tar cpf - spresso-p2v-backup-20170302.fsarchive.fsa |nc -w 5 192.168.1.95 32100
## (tar to stdout and stream a copy of the backup file to the VM's IP)

This is basically a quick-and-dirty way to transfer files over the network without resorting to FTP or slow SSH file copies – the fsarchive backup file ended up being around 3GB and transferred in less than a minute over Gig Ethernet.
Netcat REF: http://www.terminally-incoherent.com/blog/2007/08/07/few-useful-netcat-tricks/

Now to restore the backup:

(still in SystemRescueCd, in the VM)

time fsarchiver restfs spresso*.fsa id=0,dest=/dev/sdc4

And that’s half the battle right there. Now we just need to make some changes to the restored /etc/fstab so it will boot. ( If we were doing a full migration, we would also need to adjust things like /etc/network/interfaces , /etc/rc.local , /etc/hostname , and /etc/hosts )

For completeness, here’s more info on how to do a Linux bare-metal backup and restore:
REF: http://crunchbang.org/forums/viewtopic.php?id=24268

So now I ‘ reboot ‘ into the VM’s already-installed Ubuntu 14.04 and edit my Dreamlinux /etc/fstab.

mkdir /mnt/tmp
mount /dev/sdc3 /mnt/tmp
screen -aAO # PROTIP: GNU screen is invaluable for switching between virtual terminal windows
fdisk -l # check out our disks
blkid # get partition labels and UUIDs
/dev/sdb2: LABEL="swapb" UUID="f0eb7148-4ff2-4eeb-a82c-349d384a5255" TYPE="swap" PARTUUID="ff1ca075-02"
/dev/sdc3: LABEL="root" UUID="38f1d4be-3293-4272-ab79-4ad76cbd5a36" TYPE="ext4" PARTUUID="3a3b65f3-03"
/dev/sdc4: LABEL="extra" UUID="603a61fc-4436-4cb0-baac-ef9170754228" TYPE="ext4" PARTUUID="3a3b65f3-04"

NOTE that FSArchiver will by default restore the same UUID and filesystem label, so no worries – we don’t have to modify the VM’s fstab entry for the root filesystem.

( Now Hit Ctrl-A, then c to create a new Screen )

cd /mnt/tmp/etc
jstar fstab

( Use your own editor here. I happen to like Wordstar keybindings. )

Now we can switch between those 2 virtual terminal windows and even copy/paste text without using the mouse. (See ‘ man screen ‘ for more details.)

To make a long story short, I added or verified the following to SpressoVM’s /etc/fstab to enable my existing swap partition and double-check that the root filesystem would be mounted as expected.

LABEL=swapb none swap sw,pri=2 0 0
LABEL=extra /mnt/extra ext4 defaults,noatime,rw 0 2

While I’m here, I also restored the /mnt/extra files from their tar backup as well.

umount /mnt/tmp # we're done with restored root
mount /dev/sdc4 /mnt/tmp
cd /mnt/tmp
nc -l -p 32100 | tar xzpvf -

( then on my ZFS Server )

cd /mnt/bkpspace; time cat spresso-mnt-extra-bkp-20170302.tgz |nc -w 5 192.168.1.95 32100

( now back in the VM )

update-grub # make sure ubuntu knows how to boot dreamlinux
reboot

And that’s pretty much it. After that, to make a long story short (again), I went through several cycles of ‘ apt upgrade ‘ and ‘ apt-get dist-upgrade ‘ making VM Snapshots along the way, and I had to make allowances for files that were provided in more than one package.

I also upgraded the kernel to linux-image-4.9.0-1-686. The full saga is documented with errors and fixes, so email me if you want to know more (but it’s a pretty long read.)

apt-cache search 4.9.0 |awk '{print $1}'
apt-get install linux-headers-4.9.0-1-686 linux-headers-4.9.0-1-686-pae \
linux-headers-4.9.0-1-all linux-headers-4.9.0-1-common linux-support-4.9.0-1 \
linux-image-4.9.0-1-686-pae

STEP 3: I tested PXE booting with another VM using a dedicated network segment.

The end result of all this: I created a “blank” VM with the capability to boot from network (needed to modify the VM’s BIOS for this by pressing F2 at boot) and after switching the 2nd network adapter from “Host only” to “LAN segment” I successfully booted the VM from PXE! Mission Accomplished!

Now, since I’ve done all the heavy lifting in the VM and my original box is still working the same as it was (but running old software) I can use pretty much the same procedure to do a V2P (Virtual to Physical) to a spare 500GB laptop drive instead of having to repeat the upgrade all over again.

For brevity, these are the instructions I include in my bkpsys-2fsarchive script on how to Restore:

time fsarchiver restfs backup-root-sda1--ubuntu1404*.fsa id=0,dest=/dev/sdf1
Statistics for filesystem 0
* files successfully processed:....regfiles=159387, directories=25579, symlinks=49276, hardlinks=25, specials=108
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
real 4m26.116s
( 3.9GB )
 mkdir /mnt/tmp2
 mount /dev/sdf1 /mnt/tmp2
 grub-install --root-directory=/mnt/tmp2 /dev/sdf
mount -o bind /dev /mnt/tmp2/dev; mount -o bind /proc /mnt/tmp2/proc; mount -o bind /sys /mnt/tmp2/sys
chroot /mnt/tmp2 /bin/bash
update-grub
[[
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.2.0-36-generic
Found initrd image: /boot/initrd.img-4.2.0-36-generic
Found linux image: /boot/vmlinuz-3.19.0-25-generic
Found initrd image: /boot/initrd.img-3.19.0-25-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
Found Ubuntu 14.04.4 LTS (14.04) on /dev/sda1
done
]]
grub-install /dev/sdf # from chroot

^D
umount -a /mnt/tmp2/*
# DON'T FORGET TO COPY /home and adjust fstab for swap / home / squid BEFORE booting new drive!
# also adjust etc/network/interfaces , etc/rc.local , etc/hostname , etc/hosts

Restored PXE-server Spresso running in Vmware

Details of the "blank" VM that boots from PXE

BIOS details for the "client" PXE-booting VM

Once I had KNOPPIX up and running in the “blank” VM, I used ‘ GParted ‘ to make a 768MB Swap partition and used the rest as a “data” ext4 partition.

–Here’s the “client” VM booting over the network/PXE from my migrated Spresso VM:

VM booting over the network/PXE

–NOTE that while the PXE box is running a 32-bit processor and environment, CLIENT boxes can boot a 64-bit kernel and environment as long as the CLIENT processor is capable.

–The way my PXE environment is setup, clients only have a limited network connection from the PXE server and have to download everything (including OS install packages) over the Squid proxy cache running on the Spresso’s port 3128 for extra security – client boxes can’t ping out to random websites and all downloads are logged.

Squid proxy REF: https://en.wikipedia.org/wiki/Squid_%28software%29

Knoppix DVD 64-bit (selection #99) in the "blank" VM

( After booting into text mode, I issued a ‘ startx startkde ‘ at the terminal prompt to get X up and running )

Knoppix DVD 64-bit (selection #99) in the "blank" VM

STEP 4: Upgrade the ISOs in the VM to latest Knoppix and SystemRescueCd (TODO)

STEP 5: V2P: Reverse the process and upgrade the physical box with all the updates from the VM

–I haven’t done the “V2P” – Virtual to Physical – part yet, but that’s not a high priority at this point since I have everything pretty much the way I like it right now. Maybe in a future update. 😉