Poor man’s Guide to Remote Linux Desktop Support

Before anyone poo-poos this, read the ENTIRE article first. 😉

Today’s article is the third in a series I call the “Getting work done” article series.

One of the most common things we face as Linux enthusiasts is the “vacuum of despair” that comes about from switching family to Linux without fully thinking through backups and restoration. It usually starts with the dreaded phone call that they tried to “do some Linux.”

Sometimes it’s as simple as fixing a broken X server install. In other cases, it could be more complicated, like the time your Aunt Martha accidentally knuckle pounded Ctrl+H in her home directory when trying to coax the cat off the keyboard. In what suddenly feels like a bad action movie, she’s convinced herself that she’s being hacked due to the sudden changes taking place on her monitor.

“All of these weird “dot directories” just sprang to life”, she explains to you in a three hour phone call. Aunt Martha, being the ever-vigilant sort, got right to work…deleting any and all “rogue” dot directories. With “Eye of the Tiger” blasting on her 1970s Hi-Fi system in the living room, good ‘ol Aunt Martha gets right to work….deleting most of her home directory’s needed config files. She’s in the zone, on a mission – nothing can stop her. And that folks, is going to be a problem.

As she tells you her tale of heroism, all you can think about is how old her last home directory backup is. Oh yeah, that’s right – there isn’t one – she’s screwed! Sentimental pictures of Uncle Bob’s pie eating contest, email contacts for her kids who never call, expired eBay auction bookmarks for precious moments figurines – all gone! Things could definitely get ugly here.

Preventing trouble before it starts

The above dramatization may seem unlikely, but I would remind you that elements of it are not all that uncommon. Fact is, many of the best 1970s Hi-Fi systems are still in use today. In the interest preventing anyone from listening to movie-themed music best left in the past, let’s come up with a solid plan for recovery should a Linux disaster take place. Ready, okay…let’s do this.

My recommended plan of attack is a system of backup and prevention solution I’ve used successfully with a number of people. It basically comes down to the following:

- 2 Linux installations, one PC. One as the main install, the second one as a workbench for remote recovery.

- 2 Installations of TeamViewer or x2go with DynamicDNS in play

- Both installations should be set to auto-login. This will make working remotely much easier.

- Have a mutually agreeable barter arrangement in place – I recommend homemade baked goods – never try to “do the Linux” on an empty stomach.

Now, before I go any further, some of you might point out that merely backing up her home directory is all that’s really needed, right? Wrong!

Unless you plan driving across the state only hear the latest diatribe on “Mr. Whiskers’ adventures in feline incontinence.” Remember folks, we’re talking about restoring a completely trashed installation from anywhere with an Internet connection.

Personally, I’d rather fix her PC remotely from my home while binge watching Netflix in my underwear.

Note - I do not recommend rocking this dress code while setting up a backup/restore plan at your Aunt Martha’s house. It’s both highly inappropriate and unfashionably drafty. Moving on…

Set it and forget it

Now the following requires a single trip to your Aunt Martha’s house. But if you do this right, FREEDOM and then NETFLIX!

Partitions: After backing up her Linux PC’s docs, images, email, bookmarks and stuffing your face with those baked goods mentioned previously, nuke the hard drive and reformat with at least two partitions.

Save everything mentioned above to a flash drive or whatever you have handy. You’ll want to make one of the installation partitions smaller than the main installation partition, since it’s not going to be for daily use.

Pro tip - If your Aunt Martha is a baller, you might want to consider additional partitions…just because. I’ll leave that up to you to evaluate her “street cred” on that front. But at a minimum, you need to set up two dedicated Linux-ready partitions formatted in the file system of your choosing.

Installation: Install her favorite Linux distribution onto both pre-defined installation partitions.

Setup access: Install and configure TeamViewer or x2go with DynamicDNS enabled on her router.
If you decide to go with x2go, make sure to open the proper ports on the router and verify DynamicDNS is set up correctly. Personally, I’ve found that for relatives, TeamViewer is the way to go. Just set it up for unattended access on BOTH install partitions. While it’s only mandatory to have it installed on the recovery partition, including it with the main installation means you can bail out Aunt Martha should the cat jump on the keyboard and disappear one of the panels. Minor issue, but it really helps to have a remote in to fix it.

Setup GRUB: Before going any further, I recommend setting up GRUB so it’s booting the default installation if the keyboard is untouched at boot. This will ensure the right installation is being selected at each system start up.

Important note- The following method relies on using rsync to make a series of backups of system files and directories, while also maintaining their ownership and permissions. I fully realize that if I have a reliable SSH connection…attempting a LVM snapshot restoration (or other alternatives) could be doable using a variation of this tutorial. I personally think using SSH to restore LVM is a pain in the butt. If you’re bent on using LVM regardless, here’s a good tutorial to get going with LVM snapshots. Personally, I’ve never been a fan of LVM snapshots for home users.

Setup your restoration: With GRUB setup correctly, now you’re ready to get the main installation the way Aunt Martha likes it. Got it? Fantastic – now let’s prep our backup options. First, there’s our backup installation partition. It’s going to be a vanilla install that if you do your job right, it will have one job – to provide a workbench environment we can use to restore Aunt Martha’s main installation if everything goes to hell.

Our restoration partition with the smaller of the two Linux installations is really little more than a working area for our use. It won’t actually store any backup data itself. Ideally, you’ll have a separate destination drive, likely an external backup drive, that will be formatted with your choice of a Linux file system and will be labeled via GParted as something human readable like “backup drive.” I like to label drives in this fashion as it makes sure they rsync data back and forth with a static path.

I would also recommend using GParted to label your Aunt Martha’s main Linux installation partition from within the recovery partition as something more human readable. This will make setting up a crontab to backup the main installation’s system data over to the external backup drive a much simpler process.

Why? Depending on the configuration, drives tend to change their label each time they’re mounted. If you haven’t spelled things out for your other drives in your fstab, using labels will save you a lot of grief.

This includes any instance where the computer is rebooted. And since we want the label (/media/user/label/), not the device path (/dev/sdbx). Note, some distros might mount your device under /run/media/user/label/ instead. Regardless, labeling the external drive will make life much easier.

Using rsync to backup the system: One thing you’ll note is that I do NOT include the home directory in the backup for Aunt Martha’s main installation. Why? Do you really want to explain to your relatives that you can’t retrieve their home directory contents because something timed out during the rsync process? Don’t be a dork. Set up a separate directory on the backup drive and use another rsync instance for the main installation’s home directory. By keeping these two backups separate, you ensure you have a plan b should the full system backup go south on you.

Once you’ve booted into the main installation, you’re going to setup a crontab so that rysnc can backup both the system files and the home directory contents.

Let’s give this a test run! In a terminal, on the main installation – this backs up the system files:

rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/system-backup-folder

As per the Arch Wiki, you may need to exclude the swap space.

In a terminal, on the main installation – set up an ignore list and back up the home directory files:

wget https://raw.githubusercontent.com/rubo77/rsync-homedir-excludes/master/rsync-homedir-excludes.txt -O /var/tmp/ignorelist

(Review what is commented out and what is not and make sure it’s set how you want it – the list should be a good match for Aunt Martha.)

To make any changes:

 nano /var/tmp/ignorelist

Then backup the home directory:

rsync -aP --exclude-from=/var/tmp/ignorelist /home/$USER/ /path/to/home-directory-backup-folder

In both instances, the backup folders should be on the external backup hard drive (we labeled it earlier). Assuming both backups went well, we’re ready to setup a crontab to run both backups automatically.

Schedule your backups with Cron: Setting up a crontab is actually stupid easy. The hard part is figuring out the time/day/etc. If you’d like a crash course in setting up the scheduling portion of a crontab, go for it. On the other hand, if you’d like to simply get the hell out of Aunt Martha’s computer room, use this instead.

Once you’ve settled on the method you wish to select the frequency of your crontab, you’re going to be entering the rsync instances used above into TWO separate schedules.

For example to run a system backup at 4am everyday, Monday through Friday…put this into a terminal:

crontab -e

then…

* 4 * * 1-5 rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/system-backup-folder/

Then for a home directory backup:

* 1 * * 1-5 rsync -avh /home/$USER/ /path/to/home-directory-backup-folder/

Setting this up on the main installation ensures that everything is backed up, automatically at two different times of day. This was easy enough. But what about the restoration?

The recovery

So let’s say you’ve got everything set up. You’re at home, in your underwear watching Warehouse 13 on Netflix while working on your laptop. The phone rings and Aunt Martha is calling because her main install went south. You calmly tell her not to panic. Have her reboot the PC, then at the grub prompt…instead of letting it sit there, she’ll arrow down to the recovery install.

Pro tip: Take a photograph of the GRUB screen. This way you can tell here to arrow down until it highlights the right option. Then have her hit the enter key. That’s it., Now tell her to go watch her stories on TV so you can get to work. I forgot to do this once…never again.

Once the recovery partition has booted, you’re going to login via TeamViewer (remember during the setup, we elected for unattended login). If using x2go instead, then login using it from your local machine.

Now, depending on what the problem is will reflect which recovery we’re going to run. For the sake of this article, let’s assume the entire system is completely toast and needs a full restoration.

Note: Since we did exclusions previously, there’s no need to do them again for the restore.

rsync -aAX --info=progress2 /path/to/system-backup-folder /media/user/label-of-root-directory-for-main-installation/

Remember your / for your main installation will differ from the example above. If you labeled it something easy to recognize, like mainsystem, then it might look like:

/media/user/mainsystem/

Otherwise it might look something similar to this:

/media/user/6ef30603-f2d1-43be-a3a2-734080733e00

In either case, browse and verify the directory before running a restore.

Once the system files have been restored, you can also restore the backed up home directory if needed.

(Fixed - Missed this doing it from memory, thanks Joseph)

rsync -avh --info=progress2 /path/to/home-directory-backup-folder/USER/ /path/to/main-system/HOME/

This basically translates as follows:

Main installation’s user’s data is dropped back into the main installation’s home directory.

So in Martha’s example: /path/to/home-directory-backup-folder/Martha/ to /media/USER/LABEL/home/

This gives us a restored /home/Martha once we boot back into the main installation later on.

Once completed, both the system and home files should all be completed restored. Remotely reboot and after a couple minutes, login again via TeamViewer (this time on the main installation). Ta-da! Everything is as it should be!

The Takeaway

Clearly, this tutorial isn’t for the faint of heart. There are some things I want to make clear right off the bat.

1) This is for home users. I would NOT recommend this type of setup for any business.

2) Using rsync isn’t the same as creating a proper drive image or a backup with Clonezilla, dd or a LVM snapshot. At the same time, getting either of those to work remotely across the Internet is another tutorial altogether.

3) I’m not claiming this is the “best” way to handle remote backups and restorations. I am however, telling you it works for my family and it could with yours, too.

4) Classic 1970s Hi-Fi systems are awesome. I’ve had my fill of more than a few and I have nothing but respect for the amazing responsiveness of the knobs, the display layout and the warm sound these units can generate.

Why not TimeShift? The reason I opted against using TimeShift for this is that it requires backups on the same drive partition. Also, if the system is screwed up and you’re not there locally, you’re still going to need a recovery partition or a live CD in order to get things working again…unless dropping to a shell works for you. If that’s the case, then more power to you. Personally, I loathe troubleshooting and prefer a nuke and pave restore.

Problems!

Suspend or sleep mode a problem? Just disable it, just make their monitor sleep – fun fact, they’ll never know the difference! Sounds terrible, but I do it and no one even realizes it’s a thing!

Slow internet? Networking completely busted? No remote? Make sure the boot menu is set long enough for her to select the recovery partition. Once there, have her click the executable script called CLICK ME! Assuming a desktop like MATE is being used, and it’s marked as executable, you can use a simple script to nuke her other installation’s home and rsync over the backup – “automagically.”

Suggestions or alternatives?

Clearly, this wouldn’t be the Internet without folks who have their own way of handling remote restorations. Awesome! Share your techniques in the comments below (click load comments at the bottom of this page).

I firmly believe there’s more than one way to do just about anything, so if you have an approach you feel is better somehow, shoot – I’m always interested in what others are doing.

Matt Hartley

Freedom Penguin

Freedom Penguin’s founder & talking head – Matt has over a decade working with Linux desktops, his operating system experience consists of both Windows and Linux operating platforms. In addition to writing articles on Linux and open source technology for Datamation.com and OpenLogic.com/wazi, Matt also once served as a co-host for a popular Linux-centric podcast.

Matt has written about various software titles, such as Moodle, Joomla, WordPress, openCRX, Alfresco, Liferay and more. He also has additional Linux experience working with Debian based distributions, openSUSE, CentOS, and Arch Linux.