Laptop backup/sync service

The automated laptop backup service for Linux has been enhanced and re-launched for October 2016. Originally introduced in February, 2011 to supersede an even older facility based on the original silo server, it was shut down in 2014 partly through lack of use and partly because we were trying to minimise power usage (and hence heat production) in the Maths server room from mid-2014 until autumn 2016.

The service creates a copy of of your laptop home directory on the backup server in Maths and can be accessed from anywhere within college via wi-fi or a wired network connection - a wired network will usually be faster than a wireless connection. You can also use it from anywhere outside the college by using the college's VPN service.

For those impatient to get started and are familiar with editing scripts, setting up ssh private/public keypairs, cron jobs, etc and want a simple full home directory backup, you can skip this full documentation and go straight to the quick start page instead. Otherwise, read on...

Two types of backup - what's the difference?

By default, the backup script described here will back up your laptop's home directory with the laptop backup server, copying all of your files and folders the first time you use it. On subsequent backups, it will compare what you have on the laptop with the previous backup and will add any new or changed files and folders from your laptop to the backup, leaving all other files unchanged. This is the 'backup' mode.

Alternatively, you can instead choose the 'sync' mode which synchronises the backup you have with the files & folders that are currently on your laptop. This is similar to the 'backup' mode but with one important difference - with sync mode, if you delete a file or folder on your laptop, the same file or folder will be deleted from the backup. After all, you're literally synchronising the backup with your laptop, so that they both end up with exactly the same data.

This may not be what you want since if you accidentally delete something on your laptop, you might want to retrieve it from the backup and restore it to your laptop. But that accidentally deleted file or folder won't be there if your last backup used the 'sync' mode! Fortunately, you can configure the system to always use the 'backup' mode or use the 'sync' mode and you can also override this stored setting and use a different mode on a 'one off' basis when you give the command to run it; these two methods are discussed below.

To use the backup service you will need to:

• have a laptop or netbook running Linux, MacOS or any modern version of UNIX

• have an account on the backup service; if you don't already have one you can ask for one.

• have rsync installed on your laptop

• create an SSH public/private key pair on your laptop and upload the public key to your account on the backup server once your account has been set up and you have received the details

• download the backup script sync2lapback and (optionally) auto-backup2lapback to your laptop

• test the sync2lapback backup script

• if you want automated backups, edit your crontab file to use the auto-backup2lapback script so that it starts to back up shortly after you switch on your laptop

Here is a step by step guide to what you have to do:

Getting a backup account

To get an account on the backup service, simply ask for one stating your college username.

Installing rsync on your laptop

Since Ubuntu Linux seems to be the most popular Linux distribution in Maths, you can check to see if rsync is installed on your Ubuntu system with:

dpkg -l rsync

If it is installed, you should see something like this:

andy@ubuntu2:~$ dpkg -l rsync
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  rsync          3.1.0-2ubunt amd64        fast, versatile, remote (and loca

but if instead you see:

dpkg-query: no packages found matching rsync

then you can install it with:

sudo apt-get install -y rsync

If you use a different version of Linux, a Mac or some other UNIX then consult your operating system's documentation to find out how to check if rysnc is installed and how to install it if it isn't. Typing 'whereis rsync' (on Solaris or Solaris-derived systems, use 'whatis rsync' instead) is one way to find out if it is installed.

Setting up your SSH public/private keypair

To create your ssh key pair and install the public key on the backup server, do the following:

1. log into your laptop as normal and using a shell (terminal) session, check to see if you have a subdirectory called .ssh in the top level of your home directory:

   ls -ald ~/.ssh

2. if this doesn't exist, create one with:

   mkdir ~/.ssh

3. now create your ssh keypair with:

   ssh-keygen -b 1024 -t rsa

   (ssh-keygen will be on your path on most Linux systems but if you get a 'command not found' warning, on some systems you may need to prefix the ssh-keygen command with the full path to the ssh-keygen utility - this might be /usr/bin/ssh-keygen or /usr/local/bin/ssh-keygen, for example).

   You will be prompted for the filename in which to save the key and you can just hit return to accept the default, which is to put it in the .ssh subdirectory of your home directory as the file id_rsa. Next you will be prompted for a passphrase - hit return here and again at the next prompt as you don't want one. Finally ssh-keygen will save both your private and public keys as the files id_rsa and id_rsa.pub respectively as in this example:

   andy@ubuntu2:~ $ ssh-keygen -b 1024 -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/andy/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase): 
   Enter same passphrase again: 
   Your identification has been saved in /home/andy/.ssh/id_rsa.
   Your public key has been saved in /home/andy/.ssh/id_rsa.pub.
   The key fingerprint is:
   76:36:b7:e3:5c:8e:8d:02:7a:5e:39:4a:02:a6:eb:70

4. now you need to create a .ssh subdirectory in your home directory on the backup server and then copy your public ssh key to it before enabling it. First create the .ssh directory with:

   ssh username@lapback1.ma.ic.ac.uk mkdir .ssh

   where 'username' is your college username (if your username on your laptop matches your college username, you can omit the 'username@' bit in the above command); you will be prompted for your backup server password. Now copy your public ssh key to the backup server:

   scp ~/.ssh/id_rsa.pub username@lapback1.ma.ic.ac.uk:.ssh/authorized_keys

   giving your backup service password when prompted.

5. Finally, check that you can connect to the backup server without using a password:

   ssh username@lapback1.ma.ic.ac.uk

   You should find that you are logged into the server without being prompted for a password. (If you are not used to using ssh private/public keypairs for remote logins and are scared by this, don't worry! It is not as insecure as it seems as the combination of your hidden private key and the matching public key means that only you can log into the backup server, and then only from your laptop. So it is actually more secure than using a password and avoids the bother of being prompted for a password every time you do a backup).

Installing the scripts on your laptop

You need to install a script on your laptop and, optionally, a second script if you want to automate the backups to occur shortly after you switch on your laptop and connect to the college network.

1. Download the sync2lapback and (optional) auto-backup2lapback scripts

2. now edit the sync2lapbackup script and make the following two changes:

   on line 10, change 'MY_LAPTOP_HOME_DIRECTORY' to the name of your home directory on your laptop. For example, if your laptop username is 'fiona' and your home directory is '/home/fiona', then change this line so that it reads:

   LAP_HOME_DIR="/home/fiona"

   (On Linux systems your home directory will usually be /home/your_laptop_username but you may have changed this to something else and on some other UNIXes, it may be something like /export/home/your_laptop_username, /usr/users/your_laptop_username, etc. If in doubt, typing 'echo $HOME' on any Linux or UNIX system will tell you where your home directory is).

   on line 11, replace 'MY_COLLEGE_USERNAME' with your college username as this will be used for your laptop backup account; if your college username is 'fmct09', then this line should be edited so it reads:

   COLL_USERNAME="fmct09"

3. now become root (or use sudo) and copy or move these two scripts into your /usr/local/bin directory:

   sudo mv sync2lapback /usr/local/bin
   sudo mv auto-backup2lapback /usr/local/bin

4. make sure that the scripts are executable by yourself by typing the following commands in a terminal:

   sudo chmod 755 /usr/local/bin/sync2lapbackup
   sudo chmod 755 /usr/local/bin/auto-backup2lapbackup

Setting the type of backup you want - backup or sync

This step is optional - if you skip mode configuration, the system will always use 'backup' mode. However, you can override this default when you run the sync2lapback command and switch to sync mode by adding the word 'sync' after the command, as described below.

If you would like to always use 'sync' mode, you can do this as follows:

using a text editor, create the file:

$HOME/.laptop-backup/lapback-backup-mode

containing a single line with the word 'sync' and save it. Now each time you run sync2lapback, it will use sync mode.

If you would always like to use the 'backup' mode, create this file as described above for sync mode but containing the word 'backup' instead of 'sync'. This is not strictly necessary because if this configuration file is missing, sync2lapback will use the backup mode by default anyway.

Testing the sync2lapback script

You should now test the sync2lapbackup script by running it for the first time. It will do the following:

1. connect to the backup server and create the folder called 'laptop_backup' if it doesn't already exist

2. then it will backup all of your home directory to this folder as 'home/your_laptop_username'

Putting the backup into the 'laptop_backup' folder means you can move or rename this folder if you have more than one laptop you would like to back up and you can also backup other parts of your laptop(s) such as the contents of /usr/local, /opt or /etc by modifying the sync2lapback script - ask Andy Thomas if you want help adding enhancements like this to the basic script. For example, you may sometimes have a USB pendrive, or a camera or phone, or a memory card, plugged into your laptop and you might like to auto-detect and back these up too. Variants of this script are available to handle backing up additional data sources but why not have some fun and modify the provided script yourself to get exactly the behaviour you want?

To run the script in 'backup' mode, in a terminal simply type:

sync2lapback

and to run the script in 'sync' mode, in a terminal simply type:

sync2lapback sync

If you have a lot of data in your laptop's home directory, this could take a long time to do so be warned - some users in Maths have several hundreds of gigabytes of data stored on their laptops. The log file in $SHOME/.laptop-backup/laptop-backup.log will tell you when it has completed.

Automating your backups

Once you are satisfied that the sync2lapback script is working as expected, you can automate this so that it will attempt to back up your laptop 5 minutes after you switch it on and it has connected to the college network; if the connection attempt fails, it will try again at 5 minute intervals for 20 minutes.

To do this you need to edit your laptop's crontab file as follows:

1. in a shell or terminal, type:

   crontab -e

2. if this is the first time you have edited your crontab file on your laptop, on Ubuntu and Debian systems you may be prompted to choose the editor you would like to use - choose your favourite editor (the default is nano). Next time you edit your crontab, your choice of editor will be remembered so you won't be prompted for this again.
   you are now in the crontab editor and if you have never edited it before, it will be empty. On the other hand, if you use cron and have edited it before, it will have some lines in it already.

   Add the following lines to your crontab file, below any existing lines you mat have in it:

   
   #
   # start the auto-backup 5 minutes after booting
   @reboot sleep 300; /usr/local/bin/auto-backup2lapbackup
   # retry at 10, 15 and 20 minutes after booting if earlier attempts fail
   @reboot sleep 600; /usr/local/bin/auto-backup2lapbackup
   @reboot sleep 900; /usr/local/bin/auto-backup2lapbackup
   @reboot sleep 1200; /usr/local/bin/auto-backup2lapbackup
   

   Now save the file.

   Next time you switch your laptop on, it will have 5 minutes to connect to a wi-fi or wired network before it makes the first attempt to run the backup script. If for some reason it doesn't succeed in contacting the backup serve, it will try again 10, 15 and 20 minutes after being switched on before giving up. (Note: the numbers 300, 600, 900, etc that you've put into your crontab above are these wait times in seconds).