Silo data storage facilities
The Maths department has two large capacity storage servers for research users known as the silos; inspired by the original silo server set up in 2006, the present silo1 and silo2 servers each provide 10.4 Tb of storage.
Why use the silos?
- Disk space available on the ICNFS server which hosts Maths Linux home directories is limited and each user has a finite disk usage quota imposed. This can be counter-productive in a research environment - many users now work with very large data sets of many gigabytes or even terabytes in size and often there is no way of predicting in advance how much output will be produced by an experiment.
- Using the silo servers is preferable to storing data on local USB disk drives attached to your PC or laptop, for example - it is faster & more reliable and each data silo is backed up by an identical mirror server. And it is also very secure - considerably more so than some cloud storage providers - yet accessible from anywhere there is a network connection.
How do I use the silos?
- To use either of the silos you need an account on it - simply email Andy Thomas requesting an account. This will then be set up and the account details will be mailed to you along with some information on its usage. It's as simple as that.
- Accounts on the silos are local machine accounts and not directly connected to the College's central computer accounts system but usernames and the groups to which users belong will be identical to college accounts making it easy to transfer data between Maths & College computer systems and the silos, and makes possible the mounting of NFS exports from the silos onto Maths systems.
- You can copy data to and from your account on either silo using a variety of familiar methods - scp, sftp, rsync and by mounting the data silo disk(s) via NFS on a Linux or UNIX workstation or PC. You can also log into the silos using ssh and carry out the usual file and directory tasks, such as moving, renaming, copying and deleting, etc. However, the range of applications and packages installed on the silos has been deliberately kept to the bare minimum to discourage the use of this system for computational work, etc as they are strictly intended as a bulk data storage facility.
- Both silos are mounted on Maths publicly-accessible systems such as the ssh
gateways, the Maths compute cluster and the Stats Linux cluster as:
- silo1: /home/silo1
- silo2: /home/silo2
- and for many this will be the preferred way of using the silos.
Are disk quotas imposed on the silos?
- No! Per-user disk quotas are not imposed on either silo as this would defeat the object of this facility. However, there is a finite limit set by the available space on each silo server - the maximum contiguous file space that can be provided on the silos is 10.4 Tb.
Is data on the silos backed up?
- Yes, each silo is mirrored every 4 hours to an identical mirror server, each of which contain exact copies of the parent silo's filesystems. There are 4 servers in all.
Can I mount the silos directly onto my computer?
- Yes - both silos export their storage via NFS so if your computer runs Linux or some other UNIX-like operating system, or if you have a Mac, you can easily mount these on your computer so that they appear to be a part of your computer's filesystem. The storage filesystems are exported as follows:
- silo1: /maths1/home
- silo2: /maths3/home
- To permanently mount one or more of these filesystems on your computer, just follow these instructions:
silo2 data should now be accessible in /home/silo2
- first, decide where in your computers filesystem you want to mount your chosen silo1 or silo2 share; Maths systems use /home/silo1 and /home/silo2 as the mountpoints for the silos but you need not choose the same mountpoint names yourself; typical alternatives could be /mnt/silo or even /silo and /silo2.
Here, we assume you want to mount silo2's single share on the mountpoint called /home/silo2 on your computer
- create the /home/silo2 mountpoint on your computer - you'll need to be the root user to do this or alternatively, use sudo as in this example:
sudo mkdir /home/silo2
- using your favourite text editor and again working as root (or use sudo), add the new NFS mount from silo2 by adding the following lines to the bottom of your /etc/fstab file:
silo2.ma.ic.ac.uk:/maths3/home /home/silo2 nfs rw,bg,nosuid,soft,intr,rsize=32768,wsize=32768 0 0
- save your updated /etc/fstab and then mount the new mount by typing:
sudo mount -a
- (if you are doing this as root, then omit the 'sudo' from the above command)
About the silo servers
- Both silos and their mirrors are HP Microservers running FreeBSD - silo1 and mirror is a Gen8 model running FreeBSD 10.1 while silo2 and its mirror was set up in late 2012 and uses an older N40L machine running FreeBSD 9,0. Both use an internal USB pendrive as a system disk allowing all four disk bays in the server to be fitted with 4 TB disks dedicated to ZFS raidz1 storage only; almost all of the large storage servers in Maths have since been based on the same design & architecture.
Research Computing Officer
Department of Mathematics
last updated: 02.05.2015