Latest news for the Maths compute cluster
Using the Linux du command on the cluster
- Those of you who are using clustor2 for file storage may be mystified by the large discrepancy between file and folder sizes reported by the 'ls -l' and 'du' commands. This is because clustor2 uses file compression internally to improve read/write performance so that file and folder sizes as seen by the 'du' utility are the compressed sizes as stored in clustor2's ZFS disk pool, not the sizes seen by the operating system or yourself!
- To support ZFS compression-based storage systems such as that on clustor2, the latest version of du available for Ubuntu Linux now has the '--apparent-size' option which will report the actual file size, not the compressed size as seen on the clustor2 server. You can use this new option in conjunction with the existing du options such as 'h', 'm', 'c', etc.
Here is an example, using a file that is 3.2 GB in size:
ls -l reports the file size is 3.2 GB as expected:
andy@macomp001:~$ ls -l 3_2GB_test_file.tar
-rw-r--r-- 1 andy root 3343513600 Feb 2 2018 3_2GB_test_file.tar
but du shows it as less than half this size:
andy@macomp001:~$ du 3_2GB_test_file.tar
using the '--apparent-size' option to du now reports the size you would expect to see:
andy@macomp001:~$ du --apparent-size 3_2GB_test_file.tar
- Using du to find sizes of files or folders on other servers attached to the compute cluster, for example silo2 or clustor, will show very similar sizes with or without the '--apparent-size' option since they do not use compression in their underlying storage systems.
major R upgrade for Maths compute cluster completed
- The core R installation on the cluster has been upgraded from version 3.4.3 to 3.4.4 which is a very minor upgrade but at the same time, the large additional R package collection - mostly from the CRAN repository - has also been rebuilt from current sources, many for the first time ever since the cluster was introduced.
- As always, I'll be happy to answer any questions you may have.
Research Computing Manager,
Department of Mathematics
last updated: 29.08.2018