The Maths server room, May 2013
Set up in April 2012, the Maths server room is home to nearly 70 operational servers (plus about a dozen idle spare systems) and 150 TB of storage. This is where mathematicians really hammer their kit with load averages in double figures being the norm!
All equipment in this room is on UPS power with 5 large UPS units (plus one spare) fed from dedicated 32 amp power feeds with a total UPS load capacity of 25.5 kVA. Cooling is provided through two 14 kW air conditioning units to provide redundancy and better distribution of cold air. Temperature probes at strategic locations in the room continually monitor the thermal conditions in the room and any excessive rise in temperature will page the sys admin; any further temperature rise beyond an upper threshold will trigger a sequential shutdown of nearly all systems and, finally, power off all but one of the UPS units to remove the sources of heat generation. (The exceptions are webservers hosting external websites and internal server room management systems which must be kept available at all times).
Remote monitoring from a dedicated system located in the ICT data centre keeps an eye on all systems in Maths as well as the off-site backup servers hosted externally.
This server room accommodates only part of the Maths research IT 'real estate' - there are another 45 servers that have been hosted in the ICT data centre for some time.
Rack 1 contains all the 1U servers, a mix of Maths compute cluster servers, user's servers as well as the London Mathematical Society's webservers.
At the bottom are the HP desktop systems (with diskette drives replaced with 1TB second disks) that form the Fanosearch project's clustered MySQL database engine, along with staging servers used by the LMS and an internal server room management PC.
The hot water radiator next to the rack is entirely redundant!
Rack 2 contains the more interesting specialist servers - 2U servers fitted with 4 x 16-core Opteron CPUs belonging to individual researchers, the GPU cluster (the silver box with the perforated plates) and below it, the blade servers that hosts the GPU cluster (the dark box with 12 disks).
Below this is a cluster used for laminar flow research (the black 'cables' attached to the server blades are actually multimode twin fibres operating at 10 Gbit/s) and at the bottom of the rack is another dedicated GPU server containing a pair of GPUs. At the top of the rack is the network switch for the server room's internal LAN, used for internal management.
a bank of blade servers at night
the backups rack
The backups rack contains the original Maths 'silo' server - which is still in use - at the bottom and at the top, the wide black box is the Ultrium/LTO 3 tape library with SDLT 320 and DLT 4 single tape drives above it, which are used to read legacy backup tapes or to do one-off backups where the tapes need to be exported in DLT4 or SDLT formats.
The ancient-looking tower server on the floor to the left of the rack actually contains a more modern motherboard with AMD K7 CPU and is the tape backup server, controlling all 3 tape drives using the Bacula backup system. This tower server will eventually be replaced by a rack-mount server inside the rack.
The vacant space in the middle of the rack will be filled with 1U servers when the Stats Linux cluster is upgraded.
the storage servers
Each of these 12 HP MicroServers has a storage capacity of 11 TB (raw installed disk capacity is 16 TB) and along with silo is used both for general purpose departmental storage as well as for private cluster storage. The 'system disk' on these is an internal 4 GB USB stick running FreeBSD 9.0, allowing all of the 'real' hard disks to be used as a dedicated ZFS raidZ1 storage pool.
Below these servers are 8 HP desktop PCs used for the departmental SSH gateways, the SVN servers and internal server room management and beyond these, on the far side, is the Stats Linux cluster, Alex Hewson's cluster and various user's legacy PCs.
the sys admin's corner
Last but not least, somewhere for the sys admin to sit. I spend very little time in this room as it's very noisy and rather cold too - almost everything is done remotely from my office. A stack of tapes and new spare hard disks sits on the top shelf and on the right almost out of shot is a spare Sun E450 server and several other spare systems that are kept for emergency use. The table with two screens & keyboards can be wheeled anywhere in the room as all systems are headless.
Research Computing Officer, Department of Mathematics
last updated: 24.07.2013