July 14th: NextGen HPC cluster is unavailable, remaining GPU servers in Huxley switched to low power mode during the UK heatwaves, nvidia4 GPU server now available again
May 27th's power outage: lasting just 2 seconds at the South Kensington campus this has caused a lot of problems; 75% of the cooling failed in the Maths server room which in turn triggered an automated shutdown of many systems to prevent overheating while in the ICT data centre, all power including UPS power was lost to all of Maths systems.
The current situation is all systems hosted in Huxley 616 & 6M46 that are needed by users for their work are operational, while in the ICT South Kens data centre, everything except the NextGen HPC cluster is back online.
nvidia4 GPU server: this server has been repaired with a failing hard disk now replaced and is vack online and available to all users. Also, owing to ongoing issues with the 22TB local data storage always being close to full, data for users who have not used this server during the past 12 months was relocated to networked clustor4 storage over the weekend of June 12-15th) prior to the disk replacement. This will not only reduce the clutter left behind by absent users but will also speed up the rebuild of the on-server storage pool after the disk is replaced next week.
NextGen HPC: frankly, this is FUBAR (if you don't know what this means, look it up, preferably from a source that doesn't mince its words or try to bowdlerise their meaning ;-). The compute nodes in NextGen will no longer boot apparently owing to a range of software issues plus a hardware issue with the openpbs server. Fixing this individually across 35 nodes is obviously going to take a long time so instead of carrying out this year's bi-annual operating system upgrade that is normally done in late summer, we may consider doing a 'bare metal' install of Linux on the compute nodes from scratch. This will erase all traces of older installations including obsolete libraries, etc from earlier builds that could interfere with newer O/S installations.
Owing to existing commitments, work on rebuilding the compute nodes will begin week commencing Monday July 13th.
Upcoming work in July:
Originally scheduled to be undertaken in April, and then May, the jobs listed below have had to be pushed back to June and now running into July:
Relocation of GPU servers: we now have additional server rackspace in the ICT CAGB (City & Guilds Building) data centre and are planning to move the GPU servers and dedicated GPU storage facilities from the Maths server room to CAGB where they will join the NextGen HPC already hosted there.
This reorganisation will have two major benefits for Maths; one is with the NextGen cluster, the GPU servers and dedicated data storage all in close proximity to each other, the GPU and NextGen CPU facilites can communicate with each other and share data much faster than at present since we can install dedicated local network connections to link the two that are much faster than using the College network across campus between the Huxley and CAGB buildings.
The other benefit is that with the GPU servers, which produce a lot of heat (and noise), out of the Maths server room, we can at last restore some of the smaller clusters and facilities that have been powered off for a very long time owing to insufficient cooling capacity in the server room, with the popularity of GPU-based computing having taken priority over other computational methods in the past few years.