July 28th: NextGen HPC cluster is unavailable, relocation of GPU servers in Huxley to ICT data centre nearly complete, login issues with ssh gateways and some systems now resolved, nvidia4 GPU server available again
June 24th: ssh access issue for some systens ssh access to 7 systems in Maths suddenly became blocked at about 7am yesterday morning but has since been resolved today.
Relocation of GPU servers: nvidia4, nvidia5 & nvidia6 have now been relocated to the ICT CAGB data centre in South Kensington and the new nvidia8 GPU server has also been installed there. nvidia4 has been repaired and along with nvidia8 is available for use; nvidia5 and nvidia6 are awaiting delivery of some cables and then assignment of new network addresses before they too go back online. Finally, the clustor4 GPU user data storage server and the associated 10Gbit/s local network will be moved over to CAGB early next week.
May 27th's power outage: lasting just 2 seconds at the South Kensington campus this has caused a lot of problems; 75% of the cooling failed in the Maths server room which in turn triggered an automated shutdown of many systems to prevent overheating while in the ICT data centre, all power including UPS power was lost to all of Maths systems.
The current situation is all systems hosted in Huxley 616 & 6M46 that are needed by users for their work are operational, while in the ICT South Kens data centre, everything except the NextGen HPC cluster is back online.
NextGen HPC: frankly, this is FUBAR (if you don't know what this means, look it up, preferably from a source that doesn't mince its words or try to bowdlerise their meaning ;-). The compute nodes in NextGen will no longer boot apparently owing to a range of software issues plus a hardware issue with the openpbs server. Fixing this individually across 35 nodes is obviously going to take a long time so instead of carrying out this year's bi-annual operating system upgrade that is normally done in late summer, we may consider doing a 'bare metal' install of Linux on the compute nodes from scratch. This will erase all traces of older installations including obsolete libraries, etc from earlier builds that could interfere with newer O/S installations.
Owing to existing commitments, work on rebuilding the compute nodes will begin week commencing Monday July 13th.
Upcoming work in July:
Originally scheduled to be undertaken in April, and then May, the jobs listed below have had to be pushed back to June and now running into July: