With the College increasingly returning to pre-pandemic activity levels, new initiatives currently being introduced in Maths include:
OpenPBS job & GPU management is being introduced on the forrest GPU server: demand for GPU facilities on the StatML CDT's GPU server, forrest, is exceeding the four GPUs installed in this server and users have requested some sort of job/resource management be installed to manage these. Unfortunately, support for managing GPU hardware is poor or non-existent in most free/open source resource managers while commercial software that does provide these facilities is expensive. However, an open source version of PBSPro from Altair is being trialled on a pair of test servers with a view to introducing this on the forrest GPU server.
development of a new Hadoop/Spark platform: the current Bazooka Hadoop cluster has been operational since January 2017 with only minor updates having been made since. This is because the MapR Hadoop Ecosystem upon which it is based is no longer supported in it's original free community edition guise; MapR went bankrupt and what was left of it was snapped up by HP Enterprise and is now marketed as their HPE Ezmeral Data Fabric which is definitely not free! The original MapR software is Python 2.7-based but the world has moved on to Python 3.x now and this is causing all sorts of problems with adding new Hadoop-related software, system upgrades, etc which in reality is forcing us to maintain an aging cluster that cannot be updated or upgraded. To circumvent this in the short term, we are building a new Hadoop cluster from scratch using just the original Apache Hadoop componentsi; this will initially be on a single server and designed to meet the spacifications for an AI course starting in the summer term.