It is not clear what caused this, but this is how it was fixed – eventually! The main issue that resulted from the failure was that the database would not restart and there were 1000's of relay log files in the $DATADIR
- Killed all mysql processes and mysqld processes – had to use kill -9 for mysqld process
- Removed all relay bin logs from $DATADIR
- Restarted mysql using /usr/local/mysql/bin/mysqld_safe &
- Stopped slave
- Systems team stop mail
- Recreated slave from master (oook)
- Connect to database and issue 'flush logs'
- Set master log file and llog file position
- Restart slave
Everything should be working now!