Blame Murphy
Back online: hooray! We have just overcome a bad hardware failure on our backend storage server… the NIC started smoking. If smoking is bad for humans, it is for sure not very good for network adaptors either and so the card died this morning at around 4am Houston local time.
Ev1Servers did their best to get it back going, and replaced the whole server and moved the RAID5 system over to another box. We booted that one and the hardware failed again… This time the BIOS chip went away. It is not noname hardware, it is a large and well known supplier of servers, so don’t blame me for being cheap here!
So, ev1 went and got the 2nd replacement machine to repeat the procedure. As you can see, we are back on track and lost around 4 hours traffic, hopefully not the trust of our users and merchants though!
We resumed normal operation, and if you meet Murphy today, beat him up a little bit from us, please!



May 19th, 2006 at 10:23 am
PS: the picture shows the 2 guys who have been working on our servers for over 2 hours this morning…