Unlucky weeks

Quote of this entry: What doesn't you kill you, makes you stronger.

Been down with unbelievable bad lucks since last friday. An unbelievable week for me which had made me so lousy. It all happened like this.

On thursday evening last week, I did a "ping" command onto my servers to make sure that they are alive as part of my routine check. However, all the servers didn't reply to me and i was pretty worried that the server might be down.

So, on friday, i drew out the key and went in. Checked that the servers were actually up and working. But, i still couldn't interact with them digitally.

So, i initiated a shutdown on all the servers. Bad things happen when the Service of one of the servers can't startup.

Informed my colleagues and after trying for a couple of hours, we can't do anything. Then we all informed my boss over this.

From there, hell starts to break loose and bad lucks were seem to be targetting at me.

1. My colleague and I went down to Basement (where one of the many servers we had are located) and it's a long way, so i had been running here and there a lot of times each day for the past week. We spent time until 9.30 pm in there before we left our workplace.
   
   
2. Boss came and suggested that to transferred these files to the downed server. So we spent 1 hour down there trying to mount the files directly to the unix machine. After trying that much, i'm unable to do so. Then i was thinking to copy the files were copied to windows and from there, to transfer to the USB external HD. From that idea, we had to stay back and configure the necessary setting before we all left the place. It was pretty early then. 7 pm
   
   
3. In order to have more backup coverage, we decided to have more backups with different medium. 4mm tape and a bigger tape. Did that and stayed back. Forget the timing though.
   
   
4. Done with the backup tapes the next day, wanted to recover and my boss just discover that the tape library (for the bigger tape) hardware fail! Gosh. Coz i didn't realise that it has stopped. Reprimanded me a bit.
   
   
5. My boss then thought of other idea. To bring a laptop and the external HD in to send the files in as there's an Gigabit Ethernet switch in it, therefore transferring of files would be quick. But, he forgot that the device is down and thus that idea was dropped. Again, the recovering of server is delayed.
   
   
6. Decided to do the slow method by ftp the files. And meanwhile, writing of data from the 4mm tape had began and i was hoping to change to the 2nd tape on that night as well. Didn't realise that it gotta be so long and i spent the whole night on friday in the office till sat 1.30 am without changing the tape. (Thanks lm and raquel who were there waiting for me till even so late)
   
   
7. Saturday. Came back to the office. Saw the message by the system informing me to change tape. and so i changed, but to my bad luck again, the system said that there's checksum error, and was unable to proceed to writing the data to the system. Damn it. To add to the problem, i was only informed at the last minute that there was an 3 hours power shutdown-cum-maintenance which of course i couldn't do anything else!
   
   
8. I realised that, at the end of the day, it might be the fault of the patch panel which linked to our office might have spoilt which was why it was unable to communicate well. Had i discovered that earlier, the above mentioned 7 points wouldn't occur.
   
   
9. For now, the good thing is that the ftp program that i used doesn't hanged the files on me though it would disconnect from the server after long period of inactivity.

Now i hope, that there would be no more bad luck anymore for tomorrow as we are going to restore the system. Everyone please pray for me.

Nonethess, from these past 8+ days, i realised what i was missing in relation to my work performances. Thanks rebecca for chatting with me throughout the whole night when i was in the office.