My last few weeks have been pretty much all like this.
Thursday 16.: woozle, my new serverhousing box vanishes off the net. woozle is a Sun Ultra2 and runs infrastructure for me, the two flying clubs and their respective competitions and email for my exwife and daughter. woozle lives in 48N 16E. I live around 28S 153E so hands-on work is Not Feasible.
At that point we didn't have any consoles connected to that box, so I was in the dark. A powertoggle by the helpful housing team on the next day didn't do anything beyond spin up the disks.
Friday 17.: rw was planning to do the console connection but is sick. cm earns eternal gratitude by driving up to the box and having a look. It's an ex-computer. It ceased to be. Beautiful plumage and all that, but it's stone dead. cm pulls the sucker and diagnoses later that the PSU only supplies some of the required voltages. I set up the server at home to provide most of the functionality in the meantime, bend all the DNS around etc.pp. and start hunting for PSUs.
Friday 24.: rw cross-connects the consoles of his and my machines (lovely OpenPROM) and puts my disks into a temporary replacement U2 but it doesn't boot. I start figuring out whether the main disk also emitted the magic smoke or not, and manage to have woozle lock up in the bloody PROM...no break, no nothing. Weird, lockups in the PROM are rare... Anyway, this happens at about 1800 +1000, which means nobody will be able to powercycle the sucker until Monday 0900 +0200. Yay :-/
Saturday I went flying with the others, and got about 3/4 of an hour of airtime. Weather was rather lousy, and in the eveing I felt very tired. Sunday I slept until 1500 with a bad headache so I guess I was a bit on the sick side.
Monday 27.: I crawled out of bed around mid-morning, still feeling less than perfect, and worked on various stuff. In the evening they powercycle woozle, but due to the consoles now the other box vanishes: powercycles produce break events and breaks drop Suns into the PROM. Two hours later they also powercycle the other box and I can finally start fixing things. It turns out that the disks were inserted in the wrong order and thus changed SCSI ids, leading to all kinds of fun effects. I fix all this, get the box cleaned up and migrate the services back. This is now somewhat urgent as my old serverhousing contract expires on the 28. and the old box needs to be shut down. About 0100 local I'm done and go to bed.
Tuesday 28.: Everything's peachy. Except the weather: it rains. And only until 1730, when woozle goes splat and drops to the PROM without any error indication. I restart it at 1830 and start digging for evidence of a problem, and about two minutes later it goes fully catatonic: no reaction to breaks etc. Bloody fucking piece of shit! looks like this chassis has real hardware trouble, and no chance of fixing it up before next week. In disgust, I leave everything broken for the time being and wonder why oh why do I put in these efforts to provide services to others for nothing...
Today: It rains. The old serverhousing box is taken down. sniff
[root:~] date Wed Mar 1 00:30:40 MET 2006 [root:~] w 00:30:42 up 185 days, 13:29, 1 user, load average: 0.51, 0.22, 0.18 [root:~] halt
Since 2002 this aged Sun Sparcstation4-110 has been a good, reliable server. I move the essential services yet another time to the backup box at home and hope for the future. Like the future of woozle, for which I've ebayed a full Ultra2 (cheaper than a PSU alone). In 1996 Sun said
"The Ultra 2 Model 2200 Creator3D is now priced at $52,495 (US)"now I paid EU33 for the box. We'll see.