leibold MVM join:2002-07-09 Sunnyvale, CA Netgear CG3000DCR ZyXEL P-663HN-51
2 recommendations |
[Serious] No MilestonesWhen I came home from work today I was greeted with a strong strange smell, some beeping and the sound of fans starting and stopping The beeping came from one of the two APC UPS powering the computer that runs the stats programs for Team Helix (both R@H and F@H). A bad battery in a UPS (that is what the cause appears to be) shouldn't have caused any problems since there are two redundant power supplies in the computer (each attached to a different UPS). However something went horribly wrong and the computer keeps turning itself on and off continuously. This all will get fixed but it may take a while. All the data is saved on multiple RAID6 arrays and therefore hopefully survived but I don't have any other computer large enough for all the disk drives and therefore can't even access the data at this time :-( I'm sorry but no milestones for today and no ETA for when they will be back. |
|
IcarusCHAOS RULES Mod join:2000-11-08 Off Center
1 recommendation |
Icarus
Mod
2013-Oct-23 7:00 am
Sorry youre having troubles. Hope its not too much of a problem getting things back up and running. In the meantime lets blame Sorto! |
|
parkutCrunch Addict MVM join:2001-12-15 Clinton Township, MI
1 recommendation |
to leibold
said by leibold:When I came home from work today I was greeted with a strong strange smell, some beeping and the sound of fans starting and stopping Don't ya just hate it when that happens? |
|
squirtcacatne alces in silvis? MVM join:2004-04-27 Out Grazing
3 recommendations |
to leibold
oops!
thanks for all of your contributions to the stats. in the meantime, we can just make up numbers! |
|
IcarusCHAOS RULES Mod join:2000-11-08 Off Center
1 recommendation |
Icarus
Mod
2013-Oct-23 7:52 am
said by squirt:oops!
in the meantime, we can just make up numbers! 72:) |
|
suprlegAbracadabra Premium Member join:2004-05-06 Garden Grove, CA
3 recommendations |
to leibold
As soon as we have stats I'll diligently post them again...! |
|
onDvineGrown up Flower Child Premium Member join:2005-01-29 So. CA, USA
1 recommendation |
to leibold
I'm sorry you've encountered trouble and wish you good luck dealing with it. |
|
SunnyRuns from Clowns
join:2001-08-19
1 recommendation |
to leibold
Wow! Thank you for all you do for us! I had no idea you had such an elaborate setup for our milestone reporting! Please let us know if the repair gets expensive. I'm sure many of us would be happy to contribute to the cause. Interesting timing as I've been sitting here mumbling to myself about the cost to replace two UPS batteries, $179 each! History tells me they last about five years, which is long enough to face sticker shock when they need replacement again. It appears you are looking at a far more complex situation, makes my head hurt trying to even think about it. |
|
Sunny |
to Icarus
said by Icarus:In the meantime lets blame Sorto! No wonder my head hurts! It's guilt! It's all my fault!!!???!!! |
|
leibold MVM join:2002-07-09 Sunnyvale, CA Netgear CG3000DCR ZyXEL P-663HN-51
|
to Sunny
I hope I don't disappoint you, but the milestone reporting wasn't the only purpose of that computer Some time ago I decided to consolidate the ever growing number of small PCs at my home into virtual machines hosted on a more powerful and reliable server (with as much redundancy as possible, hence the dual power supplies, two UPS and two 12-channel RAID6 controllers). There were a number of reasons for that but energy efficiency played a big role in it (both the reduce the energy bill as well as to keep the home cooler in summer). Most of the VM instances aren't important since they just served for experiments (for example there are 5 instances alone just to get some practical experience with IPv6). There are however some other instances I'm very eager to get back up and running again (like my email server). I'm going to do some more troubleshooting tonight and hopefully can narrow down what is causing it. I have a huge collection of computer parts but they are all from desktop computers and are not suitable for such a big server. The worst case scenario would be to have to order some parts and having to wait for them (unfortunately that is very likely). Of course, whenever something like this happens there is the temptation to use it as an excuse to buy something bigger and better instead. |
|
Santa FeBUT.....I Digress!
join:2000-08-22 Freight Yard |
to suprleg
said by suprleg:As soon as we have stats I'll diligently post them again...! Till then you can UN-diligently post them? |
|
SunnyRuns from Clowns
join:2001-08-19
1 recommendation |
to leibold
said by leibold:I hope I don't disappoint you, but the milestone reporting wasn't the only purpose of that computer Oh! I'm in such total shock! said by leibold:
Of course, whenever something like this happens there is the temptation to use it as an excuse to buy something bigger and better instead.
Reason 2 here for my procrastination over ordering those batteries. Those two APC Smart-UPS 1000XL's are probably 10 years old, maybe more, now. There is the temptation to get newer ones with the newer bells and whistles. Still, the only thing about the newer ones I would really like to have would be the USB vs. serial smart cable connectors. Naturally I have no computers with serial connections anymore. It would be extravagant, however, since I no longer have a rec room full of computers. It is easy enough to check them physically. Remote monitoring is far from necessary these days. OK, thanks for listening. I think I have squashed the "wants" for now. |
|
signmeuptoo94Bless you Howie Premium Member join:2001-11-22 NanoParticle |
to leibold
Sorry you came home to that sort of trouble, Thomas. Sometimes even extra measures aren't enough when the squirrels go on strike!
I vote also that we help raise funds if needed, I might not have much, but I'll contribute!
You know, that new Apple Mac Pro is coming out, if we all win the lottery, we should get ya a few of them puppies! |
|
leibold MVM join:2002-07-09 Sunnyvale, CA Netgear CG3000DCR ZyXEL P-663HN-51
3 recommendations |
to suprleg
Got a little present for suprleg here: fah_daily_Tu···2013.txt 659 bytes Tuesday 22 Oct 2013
fah_daily_We···2013.txt 1043 bytes Wednesday 23 Oct 2013
Did some emergency surgery on the computer and ended up removing one power supply, two UPS and an IPMI BMC. That doesn't necessarily mean all of those parts are bad but those are the parts that were removed when I got the computer to start up again.
That 2nd power supply may be still working but one of the two power supply slots is definitely bad (and that is part of the chassis and not easily replaced).
There will be some more downtime but for now the computer is up and running.
|
|
rfharThe World Sport, Played In Every Country Premium Member join:2001-03-26 Buicktown,Mi
1 recommendation |
to Sunny
said by Sunny:Wow! Thank you for all you do for us! I had no idea you had such an elaborate setup for our milestone reporting!
Please let us know if the repair gets expensive. I'm sure many of us would be happy to contribute to the cause.
Interesting timing as I've been sitting here mumbling to myself about the cost to replace two UPS batteries, $179 each! History tells me they last about five years, which is long enough to face sticker shock when they need replacement again. It appears you are looking at a far more complex situation, makes my head hurt trying to even think about it. Do price the new UPS units before buying a battery. I took my old APC 1000 to a local repair shop to give it to him because I did not want to spend $85 for a new battery and he did not want it. He told me that the prices have really come down lately and showed me a 625 unit that cost $65, twenty dollars less than the battery. When I had Win86 and Win XP An UPS was needed but I have had power outages with this Win7 machine and no problems as for as the operating system goes. I may lose a WU in FAH or what I have on Word but no problem getting the computer running, I am not sure I am going to buy another UPS right away. Money is flying out our doors this year. |
|
nonymous (banned) join:2003-09-08 Glendale, AZ |
to leibold
Go to the cloud. Think Justin said he may be buying a used over powered server for the web site with way extra capacity. Ask him for cloud use. Heck talk him into FAH on it also. |
|
|
signmeuptoo94Bless you Howie Premium Member join:2001-11-22 NanoParticle |
to leibold
What is an IPMI BMC? Is this a rack system, or a stand alone case with redundant SMPS units?
I was up until 5 AM last night hassling with my laptop. I won't go into all the details but it is a Lenovo G580 and I just can't seem to get a 16 GB thumb drive restore "disk" to make a proper working backup restore drive. I experimented with a hard drive that someone on DSLR sold me, but the drive is defective, then I tried with a 2.5" low capacity drive and it saved ok, but I don't have a second 2.5" drive to put in the system to verify it will work. All these changes with Secure Boot and Windows 8 are frustrating (you can't make restore optical disks any longer, according to Lenovo, and they are zero help).
Computers are time vacuums, they can eat up on a person's personal time! |
|
leibold MVM join:2002-07-09 Sunnyvale, CA Netgear CG3000DCR ZyXEL P-663HN-51
|
IPMI (Intelligent Platform Management Interface) is a way to remotely manage servers. The BMC (Baseboard Management Controller) is a key component of IPMI and provides out-of-band remote communication to the hardware so the the server can be monitored and controlled (including power on and off) even if the operating system isn't running. |
|
signmeuptoo94Bless you Howie Premium Member join:2001-11-22 NanoParticle |
Oh, thanks for that. You aren't the only one having problems, my trusty Beast has started locking up on me. I suspect RAM, but don't know yet.
I'd love it, if just once, I'd build and have a system that doesn't have issues! |
|
leibold MVM join:2002-07-09 Sunnyvale, CA |
Well, that didn't last long It was still running this morning but now it is down again. |
|
SunnyRuns from Clowns
join:2001-08-19
1 recommendation |
Sunny
2013-Oct-25 2:56 pm
I hate hardware. |
|
leibold MVM join:2002-07-09 Sunnyvale, CA Netgear CG3000DCR ZyXEL P-663HN-51
3 recommendations |
I'm saving these here for suprleg in case the server goes down again (which at this point seems to be likely): fah_daily_Th···2013.txt 531 bytes Thursday 10/24
|
|
signmeuptoo94Bless you Howie Premium Member join:2001-11-22 NanoParticle |
to Sunny
Sometimes I definitely agree with you. Computer products are kinda like boats, boats are holes in the water into which you pour money. At least computers can do useful work. But some of us have had to put up with a lot of defective hardware. Back in TN I went through 5 power supplies in 3 years, for instance. |
|