dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
1019
share rss forum feed


Modus
I hate smartassery on forums
Premium
join:2005-05-02
us

Need help IT vs Facilities

Here is the is situation:

Every week 2 time a week on the same day the facilities guys conduct generation test which consist of running the generator and a load transfer. I've been at my current job about 3yrs now and come from DC environment where this sort of test wasn't done on a regular bases, what are your facilities guys doing as far as generator maintenance and fail-over testing?
--
Think Ahead. Learn More. Solve Now!


beachintech
There's sand in my tool bag
Premium
join:2008-01-06
kudos:5
Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.
--
Ex-Comcast Tech at the Beach. I speak for myself, not my former employer.


state
stress magnet
Premium,Mod
join:2002-02-08
Purgatory
kudos:6
said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer.

That's how our switching and datacenter facilities operate as well.


Modus
I hate smartassery on forums
Premium
join:2005-05-02
us
reply to beachintech
said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.

That's just it.....its not working for us. It has caused so many issues in the past, currently the A/C, standalone ups all have encountered issues. Last week one of the buildings on our campus lost power during the load transfer (of course no one told us) and my apc symmetra lx that i have in that building for my IDF network gear was on battery power. Power was restored but the symmetra died and so did the network gear, i know the generator needs to run but the load transfer test is kill us.
--
Think Ahead. Learn More. Solve Now!


DadeMurphy
Premium
join:2002-07-25
Danvers, MA
Based on that I'd say your problem isn't the testing, it's that there is something wrong with your load transfer that needs to be resolved.

BTW, we do once a month live load transfer testing and weekly no load testing of the gensets at the two data centers I have my hands in.
--
Most people don't think clearly when they're on fire.


Modus
I hate smartassery on forums
Premium
join:2005-05-02
us
said by DadeMurphy:

Based on that I'd say your problem isn't the testing, it's that there is something wrong with your load transfer that needs to be resolved.

BTW, we do once a month live load transfer testing and weekly no load testing of the gensets at the two data centers I have my hands in.

ok so far looks like the average is about once a month which is better than 2 times a week. And load transfer failing is on them, about 2yrs ago in another one of our locations the transfer switch failed could have been from ware & tear who knows....
--
Think Ahead. Learn More. Solve Now!


DadeMurphy
Premium
join:2002-07-25
Danvers, MA
I think everyone can agree that twice a week is excessive for a multitude of reasons. Can you run it up your management chain and get some pressure applied to the facilities group to modify the testing schedule. Oh and while they're modifying the test schedule they should fix the broken transfer devices. Just sayin'
--
Most people don't think clearly when they're on fire.


Modus
I hate smartassery on forums
Premium
join:2005-05-02
us
said by DadeMurphy:

I think everyone can agree that twice a week is excessive for a multitude of reasons. Can you run it up your management chain and get some pressure applied to the facilities group to modify the testing schedule. Oh and while they're modifying the test schedule they should fix the broken transfer devices. Just sayin'

That's my plan....i'm currently compiling a list of things that have occurred and trying to research other things that could come up as a result of this excessive testing
--
Think Ahead. Learn More. Solve Now!


Wily_One
Premium
join:2002-11-24
San Jose, CA
reply to DadeMurphy
Load transfer testing twice a week? Hell I'd say monthly testing is excessive. What it sounds like to me is they know they have a reliability problem and are trying to find the issue/fix that will work, hence the extra testing.


DadeMurphy
Premium
join:2002-07-25
Danvers, MA
reply to Modus
Make sure you point out that with the AC failing (not sure how often it does during the transfers) you can overheat your servers/networking gear and kill it if it is too old to have internal thermal monitoring and shutdown. Sudden power loss isn't healthy for any elctronics either and can kill them if it happens enough. (of course you know all this)

Recently we had a problem with one of the main power feeds to one of our data centers that killed all power to one side of our DC every 3 days. We couldn't get funding to repair it until we lost an entire 200TB SAN (thankfully it was mirrored with another SAN). Had the funding come in before the SAN was fried the cost to the company would have only been a day of an electricians time instead of SAN + electrician.

Don't let your management push it off until hardware starts dying.
--
Most people don't think clearly when they're on fire.


Nightfall
My Goal Is To Deny Yours
Premium,MVM
join:2001-08-03
Grand Rapids, MI
Reviews:
·ooma
·Comcast
·Callcentric
·Site5.com
reply to beachintech
said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.

This is the same here.

From the way things sound, your generator is having a problem or the procedure is an issue. I would get the attention of the executives on this one. Sounds like the facilities guy could do a better job at getting his generator working properly. Either that or the company contracted to support the generator.
--
My domain - Nightfall.net


workablob

join:2004-06-09
Houston, TX
kudos:3
Reviews:
·Comcast
reply to Modus
We moved our entire data center and DR gear to a COLO.

I remember when we had the stuff on-premises.

What a pain when power went out and we had to rely on the genny for extended periods of time.

Think IKE.

Dave
--
I may have been born yesterday. But it wasn't at night.


exocet_cm
Free at last, free at last
Premium
join:2003-03-23
New Orleans, LA
kudos:3
said by workablob:

We moved our entire data center and DR gear to a COLO.
I remember when we had the stuff on-premises.
What a pain when power went out and we had to rely on the genny for extended periods of time.
Think IKE.
Dave

Or do like us and put your fuel supplies in the basement (think Katrina). Derrrr

@Op:
We have two generators on our headquarters. They both power-on, no load, every Wednesday at like 2 AM. Load test a few times prior to hurricane season but no scheduled load test.
--
"All newspaper editorial writers ever do is come down from the hills after the battle is over and shoot the wounded." - Bruce Anderson

"I have often regretted my speech, never my silence." - Xenocrates

8744675

join:2000-10-10
Decatur, GA
reply to Modus
Once a month here, on a Saturday or Sunday when there's nobody at work.


Karride
Slower Traffic Keep Right
Premium
join:2000-04-17
Germantown, TN
reply to Modus
Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.


BigBlarg

join:2008-02-10
Longueuil, QC
kudos:1
Reviews:
·ELECTRONICBOX
reply to Modus
Even it it seems too much for some of you, these big generators are required to work on load about once a month to make sure they will work properly when there will be a real need. A few self-test or running it with no load is not enough but I can't explain exactly why.


Nightfall
My Goal Is To Deny Yours
Premium,MVM
join:2001-08-03
Grand Rapids, MI
Reviews:
·ooma
·Comcast
·Callcentric
·Site5.com
reply to Karride
said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.
--
My domain - Nightfall.net


LazMan
Premium
join:2003-03-26
canada
reply to Modus
I maintain (either directly, or through contracted partners) about 25 generators, ranging from 50kw to 2MW, in telco and DC support.

Twice a week far exceeds industry best practice, that's for sure - ours get tested monthly or quarterly, depending on the classification of the site... Test includes a transafer of site load, 15 minutes runtime on load, and a transfer back. Annually, we do a 4 hour, 100% loadbank test.

It doesn't sound like you have a generator problem; sounds like you have a standby power/UPS issue; or your transfer scheme's not right.

As for why you need to do a load test - I'll give you a real-world example...

At a site I just took over resposibility for last fall - 500kw diesel standby unit, that's been 'tested' monthly for the last few years, without a load - all the previous tech did was fire up the engine, and run it for 15-20 minutes... I put a load bank on it, and it started to overheat, and shut down in about 15 minutes. The rad is done - there was enough cooling capacity to run without a load, but as soon as the engine started working hard, it couldn't handle it... That's something you want to find out when you have time to fix it, not when you're in the dark, and the clock's ticking.


Modus
I hate smartassery on forums
Premium
join:2005-05-02
us
reply to Nightfall
said by Nightfall:

said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.

Yes i have discuss this with them and the main response was "it's always been done this way and no reason to change routine"
--
Think Ahead. Learn More. Solve Now!


exocet_cm
Free at last, free at last
Premium
join:2003-03-23
New Orleans, LA
kudos:3
said by Modus:

said by Nightfall:

said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.

Yes i have discuss this with them and the main response was "it's always been done this way and no reason to change routine"

Then send them the bill for the replacement cost of your hardware including time/cost for repairing the issue.

If you want to be a smart ass, like I would, I would also send them another invoice for a standby part(s) for the next time they were to screw it up.
--
"All newspaper editorial writers ever do is come down from the hills after the battle is over and shoot the wounded." - Bruce Anderson

"I have often regretted my speech, never my silence." - Xenocrates