dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
1071

Modus
I hate smartassery on forums
Premium Member
join:2005-05-02
us

Modus

Premium Member

Need help IT vs Facilities

Here is the is situation:

Every week 2 time a week on the same day the facilities guys conduct generation test which consist of running the generator and a load transfer. I've been at my current job about 3yrs now and come from DC environment where this sort of test wasn't done on a regular bases, what are your facilities guys doing as far as generator maintenance and fail-over testing?

beachintech
There's sand in my tool bag
Premium Member
join:2008-01-06

beachintech

Premium Member

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.

state
stress magnet
Mod
join:2002-02-08
Purgatory

state

Mod

said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer.

That's how our switching and datacenter facilities operate as well.

Modus
I hate smartassery on forums
Premium Member
join:2005-05-02
us

Modus to beachintech

Premium Member

to beachintech
said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.

That's just it.....its not working for us. It has caused so many issues in the past, currently the A/C, standalone ups all have encountered issues. Last week one of the buildings on our campus lost power during the load transfer (of course no one told us) and my apc symmetra lx that i have in that building for my IDF network gear was on battery power. Power was restored but the symmetra died and so did the network gear, i know the generator needs to run but the load transfer test is kill us.

DadeMurphy
Premium Member
join:2002-07-25
Danvers, MA

DadeMurphy

Premium Member

Based on that I'd say your problem isn't the testing, it's that there is something wrong with your load transfer that needs to be resolved.

BTW, we do once a month live load transfer testing and weekly no load testing of the gensets at the two data centers I have my hands in.

Modus
I hate smartassery on forums
Premium Member
join:2005-05-02
us

Modus

Premium Member

said by DadeMurphy:

Based on that I'd say your problem isn't the testing, it's that there is something wrong with your load transfer that needs to be resolved.

BTW, we do once a month live load transfer testing and weekly no load testing of the gensets at the two data centers I have my hands in.

ok so far looks like the average is about once a month which is better than 2 times a week. And load transfer failing is on them, about 2yrs ago in another one of our locations the transfer switch failed could have been from ware & tear who knows....

DadeMurphy
Premium Member
join:2002-07-25
Danvers, MA

DadeMurphy

Premium Member

I think everyone can agree that twice a week is excessive for a multitude of reasons. Can you run it up your management chain and get some pressure applied to the facilities group to modify the testing schedule. Oh and while they're modifying the test schedule they should fix the broken transfer devices. Just sayin'

Modus
I hate smartassery on forums
Premium Member
join:2005-05-02
us

Modus

Premium Member

said by DadeMurphy:

I think everyone can agree that twice a week is excessive for a multitude of reasons. Can you run it up your management chain and get some pressure applied to the facilities group to modify the testing schedule. Oh and while they're modifying the test schedule they should fix the broken transfer devices. Just sayin'

That's my plan....i'm currently compiling a list of things that have occurred and trying to research other things that could come up as a result of this excessive testing

Wily_One
Premium Member
join:2002-11-24
San Jose, CA

Wily_One to DadeMurphy

Premium Member

to DadeMurphy
Load transfer testing twice a week? Hell I'd say monthly testing is excessive. What it sounds like to me is they know they have a reliability problem and are trying to find the issue/fix that will work, hence the extra testing.

DadeMurphy
Premium Member
join:2002-07-25
Danvers, MA

DadeMurphy to Modus

Premium Member

to Modus
Make sure you point out that with the AC failing (not sure how often it does during the transfers) you can overheat your servers/networking gear and kill it if it is too old to have internal thermal monitoring and shutdown. Sudden power loss isn't healthy for any elctronics either and can kill them if it happens enough. (of course you know all this)

Recently we had a problem with one of the main power feeds to one of our data centers that killed all power to one side of our DC every 3 days. We couldn't get funding to repair it until we lost an entire 200TB SAN (thankfully it was mirrored with another SAN). Had the funding come in before the SAN was fried the cost to the company would have only been a day of an electricians time instead of SAN + electrician.

Don't let your management push it off until hardware starts dying.

Nightfall
My Goal Is To Deny Yours
MVM
join:2001-08-03
Grand Rapids, MI

Nightfall to beachintech

MVM

to beachintech
said by beachintech:

Once a month we have a schedule test where they spin everything up and make the live transfer. Generator does a self test weekly, with no load transfer. If everything is working you should not notice the event really.

This is the same here.

From the way things sound, your generator is having a problem or the procedure is an issue. I would get the attention of the executives on this one. Sounds like the facilities guy could do a better job at getting his generator working properly. Either that or the company contracted to support the generator.

workablob
join:2004-06-09
Houston, TX

workablob to Modus

Member

to Modus
We moved our entire data center and DR gear to a COLO.

I remember when we had the stuff on-premises.

What a pain when power went out and we had to rely on the genny for extended periods of time.

Think IKE.

Dave

exocet_cm
Writing
Premium Member
join:2003-03-23
Brooklyn, NY

exocet_cm

Premium Member

said by workablob:

We moved our entire data center and DR gear to a COLO.
I remember when we had the stuff on-premises.
What a pain when power went out and we had to rely on the genny for extended periods of time.
Think IKE.
Dave

Or do like us and put your fuel supplies in the basement (think Katrina). Derrrr

@Op:
We have two generators on our headquarters. They both power-on, no load, every Wednesday at like 2 AM. Load test a few times prior to hurricane season but no scheduled load test.
8744675
join:2000-10-10
Decatur, GA

8744675 to Modus

Member

to Modus
Once a month here, on a Saturday or Sunday when there's nobody at work.

Karride
Slower Traffic Keep Right
Premium Member
join:2000-04-17
Germantown, TN

Karride to Modus

Premium Member

to Modus
Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

PatPatate
join:2008-02-10
Sainte-Julie, QC

PatPatate to Modus

Member

to Modus
Even it it seems too much for some of you, these big generators are required to work on load about once a month to make sure they will work properly when there will be a real need. A few self-test or running it with no load is not enough but I can't explain exactly why.

Nightfall
My Goal Is To Deny Yours
MVM
join:2001-08-03
Grand Rapids, MI

Nightfall to Karride

MVM

to Karride
said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.

LazMan
Premium Member
join:2003-03-26
Beverly Hills, CA

LazMan to Modus

Premium Member

to Modus
I maintain (either directly, or through contracted partners) about 25 generators, ranging from 50kw to 2MW, in telco and DC support.

Twice a week far exceeds industry best practice, that's for sure - ours get tested monthly or quarterly, depending on the classification of the site... Test includes a transafer of site load, 15 minutes runtime on load, and a transfer back. Annually, we do a 4 hour, 100% loadbank test.

It doesn't sound like you have a generator problem; sounds like you have a standby power/UPS issue; or your transfer scheme's not right.

As for why you need to do a load test - I'll give you a real-world example...

At a site I just took over resposibility for last fall - 500kw diesel standby unit, that's been 'tested' monthly for the last few years, without a load - all the previous tech did was fire up the engine, and run it for 15-20 minutes... I put a load bank on it, and it started to overheat, and shut down in about 15 minutes. The rad is done - there was enough cooling capacity to run without a load, but as soon as the engine started working hard, it couldn't handle it... That's something you want to find out when you have time to fix it, not when you're in the dark, and the clock's ticking.

Modus
I hate smartassery on forums
Premium Member
join:2005-05-02
us

Modus to Nightfall

Premium Member

to Nightfall
said by Nightfall:

said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.

Yes i have discuss this with them and the main response was "it's always been done this way and no reason to change routine"

exocet_cm
Writing
Premium Member
join:2003-03-23
Brooklyn, NY

exocet_cm

Premium Member

said by Modus:

said by Nightfall:

said by Karride:

Have you talked to facilities about it? If not I'd try to have a friendly discussion about it before I went the manager route.

This right here. I should have mentioned this first if you haven't talked to them already.

Yes i have discuss this with them and the main response was "it's always been done this way and no reason to change routine"

Then send them the bill for the replacement cost of your hardware including time/cost for repairing the issue.

If you want to be a smart ass, like I would, I would also send them another invoice for a standby part(s) for the next time they were to screw it up.