dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2379
share rss forum feed


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:3

[request] Recovery Fund

Justin - how about setting up a Recovery Fund to help offset the massive expense of this ordeal?

I'd certainly throw some bucks in - this site is worth so much more to me than the $10 I paid eons ago ....

Got a Paypal address?



HiVolt
Premium
join:2000-12-28
Toronto, ON
kudos:21

Re: Recovery Fund

Agreed, for some of us its been 10+ years here and it was odd being away for this long.

I'd pitch in.
--
BUCK FELL ,,!,,('-'),,!,,



GlennAllen
Sunny with highs in the 80s
Premium
join:2002-11-17
Richmond, VA
reply to dennismurphy

I'd say to send the bill to whoever was supposed to be providing [uninterruptible] backup power.

Expand your moderator at work


Spike
Premium
join:2008-05-16
Toronto, ON

2 edits
reply to GlennAllen

Re: Recovery Fund

said by GlennAllen:

I'd say to send the bill to whoever was supposed to be providing [uninterruptible] backup power.

Any piece of equipment can fail, if you're not prepared to handle it then its not the datacenters fault...

Being prepared is the key.
In our case, its a 16 slot LTO3 changer with daily mysql dumps.


usa2k
Blessed
Premium,MVM
join:2003-01-26
Redford, MI
kudos:3
Reviews:
·WOW Internet and..
reply to dennismurphy

said by dennismurphy:

Justin - how about setting up a Recovery Fund to help offset the massive expense of this ordeal?

I'd certainly throw some bucks in - this site is worth so much more to me than the $10 I paid eons ago ....

Got a Paypal address?

Agreed. $10 or $20 from half the membership I bet would go a long way!
What do you think Justin?
--

Jim, VoIP 12/2002, VOIPo 2/2007
FAH-Tool~Pets~Join Artist-247

garys_2k
Premium
join:2004-05-07
Farmington, MI
reply to dennismurphy

Works for me, sure.



GlennAllen
Sunny with highs in the 80s
Premium
join:2002-11-17
Richmond, VA
reply to Spike

Well, yeah--that's my point. When those you pay for "preparedness" don't deliver on what you're paying them for, it is their fault.


graniterock

join:2003-03-14
London, ON
Reviews:
·WIND Mobile
·TekSavvy Cable
reply to usa2k

said by usa2k:

Agreed. $10 or $20 from half the membership I bet would go a long way!
What do you think Justin?

I remember correctly I think he said he had 1 000 000 registrations of whiche 10% were active members?

1 000 000 * .1 / 2 * $10 = $500 000

But yeah... I would contribute to the virtual hat. If not perhaps once the site is running again buying some tool points or upgrading to premium might be the best way to support the site.


Arne Bolen
Happy Anveo customer
Premium
join:2009-06-21
Planet Earth
kudos:4
reply to dennismurphy

I suggest Justin adds the Flattr crowd funding system to the DSLR site.
 
--
Voip News



snappakazoo
Inconceivable

join:2000-09-10
Wyckoff, NJ
reply to dennismurphy

Add me to the list of minimally active users willing to donate.
--
Let me 'splain. [pause] No, there is too much. Let me sum up.


iknow_t

join:2012-05-03
reply to dennismurphy

how about someone with experience with an MD 3000 help justin out recovering the good copy of the site, it was on a raid 10 setup.. no one responded so far, so he had to spend thousands of dollars to get what he got so far, one would think someone would have experience with this setup..



JJJohnson

join:2001-08-25
Fort Collins, CO
reply to dennismurphy

I'll help him find his daily backup of the database. Does he remember approximately where he last saw it?


iknow_t

join:2012-05-03
reply to Spike

said by Spike:

said by GlennAllen:

I'd say to send the bill to whoever was supposed to be providing [uninterruptible] backup power.

Any piece of equipment can fail, if you're not prepared to handle it then its not the datacenters fault...

Being prepared is the key.
In our case, its a 16 slot LTO3 changer with daily mysql dumps.

actually, this is the THIRD time they lost all power because the battery for the generator failed. the other TWO times the servers were brought back online quickly. this time, there was corruption in the SQL database.. »www.webhostingtalk.com/showthrea···t=614624 this link is one of them, there is another, too. the fault lies with the datacenter for not maintaining the batteries properly.


JJJohnson

join:2001-08-25
Fort Collins, CO

said by iknow_t:

actually, this is the THIRD time they lost all power because the battery for the generator failed. the other TWO times the servers were brought back online quickly. this time, there was corruption in the SQL database.

»www.webhostingtalk.com/showthrea···t=614624 this link is one of them, there is another, too. the fault lies with the datacenter for not maintaining the batteries properly.

Fool me once, shame on you...

Should have been out of there within days after the second incident.


TamaraB
Question The Current Paradigm
Premium
join:2000-11-08
Da Bronx
Reviews:
·Optimum Online
·Clearwire Wireless
reply to GlennAllen

said by GlennAllen:

I'd say to send the bill to whoever was supposed to be providing [uninterruptible] backup power.

When I co-located servers at data centers, I always installed my own UPS'. If power to my equipment failed, indicating something really really bad has happened, my UPS would kick in and trigger an immediate shutdown, then wait cold for me to physically bring them back up.

This saved my butt twice when the rack my servers were in was cut off from AC power; once due to an electrician shorting out a cable under the floor, and another time when some datacenter tech tripped the wrong breaker. Neither case involved a general datacenter power fail.

Bottom line; the server owner is ultimately responsible for the safety of the co-located hardware. If your servers are unnaware of an imminent power failure, you are SOL!

Bob
--
Would you ever go over to Czechoslovakia, and marry me daughter for me?"


GlennAllen
Sunny with highs in the 80s
Premium
join:2002-11-17
Richmond, VA
Reviews:
·Comcast
·Verizon FiOS

Then if your UPS' failed, it would be the fault of the UPS manufacturer--which would invoke their warranty I presume (unless, of course, it/they failed due to your own failure to replace an old battery). My point, however, is still the same: if you're paying someone to provide such uninterruptible backup power, and that someone fails to provide such power when it becomes needed, then--bottom lilne--it's that someone's fault for not providing that power as contracted.



BronsCon

join:2003-10-24
Walnut Creek, CA
reply to dennismurphy

I'm pretty broke right now, in the middle of moving, but I'd chip in $5. More after I'm done unpacking boxes, if the fund is still open.



TamaraB
Question The Current Paradigm
Premium
join:2000-11-08
Da Bronx
Reviews:
·Optimum Online
·Clearwire Wireless
reply to GlennAllen

said by GlennAllen:

.... if that someone fails to provide such power when it becomes needed, then--bottom lilne--it's that someone's fault for not providing that power as contracted.

Fault is irrelevant, as is blame. Every colo contract I have ever seen absolves the colo facility of any damages if they fail to live up to anything. If BBR had a few UPS' on their rack, the cost would be nothing compared to what is now being incurred.

I doubt Nac.net can be sued for the damage, they, I am sure, have their asses covered.

Bob
--
Would you ever go over to Czechoslovakia, and marry me daughter for me?"


GlennAllen
Sunny with highs in the 80s
Premium
join:2002-11-17
Richmond, VA

You're just making the assumption that BBR didn't contract with anyone to provide uninterruptible backup power. I make no such assumption, so I still stand by my original statement.



TamaraB
Question The Current Paradigm
Premium
join:2000-11-08
Da Bronx
Reviews:
·Optimum Online
·Clearwire Wireless

said by GlennAllen:

You're just making the assumption .....

No assumption. Net Access provides battery UPS, then fail-over to generator power as part of the Colo contract, see: »www.nac.net/business/Colo.asp?page=intro

That "contracted" UPS failed see: »www.webhostingtalk.com/showthrea···t=614624

When Net Access' UPS batteries went dead, their generator failed to start. That's when BBR went dark, and the damage occurred. See the Monday 16th April entry here: »docs.google.com/document/d/1kll8···sle=true

What am I assuming?

Bob
--
Would you ever go over to Czechoslovakia, and marry me daughter for me?"


GlennAllen
Sunny with highs in the 80s
Premium
join:2002-11-17
Richmond, VA

First you said BBR didn't contract for UPS; now you're saying they did. Flip-flop. (You're just wasting my time.)



TamaraB
Question The Current Paradigm
Premium
join:2000-11-08
Da Bronx

PLONK!



Koil
Premium
join:2002-09-10
Irmo, SC
kudos:2

3 edits
reply to dennismurphy

Our arrays are designed to have redundant local UPS' within the cabinet. The incoming power to the cabinet is split on two rails from separately fed PSU's within the data center. Preferably, each of those PSU's not only has a dedicated UPS, but a dedicated generator, but this last piece isn't always the case.

Now, we have thousands of customer's across the globe, and we RARELY ever reach the scenario given here, and I'm pretty sure the setup that I describe is pretty standard, to a degree, so I still fail to see how this perfect storm emerged.

In my mind, what SHOULD have happened, is that once the power loss was detected within the cabinet, and we would be talking both rails here, so this is a FATAL power loss situation, the Server Management should have detected the power loss, the local UPS within the cabinet kicked in and the servers SHUT DOWN GRACEFULLY as there is no power coming in at all. Better to be safe and take the outage at this point.

Assuming that the generators will kick in to provide the power before the batteries die is a fail point, imo, that can be minimized by the UPS at PSU, as this would keep the cabinet from shutting down immediately in a fatal power loss scenario, as the cabinet would never feel a loss of power, and remain up giving the generators time to come up. If the generators don't come up, you've got PSU UPS power until the batteries go, and again at this point, the cabinet sees the fatal power loss and goes down gracefully.

Maybe I'm missing something, but this is a facility design problem in power redundancy and/or bad server management power policy, imo.
--
DSLR Mafia
My Blog - Raising Connor
WoW: Mal'Ganis : Aftershock : Krimdal



dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:3
Reviews:
·Verizon FiOS
reply to dennismurphy

Guys,

Can I Please ask that we move the co-lo discussion to, say, the sysadmins forum? I'd really like to concentrate on gathering funds to help offset the recovery costs. I haven't heard back from Justin - I'm sure he's a little busy - so if someone can point him to this thread and ask him to setup a payment mechanism (like, say, PayPal) I would greatly appreciate it.

Thanks!


Zach1
Premium
join:2006-11-26
NW Minnesota

I can spare a few bucks so pass the collection plate this way. Regardless of what happened or who's at fault, I'm soooo glad to see the site coming back to life and returning to normal.
--
Zach



Jackarino
YacCity
Premium
join:2006-12-28
Allendale, NJ
kudos:1
reply to dennismurphy

I would be willing to donate some $$



neuronbob
THERE ARE NO SHORT CUTS. NONE.

join:2000-03-30
Bedford, OH
reply to dennismurphy

Willing to donate a few bucks. Just say when, where, and how, Justin. Been on here forever and ever and ever and was sorry to see it go down.
--
neuronbob.com