dslreports logo
 
    All Forums Hot Topics Gallery
spc
view:
topics flat nest 
Comments on news posted 2012-10-31 17:28:11: After Sandy flooded their 75 Broad Street offices and took out power, a hosting provider by the name of Peer 1 Hosting was forced to lug buckets of diesel fuel up seventeen flights of stairs to keep their backup generators operational. ..


ITALIAN926
join:2003-08-16

ITALIAN926

Member

Buckets?

Sounds like they were using non-approved gasoline containers? Buckets?

twizlar
I dont think so.
Premium Member
join:2003-12-24
Brantford, ON

twizlar

Premium Member

Re: Buckets?

said by ITALIAN926:

Sounds like they were using non-approved gasoline containers? Buckets?

True, but its not gas its diesel.

ITALIAN926
join:2003-08-16

ITALIAN926

Member

Re: Buckets?

Ahhh , Diesel isnt explosive like gasoline, I never knew that.

cdru
Go Colts
MVM
join:2003-05-14
Fort Wayne, IN

cdru

MVM

Re: Buckets?

said by ITALIAN926:

Ahhh , Diesel isnt explosive like gasoline, I never knew that.

Diesel only really burns very rapidly if it's atomized and then compressed, as in an engine. You can extinguish a cigarette or match if you drop it in a bucket of diesel fuel. Dumping it on an established fire or dropping a significant burning object (e.g. a rag lit on fire) will ignite it, but not the whooof that gasoline would cause. It just spreads around relatively slowly (as compared to gas) since it doesn't vaporize very fast.
Kearnstd
Space Elf
Premium Member
join:2002-01-22
Mullica Hill, NJ

Kearnstd to ITALIAN926

Premium Member

to ITALIAN926
technically even gasoline is not that flammable. The problem is that its vapors are explosive. In theory a bucket filled with gas on a windy roof would put out a match just like diesel.

But yes Diesel is likely the safest liquid fuel that is made. There is a reason it is used for heating and the transport networks.

Diesel generators also have long life spans because by nature a diesel engine runs very slow compared to its other fueled counterparts. The main kink being is that the fuel takes up lots of space. Like water you cannot compress it.

bbrlogue
Learning New Things Daily
Premium Member
join:2003-12-07
Alexandria, VA

bbrlogue

Premium Member

Re: Buckets?

Diesel engines also have long life spans because by necessities they are also build much higher compression ratio.

Anonymous_
Anonymous
Premium Member
join:2004-06-21
127.0.0.1

Anonymous_ to ITALIAN926

Premium Member

to ITALIAN926
said by ITALIAN926:

Sounds like they were using non-approved gasoline containers? Buckets?

diesel needs high compression to ignite
You can not just light a match and expect it to burn

KrK
Heavy Artillery For The Little Guy
Premium Member
join:2000-01-17
Tulsa, OK
Netgear WNDR3700v2
Zoom 5341J

KrK

Premium Member

I admire the tenacity to even bother.

My guess is now they will look for a more permanent solution such as relocation of fuel pumps from the basement level. Most likely outside and elevated while enclosed.

Their updates suggest their fuel supply was exhausted and that they would be failing shortly. This was yesterday afternoon, so I assume they went dark.

twizlar
I dont think so.
Premium Member
join:2003-12-24
Brantford, ON

twizlar

Premium Member

Re: I admire the tenacity to even bother.

They are still up and running, they have been carrying buckets up 17 floors for the past 28 hours or so.

KrK
Heavy Artillery For The Little Guy
Premium Member
join:2000-01-17
Tulsa, OK

KrK

Premium Member

Re: I admire the tenacity to even bother.

Wow.

Good for them.
talz13
join:2006-03-15
Avon, OH

talz13 to KrK

Member

to KrK
Yes, the redundant (load balanced?) servers should be hosted in other areas of the country.

Just for my own education (we were working with some load balancer stuff at work), can you load balance load balancers? What happens if your load balancer goes offline? Is that something that you can really distribute across a geographic area?

Guspaz
Guspaz
MVM
join:2001-11-05
Montreal, QC

Guspaz

MVM

Re: I admire the tenacity to even bother.

The most basic form of load balancing is DNS round-robin, and DNS requires multiple redundant name servers for any domain by default (two is the minimum, I believe). So if you're using DNS round-robin (which is as simple as just specifying more than one IP address for a hostname), you can already survive one of your "load balancers" going down.

rchandra
Stargate Universe fan
Premium Member
join:2000-11-09
14225-2105
ARRIS ONT1000GJ4
EnGenius EAP1250

rchandra to talz13

Premium Member

to talz13
DNS round robin is one way, as another poster has pointed out. There are lots of references on the Web which you can check out. The Ultramonkey project is one such that makes it easier to set up load balancing on RPM based Linux systems. It is basically a "distro" of LVS (Linux Virtual Server). LVS not only checks the health of the servers being load balanced but also (optionally) has a hot standby LVS machine. The key here is there is some way of periodically sending an "are you working?" message to other LVS machine(s) (could be over the network, could be a serial link), and failing over if the other LVS machine seems dead (IP address/MAC address/ARP takeover, that sort of thing). In the router realm, there is HSRP and VRRP, which is similar in concept of having hot standbys. There is also anycasting, which is used, among other things, for the root nameservers. Some root nameserver goes offline or gets DDoSed? No problem, just update the routing tables (BGP I assume) to send the traffic other places, or even to servers which are network-wise closest to the client which is querying.

sk1939
Premium Member
join:2010-10-23
Frederick, MD

1 recommendation

sk1939

Premium Member

Re: I admire the tenacity to even bother.

The only issue with that though is that there will still be downtime unless you have the backup site designated as a hot-standby or is always online as a failover since BGP is notoriously slow to update routes (important for the internet).

rchandra
Stargate Universe fan
Premium Member
join:2000-11-09
14225-2105
ARRIS ONT1000GJ4
EnGenius EAP1250

rchandra

Premium Member

Re: I admire the tenacity to even bother.

Oh sure. There's almost always going to be some downtime. But hopefully it can be measured in seconds or minutes as compared to days or weeks for something like a (utility) power outage. It all depends on the engineering chosen.

BTW, thanks for that. I knew of the technology (BGP), but I have no practical experience in having to reroute traffic.

sk1939
Premium Member
join:2010-10-23
Frederick, MD
ARRIS SB8200
Ubiquiti UDM-Pro
Juniper SRX320

1 edit

sk1939

Premium Member

Re: I admire the tenacity to even bother.

said by rchandra:

Oh sure. There's almost always going to be some downtime. But hopefully it can be measured in seconds or minutes as compared to days or weeks for something like a (utility) power outage. It all depends on the engineering chosen.

BTW, thanks for that. I knew of the technology (BGP), but I have no practical experience in having to reroute traffic.

No problem. For BGP failover it depends on the peers (have to match timing) and varies depending on provider. BGP still fails-over in a matter of minutes though for the most part (Cisco's default timer is 3 minutes I believe), but is still slower than say EIGRP or OSPF which is almost instantaneous. A lot of organizations also run BGP internally if they have a large enough network.

Edit: »www.enterprisenetworking ··· ting.htm

»www.petri.co.il/csc_what ··· _bgp.htm

rcdailey
Dragoonfly
Premium Member
join:2005-03-29
Rialto, CA

rcdailey to KrK

Premium Member

to KrK
Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.

sk1939
Premium Member
join:2010-10-23
Frederick, MD
ARRIS SB8200
Ubiquiti UDM-Pro
Juniper SRX320

sk1939

Premium Member

Re: I admire the tenacity to even bother.

said by rcdailey:

Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.

It was only a small hopsital too *sarcasm* (New York University Hospital) which required 215 patients to be evacuated, tying up dozens of ambulances. Coney Island Hospital also lost backup power.

AVD
Respice, Adspice, Prospice
Premium Member
join:2003-02-06
Onion, NJ

AVD to rcdailey

Premium Member

to rcdailey
said by rcdailey:

Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.

the fuel pumps went underwater

battleop
join:2005-09-28
00000

battleop

Member

Small hosting provider?

20+ data centers is hardly small.

pfak
Premium Member
join:2002-12-29
Vancouver, BC

pfak

Premium Member

Peer 1 (NOT Peer one) in't small, Karl

They're a pretty large provider and have a number of data centres in North America. They also host some pretty major sites.
daake07
join:2011-06-28
Kearney, NE

daake07

Member

Poor planning on behalf of the hosting company.

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

twizlar
I dont think so.
Premium Member
join:2003-12-24
Brantford, ON

twizlar

Premium Member

Re: Poor planning on behalf of the hosting company.

Redundancy for what? They have multiple datacenters throughout N/A. This is one site that is having the issues due to the flood.
mmay149q
Premium Member
join:2009-03-05
Dallas, TX

1 recommendation

mmay149q to daake07

Premium Member

to daake07
said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt
Seaboogers
join:2004-11-01
Sarasota, FL

Seaboogers

Member

Re: Poor planning on behalf of the hosting company.

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

Uh...did you actually think before you typed this reponse? If I have to point out the ridiculousness of it, you have bigger problems to worry about.
mmay149q
Premium Member
join:2009-03-05
Dallas, TX

mmay149q

Premium Member

Re: Poor planning on behalf of the hosting company.

said by Seaboogers:

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

Uh...did you actually think before you typed this reponse? If I have to point out the ridiculousness of it, you have bigger problems to worry about.

Yes I did, and yes I understand that these services MAY be accessed all over the entire world or even by the astronauts in the space station orbiting the planet..... But we don't know what all is hosted from this one particular site, and how do you know that west coast routing is routing the traffic directly to this data center? When logically it's better to route it to a data center closer on the west coast? See I took that all into play before I typed my sentence, since probably 95% of people on this website don't know everything hosted through that data center, what I said could have a logical backing to it, if it's following the logical flow of how routing on the internet works.

Matt

TomS_
Git-r-done
MVM
join:2002-07-19
London, UK

TomS_

MVM

Re: Poor planning on behalf of the hosting company.

Youre also assuming that data is replicated amongst all of these DCs. Id wager the vast majority of it isnt.

This "cloud" thingy is not ubiquitous or omnipresent, there are still plenty of websites and services that will be affected by a single datacentre outage.
watice
join:2008-11-01
New York, NY

watice to mmay149q

Member

to mmay149q
said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

not sure if this is a joke or not. Are you saying only east coast ppl without power normally access this hosting providers services? You are aware that the internet is worldwide, right? Bit confused here.
mmay149q
Premium Member
join:2009-03-05
Dallas, TX

mmay149q

Premium Member

Re: Poor planning on behalf of the hosting company.

said by watice:

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

not sure if this is a joke or not. Are you saying only east coast ppl without power normally access this hosting providers services? You are aware that the internet is worldwide, right? Bit confused here.

No I'm just looking at the standard logic behind routing, which is shortest distance for lowest ping, if up to even 30% of the traffic is coming from the Midwest/West Coast and the other 60% is all East coast, then what I said stands pretty well, but since no one really knows where all their traffic comes from, it's just a logical thought process for looking at the situation.

Matt

battleop
join:2005-09-28
00000

1 recommendation

battleop

Member

Re: Poor planning on behalf of the hosting company.

Routing decisions are based on shortest path (not physical distance) and often by Geo Location databases. It's possible for you to be on the West Coast and have a shorter path to a server in New York than one in Las Vegas. Not to mention that Geo Location databases are very often inaccurate.

Even with that the vast majority of websites are not using complex hosting from multiple datacenters or CDNs.

SimbaSeven
I Void Warranties
join:2003-03-24
Billings, MT
·StarLink

SimbaSeven to daake07

Member

to daake07
said by daake07:

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

Heh.. We don't have that issue, yet no one builds them here.
MaynardKrebs
We did it. We heaved Steve. Yipee.
Premium Member
join:2009-06-17

MaynardKrebs

Premium Member

Re: Poor planning on behalf of the hosting company.

said by SimbaSeven:

said by daake07:

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

Heh.. We don't have that issue, yet no one builds them here.

Old missile silos make decent data centers.
Open the blast door to let out all the server heat and from space it looks just like a missile launch in the IR spectrum

michieru
Premium Member
join:2009-07-25
Denver, CO

michieru to daake07

Premium Member

to daake07
Your right down time is not acceptable and they are doing what they can to keep the data center up and running. So a round of applause for a couple of nerds running up and down stairs who probably never exercised a day in their life to keep their customers running.

Even with all the redundancy in the world you cannot anticipate every single variable that can happen towards your data center regardless of size or financial power. So you can stop being a prick and give some praise for employees who are keeping services running so the rest of america can sit on the computer and complain about data center redundancy.

battleop
join:2005-09-28
00000

battleop

Member

Re: Poor planning on behalf of the hosting company.

You can only become redundant to a certain point once you cross that point you are losing money and it won't matter very long anyway.

rchandra
Stargate Universe fan
Premium Member
join:2000-11-09
14225-2105
ARRIS ONT1000GJ4
EnGenius EAP1250

rchandra to daake07

Premium Member

to daake07
Uhhhh...no. Just ask the Fukushima Daiichi people about engineering stuff to withstand anything. They had pretty good engineering, but there will always be that tornado/tsunami/earthquake/whatever which is stronger than ever encountered in human history. That's also not to mention all dependencies, such as data links and such which will have a tough time surviving storms, earthquakes, and such. The only complete answer is geographic diversity; some even maybe on a coast, some inland, some in the mountains, you name it.

battleop
join:2005-09-28
00000

battleop

Member

Re: Poor planning on behalf of the hosting company.

What you are talking about costs a lot of money. For some companies it's cheaper to take a hit for a few days than it is to spend 2-3 times the money on a "what if" scenario.

Harddrive
Proud American and Infidel since 1968.
Premium Member
join:2000-09-20
Fort Worth, TX

1 recommendation

Harddrive

Premium Member

Hurricane of 1938.

Due to the Hurricane of 1938, most businesses in Providence RI moved their telco to the 2nd floor or higher. Maybe NYC businesses will learn from this storm and do the same because it will happen again.

•••••••

exocet_cm
Writing
Premium Member
join:2003-03-23
Brooklyn, NY

exocet_cm

Premium Member

We learned...

Our offices are on the fourth floor of the NOPD building with generators and fuel supplies on the roof.

Subaru
1-3-2-4
Premium Member
join:2001-05-31
Greenwich, CT

Subaru

Premium Member

wait..

Is this our provider now? The way you worded it sounds like it
iansltx
join:2007-02-19
Austin, TX

iansltx

Member

Re: wait..

Nope. DSLR is still hosted at nac.net.

diablo18926
R.I.P. Donald Lee Wise
join:2011-04-21
Friendly, WV

diablo18926

Member

Thats a bad idea

I work in some very heavy fumes off and on of kerosene burning, oil, gasoline and even diesel.. The only burning source is the purest and cleanest is propane.. Burning diesel is a bit too heavy for some...
I hope someone will be getting ahold of human services and get this problem taken care of, if the exhaust system is hocked up right then that might work.

•••

SimbaSeven
I Void Warranties
join:2003-03-24
Billings, MT
·StarLink

SimbaSeven

Member

Few data centers here?

I'm kinda surprised there are very few data centers here in Montana. Nothing major happens here in Billings.

Heck, we don't get tornadoes, floods, earthquakes, hurricanes, tsunamis. We might get the occasional blizzard or two.

We *might* get an occasional fire or two.. that's about it.

Oh_No
Trogglus normalus
join:2011-05-21
Chicago, IL

Oh_No

Member

Natural Gas??

Have these people not heard of natural gas generators???
Why would anyone waste time with diesel generators when have unlimited natural gas piped right into your building already.

••••••••••••••••••••••••
tmc8080
join:2004-04-24
Brooklyn, NY

tmc8080

Member

?

If you have a generator providing POWER to the building, you'd think they'd only have to lug just enough fuel to get the generator going so they could use and freight elevator, no?

of course it would SUCK if you get stuck in an elevator full of fuel... buildings will have to be redesigned to have a fuel siphoning line (for emergency use ONLY).

I'm hoping people will finally kick gasoline to the curb and go for alternatives.. natural gas, ethanol, hydrogen, capacitors and battery arrays.

BTW, thre are some current and new "ZONE A" real estate that might get another look at how much it's valued in the coming year or so..

•••••••••

lordfly
join:2000-10-12
Homestead, FL

lordfly

Member

Data Center in Hurricane Zone

I happened to come across this provider, whose data center is located in Miami. They have gone to lots of trouble to make it safe and secure.

»www.terremark.com/data-c ··· cas.aspx

FiReSTaRT
Premium Member
join:2010-02-26
Canada

FiReSTaRT

Premium Member

NG?

What about running Natural Gas generators? No need to haul fuel.

AVD
Respice, Adspice, Prospice
Premium Member
join:2003-02-06
Onion, NJ

AVD

Premium Member

Re: NG?

at least they can form a bucket brgade to run the generator, what happens to a NG generator when the as goes out?

Doctor Olds
I Need A Remedy For What's Ailing Me.
Premium Member
join:2001-04-19
1970 442 W30

Doctor Olds

Premium Member

Re: NG?

said by AVD:

at least they can form a bucket brgade to run the generator, what happens to a NG generator when the as goes out?

Use a emergency CNG tank and switch-over to it if the NG gas mains lose pressure.