site Search:


 
   
story category
Hosting Provider Hauled Diesel up 17 Flights to Stay Online
Recruiting The Help of Friends and Customers
by Karl Bode Wednesday 31-Oct-2012 tags: business · bandwidth · trouble · networking
After Sandy flooded their 75 Broad Street offices and took out power, a hosting provider by the name of Peer 1 Hosting was forced to lug buckets of diesel fuel up seventeen flights of stairs to keep their backup generators operational. The company had to enlist the help of friends and customers to form the bucket brigade that is currently keeping their services online. The company has been keeping an ongoing log of their troubles over at their company forums. Other providers weren't so lucky, like hosting provider Datagram, whose basement flooded and pumps failed, resulting in numerous major websites (like Gawker.com and the Huffington Post) going offline. 75 Broad Street has of couse been a data center of one sort or another for over 100 years, once housing the offices of International Telephone & Telegraph corporation. As severe weather events become the "new normal" there will likely be some significant reconsideration of that particular location's worth as a data hub.

view: topics flat text 
Post a:
ITALIAN926

join:2003-08-16
kudos:1

Buckets?

Sounds like they were using non-approved gasoline containers? Buckets?

twizlar
I dont think so.
Premium
join:2003-12-24
Brantford, ON
kudos:3

Re: Buckets?

said by ITALIAN926:

Sounds like they were using non-approved gasoline containers? Buckets?

True, but its not gas its diesel.
--
Broadline Networks Inc.
ITALIAN926

join:2003-08-16
kudos:1

Re: Buckets?

Ahhh , Diesel isnt explosive like gasoline, I never knew that.

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
kudos:7

Re: Buckets?

said by ITALIAN926:

Ahhh , Diesel isnt explosive like gasoline, I never knew that.

Diesel only really burns very rapidly if it's atomized and then compressed, as in an engine. You can extinguish a cigarette or match if you drop it in a bucket of diesel fuel. Dumping it on an established fire or dropping a significant burning object (e.g. a rag lit on fire) will ignite it, but not the whooof that gasoline would cause. It just spreads around relatively slowly (as compared to gas) since it doesn't vaporize very fast.
Kearnstd
Elf Wizard
Premium
join:2002-01-22
Mullica Hill, NJ
technically even gasoline is not that flammable. The problem is that its vapors are explosive. In theory a bucket filled with gas on a windy roof would put out a match just like diesel.

But yes Diesel is likely the safest liquid fuel that is made. There is a reason it is used for heating and the transport networks.

Diesel generators also have long life spans because by nature a diesel engine runs very slow compared to its other fueled counterparts. The main kink being is that the fuel takes up lots of space. Like water you cannot compress it.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports

bbrlogue
Learning New Things Daily
Premium
join:2003-12-07
Alexandria, VA

Re: Buckets?

Diesel engines also have long life spans because by necessities they are also build much higher compression ratio.

Anonymous_
Anonymous
Premium
join:2004-06-21
127.0.0.1
kudos:2
Reviews:
·RoadRunner Cable
said by ITALIAN926:

Sounds like they were using non-approved gasoline containers? Buckets?

diesel needs high compression to ignite
You can not just light a match and expect it to burn
--
Well, does your car at least turn into something else? Sometimes I turn it into a trashcan. Hmm...

KrK
Heavy Artillery For The Little Guy
Premium
join:2000-01-17
Tulsa, OK

I admire the tenacity to even bother.

My guess is now they will look for a more permanent solution such as relocation of fuel pumps from the basement level. Most likely outside and elevated while enclosed.

Their updates suggest their fuel supply was exhausted and that they would be failing shortly. This was yesterday afternoon, so I assume they went dark.

--
"Fascism should more properly be called corporatism because it is the merger of state and corporate power." -- Benito Mussolini

twizlar
I dont think so.
Premium
join:2003-12-24
Brantford, ON
kudos:3

Re: I admire the tenacity to even bother.

They are still up and running, they have been carrying buckets up 17 floors for the past 28 hours or so.
--
Broadline Networks Inc.

KrK
Heavy Artillery For The Little Guy
Premium
join:2000-01-17
Tulsa, OK

Re: I admire the tenacity to even bother.

Wow.

Good for them.
talz13

join:2006-03-15
Avon, OH
Yes, the redundant (load balanced?) servers should be hosted in other areas of the country.

Just for my own education (we were working with some load balancer stuff at work), can you load balance load balancers? What happens if your load balancer goes offline? Is that something that you can really distribute across a geographic area?

Guspaz
Guspaz
Premium,MVM
join:2001-11-05
Montreal, QC
kudos:20

Re: I admire the tenacity to even bother.

The most basic form of load balancing is DNS round-robin, and DNS requires multiple redundant name servers for any domain by default (two is the minimum, I believe). So if you're using DNS round-robin (which is as simple as just specifying more than one IP address for a hostname), you can already survive one of your "load balancers" going down.
--
Developer: Tomato/MLPPP, Linux/MLPPP, etc »fixppp.org

rchandra
Stargate Universe fan
Premium
join:2000-11-09
14225-2105
DNS round robin is one way, as another poster has pointed out. There are lots of references on the Web which you can check out. The Ultramonkey project is one such that makes it easier to set up load balancing on RPM based Linux systems. It is basically a "distro" of LVS (Linux Virtual Server). LVS not only checks the health of the servers being load balanced but also (optionally) has a hot standby LVS machine. The key here is there is some way of periodically sending an "are you working?" message to other LVS machine(s) (could be over the network, could be a serial link), and failing over if the other LVS machine seems dead (IP address/MAC address/ARP takeover, that sort of thing). In the router realm, there is HSRP and VRRP, which is similar in concept of having hot standbys. There is also anycasting, which is used, among other things, for the root nameservers. Some root nameserver goes offline or gets DDoSed? No problem, just update the routing tables (BGP I assume) to send the traffic other places, or even to servers which are network-wise closest to the client which is querying.
--
English is a difficult enough language to interpret correctly when its rules are followed, let alone when a writer chooses not to follow those rules.

Jeopardy! replies and randomcaps REALLY suck!
sk1939
Premium
join:2010-10-23
Washington, DC
kudos:9

Re: I admire the tenacity to even bother.

The only issue with that though is that there will still be downtime unless you have the backup site designated as a hot-standby or is always online as a failover since BGP is notoriously slow to update routes (important for the internet).

rchandra
Stargate Universe fan
Premium
join:2000-11-09
14225-2105

Re: I admire the tenacity to even bother.

Oh sure. There's almost always going to be some downtime. But hopefully it can be measured in seconds or minutes as compared to days or weeks for something like a (utility) power outage. It all depends on the engineering chosen.

BTW, thanks for that. I knew of the technology (BGP), but I have no practical experience in having to reroute traffic.
--
English is a difficult enough language to interpret correctly when its rules are followed, let alone when a writer chooses not to follow those rules.

Jeopardy! replies and randomcaps REALLY suck!
sk1939
Premium
join:2010-10-23
Washington, DC
kudos:9
Reviews:
·T-Mobile US

1 edit

Re: I admire the tenacity to even bother.

said by rchandra:

Oh sure. There's almost always going to be some downtime. But hopefully it can be measured in seconds or minutes as compared to days or weeks for something like a (utility) power outage. It all depends on the engineering chosen.

BTW, thanks for that. I knew of the technology (BGP), but I have no practical experience in having to reroute traffic.

No problem. For BGP failover it depends on the peers (have to match timing) and varies depending on provider. BGP still fails-over in a matter of minutes though for the most part (Cisco's default timer is 3 minutes I believe), but is still slower than say EIGRP or OSPF which is almost instantaneous. A lot of organizations also run BGP internally if they have a large enough network.

Edit: »www.enterprisenetworkingplanet.c···ting.htm

»www.petri.co.il/csc_what_is_bgp.htm

rcdailey
Dragoonfly
Premium
join:2005-03-29
Rialto, CA
Reviews:
·RoadRunner Cable
Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.
--
It is easier for a camel to put on a bikini than an old man to thread a needle.
sk1939
Premium
join:2010-10-23
Washington, DC
kudos:9
Reviews:
·T-Mobile US

Re: I admire the tenacity to even bother.

said by rcdailey:

Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.

It was only a small hopsital too *sarcasm* (New York University Hospital) which required 215 patients to be evacuated, tying up dozens of ambulances. Coney Island Hospital also lost backup power.

AVD
Respice, Adspice, Prospice
Premium
join:2003-02-06
Onion, NJ
kudos:1
said by rcdailey:

Then they might need backup generators for the fuel pumps or those wouldn't work in the event of a power outage. That happened to a hospital in NYC.

the fuel pumps went underwater
--
* seek help if having trouble coping
--Standard disclaimers apply.--

battleop

join:2005-09-28
00000

Small hosting provider?

20+ data centers is hardly small.

pfak
Premium
join:2002-12-29
Vancouver, BC

Peer 1 (NOT Peer one) in't small, Karl

They're a pretty large provider and have a number of data centres in North America. They also host some pretty major sites.
--
The more I C, the less I see.
daake07

join:2011-06-28
Kearney, NE

Poor planning on behalf of the hosting company.

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

twizlar
I dont think so.
Premium
join:2003-12-24
Brantford, ON
kudos:3

Re: Poor planning on behalf of the hosting company.

Redundancy for what? They have multiple datacenters throughout N/A. This is one site that is having the issues due to the flood.
--
Broadline Networks Inc.

mmay149q
Premium
join:2009-03-05
Dallas, TX
kudos:48
said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt
--
Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning. -Albert Einstein
Seaboogers

join:2004-11-01
Sarasota, FL

Re: Poor planning on behalf of the hosting company.

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

Uh...did you actually think before you typed this reponse? If I have to point out the ridiculousness of it, you have bigger problems to worry about.

mmay149q
Premium
join:2009-03-05
Dallas, TX
kudos:48

Re: Poor planning on behalf of the hosting company.

said by Seaboogers:

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

Uh...did you actually think before you typed this reponse? If I have to point out the ridiculousness of it, you have bigger problems to worry about.

Yes I did, and yes I understand that these services MAY be accessed all over the entire world or even by the astronauts in the space station orbiting the planet..... But we don't know what all is hosted from this one particular site, and how do you know that west coast routing is routing the traffic directly to this data center? When logically it's better to route it to a data center closer on the west coast? See I took that all into play before I typed my sentence, since probably 95% of people on this website don't know everything hosted through that data center, what I said could have a logical backing to it, if it's following the logical flow of how routing on the internet works.

Matt
--
Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning. -Albert Einstein

TomS_
Git-r-done
Premium,MVM
join:2002-07-19
London, UK
kudos:4

Re: Poor planning on behalf of the hosting company.

Youre also assuming that data is replicated amongst all of these DCs. Id wager the vast majority of it isnt.

This "cloud" thingy is not ubiquitous or omnipresent, there are still plenty of websites and services that will be affected by a single datacentre outage.
watice

join:2008-11-01
New York, NY
said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

not sure if this is a joke or not. Are you saying only east coast ppl without power normally access this hosting providers services? You are aware that the internet is worldwide, right? Bit confused here.

mmay149q
Premium
join:2009-03-05
Dallas, TX
kudos:48

Re: Poor planning on behalf of the hosting company.

said by watice:

said by mmay149q:

said by daake07:

This is what they deserve for not creating redundancy. I don't care how small you are, if you are hosting any data you must assume it is mission critical for someone and down time is not acceptable.

Yeah, because you know, all those people on the coast without power, which is what is required to use the internet will really be dying to access all the websites this hosting company supports and hosts......

Just curious, do you think before you type?

Matt

not sure if this is a joke or not. Are you saying only east coast ppl without power normally access this hosting providers services? You are aware that the internet is worldwide, right? Bit confused here.

No I'm just looking at the standard logic behind routing, which is shortest distance for lowest ping, if up to even 30% of the traffic is coming from the Midwest/West Coast and the other 60% is all East coast, then what I said stands pretty well, but since no one really knows where all their traffic comes from, it's just a logical thought process for looking at the situation.

Matt
--
Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning. -Albert Einstein

battleop

join:2005-09-28
00000

Re: Poor planning on behalf of the hosting company.

Routing decisions are based on shortest path (not physical distance) and often by Geo Location databases. It's possible for you to be on the West Coast and have a shorter path to a server in New York than one in Las Vegas. Not to mention that Geo Location databases are very often inaccurate.

Even with that the vast majority of websites are not using complex hosting from multiple datacenters or CDNs.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT
said by daake07:

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

Heh.. We don't have that issue, yet no one builds them here.
MaynardKrebs
Premium
join:2009-06-17
kudos:4

Re: Poor planning on behalf of the hosting company.

said by Simba7:

said by daake07:

This is why people are trying to relocate data centers inland where the only issue is tornadoes and you can make a building to withstand that.

Heh.. We don't have that issue, yet no one builds them here.

Old missile silos make decent data centers.
Open the blast door to let out all the server heat and from space it looks just like a missile launch in the IR spectrum

michieru
Premium
join:2009-07-25
Miami, FL
Reviews:
·Comcast Business..
·AT&T U-Verse
·Clearwire Wireless
·AT&T Southeast
Your right down time is not acceptable and they are doing what they can to keep the data center up and running. So a round of applause for a couple of nerds running up and down stairs who probably never exercised a day in their life to keep their customers running.

Even with all the redundancy in the world you cannot anticipate every single variable that can happen towards your data center regardless of size or financial power. So you can stop being a prick and give some praise for employees who are keeping services running so the rest of america can sit on the computer and complain about data center redundancy.

battleop

join:2005-09-28
00000

Re: Poor planning on behalf of the hosting company.

You can only become redundant to a certain point once you cross that point you are losing money and it won't matter very long anyway.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

rchandra
Stargate Universe fan
Premium
join:2000-11-09
14225-2105
Uhhhh...no. Just ask the Fukushima Daiichi people about engineering stuff to withstand anything. They had pretty good engineering, but there will always be that tornado/tsunami/earthquake/whatever which is stronger than ever encountered in human history. That's also not to mention all dependencies, such as data links and such which will have a tough time surviving storms, earthquakes, and such. The only complete answer is geographic diversity; some even maybe on a coast, some inland, some in the mountains, you name it.
--
English is a difficult enough language to interpret correctly when its rules are followed, let alone when a writer chooses not to follow those rules.

Jeopardy! replies and randomcaps REALLY suck!

battleop

join:2005-09-28
00000

Re: Poor planning on behalf of the hosting company.

What you are talking about costs a lot of money. For some companies it's cheaper to take a hit for a few days than it is to spend 2-3 times the money on a "what if" scenario.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

Harddrive
Proud American and Infidel since 1968.
Premium
join:2000-09-20
Phone Room
kudos:2

Hurricane of 1938.

Due to the Hurricane of 1938, most businesses in Providence RI moved their telco to the 2nd floor or higher. Maybe NYC businesses will learn from this storm and do the same because it will happen again.

See 7 replies to this post

exocet_cm
I am the law - Judge Dredd
Premium
join:2003-03-23
New Orleans, LA
kudos:2

We learned...

Our offices are on the fourth floor of the NOPD building with generators and fuel supplies on the roof.

Subaru
1-3-2-4
Premium
join:2001-05-31
Greenwich, CT

wait..

Is this our provider now? The way you worded it sounds like it
iansltx

join:2007-02-19
Golden, CO
kudos:2

Re: wait..

Nope. DSLR is still hosted at nac.net.

diablo1892
Say hello to my little friend

join:2011-04-21
Friendly, WV
Reviews:
·HughesNet Satell..
·Dialup USA, Inc.
·EarthLink

Thats a bad idea

I work in some very heavy fumes off and on of kerosene burning, oil, gasoline and even diesel.. The only burning source is the purest and cleanest is propane.. Burning diesel is a bit too heavy for some...
I hope someone will be getting ahold of human services and get this problem taken care of, if the exhaust system is hocked up right then that might work.
--
HN7000S/ 1 watt/ pro plus edition/ 4 pc's on a D-Link wired router/ wireless D-Link router with password
Support only the gaming company's that matter the most, pay for something that actually is worth buying or has a good reason for how much it's worth.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Re: Thats a bad idea

I think they have the exhaust system working perfectly or they'd be having some major issues with the generator.

I'd have to say, "Bravo Peer1", but they really need to offload to another site or tell all their servers to reduce power consumption. Me, I'd use the multi-gigabit trunks to move data to a secondary location just in case. That way, if the generator does fail, it *should* be easy to shift the traffic to the secondary location.

BTW: I heard WV got a few feet of snow. I'm surprised you're online (HughesNet).
--
Bresnan 30M/5M | CenturyLink 5M/896K
MyWS[PnmIIX3@3.2G,8G RAM,500G+1.5T+2T HDDs,Win7]
WifeWS[A64@2G,2G RAM,120G HDD,Win7]
Router[2xP3@1G,2G RAM,18G HDD,Allied Telesyn AT2560FX,2xDigital DE504,Sun X1034A,2xSun X4444A,SMC 8432BTA,Gentoo]

diablo1892
Say hello to my little friend

join:2011-04-21
Friendly, WV
Reviews:
·HughesNet Satell..
·Dialup USA, Inc.
·EarthLink

Re: Thats a bad idea

wow, finally I managed to make it back to this page.. Yes Simba7 you heard right but my area only got flurries for a couple of hours or even less than that.. However those "flurries" are NOT my cause for my connection being so slow, its the service company. All day today i could not load not one page successfully without any errors, any broken pictures or anything like that, I can't even stay logged into my skype while loading google's homepage.. It takes me a half an hour to load up google homepage, are you seriuos??? and HN if you are listening to this (reeding) then PLEASE F@CK OFF!! I can post what I want about you, I tried my damndest hardest to give you everything i could about my issue.. what did you do? deleted most of my posts.

Doctor Olds
I Need A Remedy For What's Ailing Me.
Premium,VIP
join:2001-04-19
1970 442 W30
kudos:18
said by diablo1892:

The only burning source is the purest and cleanest is propane..

Wrong. Natural Gas is much cleaner even when it is wet with up to 20% Propane in it. Dry Natural Gas as it is called after refining is pure Methane. Much cleaner burning than Propane.

»www.naturalgas.org/overview/background.asp
--
What’s the point of owning a supercar if you can’t scare yourself stupid from time to time?

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Few data centers here?

I'm kinda surprised there are very few data centers here in Montana. Nothing major happens here in Billings.

Heck, we don't get tornadoes, floods, earthquakes, hurricanes, tsunamis. We might get the occasional blizzard or two.

We *might* get an occasional fire or two.. that's about it.
--
Bresnan 30M/5M | CenturyLink 5M/896K
MyWS[PnmIIX3@3.2G,8G RAM,500G+1.5T+2T HDDs,Win7]
WifeWS[A64@2G,2G RAM,120G HDD,Win7]
Router[2xP3@1G,2G RAM,18G HDD,Allied Telesyn AT2560FX,2xDigital DE504,Sun X1034A,2xSun X4444A,SMC 8432BTA,Gentoo]

Oh_No
Trogglus normalus

join:2011-05-21
Chicago, IL

Natural Gas??

Have these people not heard of natural gas generators???
Why would anyone waste time with diesel generators when have unlimited natural gas piped right into your building already.

See 24 replies to this post
tmc8080

join:2004-04-24
Brooklyn, NY
Reviews:
·ooma
·Optimum Online
·Verizon FiOS

?

If you have a generator providing POWER to the building, you'd think they'd only have to lug just enough fuel to get the generator going so they could use and freight elevator, no?

of course it would SUCK if you get stuck in an elevator full of fuel... buildings will have to be redesigned to have a fuel siphoning line (for emergency use ONLY).

I'm hoping people will finally kick gasoline to the curb and go for alternatives.. natural gas, ethanol, hydrogen, capacitors and battery arrays.

BTW, thre are some current and new "ZONE A" real estate that might get another look at how much it's valued in the coming year or so..

See 9 replies to this post

lordfly

join:2000-10-12
Homestead, FL
Reviews:
·SkyNet360

Data Center in Hurricane Zone

I happened to come across this provider, whose data center is located in Miami. They have gone to lots of trouble to make it safe and secure.

»www.terremark.com/data-centers/a···cas.aspx

FiReSTaRT
Premium
join:2010-02-26
Canada

NG?

What about running Natural Gas generators? No need to haul fuel.

AVD
Respice, Adspice, Prospice
Premium
join:2003-02-06
Onion, NJ
kudos:1

Re: NG?

at least they can form a bucket brgade to run the generator, what happens to a NG generator when the as goes out?
--
* seek help if having trouble coping
--Standard disclaimers apply.--

Doctor Olds
I Need A Remedy For What's Ailing Me.
Premium,VIP
join:2001-04-19
1970 442 W30
kudos:18

Re: NG?

said by AVD:

at least they can form a bucket brgade to run the generator, what happens to a NG generator when the as goes out?

Use a emergency CNG tank and switch-over to it if the NG gas mains lose pressure.
--
What’s the point of owning a supercar if you can’t scare yourself stupid from time to time?

Wednesday, 22-May 16:35:57 Terms of Use & Privacy | feedback | contact | Hosting by nac.net - DSL,Hosting & Co-lo
over 13.5 years online © 1999-2013 dslreports.com.