dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
1441
share rss forum feed

HELLFIRE
Premium
join:2009-11-25
kudos:18

How did Hurricane Sandy disrupt your week so far?

Just putting this out to see what Sandy has (not) done for you so far.

For me personally :

- Monday : gathering and inventorying licence / user counts for the VPN headends. One or two tickets from users advising DR connectivity does not work and needs "emergency" changes to fix.

- Monday night : first indication the hurricane made landfall is the auto-generated tickets for circuits down, first report started of affected sites / clients. This continues throughout the night.

- Tuesday AM / night : high level conf calls and escalations to figure out what is and is not related to the hurricane. The usual managerial "request status check" requests as well, plus finding one or two locations with primary and secondary connectivity
located in NY and NJ that are hard down with no ETA to restoration.

- Wednesday : Sierra Squared, Delta Squared.

Personal take away from this so far? Don't buy circuit / datacenter space in NY or NJ

Regards


Wily_One
Premium
join:2002-11-24
San Jose, CA

1 edit
Won't go into the details but I'll just say things start to break when both the primary and backup transatlantic links go down at the same time. Imagine that.


DC DSL
There's a reason I'm Command.
Premium
join:2000-07-30
Washington, DC
kudos:2
reply to HELLFIRE
said by HELLFIRE:

Personal take away from this so far? Don't buy circuit / datacenter space in NY or NJ

There is no place on Earth that is not at risk for some peril that can ruin your day. The more critical your data center requirements, the more critical it is that you have geographically diverse backup sites that are nowhere near each other. Any major enterprise should have hot, or at least warm, standby centers ready to pick up within not more than a few hours if something knocks out the primary.
--
"Dance like the photo isn't being tagged; love like you've never been unfriended; and tweet like nobody is following."


Archivis
Your Daddy
Premium
join:2001-11-26
Earth
kudos:19
reply to HELLFIRE
Just VPN'd in from home both days. It was like a free day off work because all of our clients were off work and they weren't breaking anything.

The data centers held up without issue.
--
A government big enough to give you everything you want, is strong enough to take everything you have. -MLK


The WeaseL
Premium
join:2001-12-03
Minnesota
reply to HELLFIRE
So far dealing with a new round of spam regarding the hurricane.

Also had a conference call scheduled yesterday with someone in Boston which should have taken about 20 minutes. After an hour and 20 minutes of dealing with poor cell reception due to towers being out, we rescheduled.
--
How lucky am I to have known someone who is so hard to say good-bye to.


DarkLogix
Texan and Proud
Premium
join:2008-10-23
Baytown, TX
kudos:3
reply to HELLFIRE
got news about NY, that's all that changed.


BBBanditRuR
Dingbits

join:2009-06-02
Parachute, CO
reply to HELLFIRE
Shipping was disrupted - though the impact to us was minimal.


donoreo
Premium
join:2002-05-30
North York, ON

1 recommendation

reply to HELLFIRE
Several TV shows were not on as scheduled Monday night.


XANAVirus
Premium
join:2012-03-03
Lavalette, WV
Reviews:
·Callcentric
My Primary VoIP service was down (but the cable VoIP backup was still online), as such I couldn't get calls and I had to use the backup a lot - not to mention my cellphone with only 1400 minutes/month.

In general, it was very disruptive since I had to explain to a lot of non-technical people (who remember you/me by the phone number that appears on their display):

"My phone service is down, so you've gotta call me on this backup line [backup number] or text me at [this other number]."

"My phone provider is offline cause of the hurricane, you've gotta use this or this number for the time being."

Even though I say things like that, I still get people who are like "Huh?" and they have no idea about having backup phone lines (aside from cell phones) or having to use a different number they never heard of to reach me or heaven forbid me calling them from a new number.

--- And that was just that.
For me personally it was very hard to get around in my area of WV. Where I'm at we've got only one way out of my town that leads to my workplace, and it was blocked cause of tons of accidents over these several days.

Not to mention I was still expected to show up regardless of all other factors (being 'essential' isn't the best thing a lot of the time).

And of course there was all this cold and snow and rain which made it hard to travel any sort of distance without seeing an accident.


Camelot One
Premium,MVM
join:2001-11-21
Greenwood, IN
kudos:2
reply to HELLFIRE
I experienced no direct problems from the storm, other than a part I need not getting shipped out. None of my clients were directly effected, but with with NYC shut down, they took a few days off. (they rely on wallstreet being open)

So I got a couple days off.
--
Intel i7-2600k /ASRock P67 Extreme4 /4x 4Gb G.Skill /2x Intel 510 series 250Gb SSD /3x WD20EADS 2TB /2x PNY GTX 260 /Silverstone 850W /Custom water cooler /Antec Twelve-Hundred


Wily_One
Premium
join:2002-11-24
San Jose, CA
Reviews:
·AT&T U-Verse
reply to DC DSL
Click for full size
said by DC DSL:

The more critical your data center requirements, the more critical it is that you have geographically diverse backup sites that are nowhere near each other.

Doesn't help if your European customers can't reach your US-based but still geographically diverse services. Not a lot you can do if the links from Europe to North America all basically go through the New York City area.


DC DSL
There's a reason I'm Command.
Premium
join:2000-07-30
Washington, DC
kudos:2
said by Wily_One:

said by DC DSL:

The more critical your data center requirements, the more critical it is that you have geographically diverse backup sites that are nowhere near each other.

Doesn't help if your European customers can't reach your US-based but still geographically diverse services. Not a lot you can do if the links from Europe to North America all basically go through the New York City area.

Um, you're forgetting one of the basic tenets of the Internet's design: Survivability. Transit times will suck but traffic can be rerouted. If your DC has no power, is flooded, or outright destroyed, it doesn't matter how close it is to your customers or a backbone.
--
"Dance like the photo isn't being tagged; love like you've never been unfriended; and tweet like nobody is following."


Wily_One
Premium
join:2002-11-24
San Jose, CA
Reviews:
·AT&T U-Verse
Did I say we lost connection to customers over the Internet? In this case the "customers" I'm referring to are internal users and systems in Europe, which cross dedicated WAN links connecting our LANs. Thanks to Sandy, they lost both the primary and the backup link for a brief period of time.

But please, keep pontificating.

HELLFIRE
Premium
join:2009-11-25
kudos:18
reply to HELLFIRE
Ahh, thanks for that image Wily_One. The stupid thing that graphic shows is that the two major exchange points
on the CONUS east coast are right in the hurricane landing zone. D'oh!

But I agree, there's no perfect solution.

Regards


DC DSL
There's a reason I'm Command.
Premium
join:2000-07-30
Washington, DC
kudos:2
reply to Wily_One
said by Wily_One:

Did I say we lost connection to customers over the Internet? In this case the "customers" I'm referring to are internal users and systems in Europe, which cross dedicated WAN links connecting our LANs. Thanks to Sandy, they lost both the primary and the backup link for a brief period of time.

But please, keep pontificating.

It doesn't matter what protocol you are using. Your diagram is of transoceanic Internet cables, so you implied that your connections are TCP/IP whether they are open to the general public or only those with engraved invitations. Even in 1970s, when I started in the industry and we used point-to-point connections, the underlying transports could take secondary routes if the primary was out. Heck, even the ancient long distance and overseas voice services could choose alternate intermediate routes to reach a destination.

No matter what the topology, if everything could go kaboom because of NY, that is a deficient design for something considered "critical." There should be at least a secondary data center somewhere other than the NY area. And your routing should be able to reach it or the primary by going over EMEAP routes.

If you want to take my calling this fundamental design flaw as pontificating, so be it.
--
"Dance like the photo isn't being tagged; love like you've never been unfriended; and tweet like nobody is following."

ke4pym
Premium
join:2004-07-24
Charlotte, NC
reply to HELLFIRE
I was going to say nothing other than my normal tv programming.

But we did have 1 of 5 new GPS based NTP master clocks show up 2 days late due to "issues beyond UPS' control".

Pretty easy. Comparitivly speaking.


Wily_One
Premium
join:2002-11-24
San Jose, CA
Reviews:
·AT&T U-Verse
reply to DC DSL
I'm not going to divulge my company's internal architecture, but we don't have Data Centers anywhere near NY. I said the links between the U.S. and Europe go through there, as the graphic showed. Of course we have a backup circuit via a different provider, but it went down too. Yes once they got alerted they rerouted it and the loss of connectivity lasted a little over 1 hour.

Maybe it's my imagination, but you come across as lecturing me. It's not my design. It's simply a fact most of the undersea cables to Europe run through the NY area.


NOCTech75
Premium
join:2009-06-29
Marietta, GA
Reviews:
·Comcast
reply to HELLFIRE
About a half dozen calls into our call center about cell customers having garbled communications or one way communications, all in NJ. Had to explain to CS that whole hurricane thingy and the clusterfuck it created.