dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
19
verix
join:2004-12-30
Oakland, CA

2 recommendations

verix to rblizz

Member

to rblizz

Re: Back to worst-case-scenario to start with!

said by rblizz:

I don't know about Oakland, but in most of the world "reality happens." That's why they call the call earthquakes and storms like Sandy "natural disasters" ... "disasters" being the key word here. I don't know why so many here seem to think we live in a La La World where every inconvenience can always be averted -- and, if it can't, someone's head must roll.

You didn't answer my questions. It's like you're trying to conflate my sympathies for Sandy victims to some sort of acceptance of CC's lack of service, when every other provider did not experience significant downtime. Even those that have a presence in NYC. And thank you for your lecture on natural disasters. That's why I'm waiting to hear how CC responds before I make a decision. Did I say I was leaving?

Personally I'm happy it only took 3 days for CallCentric to get back up.

Glad you're happy. A lot of people aren't.
rblizz
join:2001-12-16
North Richland Hills, TX

rblizz

Member

said by verix:

Glad you're happy. A lot of people aren't.

A lot of people like to bitch and confuse that with with "doing something." No one is happy about the downtime -- but I realize it could have been much, much worse -- so I'm happy that it wasn't.

It's also plain to me that CallCentric is proud of the time and effort they put into their infrastructure. There's a reason you and I chose CallCentric in the first place. It was their reputation for quality. So now they've been attacked from the outside with an unrelentlessness DDoS barrage (which they beat back) and a once in a century storm (that also took down the NYSE) and you're demanding they "explain themselves." They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.
verix
join:2004-12-30
Oakland, CA

verix

Member

said by rblizz:

They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?
Expand your moderator at work
Iscream
Premium Member
join:2009-02-17
New York, NY

1 recommendation

Iscream to verix

Premium Member

to verix

Re: Back to worst-case-scenario to start with!

Before I'm going to sleep a couple hours:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

- speaking of MAJOR national and world-wide carriers, about those who were and still are down - here is not compete list:

Belgacom, Telia, KPN (including former iBasis), TATA (not just BIG - this is THE LARGEST CARRIER in the world, no one is LARGER - TATA has TWO POPs in NY State - both were down), Verizon - still down, no even sign of life, ATT, T-Mobile, Sprint - either fully or still partially down along the way from Staten Island, through Brooklyn and Lower Manhattan including our offices. Speaking of "smaller" ones - Broadvox, iNetworks (that's Bandwidth and Dash-911 and Republic Wireless). Want even smaller ones - how about VoxBone ? Huh?

For those who want more detailed info - details available on the Net and specialized subscription lists.

Regarding NY Free DIDs - I wrote in my previous messages that those are currently NOT available, but NOT due to Callcentric's issues - because all TDM interconnects in LATA 132 (Big Apple) are still dead including SS7 network - at least in Lower Manhattan. We'll bring those DIDs up (as a matter of fact - they'll come up just by themselves) a soon as the PSTN network in New York restores.
rblizz
join:2001-12-16
North Richland Hills, TX

rblizz

Member

said by Iscream:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

Sorry, I'm the one who's been messing that one up. So, just slightly over two days for full restoration of service.
rblizz

rblizz to Anon

Member

to Anon
said by rblizz:

Yep. Time for me to leave this thread for now. I'm allergic to the unreasonable.

Just to give some folks an idea of the scale of what's going on in New York (and elsewhere) ...
quote:
As blackouts linger, Northeasterners try to adjust

Sandy blacked out some of the nation's most densely populated cities and suburbs, instantly taking away modern conveniences from Virginia to Massachusetts and as far west as the Great Lakes.

For power companies, the scale of the destruction was unmatched -- more widespread than any blizzard or ice storm and worse than the aftermath of Hurricane Katrina.

"It's unprecedented: fallen trees, debris, the roads, water, snow. It's a little bit of everything," said Brian Wolff, senior vice president of the Edison Electric Institute, a group that lobbies for utilities.

Initially, about 60 million people were without power in 8.2 million homes and businesses. By Wednesday night, that number had fallen to roughly 44 million people in 6 million households and businesses.

Even as power slowly returned to some pockets, a new headache emerged: Backup batteries and generators running cellphone towers were running out of juice. One out of every five towers was down, according to the Federal Communications Commission.

»news.yahoo.com/blackouts ··· 751.html
daneli
join:2012-10-23
Ann Arbor, MI

2 recommendations

daneli to Iscream

Member

to Iscream
I want to try to be constructive in my comments. Full redundancy with instant failover can be a very expensive and technically difficult challenge to meet. If a business can tolerate a longer time window (~ 8 hours) and can concentrate on only a subset of functions, then it can be much cheaper and easier to get some services back up and running in the face of a disruption. Certainly it should be possible for callcentric to provide a backup solution for its website and email that is more robust than the current solution. Perhaps a subset of functions - like voicemail or did forwarding - could also be made available from a remote location if the time window is extended to 12 hours or so?

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

After the dust settles, I think it is reasonable for us to ask callcentric to be transparent about what steps if any it plans to take to mitigate these kinds of risks in the future. The risks I have in mind are not those associated with a "storm of the century" but the usual kinds of things - flood, fire, theft, etc - that businesses try to anticipate.
rblizz
join:2001-12-16
North Richland Hills, TX

rblizz

Member

said by daneli:

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

Unfortunately it doesn't appear that a generator was an option in this building. But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.

Davesnothere
Change is NOT Necessarily Progress
Premium Member
join:2009-06-15
Canada

Davesnothere to daneli

Premium Member

to daneli
said by daneli:

I want to try to be constructive in my comments....

 
Absolutely, and I agree with pretty well all of them.

BTW, their UPS runtime might have been longer BEFORE they added a bunch of extra servers last month to help handle the DDoS attacks more effectively.
Davesnothere

2 edits

Davesnothere to rblizz

Premium Member

to rblizz
said by rblizz:

Unfortunately it doesn't appear that a generator was an option in this building.

But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.

 
Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.
tr0910
join:2006-01-03
Kansas City, KS

tr0910

Member

said by Davesnothere:

Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.

They were just down again briefly. Line went dead, and website went dead.

I blame myself for trusting them, but it was based on recommendations from you all here. However I only trusted them with 1/2 our lines, so we were not completely shut down. Still, I think a healthy discussion is warranted, and they need to have a asterisk next to their name until they can prove they really have a disaster recovery plan, and have tested it out.

Does anybody have a fly swatter handy? We need to get rid of that that (buzz, buzz, buzz) that keeps flying around our ears....
royrogers
join:2012-10-17

royrogers to Davesnothere

Member

to Davesnothere
So the NYSE is still running on generators, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

My theory is this: Callcentric is located in a building full of ISP's. ConEd and NYSE and everyone else recognized that the first priority is getting power to that building, even before the NYSE.

Iscream, how far off am I?

Davesnothere
Change is NOT Necessarily Progress
Premium Member
join:2009-06-15
Canada

Davesnothere

Premium Member

said by royrogers:

So the NYSE is still running on battery backup, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

 
One Word :

RATS

They have them running on wheels to operate power generators.

Letterman told me so.

NYC bylaws say nothing against doing THAT !
29393955 (banned)
Always the green wire
join:2011-09-11
Mount Juliet, TN

1 recommendation

29393955 (banned) to verix

Member

to verix
said by verix:

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?

I just HAVE to respond here - I'm pretty sure the 'so-far' verified 75+ people who have died as a result of this storm aren't "happy" either.... I'm pretty sure the millions still without power aren't "happy" that this storm "occurred in the first place".... I'd bet that the thousands who have lost everything they owned aren't "happy" about that either....

If you expect 100% uptime on your VOIP service, and you didn't have a backup provider and plan already in place, then kick your OWN butt for being at LEAST as poorly prepared as CC, and quit whining about your budget phone service failing. 100% uptime IS available; just whip out that check book and start writing!
royrogers
join:2012-10-17

royrogers

Member

said by 29393955:

I just HAVE to respond here - I'm pretty sure the 'so-far' verified 75+ people who have died as a result of this storm aren't "happy" either.... I'm pretty sure the millions still without power aren't "happy" that this storm "occurred in the first place".... I'd bet that the thousands who have lost everything they owned aren't "happy" about that either....

If you expect 100% uptime on your VOIP service, and you didn't have a backup provider and plan already in place, then kick your OWN butt for being at LEAST as poorly prepared as CC, and quit whining about your budget phone service failing. 100% uptime IS available; just whip out that check book and start writing!

BINGO!

VexorgTR
join:2012-08-27
Sheffield Lake, OH

VexorgTR

Member

Hmm... well I was hoping that by Morning everything would be gold... but not quite. it could be much worse however.

Neither of my DID's worked by midnight last night. one did by 9:30 AM, and both did by 10:30 AM EST.... my 3CX box isn't registering with CC just yet, however the forward to another DID when not registered rule works... so calls arrive at my phone system again.

If you're not up totally yet, I suggest posting a ticket for help, but it's coming up a little at a time. I've been following this for several clients. It's MOSTLY up.

lowdough
@att.net

lowdough to Iscream

Anon

to Iscream
said by Iscream:

- speaking of MAJOR national and world-wide carriers, about those who were and still are down - here is not compete list:

Belgacom, Telia, KPN (including former iBasis), TATA (not just BIG - this is THE LARGEST CARRIER in the world, no one is LARGER - TATA has TWO POPs in NY State - both were down), Verizon - still down, no even sign of life, ATT, T-Mobile

====================end of quote ==========

Those are all good points, Iscream.

I'm gonna have to call just the teeniest little bit of BS in your claim that ATT and T-Mobile "were and still are down," though. I use both of those services daily and both continued to work flawlessly all week.

I think nobody expects much to work in the area that's under water. I think we all understand that things are really tough where the hurricane hit and I don't hear anybody diminishing any of that one bit.

The problem is that we didn't realize that a hit to one spot (even a REALLY BIG hit to one spot - on an island) would bring down all of your service everywhere in the world.

Sure, we may have lacked due diligence in determining that you have all your eggs (and our phone service) in one geographic basket. True enough.

But implying that your customers are to blame for our failure to see through some somewhat ambiguous, obscure, technical, jargon-filled sales puffery to determine your geographical vulnerability, while it may be true, is unlikely to engender any customer loyalty.

You were able, in this case, through no small measure of heroic efforts on the part of Callcentric employees, to get the ship righted in a few days. GREAT JOB ON THE RECOVERY.

But what if the office burned, instead of experiencing a (brief?) power outage? Or if the fire triggered the sprinklers in the "server room"? Or what if someone physically stole some (or all!) of the critical pieces of equipment? Those are just risks that, now that they're exposed, some of us are no longer willing to accept.

Sooo, great job on the heroic recovery. Please let us know in clear language when you have some meaningful geographical redundancy.
travelguy
join:1999-09-03
Bismarck, ND
Asus RT-AC68
Ubiquiti NSM5

travelguy to VexorgTR

Member

to VexorgTR
said by VexorgTR:

It's MOSTLY up.

Unfortunately my DID 631 251 XXXX is still returning a "cannot complete" recording to callers. Ticket submitted several hours ago.
bigpapae35
join:2002-10-25
New York, NY

bigpapae35

Member

said by travelguy:

said by VexorgTR:

It's MOSTLY up.

Unfortunately my DID 631 251 XXXX is still returning a "cannot complete" recording to callers. Ticket submitted several hours ago.

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.
travelguy
join:1999-09-03
Bismarck, ND
Asus RT-AC68
Ubiquiti NSM5

travelguy

Member

said by bigpapae35:

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.

Missed that. Also missed any tweet or web notice about that. All they tweeted was that service was restored and to submit a ticket if something isn't working. Did just find the iscream post - sounds like he is doing a fantastic job.
bigpapae35
join:2002-10-25
New York, NY

bigpapae35

Member

said by travelguy:

said by bigpapae35:

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.

Missed that. Also missed any tweet or web notice about that. All they tweeted was that service was restored and to submit a ticket if something isn't working. Did just find the iscream post - sounds like he is doing a fantastic job.

I see it listed in the bulletin when you sign in. But there is soo much content there now, its easy to miss.

this is what is shown in the bulletin:
"-Free NY DID Service Alert-

Engineers are aware of issues affecting Free NY DIDs, resulting from Hurricane Sandy flooding of portions of the NY/NJ PSTN infrastructure. "