dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
4
share rss forum feed

verix

join:2004-12-30
Oakland, CA

2 recommendations

reply to rblizz

Re: Back to worst-case-scenario to start with!

said by rblizz:

As if New Yorkers (and others along the Eastern Coast) don't need their power. Is this the first time you've ever heard a natural disaster? Did everything work without a glitch in Oakland after the 1989 earthquake?

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

rblizz

join:2001-12-16
North Richland Hills, TX
said by verix:

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

I don't know about Oakland, but in most of the world "reality happens." That's why they call the call earthquakes and storms like Sandy "natural disasters" ... "disasters" being the key word here. I don't know why so many here seem to think we live in a La La World where every inconvenience can always be averted -- and, if it can't, someone's head must roll.

Personally I'm happy it only took 3 days for CallCentric to get back up. Even the New York Stock Exchange, the world center of capitalism with nearly unlimited resources, was down for nearly two days. And this is the first time the NYSE was down for two days or more because of weather for over a century. This should tell you something about how powerful this storm was. I.E., it could have been a hell of a lot worse for CallCentric customers.

verix

join:2004-12-30
Oakland, CA

2 recommendations

said by rblizz:

I don't know about Oakland, but in most of the world "reality happens." That's why they call the call earthquakes and storms like Sandy "natural disasters" ... "disasters" being the key word here. I don't know why so many here seem to think we live in a La La World where every inconvenience can always be averted -- and, if it can't, someone's head must roll.

You didn't answer my questions. It's like you're trying to conflate my sympathies for Sandy victims to some sort of acceptance of CC's lack of service, when every other provider did not experience significant downtime. Even those that have a presence in NYC. And thank you for your lecture on natural disasters. That's why I'm waiting to hear how CC responds before I make a decision. Did I say I was leaving?

Personally I'm happy it only took 3 days for CallCentric to get back up.

Glad you're happy. A lot of people aren't.

rblizz

join:2001-12-16
North Richland Hills, TX
said by verix:

Glad you're happy. A lot of people aren't.

A lot of people like to bitch and confuse that with with "doing something." No one is happy about the downtime -- but I realize it could have been much, much worse -- so I'm happy that it wasn't.

It's also plain to me that CallCentric is proud of the time and effort they put into their infrastructure. There's a reason you and I chose CallCentric in the first place. It was their reputation for quality. So now they've been attacked from the outside with an unrelentlessness DDoS barrage (which they beat back) and a once in a century storm (that also took down the NYSE) and you're demanding they "explain themselves." They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.

verix

join:2004-12-30
Oakland, CA
said by rblizz:

They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?
Expand your moderator at work

Iscream
Premium
join:2009-02-17
New York, NY
kudos:6
Reviews:
·Verizon FiOS

1 recommendation

reply to verix

Re: Back to worst-case-scenario to start with!

Before I'm going to sleep a couple hours:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

- speaking of MAJOR national and world-wide carriers, about those who were and still are down - here is not compete list:

Belgacom, Telia, KPN (including former iBasis), TATA (not just BIG - this is THE LARGEST CARRIER in the world, no one is LARGER - TATA has TWO POPs in NY State - both were down), Verizon - still down, no even sign of life, ATT, T-Mobile, Sprint - either fully or still partially down along the way from Staten Island, through Brooklyn and Lower Manhattan including our offices. Speaking of "smaller" ones - Broadvox, iNetworks (that's Bandwidth and Dash-911 and Republic Wireless). Want even smaller ones - how about VoxBone ? Huh?

For those who want more detailed info - details available on the Net and specialized subscription lists.

Regarding NY Free DIDs - I wrote in my previous messages that those are currently NOT available, but NOT due to Callcentric's issues - because all TDM interconnects in LATA 132 (Big Apple) are still dead including SS7 network - at least in Lower Manhattan. We'll bring those DIDs up (as a matter of fact - they'll come up just by themselves) a soon as the PSTN network in New York restores.

rblizz

join:2001-12-16
North Richland Hills, TX
said by Iscream:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

Sorry, I'm the one who's been messing that one up. So, just slightly over two days for full restoration of service.

rblizz

join:2001-12-16
North Richland Hills, TX
reply to Anon
said by rblizz:

Yep. Time for me to leave this thread for now. I'm allergic to the unreasonable.

Just to give some folks an idea of the scale of what's going on in New York (and elsewhere) ...

quote:
As blackouts linger, Northeasterners try to adjust

Sandy blacked out some of the nation's most densely populated cities and suburbs, instantly taking away modern conveniences from Virginia to Massachusetts and as far west as the Great Lakes.

For power companies, the scale of the destruction was unmatched -- more widespread than any blizzard or ice storm and worse than the aftermath of Hurricane Katrina.

"It's unprecedented: fallen trees, debris, the roads, water, snow. It's a little bit of everything," said Brian Wolff, senior vice president of the Edison Electric Institute, a group that lobbies for utilities.

Initially, about 60 million people were without power in 8.2 million homes and businesses. By Wednesday night, that number had fallen to roughly 44 million people in 6 million households and businesses.

Even as power slowly returned to some pockets, a new headache emerged: Backup batteries and generators running cellphone towers were running out of juice. One out of every five towers was down, according to the Federal Communications Commission.

»news.yahoo.com/blackouts-linger- ··· 751.html


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

4 edits

1 recommendation

reply to rblizz
said by rblizz:

said by verix:

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

....it could have been a hell of a lot worse for CallCentric customers.

 
OR a lot BETTER, if their skinflint absentee owner(s) had kicked in the bucks to set up either a 2nd server farm in another city, or rent from at least one landlord who provides a proper generator - or do BOTH of these things.

Iscream was treading a fne line a few posts ago, when he spoke of budgetary constraints. - He also has his job to consider. - HE knows what it would take to make this right for the next time.

rblizz

join:2001-12-16
North Richland Hills, TX
said by Davesnothere:

OR, a lot BETTER, if their skinflint owner(s) kicked in the bucks to set up either a 2nd server farm in another city, or rent from at least one landlord who provides a proper generator - or do BOTH of these things.

If a company with a second server farm is important to you, then by all means, look for one -- I thought you already had moved on. Personally I don't care. I know how extremely rare an extended electrical outage is in lower Manhattan and especially one involving the New York Stock Exchange's grid. I don't expect another "storm of the century" to happen any time soon -- so I'm quite willing to "take my chances."


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

2 edits

1 recommendation

said by rblizz:

If a company with a second server farm is important to you, then by all means, look for one -- I thought you already had moved on. Personally I don't care. I know how extremely rare an extended electrical outage is in lower Manhattan and especially one involving the New York Stock Exchange's grid. I don't expect another "storm of the century" to happen any time soon -- so I'm quite willing to "take my chances."

 
IMNSHO, this is NOT about 'Playing the Odds'.

It's about customer service - at ALL levels of the operation, including planning and investment.

Below is part of what I said to someone yesturday in an IM :
quote:
I indeed AM angry that a company which is so professional in SO many other ways - can leave themselves WIDE open for such an epic fail in this regard.

One thing that this particular debacle illustrates and drives home - is the need for an even greater sophistication of process in doing our 'due diligence' when choosing a VoIP provider than most of us ever thought, as one poster [you, I believe] suggested to me.

I continue to have a lot of respect for CallCentric as a company - just not about THIS event's handling.

= = = = = = =

And BTW, in response to your reply to another poster :

What it would take to make ME shut my pie hole would be an open public announcement from Callcentric management that they COULD have done better if they had handled this event like VOIP.MS did in the same city, and what they will be doing in the next few months to CATCH UP with those CANUCKS in that regard.

rblizz

join:2001-12-16
North Richland Hills, TX

1 edit
said by Davesnothere:

This is not about 'Playing the Odds'.

It's about customer service - at ALL levels of the operation, including planning and investment.

I'm happy with the quality of service CallCentric has provided to me and I'm happy with their single server farm and the server redundancy they have there. It's not "playing the odds" -- the odds are in CallCentric's favor in my opinion. Until the DDoS attack and this storm, I've been blissfully using CallCentric and noting other provider's outages -- especially issues that seem to plague one of the companies that has multiple servers. Redundancy is nice, but it appears it can also introduce a whole new set of complications that can impact reliability.

You should also note "take my chances" was in quotes. In other words, I really don't think of it as taking a chance.

daneli

join:2012-10-23
Ann Arbor, MI

2 recommendations

reply to Iscream
I want to try to be constructive in my comments. Full redundancy with instant failover can be a very expensive and technically difficult challenge to meet. If a business can tolerate a longer time window (~ 8 hours) and can concentrate on only a subset of functions, then it can be much cheaper and easier to get some services back up and running in the face of a disruption. Certainly it should be possible for callcentric to provide a backup solution for its website and email that is more robust than the current solution. Perhaps a subset of functions - like voicemail or did forwarding - could also be made available from a remote location if the time window is extended to 12 hours or so?

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

After the dust settles, I think it is reasonable for us to ask callcentric to be transparent about what steps if any it plans to take to mitigate these kinds of risks in the future. The risks I have in mind are not those associated with a "storm of the century" but the usual kinds of things - flood, fire, theft, etc - that businesses try to anticipate.

rblizz

join:2001-12-16
North Richland Hills, TX
said by daneli:

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

Unfortunately it doesn't appear that a generator was an option in this building. But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7
reply to daneli
said by daneli:

I want to try to be constructive in my comments....

 
Absolutely, and I agree with pretty well all of them.

BTW, their UPS runtime might have been longer BEFORE they added a bunch of extra servers last month to help handle the DDoS attacks more effectively.


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

2 edits
reply to rblizz
said by rblizz:

Unfortunately it doesn't appear that a generator was an option in this building.

But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.

 
Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.

tr0910

join:2006-01-03
MI/USA
said by Davesnothere:

Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.

They were just down again briefly. Line went dead, and website went dead.

I blame myself for trusting them, but it was based on recommendations from you all here. However I only trusted them with 1/2 our lines, so we were not completely shut down. Still, I think a healthy discussion is warranted, and they need to have a asterisk next to their name until they can prove they really have a disaster recovery plan, and have tested it out.

Does anybody have a fly swatter handy? We need to get rid of that that (buzz, buzz, buzz) that keeps flying around our ears....

royrogers

join:2012-10-17
reply to Davesnothere
So the NYSE is still running on generators, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

My theory is this: Callcentric is located in a building full of ISP's. ConEd and NYSE and everyone else recognized that the first priority is getting power to that building, even before the NYSE.

Iscream, how far off am I?


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7
said by royrogers:

So the NYSE is still running on battery backup, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

 
One Word :

RATS

They have them running on wheels to operate power generators.

Letterman told me so.

NYC bylaws say nothing against doing THAT !


airwavz
Always the green wire

join:2011-09-11
Mount Juliet, TN
kudos:1

1 recommendation

reply to verix
said by verix:

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?

I just HAVE to respond here - I'm pretty sure the 'so-far' verified 75+ people who have died as a result of this storm aren't "happy" either.... I'm pretty sure the millions still without power aren't "happy" that this storm "occurred in the first place".... I'd bet that the thousands who have lost everything they owned aren't "happy" about that either....

If you expect 100% uptime on your VOIP service, and you didn't have a backup provider and plan already in place, then kick your OWN butt for being at LEAST as poorly prepared as CC, and quit whining about your budget phone service failing. 100% uptime IS available; just whip out that check book and start writing!

royrogers

join:2012-10-17
said by airwavz:

I just HAVE to respond here - I'm pretty sure the 'so-far' verified 75+ people who have died as a result of this storm aren't "happy" either.... I'm pretty sure the millions still without power aren't "happy" that this storm "occurred in the first place".... I'd bet that the thousands who have lost everything they owned aren't "happy" about that either....

If you expect 100% uptime on your VOIP service, and you didn't have a backup provider and plan already in place, then kick your OWN butt for being at LEAST as poorly prepared as CC, and quit whining about your budget phone service failing. 100% uptime IS available; just whip out that check book and start writing!

BINGO!


VexorgTR

join:2012-08-27
Sheffield Lake, OH
kudos:1
Reviews:
·voip.ms
Hmm... well I was hoping that by Morning everything would be gold... but not quite. it could be much worse however.

Neither of my DID's worked by midnight last night. one did by 9:30 AM, and both did by 10:30 AM EST.... my 3CX box isn't registering with CC just yet, however the forward to another DID when not registered rule works... so calls arrive at my phone system again.

If you're not up totally yet, I suggest posting a ticket for help, but it's coming up a little at a time. I've been following this for several clients. It's MOSTLY up.


lowdough

@att.net
reply to Iscream
said by Iscream:

- speaking of MAJOR national and world-wide carriers, about those who were and still are down - here is not compete list:

Belgacom, Telia, KPN (including former iBasis), TATA (not just BIG - this is THE LARGEST CARRIER in the world, no one is LARGER - TATA has TWO POPs in NY State - both were down), Verizon - still down, no even sign of life, ATT, T-Mobile

====================end of quote ==========

Those are all good points, Iscream.

I'm gonna have to call just the teeniest little bit of BS in your claim that ATT and T-Mobile "were and still are down," though. I use both of those services daily and both continued to work flawlessly all week.

I think nobody expects much to work in the area that's under water. I think we all understand that things are really tough where the hurricane hit and I don't hear anybody diminishing any of that one bit.

The problem is that we didn't realize that a hit to one spot (even a REALLY BIG hit to one spot - on an island) would bring down all of your service everywhere in the world.

Sure, we may have lacked due diligence in determining that you have all your eggs (and our phone service) in one geographic basket. True enough.

But implying that your customers are to blame for our failure to see through some somewhat ambiguous, obscure, technical, jargon-filled sales puffery to determine your geographical vulnerability, while it may be true, is unlikely to engender any customer loyalty.

You were able, in this case, through no small measure of heroic efforts on the part of Callcentric employees, to get the ship righted in a few days. GREAT JOB ON THE RECOVERY.

But what if the office burned, instead of experiencing a (brief?) power outage? Or if the fire triggered the sprinklers in the "server room"? Or what if someone physically stole some (or all!) of the critical pieces of equipment? Those are just risks that, now that they're exposed, some of us are no longer willing to accept.

Sooo, great job on the heroic recovery. Please let us know in clear language when you have some meaningful geographical redundancy.

travelguy

join:1999-09-03
Santa Fe, NM
reply to VexorgTR
said by VexorgTR:

It's MOSTLY up.

Unfortunately my DID 631 251 XXXX is still returning a "cannot complete" recording to callers. Ticket submitted several hours ago.

bigpapae35

join:2002-10-25
united state
kudos:1
said by travelguy:

said by VexorgTR:

It's MOSTLY up.

Unfortunately my DID 631 251 XXXX is still returning a "cannot complete" recording to callers. Ticket submitted several hours ago.

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.

travelguy

join:1999-09-03
Santa Fe, NM
said by bigpapae35:

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.

Missed that. Also missed any tweet or web notice about that. All they tweeted was that service was restored and to submit a ticket if something isn't working. Did just find the iscream post - sounds like he is doing a fantastic job.

bigpapae35

join:2002-10-25
united state
kudos:1
said by travelguy:

said by bigpapae35:

is that a free DID, because if it is, iscream already stated the free DID's are currently not working.

Missed that. Also missed any tweet or web notice about that. All they tweeted was that service was restored and to submit a ticket if something isn't working. Did just find the iscream post - sounds like he is doing a fantastic job.

I see it listed in the bulletin when you sign in. But there is soo much content there now, its easy to miss.

this is what is shown in the bulletin:
"-Free NY DID Service Alert-

Engineers are aware of issues affecting Free NY DIDs, resulting from Hurricane Sandy flooding of portions of the NY/NJ PSTN infrastructure. "