If not for the unfortunate DDoS attack against CC, I believe that even if the entire outage that Hurricane Sandy caused for CC customers for a few days after the thunderstorm would have occurred all the same, it would have been almost entirely forgiven and forgotten by nearly all their customers considering their long history of reliability.
It was only due to the fact that the DDoS attack and the hurricane outage were back-to-back that there was that brouhaha that we saw.
"As detailed below, Callcentric does have fully redundant facilities by way of multiple power feeds"
A few or more very large business have placed their datacenters where the multiple feeds are from different substations feeding the building from different poles or trenches and generator backup. This is Phoenix metro. Mulitple feeds with a single failure point is not multiple. Yes stickler.
I'm part of the way through this and would like to thank Callcentric and Iscream for posting this information. It sounds like they handled the situation as well as possible given the situation. I'm sure it was stressful for everyone involved, and am glad to see that Callcentric has returned to their normal stability recently - after the DDoS attacks then the outage from Sandy.
Looking forward to read the rest of this... just thought I should go ahead and post this before any more flamewars start.
I'm not sure, if this is true but: does a competitor who is well known for its geo-redundancy, still depend on a common "brain" to steer their DIDs to each of their "redundant" destinations? Is this a single-point of failure?
All of your other comments about VoIP.ms aside, this topic has been beaten absolutely to death on this forum.
Yes the other business were idiots for either placing the generators or the fuel and associted pumps in a basement of subbasement. Emergency and bad things can mean flooding thus basement are bad for that. How many splice cases opened by a thirty year supposedly 5 9's tech put back together quick as oh well in it again next week to fix something. A one at a time fix plus too lazy to reseal it each time properly. Or break the case and if outside put a plastic slicker over it and send to someone else. Then a major weather thing. Not our fault.
Sure they want a sweet address and live in New York. Maybe a time for a move.
I tried to read and understand it all. One line near the end made me laugh: "The solution isnt a $40 a shrink-wrapped copy of Norton or McAfee AntiVirus." -- ...Who, What, When, Where, How... Why? Why Not?
The article is probably too long to be accepted on this forum.
This document, which is essentially a schematic of the Callcentric facility along with a detailed, blow-by-blow account of the DDoS attack and impact from Hurricane Sandy, is an amazing read.
The part that I found the most riveting was the imagery and events of the three Callcentric techs, evacuated from their Lower Manhattan facility, and remotely administering the facility from their homes on Staten Island and Long Island. As the storm was bearing down on them, they were stunned to learn that both redundant sources of commercial power had failed at Callcentric. Knowing that a disorderly "shutdown" of the facility would have been problematic, and knowing that their own power, Internet, and/or communications could fail at any moment, they had a difficult decision to make - and quickly. Just like in action-thriller movies, the power and Internet failed at Iscream's home just minutes after that shutdown was complete.
"Most our TDM network infrastructure is based on former Telicas Plexus 9000 switch (now Lucent-Alcatel Compact Switch ACS-9000) - this baby handles, in one chassis: 6 by 8 x DS3 cards, 2 by 28 x DS1 cards (primarily for SS7 links), non-blocking switching fabric (SF) for 120,000 calls and a number of optional SIP Server cards (each for 7000 simultaneous calls)."
Holy crap! And I thought I worked on some big telco switches in my previous career. That is very impressive.
An interesting look at the behind-the-scenes stuff of a voip provider.
The writer has a poor opinion of his main competitor's setup:
Thats right - unfortunately, Callcentric is not in the position to invest the above amounts now while we cannot allow ourselves to do it in [crappy] way like other companies have accomplished the same....
Im sorry - discussed earlier, on those boards, mentioned geo-redundancy of providers considered as our competitors was a joke by itself, because it either doesnt provide for the service level to be qualified as PSTN-like/phone replacement or the redundancy itself doesnt happen automatically,
It would be nice if the competitor's failover to a different server happened automatically, but taking a minute to change the ata settings to another server is better than having no phone service for several days.
Mr. Scream does not convince me that Callcentric is useful as more than a backup.
I just finished reading the whole article. Thanks for taking the time to write it.
It sounds like you did have generator backup available through one of your power feeds (even if you didn't own and operate the generator yourself).
And that's the main thing. With generators available at least there's a fighting chance to deal with power outages. In this case it didn't work out so well due to the flooding. But it sounds like it still allowed services to return earlier (on generator power) than they otherwise would have.
....The part that I found the most riveting was the imagery and events of the three Callcentric techs, evacuated from their Lower Manhattan facility, and remotely administering the facility from their homes on Staten Island and Long Island.
As the storm was bearing down on them, they were stunned to learn that both redundant sources of commercial power had failed at Callcentric.
Knowing that a disorderly "shutdown" of the facility would have been problematic, and knowing that their own power, Internet, and/or communications could fail at any moment, they had a difficult decision to make - and quickly.
Just like in action-thriller movies, the power and Internet failed at Iscream's home just minutes after that shutdown was complete....
[deep omimous movie trailer announcer voice] "Coming soon, to a theatre near you...." [/deep omimous movie trailer announcer voice]
But seriously, I did find Iscream's report to be both informative and somewhat entertaining.
It's decent of him to have written and posted it (even though it may lead to further questions from some of us).
....It would be nice if the competitor's failover to a different server happened automatically, but taking a minute to change the ata settings to another server is better than having no phone service for several days....
That's true, but it's still SOMETHING, just as CallCentric's generally known and accepted superior audio quality is also SOMETHING.
No [single] VoIP solution has every criteria and consideration covered completely, but for the price of BYOD VoIP, what SHOULD we expect for our small cost as subscribers ?
Each VoIP provider has its strong and weak points, and they are not all the same points.
We have recently learned a lot about which provider has which of these, and SO HAVE THE PROVIDERS.
I just finished reading the whole article. Thanks for taking the time to write it.
It sounds like you did have generator backup available through one of your power feeds (even if you didn't own and operate the generator yourself)....
Yes, I noticed (and jumped at) that revelation too.
It's unfortunate that a 'subcontractor/partner' let CallCentric down in that regard - though that company's OWN service (The NYSE) also fell simultaneously.
= = = = = = = = =
It might (as a corollary to the limited communications which we received during the storm-induced outage) have been useful for CallCentric to have stated at the time that they in fact DID have a source of local/neighbourhood generator power, and that such power had also failed.
Numerous critics at the time (myself included) might have said less (or nothing at all) about backup power solutions if we had been made aware that one done by generator had actually existed (though it had failed).
This (as I said above) is one more item of info which (if we had simply been informed) would have mattered to our peace of mind and trust about how things were being handled during the crisis, though I acknowledge that CC has stated the same, now that it has passed.
= = = = = = = = =
As many successful comedians profess (and I believe that it applies here too) :
"It's all about TIMING."
And, as I frequently say to folks in all circles of life :
Iscream's document is an impressive and highly detailed account of Callcentric's operations and provides the context in which to understand why full geographical redundancy is not a realistic/affordable goal in the near future. I also appreciate his statement that Callcentric's web and email operations will be strengthened so that customer communications can be made more robust in the face of any future trouble at the Broadway site.
What I would like to know - and I am trying to ask again in a respectful manner - is this: Can DID forwarding and/or voicemail be made redundant in a relatively inexpensive fashion so that these two important functions could survive some future trouble or disaster at the Broadway site? In other words: Iscream has carefully explained how the Broadway center allows Callcentric to interconnect with the PSTN, etc, and hence to provide resilient and high quality service. Are DID forwarding and/or voicemail dependent on the Broadway site in the same way as other functions? Could DID forwarding or voicemail be given the same kind of additional redundancy that is now being planned for the web and email functions?
I understand that DID forwarded calls at the time of a disaster might not meet the usual high standards for call quality we have come to expect from Callcentric, but I think many people would prefer to have lower quality calls available in the event of a disaster than no calls at all. This suggests a further question: If high-quality calling cannot realistically be made "geo-reduntant" than is it possible that a low-quality, cheaper-to-develop-and-maintain alternative could be available in place in the event of a disaster?
Of course, I'll leave it to Callcentric to discuss their specific business strategies, so my comments are of a general nature only.
A cheaper-to-develop-and-maintain alternative is certainly technically possible, but why settle for low quality? I know I can't tell the difference in audio between a DID from a VoIP provider with PSTN interconnect, and a DID from a VoIP provider that's delivered to the service provider by SIP. (That is - assuming all other variables like codec are equal.) I have one of each and I use both regularly.
Certainly - there are other factors to consider besides audio quality, but my point is that there is nothing wrong with using SIP. Everyone here does that every day
A short answer to your question would be - no, unfortunately this is not possible.
A longer one - has variants and extensions and can be answered as "yes" and "no" depending on some conditions and circumstances.
I'll try to explain both, but a bit later, perhaps tonight - when time permits. I want to do it in this way because there are some basic technical things which must be explained first, prior the answer will be made simpler to understand.