PX EliezerPremium
join:2008-08-09
Hutt River
kudos:13 Reviews:
 ·callwithus
·voip.ms
·Optimum Voice
·Vitelity VOIP
·Gizmo5
| CallCentric posts commentary on many crucial issues. CallCentric has long been a "quiet company" with a long track record for quality and reliability.
Recent events placed it front and center as a topic for active discussion.
This occurred because of a brutal and prolonged DDoS attack, and the events associated with the storm Sandy which devastated this area and led to an emergency shutdown of its network.
CallCentric's Iscream has written a detailed commentary on many significant issues. To my knowledge this is the most detailed information ever posted by a VoIP provider on its internal operations.
It can be found here, on Arne Bolen's "My VoIP News".
»myvoipnews.com/post/36178427453/···multiple
[This has been a significant topic of interest to many on this forum. Thank you to DSLR and VoIP Tech Chat forum, for graciously allowing this link to be posted]. |
|
 Arne BolenHappy Anveo customerPremium
join:2009-06-21
Planet Earth
kudos:4
·Anveo
·callwithus
·Callcentric
·voip.ms
| said by PX Eliezer:[This has been a significant topic of interest to many on this forum. Thank you to DSLR and VoIP Tech Chat forum, for graciously allowing this link to be posted].
The article is probably too long to be accepted on this forum. 
--
My VoIP News |
|
|
|
Reviews:
·VoicePulse
| reply to PX Eliezer
If not for the unfortunate DDoS attack against CC, I believe that even if the entire outage that Hurricane Sandy caused for CC customers for a few days after the thunderstorm would have occurred all the same, it would have been almost entirely forgiven and forgotten by nearly all their customers considering their long history of reliability.
It was only due to the fact that the DDoS attack and the hurricane outage were back-to-back that there was that brouhaha that we saw. |
|
nonymousPremium
join:2003-09-08
Glendale, AZ Reviews:
·Callcentric
| reply to PX Eliezer
"As detailed below, Callcentric does have fully redundant facilities by way of multiple power feeds"
A few or more very large business have placed their datacenters where the multiple feeds are from different substations feeding the building from different poles or trenches and generator backup. This is Phoenix metro.
Mulitple feeds with a single failure point is not multiple.
Yes stickler. |
|
PX EliezerPremium
join:2008-08-09
Hutt River
kudos:13 | reply to PX Eliezer
I'm a very fast reader, and it took me a good half hour to read and digest what Iscream wrote.
I hope that most people will at least read what he wrote---ALL of what he took the time to write---before taking out bits and pieces to respond to. |
|
gweidenh
join:2002-05-18
Houston, TX
kudos:1 | reply to PX Eliezer
Excellent write up.
While reading, it reminded me of this video....
»www.youtube.com/watch?v=oCZMoY3q2uM |
|
 jimkPremium
join:2006-04-15
Raleigh, NC
 ·RoadRunner Cable
·voip.ms
| reply to PX Eliezer
I'm part of the way through this and would like to thank Callcentric and Iscream  for posting this information. It sounds like they handled the situation as well as possible given the situation. I'm sure it was stressful for everyone involved, and am glad to see that Callcentric has returned to their normal stability recently - after the DDoS attacks then the outage from Sandy.
Looking forward to read the rest of this... just thought I should go ahead and post this before any more flamewars start. |
|
Mangowww.toao.net
join:2008-12-25
Alberta
kudos:11 Reviews:
·Anveo
·Shaw
·AcroVoice
·Callcentric
·callwithus
·voip.ms
·FreePhoneLine
 ·TELUS
| I'm not sure, if this is true but: does a competitor who is well known for its geo-redundancy, still depend on a common "brain" to steer their DIDs to each of their "redundant" destinations? Is this a single-point of failure?
All of your other comments about VoIP.ms aside, this topic has been beaten absolutely to death on this forum.
No, VoIP.ms does not depend on a central site. |
|
nonymousPremium
join:2003-09-08
Glendale, AZ Reviews:
·Callcentric
| reply to PX Eliezer
Yes the other business were idiots for either placing the generators or the fuel and associted pumps in a basement of subbasement.
Emergency and bad things can mean flooding thus basement are bad for that.
How many splice cases opened by a thirty year supposedly 5 9's tech put back together quick as oh well in it again next week to fix something. A one at a time fix plus too lazy to reseal it each time properly. Or break the case and if outside put a plastic slicker over it and send to someone else. Then a major weather thing. Not our fault.
Sure they want a sweet address and live in New York. Maybe a time for a move. |
|
DavesnothereNo-BHELL-ity DOES have its Advantages
join:2009-06-15
START&Cogeco
kudos:6 | reply to PX Eliezer
Hmmmm....
Might need 'reading' glasses for this one. 
Will examine it and post back ASAP. |
|
DavesnothereNo-BHELL-ity DOES have its Advantages
join:2009-06-15
START&Cogeco
kudos:6 | reply to Arne Bolen
said by Arne Bolen:The article is probably too long to be accepted on this forum.
Nor on the front page of the CallCentric web portal as a bulletin. 
Just skimmed the article.
I'd better go and make me-self a pot of COFFEE, so that I can stay up to read the about the TEA leaves.  |
|
Arne BolenHappy Anveo customerPremium
join:2009-06-21
Planet Earth
kudos:4 Reviews:
·Anveo
·callwithus
·Callcentric
·voip.ms
| said by Davesnothere:said by Arne Bolen:The article is probably too long to be accepted on this forum.
Nor on the front page of the CallCentric web portal as a bulletin. Probably too advanced for the average customer.
--
My VoIP News |
|
DavesnothereNo-BHELL-ity DOES have its Advantages
join:2009-06-15
START&Cogeco
kudos:6
1 edit | said by Arne Bolen:Probably too advanced for the average customer.
Yeah, they might say something like "What happened ? - WHERE's the web portal ?!"
AND, we would't want the DSLR board software to choke either. 
I wonder how many characters long that that omnibus post is ? |
|
OmagicQPosting in a thread near you
join:2003-10-23
Bakersfield, CA
kudos:1 | reply to PX Eliezer
I tried to read and understand it all. One line near the end made me laugh: "The solution isn’t a $40 a shrink-wrapped copy of Norton or McAfee AntiVirus."
--
...Who, What, When, Where, How... Why? Why Not? |
|
Arne BolenHappy Anveo customerPremium
join:2009-06-21
Planet Earth
kudos:4 Reviews:
·Anveo
·callwithus
·Callcentric
·voip.ms
| reply to Davesnothere
said by Davesnothere:I wonder how many characters long that that omnibus post is ?
--
My VoIP News |
|
nonymousPremium
join:2003-09-08
Glendale, AZ Reviews:
·Callcentric
| reply to PX Eliezer
Plus I read they are fully insured. great. By the time they collect and maybe if never rebuild i will have switched.
Tecos cable etc are so big even if lose a switch rebuild themselves first even if have insurance no wait. Most may be self insured against most stuff so doesnt matter. |
|
Reviews:
·Callcentric
| reply to Arne Bolen
said by Arne Bolen:The article is probably too long to be accepted on this forum.
This document, which is essentially a schematic of the Callcentric facility along with a detailed, blow-by-blow account of the DDoS attack and impact from Hurricane Sandy, is an amazing read.
The part that I found the most riveting was the imagery and events of the three Callcentric techs, evacuated from their Lower Manhattan facility, and remotely administering the facility from their homes on Staten Island and Long Island. As the storm was bearing down on them, they were stunned to learn that both redundant sources of commercial power had failed at Callcentric. Knowing that a disorderly "shutdown" of the facility would have been problematic, and knowing that their own power, Internet, and/or communications could fail at any moment, they had a difficult decision to make - and quickly. Just like in action-thriller movies, the power and Internet failed at Iscream's home just minutes after that shutdown was complete.
Thanks Iscream, for this detailed report. |
|
Arne BolenHappy Anveo customerPremium
join:2009-06-21
Planet Earth
kudos:4 Reviews:
·Anveo
·callwithus
·Callcentric
·voip.ms
| said by engineerdan:This document, which is essentially a schematic of the Callcentric facility along with a detailed, blow-by-blow account of the DDoS attack and impact from Hurricane Sandy, is an amazing read. This document is likely to be studied by most VoIP providers in the USA and Canada. I believe the content is of great interest for people in the industry.
said by engineerdan:Just like in action-thriller movies, the power and Internet failed at Iscream's home just minutes after that shutdown was complete. I can imagine what pressure Iscream was under knowing that he had only minutes to avoid a disaster. And what relief it must have been when he succeeded.
--
My VoIP News |
|
| reply to PX Eliezer
Bravo, Iscream. Very good report. It answered a lot of my questions.
It also reminded me of reading that blog from an IT guy in a NOC in New Orleans during Katrina. It was almost surreal.
I'd gladly deliver beer to the NOC team for a job well done, but can I still get my boat through Battery Park, or do I need a submarine to get through the Tunnel?
Maybe someone from up north will send on some Pabst Blue Ribbon, or Molson?
Carl |
|
| reply to PX Eliezer
"Most our TDM network infrastructure is based on former Telica’s Plexus 9000 switch (now Lucent-Alcatel Compact Switch ACS-9000) - this “baby” handles, in one chassis: 6 by 8 x DS3 cards, 2 by 28 x DS1 cards (primarily for SS7 links), non-blocking switching fabric (SF) for 120,000 calls and a number of optional SIP Server cards (each for 7000 simultaneous calls)."
Holy crap! And I thought I worked on some big telco switches in my previous career. That is very impressive. |
|
