dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
13
share rss forum feed

verix

join:2004-12-30
Oakland, CA

2 recommendations

reply to Iscream

Re: Back to worst-case-scenario to start with!

said by Iscream:

Callcentric is 100% redundant locally, in all possible means - lines, equipment, databases, etc., but is NOT multi-site'd - it doesn't have geographical redundancy.

I look forward to the upcoming blog post/e-mail notice/press release/whatever before I decide on further action. Sorry this storm was unprecedented and that there may be budgetary reasons for your infrastructure. But, we need our lines.


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

said by verix:

said by Iscream:

Callcentric is 100% redundant locally, in all possible means - lines, equipment, databases, etc., but is NOT multi-site'd - it doesn't have geographical redundancy.

I look forward to the upcoming blog post/e-mail notice/press release/whatever before I decide on further action.

Sorry this storm was unprecedented and that there may be budgetary reasons for your infrastructure. But, we need our lines.

 
Agreed on that.

And WHERE is Tower of Power when we need them ?!

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to verix

said by verix:

I look forward to the upcoming blog post/e-mail notice/press release/whatever before I decide on further action. Sorry this storm was unprecedented and that there may be budgetary reasons for your infrastructure. But, we need our lines.

As if New Yorkers (and others along the Eastern Coast) don't need their power. Is this the first time you've ever heard a natural disaster? Did everything work without a glitch in Oakland after the 1989 earthquake?

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to verix

said by verix:

said by Iscream:

Callcentric is 100% redundant locally, in all possible means - lines, equipment, databases, etc., but is NOT multi-site'd - it doesn't have geographical redundancy.

I look forward to the upcoming blog post/e-mail notice/press release/whatever before I decide on further action. Sorry this storm was unprecedented and that there may be budgetary reasons for your infrastructure. But, we need our lines.

It was no secret that CallCentric had one server location. It's been discussed here several times.

verix

join:2004-12-30
Oakland, CA

2 recommendations

reply to rblizz

said by rblizz:

As if New Yorkers (and others along the Eastern Coast) don't need their power. Is this the first time you've ever heard a natural disaster? Did everything work without a glitch in Oakland after the 1989 earthquake?

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

verix

join:2004-12-30
Oakland, CA

2 recommendations

reply to rblizz

said by rblizz:

It was no secret that CallCentric had one server location. It's been discussed here several times.

I check this forum quite often and had no idea that CC had one server location until this event. Don't make assumptions of what people know.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to verix

said by verix:

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

I don't know about Oakland, but in most of the world "reality happens." That's why they call the call earthquakes and storms like Sandy "natural disasters" ... "disasters" being the key word here. I don't know why so many here seem to think we live in a La La World where every inconvenience can always be averted -- and, if it can't, someone's head must roll.

Personally I'm happy it only took 3 days for CallCentric to get back up. Even the New York Stock Exchange, the world center of capitalism with nearly unlimited resources, was down for nearly two days. And this is the first time the NYSE was down for two days or more because of weather for over a century. This should tell you something about how powerful this storm was. I.E., it could have been a hell of a lot worse for CallCentric customers.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to verix

said by verix:

[I check this forum quite often and had no idea that CC had one server location until this event. Don't make assumptions of what people know.

I like to know who I'm dealing with. One of CallCentric's big selling points was that it was on the same electrical grid as the NYSE.

From their promotional material ...

quote:
"CallCentric retail is backed by the resources of CallCentric Wholesale----

"Callcentric Wholesale, Inc. (parent of Callcentric) owns and operates its own TDM/SS7 network interconnected with major (as well as small ones) carriers, networks and voice exchanges....

"Callcentric Wholesale is also an ARIN registered ISP, operating its own IP network blocks and exchanging BGP-4 routing information with most ISP backbones (also nationally and internationally) via its own managed fiber optical network thus guarantying the best IP packet delivery to virtually anywhere in the world....

"We're NOT co-located with any providers - we operate our own data-center (with all underlying infrastructure, under 24x7 video surveillance) with redundant air-conditioning and AC power-plants; our dual-redundant AC sources reside on the same AC power grid with NY stock exchange (which is just across Broadway from us). In fact, we provide co-location, IP and TDM connection as well as origination and termination services to many international operators....."
Can you name the last time the NY Stock Exchange's electrical grid went down? I can't.

verix

join:2004-12-30
Oakland, CA

2 recommendations

reply to rblizz

said by rblizz:

I don't know about Oakland, but in most of the world "reality happens." That's why they call the call earthquakes and storms like Sandy "natural disasters" ... "disasters" being the key word here. I don't know why so many here seem to think we live in a La La World where every inconvenience can always be averted -- and, if it can't, someone's head must roll.

You didn't answer my questions. It's like you're trying to conflate my sympathies for Sandy victims to some sort of acceptance of CC's lack of service, when every other provider did not experience significant downtime. Even those that have a presence in NYC. And thank you for your lecture on natural disasters. That's why I'm waiting to hear how CC responds before I make a decision. Did I say I was leaving?

Personally I'm happy it only took 3 days for CallCentric to get back up.

Glad you're happy. A lot of people aren't.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric

said by verix:

Glad you're happy. A lot of people aren't.

A lot of people like to bitch and confuse that with with "doing something." No one is happy about the downtime -- but I realize it could have been much, much worse -- so I'm happy that it wasn't.

It's also plain to me that CallCentric is proud of the time and effort they put into their infrastructure. There's a reason you and I chose CallCentric in the first place. It was their reputation for quality. So now they've been attacked from the outside with an unrelentlessness DDoS barrage (which they beat back) and a once in a century storm (that also took down the NYSE) and you're demanding they "explain themselves." They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.

verix

join:2004-12-30
Oakland, CA

said by rblizz:

They've already explained themselves. What exactly would make you happy at this point? I'm guessing nothing.

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?


Besant

@uwaterloo.ca
reply to verix

said by verix:

said by rblizz:

It was no secret that CallCentric had one server location. It's been discussed here several times.

I check this forum quite often and had no idea that CC had one server location until this event.

I did not know that either. All I recall is seeing 'redundant' in CC's description of its servers and assumed the redundant servers were geographically remote from the main ones.

For me it is not important: I use CC as a backup and have no DID with them; but if I had a business I would give them a wide berth until the day that real redundancy is finally in their budget.
Expand your moderator at work

Iscream
Premium
join:2009-02-17
New York, NY
kudos:6
Reviews:
·Verizon FiOS

1 recommendation

reply to verix

Re: Back to worst-case-scenario to start with!

Before I'm going to sleep a couple hours:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

- speaking of MAJOR national and world-wide carriers, about those who were and still are down - here is not compete list:

Belgacom, Telia, KPN (including former iBasis), TATA (not just BIG - this is THE LARGEST CARRIER in the world, no one is LARGER - TATA has TWO POPs in NY State - both were down), Verizon - still down, no even sign of life, ATT, T-Mobile, Sprint - either fully or still partially down along the way from Staten Island, through Brooklyn and Lower Manhattan including our offices. Speaking of "smaller" ones - Broadvox, iNetworks (that's Bandwidth and Dash-911 and Republic Wireless). Want even smaller ones - how about VoxBone ? Huh?

For those who want more detailed info - details available on the Net and specialized subscription lists.

Regarding NY Free DIDs - I wrote in my previous messages that those are currently NOT available, but NOT due to Callcentric's issues - because all TDM interconnects in LATA 132 (Big Apple) are still dead including SS7 network - at least in Lower Manhattan. We'll bring those DIDs up (as a matter of fact - they'll come up just by themselves) a soon as the PSTN network in New York restores.


rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric

said by Iscream:

- took only 2 days (53 hours) to restore the service. NASDAQ and NYSE's floors are still down for general public. Perhaps Thursday or Friday.

Sorry, I'm the one who's been messing that one up. So, just slightly over two days for full restoration of service.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to Anon

said by rblizz:

Yep. Time for me to leave this thread for now. I'm allergic to the unreasonable.

Just to give some folks an idea of the scale of what's going on in New York (and elsewhere) ...

quote:
As blackouts linger, Northeasterners try to adjust

Sandy blacked out some of the nation's most densely populated cities and suburbs, instantly taking away modern conveniences from Virginia to Massachusetts and as far west as the Great Lakes.

For power companies, the scale of the destruction was unmatched -- more widespread than any blizzard or ice storm and worse than the aftermath of Hurricane Katrina.

"It's unprecedented: fallen trees, debris, the roads, water, snow. It's a little bit of everything," said Brian Wolff, senior vice president of the Edison Electric Institute, a group that lobbies for utilities.

Initially, about 60 million people were without power in 8.2 million homes and businesses. By Wednesday night, that number had fallen to roughly 44 million people in 6 million households and businesses.

Even as power slowly returned to some pockets, a new headache emerged: Backup batteries and generators running cellphone towers were running out of juice. One out of every five towers was down, according to the Federal Communications Commission.

»news.yahoo.com/blackouts-linger-···751.html


fukitol
Solon for President
Premium
join:2001-06-11
PonziWorld

1 recommendation

reply to Besant

Yeah, I don't buy the "it happened to a whole bunch of others, too" BULLSHIT. I don't give a flying fuck if a whole bunch of other companies didn't build a properly redundant network, either. If I ever used an excuse like that for bad behavior when I was growing up, my parents would ask me, "Well, if your friends jumped off a bridge, would you do it too?" I certainly hope that no one needed the E911 service for which they paid extra to CC over the past month. Oh, and where are the refunds? Hmm?
--
Vote Lucifer in 2012. It's real change you can believe in!
The truth is that the State is a conspiracy designed not only to exploit, but above all to corrupt its citizens. - Tolstoy



Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

4 edits

1 recommendation

reply to rblizz

said by rblizz:

said by verix:

Condolences to everyone affected by Sandy, but why should people nowhere near it have to suffer? What about the businesses in CA that have lines with CC?

....it could have been a hell of a lot worse for CallCentric customers.

 
OR a lot BETTER, if their skinflint absentee owner(s) had kicked in the bucks to set up either a 2nd server farm in another city, or rent from at least one landlord who provides a proper generator - or do BOTH of these things.

Iscream was treading a fne line a few posts ago, when he spoke of budgetary constraints. - He also has his job to consider. - HE knows what it would take to make this right for the next time.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to fukitol

said by fukitol:

Yeah, I don't buy the "it happened to a whole bunch of others, too" BULLSHIT. I don't give a flying fuck if a whole bunch of other companies didn't build a properly redundant network, either. If I ever used an excuse like that for bad behavior when I was growing up, my parents would ask me, "Well, if your friends jumped off a bridge, would you do it too?" I certainly hope that no one needed the E911 service for which they paid extra to CC over the past month. Oh, and where are the refunds? Hmm?

You know, I really don't give a flying fuck about what you don't give a flying fuck about. The narrow-minded, self contentedness here is beginning to piss me off. CallCentric never claimed to have geographic redundancy. Not once. Not ever. And it's not "bad behavior" not to have what they never advertised. What they always claimed was that their server farm was on the same electrical grid as the New York Stock Exchange. How often has the power gone out to the NYSE in the last thirty years? This is an extremely rare event and a once in a century storm. But stuff like this happens (rarely), regardless of how much posturing and huffing and puffing armchair quarterbacks spit out after the fact. And the reason the problems that the other companies are having is important to note, is because it this is a point of reference to show the scale of this storm and the damage it caused. It looks like the obvious is falling on deaf ears.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric
reply to Davesnothere

said by Davesnothere:

OR, a lot BETTER, if their skinflint owner(s) kicked in the bucks to set up either a 2nd server farm in another city, or rent from at least one landlord who provides a proper generator - or do BOTH of these things.

If a company with a second server farm is important to you, then by all means, look for one -- I thought you already had moved on. Personally I don't care. I know how extremely rare an extended electrical outage is in lower Manhattan and especially one involving the New York Stock Exchange's grid. I don't expect another "storm of the century" to happen any time soon -- so I'm quite willing to "take my chances."


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

2 edits

1 recommendation

said by rblizz:

If a company with a second server farm is important to you, then by all means, look for one -- I thought you already had moved on. Personally I don't care. I know how extremely rare an extended electrical outage is in lower Manhattan and especially one involving the New York Stock Exchange's grid. I don't expect another "storm of the century" to happen any time soon -- so I'm quite willing to "take my chances."

 
IMNSHO, this is NOT about 'Playing the Odds'.

It's about customer service - at ALL levels of the operation, including planning and investment.

Below is part of what I said to someone yesturday in an IM :
quote:
I indeed AM angry that a company which is so professional in SO many other ways - can leave themselves WIDE open for such an epic fail in this regard.

One thing that this particular debacle illustrates and drives home - is the need for an even greater sophistication of process in doing our 'due diligence' when choosing a VoIP provider than most of us ever thought, as one poster [you, I believe] suggested to me.

I continue to have a lot of respect for CallCentric as a company - just not about THIS event's handling.

= = = = = = =

And BTW, in response to your reply to another poster :

What it would take to make ME shut my pie hole would be an open public announcement from Callcentric management that they COULD have done better if they had handled this event like VOIP.MS did in the same city, and what they will be doing in the next few months to CATCH UP with those CANUCKS in that regard.

rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric

1 edit

said by Davesnothere:

This is not about 'Playing the Odds'.

It's about customer service - at ALL levels of the operation, including planning and investment.

I'm happy with the quality of service CallCentric has provided to me and I'm happy with their single server farm and the server redundancy they have there. It's not "playing the odds" -- the odds are in CallCentric's favor in my opinion. Until the DDoS attack and this storm, I've been blissfully using CallCentric and noting other provider's outages -- especially issues that seem to plague one of the companies that has multiple servers. Redundancy is nice, but it appears it can also introduce a whole new set of complications that can impact reliability.

You should also note "take my chances" was in quotes. In other words, I really don't think of it as taking a chance.

daneli

join:2012-10-23
Ann Arbor, MI

2 recommendations

reply to Iscream

I want to try to be constructive in my comments. Full redundancy with instant failover can be a very expensive and technically difficult challenge to meet. If a business can tolerate a longer time window (~ 8 hours) and can concentrate on only a subset of functions, then it can be much cheaper and easier to get some services back up and running in the face of a disruption. Certainly it should be possible for callcentric to provide a backup solution for its website and email that is more robust than the current solution. Perhaps a subset of functions - like voicemail or did forwarding - could also be made available from a remote location if the time window is extended to 12 hours or so?

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

After the dust settles, I think it is reasonable for us to ask callcentric to be transparent about what steps if any it plans to take to mitigate these kinds of risks in the future. The risks I have in mind are not those associated with a "storm of the century" but the usual kinds of things - flood, fire, theft, etc - that businesses try to anticipate.


rblizz

join:2001-12-16
North Richland Hills, TX
Reviews:
·Callcentric

said by daneli:

What surprised me about Iscream's last post before the power went out was the revelation that callcentric has less than 60 minutes of backup power available in the event of a power failure. A diesel-powered generator isn't a perfect solution - recent events in New York confirm this - but it would go a long way to mitigate most of the risks associated with future power outages. (It isn't far-fetched to think that another power outage is likely in the future and that it could last more than 60 minutes.)

Unfortunately it doesn't appear that a generator was an option in this building. But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7
reply to daneli

said by daneli:

I want to try to be constructive in my comments....

 
Absolutely, and I agree with pretty well all of them.

BTW, their UPS runtime might have been longer BEFORE they added a bunch of extra servers last month to help handle the DDoS attacks more effectively.


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

2 edits
reply to rblizz

said by rblizz:

Unfortunately it doesn't appear that a generator was an option in this building.

But, maybe now, CallCentric can use this event as leverage with their landlords and get permission to install a generator.

 
Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.

tr0910

join:2006-01-03
MI/USA

said by Davesnothere:

Yes, and I would be more than willing to put my name on a petition to help them make that happen, whether or not I am still a customer at the time.

They were just down again briefly. Line went dead, and website went dead.

I blame myself for trusting them, but it was based on recommendations from you all here. However I only trusted them with 1/2 our lines, so we were not completely shut down. Still, I think a healthy discussion is warranted, and they need to have a asterisk next to their name until they can prove they really have a disaster recovery plan, and have tested it out.

Does anybody have a fly swatter handy? We need to get rid of that that (buzz, buzz, buzz) that keeps flying around our ears....

royrogers

join:2012-10-17
reply to Davesnothere

So the NYSE is still running on generators, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

My theory is this: Callcentric is located in a building full of ISP's. ConEd and NYSE and everyone else recognized that the first priority is getting power to that building, even before the NYSE.

Iscream, how far off am I?



Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

said by royrogers:

So the NYSE is still running on battery backup, and has no connectivity on the floor. That was just reported on Squawk on the Street as I type this.

Where is the juice for Callcentric coming from?

 
One Word :

RATS

They have them running on wheels to operate power generators.

Letterman told me so.

NYC bylaws say nothing against doing THAT !


airwavz
Always the green wire

join:2011-09-11
Mount Juliet, TN
kudos:1

1 recommendation

reply to verix

said by verix:

I'm happy they restored service. I'm not happy this occurred in the first place. Happy?

I just HAVE to respond here - I'm pretty sure the 'so-far' verified 75+ people who have died as a result of this storm aren't "happy" either.... I'm pretty sure the millions still without power aren't "happy" that this storm "occurred in the first place".... I'd bet that the thousands who have lost everything they owned aren't "happy" about that either....

If you expect 100% uptime on your VOIP service, and you didn't have a backup provider and plan already in place, then kick your OWN butt for being at LEAST as poorly prepared as CC, and quit whining about your budget phone service failing. 100% uptime IS available; just whip out that check book and start writing!