dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2235
share rss forum feed

nitzan
Premium,VIP
join:2008-02-27
kudos:8

2 recommendations

Business continuity planning

Leaving the drama aside, I think it would be helpful for users to understand exactly what (and if!) their provider is doing to protect them from total loss of service in case of a disaster. Hypothetical example: a data-center fire destroys all equipment in it - what happens next?

As you all know, I run Future Nine. In our case we have servers in one data center in Miami, and another data center in Amsterdam. If either data center is destroyed, service will automatically fail over to the other data center within 5-15 minutes. This is done automatically and most users won't even notice service was down for a few minutes. Everything is replicated in real time so there's no issue of needing to bring anything back up - we could continue business as usual even if the entire data center is destroyed.

In Cheap Voip Inc's case, it currently only has one data center, in Germany. We are working on bringing up US servers and of course they will be replicated in real time and completely redundant. I personally think it's insane of a telephone service provider to NOT provide this basic level of redundancy. CVI is a new entity with few users currently, but going forward we want to have redundancy in place.

I'll leave it to other providers to describe their own plans for complete and utter data center destruction, although I encourage you (users) to inquire and think about this possibility - if your provider has no business continuity plan they could be gone overnight in case of a data center fire or other such disaster.

Mango
What router are you using?
Premium
join:2008-12-25
www.toao.net
kudos:13
Reviews:
·AcroVoice
·Callcentric
·Anveo
·Shaw
Thanks for your post. I have a related question.

Assuming all other variables are equal, would it cost you more, less, or the same, to build one single site that provides 99.999% uptime, vs two sites that each provide 99.95% uptime and are unlikely to have issues at the same time?

userofdsl

join:2000-07-31
Brighton, MA

2 edits
reply to nitzan
From the end-user perspective, the risk of using a provider with one data center can be reduced by maintaining a functioning T-Mobile prepaid phone and account, to which a number can quickly be ported in the event of a rare disaster.

Spending $100 on prepaid minutes causes them to last one year; minutes can be renewed for another year by spending any additional amount, such as $10.

EDIT: With an existing phone/account, porting requires only a phone call. But one's disaster plan can more cheaply be to quickly buy a phone and/or account/SIM chip, perhaps from a local T-Mobile store.

nitzan
Premium,VIP
join:2008-02-27
kudos:8

1 recommendation

reply to Mango
said by Mango:

Assuming all other variables are equal, would it cost you more, less, or the same, to build one single site that provides 99.999% uptime, vs two sites that each provide 99.95% uptime and are unlikely to have issues at the same time?

99.999% uptime is not realistic and not possible for VOIP. 99.99% might be, but not at a single location - even if you provide 99.99% for years, a single prolonged outage at the data center will put you above 99.99%. Example: a provider normally has 100% uptime, but in a 5 year period suffers 2 prolonged outages at their single location which total 60 hours. That's 12 hours a year on average, or in other words somewhere around 99.86%.

I guess this point is debatable, but in my opinion it's better to have more short outages than less long ones. For example I'd rather have 3 fifteen minute outages a year than a 10-hour outage every 3 years. To that point, IMO, creating two less reliable locations which backup each other is more reliable than creating one single more reliable location.

Cost-wise, it totally depends what type of equipment and software you use. If you use proprietary stuff like Solaris etc. than cost would skyrocket but it'll probably be slightly more reliable assuming the whole data center isn't offline. If you use commodity servers and open source software then cost isn't even an issue and you can have as many locations as you'd like. Again this is a matter of preference but in general IMO a multi-location setup will always result in less downtime overall.

Case in point: Future Nine had 11 hours of outage in it's first year of business - 7 hours of which due to not having a backup location at the time, and another 4 hours due to our failover scripts failing to automatically move things over to the failover location. In years 2, 3, 4, and 5 combined we've only had 3 hours downtime total. That's 99.99% availability.

MartinM
VoIP.ms
Premium,VIP
join:2008-07-21
kudos:3

1 edit

2 recommendations

reply to Mango
said by Mango:

Thanks for your post. I have a related question.

Assuming all other variables are equal, would it cost you more, less, or the same, to build one single site that provides 99.999% uptime, vs two sites that each provide 99.95% uptime and are unlikely to have issues at the same time?

This is my point of view in regard to VoIP.ms

In VoIP.ms case, it would cost much less to run a single site, and much easier to operate. The man power required to administer multiple sites is greater. There's a lot of complexity added in managing inbound calls and keeping everything working in an independant manner without depending on central or "main" proxies.

You have to find providers that will accept multiple connections from different sources and are also able to send incoming calls to different source.

I also fully agree with Nitzan that multiple short downtimes are much better than long down times for 100% of your user base. It impact the support response time and affect your global uptime much more.

Upsides of central setup:

- Easier to maintain
- Cheaper by not having to rent rack space in multiple locations
- Cheaper in human resources to maintain
- Easier for non technical users
- Less "small downtime". If a single site is likely to have small hiccups from time to time, and you add over 13 locations, you do the math. However this only affect the customers of the affected POP, but the amount of incidents when viewed as a whole can turn off potential customers.
- Can use "physical connection" to the PSTN instead of SIP Trunking with providers.

Upsides of multiple locations setup (VoIP.ms)

- Can move users around when a Point of Presence (POP) goes down via DNS
- Can move Website around via DNS
- Support not overwhelmed by customers if a site goes down (In our case, that represent more or less 5% to 12% of the user base depending on which location goes down.
- Internet is not perfect, and certain users might have routing problems with a specific data center. If they have the opportunity to use a different location to connect and fix that issue, that's a plus.
- Minimize latency. This of course doesn't affect everyone, but it's useful for people with lesser connections, international customers etc.

The downsides of multiple location is you can have your setup bashed repeatedly and publicly by the competition. j/k
--
Martin - VoIP.ms

PX Eliezer70
Premium
join:2008-08-09
Hutt River
kudos:13
Reviews:
·callwithus
·voip.ms
Interesting analysis.

I do have one question and one observation:

1) What is the happy medium? If you feel that too few locations are a problem, what of too many? Currently I think VoIP.MS have 11 server locations. By contrast, Anveo (for example) has three (Dallas, Montreal, Nuremberg).

When does it get to be too much in terms of monitoring, maintaining, and paying for them?

And, as with kids, 2 or 3 or 5 may be ok, but why 11?

2) CallCentric's Iscream recently said that they literally might have not been able to survive the DDoS episode if they were in multiple locations. They had to analyze and repel tsunamis of garbage data in real time. Every company has different architecture and systems, so I don't want to fully compare poutine and knishes, but it worth noting that in CC view having everything integrated rather than spread out, was lifesaving.

Yes, if you want to say it, the SAME structure that saved them in the DDoS ended up causing a problem in the hurricane that immediately followed. Life is ironic....!

---------------------------------------------

For those who like SF and who want to see a parallel to how a DDoS attack might have been met in real time, I suggest [The Game of Rat and Dragon] by Cordwainer Smith.

»www.gutenberg.org/files/29614/29···14-h.htm

Anyone who does not like SF, [don't] go there, you won't get it.

(Cordwainer Smith was actually Paul Linebarger, a science fiction author, US Army expert on propaganda, White House advisor especially to JFK, friend of Chiang Kai-shek, a consultant to the OSS and the CIA, and a Professor of Political Science at Duke and at Johns Hopkins.)

nitzan
Premium,VIP
join:2008-02-27
kudos:8
said by PX Eliezer70:

1) What is the happy medium? If you feel that too few locations are a problem, what of too many? Currently I think VoIP.MS have 11 server locations. By contrast, Anveo (for example) has three (Dallas, Montreal, Nuremberg).
When does it get to be too much in terms of monitoring, maintaining, and paying for them?
And, as with kids, 2 or 3 or 5 may be ok, but why 11?

In terms of reliability/uptime, IMO 2-3 locations are the same as 11 if they are geographically dispersed. The chances of a US data center and a European data center getting completely knocked out at the same time by a natural disaster are extremely low. If you want to be paranoid you could add in a third geo location, maybe somewhere in Asia - although arguably if both the US and Europe are down then none of your calls will work anyway because all your carriers are down too.

Having more than 3 locations is overkill and not required in terms of reliability (IMO). If you want to provide lower latency and of course for marketing purposes then yes having 11 locations might be desirable - but the downside is extra maintenance and extra potential for human error and other "random" problems.

So personally I think the magic number is 2-3 locations.

PX Eliezer70
Premium
join:2008-08-09
Hutt River
kudos:13
Thank you Nitzan, and I hope that you had a good holiday.

sokhapkin
Premium
join:2003-05-08
North Fort Myers, FL
reply to nitzan
said by nitzan:

So personally I think the magic number is 2-3 locations.

2-3 is too few. Dumb customers want ping time to the closest server to be less than 15 ms. You need to provide at least 30 servers on US east coast only

Ping time from a server to PSTN gateways doesn't matter
--
»www.callwithus.com

Will Hill

join:2009-06-07

1 recommendation

reply to nitzan
What is the plan if a VoIP provider goes out of business suddenly and without any warning or notification to its customers?


Trev
IP Telephony Addict
Premium
join:2009-06-29
Victoria, BC
kudos:6
said by Will Hill:

What is the plan if a VoIP provider goes out of business suddenly and without any warning or notification to its customers?

That would be part of the customer's business continuity planning. At that point the provider has decided to cease continuations altogether
--
Wondering what I do? Find out at »www.digitalcon.ca
Get your Obihai ATA in Canada.

Mango
What router are you using?
Premium
join:2008-12-25
www.toao.net
kudos:13
Reviews:
·AcroVoice
·Callcentric
·Anveo
·Shaw
reply to Will Hill
I believe service providers are legally required to provide notice before suspending service. In the event that this is legitimately not possible, you would experience downtime for the length of time it would take you to port to another carrier.

Fortunately, there are very few cases of this happening.

PX Eliezer70
Premium
join:2008-08-09
Hutt River
kudos:13
Reviews:
·callwithus
·voip.ms
1) As Mango said, FCC has rules about such things requiring notifications of companies that are shutting down. I don't know if the Canadian CRTC also has them.

Of course, if someone skips town, then any rules against skipping town are hard to enforce. That's why it's a good reason to stay with providers that have an established record, a physical office, and legal incorporation by a state or province.

Two providers (there are MANY others) that meet that are CallCentric (aka Accatel) and Voip.MS (aks Swiftvox and 9171-5573 Quebec inc).

A while back there was a provider (have not seen them lately) who stated that their home base was "the internet" and that they were subject to regulation by no jurisdiction.

Another former provider noted that they were domiciled in Panama.

I'd avoid both such....

2) A few years ago, Brookhaven National Laboratory on Long Island was going to conduct an experiment that a few people were worried might destroy the universe. [This was reported in major news media at the time, it's not an urban myth].

The lab said that at SOME point in billions of years and on billions of planets, someone must have already done the experiment. Because the universe was still here, it proved there was no danger.

[However, for a contrary view, see William Tenn's SF story "Brooklyn Project"].

At ANY rate, because of the challenging economy in recent years it is quite likely that any VoIP company that was going to fail, already HAS failed.

The list here is outdated, but the point is, the slower antelope in the herd have already been eaten by the lions and hyenas:

VoIP Graveyard
»www.myvoipprovider.com/en/VoIP_P···raveyard

nitzan
Premium,VIP
join:2008-02-27
kudos:8

2 recommendations

said by PX Eliezer70:

That's why it's a good reason to stay with providers that have an established record, a physical office, and legal incorporation by a state or province.

Historically speaking, you're completely wrong. Let's look at recent providers going out of business that come to mind:

AT&T CallVantage
Verizon VoiceWing
VoiceStick
SunRocket

What's common between all of these? they were all well-funded, had a physical office (aka rent!), a large team of employees (aka lots of salaries to pay!) and were legally incorporated in the US. Obviously these factors are no guarantee and have nothing to do with long-term stability.


XCOM
digitalnUll
Premium
join:2002-06-10
Spring, TX
reply to nitzan
Good information all around.

Thanks.
--
[nUll@dcypher ~]$

PX Eliezer70
Premium
join:2008-08-09
Hutt River
kudos:13
Reviews:
·callwithus
·voip.ms
reply to nitzan
said by nitzan:

AT&T CallVantage
Verizon VoiceWing
VoiceStick
SunRocket

What's common between all of these? they were all well-funded, had a physical office (aka rent!), a large team of employees (aka lots of salaries to pay!) and were legally incorporated in the US.

Obviously these factors are no guarantee and have nothing to do with long-term stability.

Ah, when I was writing my post in real-time, I knew that you'd be responding thusly.

BUT:

Will Hill did NOT ask about long-term stability, but RATHER he asked about skipping town without notice.

said by nitzan:

AT&T CallVantage
Verizon VoiceWing
VoiceStick
SunRocket

AT&T CallVantage and Verizon VoiceWing gave lots of notice.

Even VoiceStick gave notice. IIRC the new FCC rules were in effect then, and they [did] comply.

So it's only SunRocket. I would agree that SunRocket was a shame and also shameful, but that's pretty much ancient history in the VoIP timeframe.

[Ironically the woman who ran SunRocket (and ran it into the ground) now runs Neustar and as such wields authority over all of us because she hands out the phone numbers. In May 2011 President Obama even appointed her to the National Security Telecommunications Advisory Committee. But I digress].

said by nitzan:

Obviously these factors are no guarantee....

Never said they were an absolute guarantee. Nothing is guaranteed but death and spam faxes.

said by nitzan:

....and have nothing to do with long-term stability.

Well, to say they have NOTHING to do with long-term stability is going to the other extreme.

Point is, if I have two providers who are identical in all respects except that one has a physical office and the other doesn't, IMHO that would be a factor to look at. Not the only factor, but a factor.

But the physical office was just one of three things that I mentioned, the other two being a track record and legal incorporation in a state or province. I don't think anyone can argue with that....

Finally, I was not talking so much about the huge companies that exited the business (such as AT&T CallVantage etc) but rather about the hundreds of providers listed in the "VoIP Graveyard" listing, most of whom were just websites and little else.

-------------------------------------------------

One thing we can probably agree on is to avoid VoIP providers recommended by Howard Stern.

TomatoVine VoIP was one such.

»User reviews - tomatovine.com

IPfaxer

join:2010-10-24
reply to nitzan
Interesting post - thank you.

What are the annual sales of Future Nine and how many full-time people are on staff?

What is your connection to the public telco infrastructure/POTS. Is it with a 3rd party like XO? Please describe in some detail.

That you are able to post here shows you have some free time due to your system running smoothly. Please provide some more info about Future Nine. I am not familiar with it and would like to know more as I bet a lot of other readers would as well. There is a lot of room for growth in the VOIP business. We connect lots of businesses to VOIP and would like to know about vendors like yourself that we have not been familiar with.

Thanks


dbmaven
There's no shortage
Premium,Mod
join:1999-10-26
Sty in Sky
kudos:3
reply to nitzan

(topic move) Business continuity planning

Moderator Action
The post that was here (and all 4 followups to it), has been moved to a new topic .. »Split - Websites


espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN
kudos:2

1 recommendation

reply to nitzan

Re: Business continuity planning

said by nitzan:

Leaving the drama aside, I think it would be helpful for users to understand exactly what (and if!) their provider is doing to protect them from total loss of service in case of a disaster. Hypothetical example: a data-center fire destroys all equipment in it - what happens next?

Since you're mentioning your BCP plans, how do you intend to restore service if your upstream DID provider's facility burns down? Can you publish the BCP plans of your vendors supplying the PSTN interconnect?

nitzan
Premium,VIP
join:2008-02-27
kudos:8

1 recommendation

said by espaeth:

Since you're mentioning your BCP plans, how do you intend to restore service if your upstream DID provider's facility burns down? Can you publish the BCP plans of your vendors supplying the PSTN interconnect?

Good point - there are of course single points of failure other than our own network - but that is true with any provider. At least on our end we can ensure business continuity. To be more specific: since we work with several carriers, outgoing service would not be impacted much even if a carrier goes down - calls will just be re-routed to another carrier. Incoming DIDs would be impacted if a carrier goes down, but there's nothing that can be done about that.

On the user end if a business it would be wise to purchase phone numbers from two different providers and/or that are coming from two different carriers and/or two different geographic locations. If you have two phone numbers listed and one doesn't work, customers can just call the second number - not ideal, but definitely better than being down for days/weeks. (in case of a fire etc.)


engineercarl
Premium
join:2003-02-24
20026-4474
Here is a simple idea to work on. Since you operate both F9 & CVI, why not co-locate their backups at alternate sites? For example, place your backup for CVI in Miami (with F9) and Amsterdam.

Likewise, locate F9's redundant servers in Nurenberg. You'll have three sites, with both businesses at all of them.

I would take out one of the european servers and replace it with something else, like los angeles, seattle or east asia.

Anybody see any holes in this?

nitzan
Premium,VIP
join:2008-02-27
kudos:8
We have plans to add a 3rd location for Future Nine, and a 2nd location for CVI in the very near future. Perhaps New York (heh!) but that's not finalized yet.

PX Eliezer70
Premium
join:2008-08-09
Hutt River
kudos:13
Reviews:
·callwithus
·voip.ms
That will be nice.

-----------------

I think that my cousin helped to make "Casablanca" so I always enjoy quoting from that film:

Heinz: Can you imagine us in London?

Rick: When you get there, ask me!

Captain Renault: Hmmh! Diplomatist!

Major Strasser: How about New York?

Rick: Well there are certain sections of New York, Major, that I wouldn't advise you to try to invade.


garys_2k
Premium
join:2004-05-07
Farmington, MI
Reviews:
·Callcentric

1 recommendation

reply to nitzan
I think the real lesson for those needing their phone to work is to have their OWN continuity planning. If you have a business to support publish two phone numbers and carry each with its own different VoSP.

It's easy to implement a contingency plan for outgoing calls -- Callwithus worked great for me and allowed me to set my own caller ID. Incoming calls are always the weak link.

EVERY provider depends on another party to handle incoming calls. From the number's C.O. to whoever hands it off to the provider, whether via S7 switch or by the Internet, that provider can fail.

Face it, we all learned that having at least two numbers is important if you NEED that phone to work. If your business depends on incoming calls then an active backup is essential no matter who you use.

OmagicQ
Posting in a thread near you

join:2003-10-23
Bakersfield, CA
kudos:1
reply to nitzan
said by nitzan:

We have plans to add a 3rd location for Future Nine, and a 2nd location for CVI in the very near future. Perhaps New York (heh!) but that's not finalized yet.

What ever happened to those servers gathering dust. in california?
--
...Who, What, When, Where, How... Why? Why Not?


Davesnothere
No-BHELL-ity DOES have its Advantages
Premium
join:2009-06-15
START Today!
kudos:7

4 edits
reply to garys_2k
said by garys_2k:

I think the real lesson for those needing their phone to work is to have their OWN continuity planning....

....Face it, we all learned that having at least two numbers is important if you NEED that phone to work....

....an active backup is essential no matter who you use.

 
Yes, I agree that THAT really IS the bottom line here, the 'Moral of the Story', as it were.

When we consider how extremely much less that almost EVERYONE posting here already pays for VoIP service - compared to what Verizon, AT&T, or whichever other incumbent would charge us even today for a less sophisticated form of phone service - surely we each could justify keeping TWO lines of VoIP happening, each from a different VoIP provider, even if we might only use one of them most of the time.

As much as we might feel justified in advising various VoIP providers as to how they could do things better (with the exception of sometimes having better organized and/or more informational websites - and those two factors have nothing at all to do with geo-redundancy and onsite power generators ), maintaining TWO VoIP services would still cost any of us a small fraction of one incumbent phone line !

Even BEFORE the fit hit the shan recently, I had TWO VoIP lines happening, and though not a business myself, the cost of doing so did not break me, and going forward it still won't.

nitzan
Premium,VIP
join:2008-02-27
kudos:8
reply to OmagicQ
said by OmagicQ:

said by nitzan:

We have plans to add a 3rd location for Future Nine, and a 2nd location for CVI in the very near future. Perhaps New York (heh!) but that's not finalized yet.

What ever happened to those servers gathering dust. in california?

We had extreme problems with the company running them (downtime, billing errors, "lost" a server, etc.) so we never felt comfortable bringing them online full-scale and finally decided to abort altogether.

Another facility I've been considering for a long time is Hurricane Electric in Fremont - but their minimum is a whole rack which is a bit overkill for our needs.


ABGoogle

@174.128.230.x

We had extreme problems with the company running them (downtime, billing errors, "lost" a server, etc.) so we never felt comfortable bringing them online full-scale and finally decided to abort altogether.

Another facility I've been considering for a long time is Hurricane Electric in Fremont - but their minimum is a whole rack which is a bit overkill for our needs.

Ask voip.ms. They know a lot about datacenters and might give you the details out of professional courtesy.
Expand your moderator at work

adatech

join:2010-04-23
reply to nitzan

Re: Business continuity planning

I think the vast difference in ping times between voip.ms's servers is indication enough that the servers are actually located in the locations they specify. I'm on the West Coast. LA and Seattle have low pings, Chicago and Dallas somewhere in the middle, NYC and Tampa the highest.

That, and the traceroutes that show a route from my home Comcast account, to Comcast's Seattle backbone, then to the Seattle voip.ms server, for example.