VOIP Tech Chat → [SunRocket] SunRocket Network Disruption Email

Dear SunRocket Member,

On behalf of SunRocket and all of our dedicated employees, I would like to apologize for any inconvenience that resulted from a SunRocket network disruption that occurred on Monday, July 11. The disruption was caused by recent surges in subscribers and call volumes. Unfortunately, one of the partitions in our network was unable to handle the demand, and it took several hours for our engineers to fully restore the network. While thousands of subscribers remained in service during the disruption, we failed to meet our reliability standards. Last week, we activated a new partition that doubled our capacity, but we had not begun the process of rebalancing the system load between the partitions prior to Monday's disruption. While that rebalancing is now underway, we will be activating a third partition within the next week to further augment our system capacity and improve the reliability of the SunRocket network.

We also recognize that we need to improve our ability to provide real-time status and notices during network disruptions, though our primary goal is to ensure that our network is fully operational at all times. You expect and deserve reliable phone service, and SunRocket remains committed to providing exceptional service at an exceptional value.

As an expression of our regret for any inconvenience you may have experienced, we will add 10 points into your RocketRewards account. Thousands of SunRocket members have taken advantage of the RocketRewards Referral Program, and we greatly appreciate your contributions in spreading the word about SunRocket to friends, family, neighbors and colleagues. We are also expanding your opportunities for earning RocketRewards points, as well as introducing more redemption choices. You can learn more about RocketRewards at www.sunrocket.com/community/referrals.

We are continuously striving to improve your service and enhance our ability to serve you better. Thank you for being a valued SunRocket member.

Sincerely,

Kevin Bennis
Chief Executive Officer

Makes sense to me. Did anyone here working service during the outage?

Just came here to post this, you beat me to it.

What's a 'partition' ?

dang you guys are fast.

anyway, let's hope this new partition solves the problems they've been having of late, and that they keep on communicating with us like this.

My service remained working the entire day.
St. Louis, MO.

I got that e-mail a little bit ago

They must do some sort of partitioning of coverage areas, area codes, etc. somehow on their network. Now if the partitions had failover to some sort of standbye or hot swappable partitions, that would be ideal.

don't want to sound too sarcastic, but my guess is that we will all have to "open a ticket" to get the 10 rocket reward points SR gave us

Again... growing pains and a work in progress...

...but they can only survive a certain amount of this sort of pain... The most encouraging aspect of all of this is their communication with us and the acknowledgement that it's important and they need to do more toward that fact (like a network status page).

I like the "third partition" mention... In *anticipation* of growth... not in response to it... more $ up front, yes, but a good move to keep us on board...

That e-mail is all very well and good... however I'm still waiting for some kind of acknowledgment that SR will be putting some kind of failover system (auto reroute to another number I specify or to voicemail) in place sooner rather than later.

My biggest beef is having my number go into limbo (dead air/fast busy/recorded msg) whenever their network fails. That should just not happen.

I am glad I'm getting 10 rocket rewards points. Now I can choose from a whole varienty of options...oh yeah, their minimum is 25 to get anything.

Quoted: " I am glad I'm getting 10 rocket rewards points. Now I can choose from a whole varienty of options...oh yeah, their minimum is 25 to get anything..."

But, didn't read the mesg in their redeem rewards section? They are talking about rewards less than 25 wil be available soon!!!

My guess is 10 point reward for some restaurant certificate worth $10... Not a bad deal for 6 hours of outage

Man, there is no pleasing some people. I personally think that Sunrocket has raised the bar in terms of the industry standard for dealing with this sort of thing. You asked for an email apology and they delivered. Additionally, they gave you (automatically, check your account) some form of compensation. Tell me, what other companies have sent apology notices, much less compensation, for network outages?

Thank you Sunrocket. Not only do you offer the most service for the best price, I truly believe you are working hard to build a solid, first-rate VoIP company with customer concerns and desires in mind.

I agree.

I'm pleased of this update and apology, although 4 days after the outage. But that, I can deal with.

Thanks SR.

Maybe I needed to add a /SARCASM on the end of my message. I love my SunRocket service, and am glad they finally made some sort of explanation/apology.

I agree with this statement 100 percent:)

Instead of adding "partitions" SunRocket needs to open POPs throughout the country.

As it stands now, all traffic is routed to their ONE AND ONLY data center near D.C. This represents a "single point of failure".

Having a presence in several regions would not only allow them to re-route traffic should a problem arise it may also decrease the latency of callers in that particular region.

Another thing to consider is the fact that a large Denial of Service attack could easily take us all down. SR is currently showing two transit providers - Qwest and Global Crossing. In my experience, neither of these providers is very helpful or clueful when dealing with attacks.

If you have the time, would you mind explaining what a 'POP' and 'partition' are? This may help facilitate the discussion.

Was this sent to all customers? Any significance to not receiving it?

check your bulk folder also. I found mine in the bulk folder (yahoo mail)

POP = Point of Presence

As mentioned above, it's called Point of Presence. Most big name companies today who do a lot of data storage or high availability systems will have more than one point of presence in the US or abroad. For VoIP redundancy to be effective, it would require that they have a similar setup that can handle the capacity in another location.

So a POP is an actual physical presence, such as a server or farm, droped here and there about the country. I assume then that each can act independently with some sort of coordinated distribution control. I wonder then, what the heck is a "partition?"

My assumption would bee partitions are in a way, an alternative to actual POP's, in that, traffic is distributed and routed through them based on geographical location of actual subscribers.

This way, there is some kind of logical routing and separation (hence partition) between areas.
So that not all subscribers are being routed by one piece of hardware, which would be disastrous.

Possibly this?

»www.voip-news.com/sp/ng/ ··· tion.htm

It seems counterintuitive to make referrals to bring more customers into the SunRocket network, since SunRocket can't handle its current customer load.

E-mail and rewards points are nice. It would be nice for noodler or zznate to tell us what to expect as far as feature upgrades,which are part of this upgrade (as per noodler.) I think transparency goes a long way. Had we known that SR is in the midst of a major capacity upgrade, we could have anticipated some potential problems with it.

Nicegirl-

From the e-mail SR sent it looks like capacity has been doubled and they are in the process of tripling it. So, it seems there is now the capacity to deal with new customers. I have 3 referrals that I am looking to get online with SR but will wait a week to make sure the capacity upgrade has gone without a hitch. According to noodler (Sr's Software Architect) that should be after this coming week. If the system is stable through that I'll go ahead with the 3 referral signups.

I also suggested to builditbetter@sunrocket.com to get an email address for us to email level 2 (or 3) support directly. Here is the response I got:So my guess is they're a little light on the experienced technical support side. Hopefully they make a tripple expansion in this area as well...

You can get at least five Level 3 techs in India for the price of one in Vienna VA. World connectivity is a double edge sword for American programmers & techs.