dslreports logo
 story category
Massive BlackBerry Blackout
Users in our forums note that the Research In Motion Blackberry network suffered a massive outage late yesterday. According to some news reports, the company's infrastructure failed around 8 pm Tuesday and was down until about 7 am Wednesday. While the network is back up, the company says they're holding back much of the backlogged data to help control network strain.
view:
topics flat nest 

manfmmd
Premium Member
join:2003-01-14
Earth, TX

manfmmd

Premium Member

Database Failure...

from what I hear...

It happens.

cdru
Go Colts
MVM
join:2003-05-14
Fort Wayne, IN

cdru

MVM

Re: Database Failure...

Yeah database servers fail. The question I would raise is how exactly did it fail and why wasn't there a backup? You'd think that if your entire business was reliant on that data you would want to guarantee high availablity no matter what...

kapil
The Kapil
join:2000-04-26
Chicago, IL

kapil

Member

...Aaaah

....that's why my BB has been so quiet since yesterday.
caco
Premium Member
join:2005-03-10
Whittier, AK

caco

Premium Member

Re: ...Aaaah

I got bombarded with email around 730 this morning. I was like WTF. This explains it.
dariena
join:2005-04-24
Germantown, MD

dariena

Member

Not back up all the way....

Unfortunately only 75% of my org is back up, and the most important execs are not.

FFH5
Premium Member
join:2002-03-03
Tavistock NJ

FFH5

Premium Member

Back-up plans ?? - switch from Blackberry

RIM officials recommend all who depend on their BlackBerry as a major way of communication should make some back-up plans in case more problems occur Wednesday morning.
And one of the back-up plans their users should consider is evaluating alternatives to RIM. There are plenty of competitors out there both in hardware and in networks and their users should maybe look elsewhere.
dariena
join:2005-04-24
Germantown, MD

dariena

Member

Re: Back-up plans ?? - switch from Blackberry

Unfortunately none of the competing products work quite as smoothly...
bigddybn
join:2000-10-18
Stuart, FL

bigddybn

Member

Re: Back-up plans ?? - switch from Blackberry

My WM5 PDA phone synchs directly to exchange via Direct Push. How much smoother can you get?

jay608
Going Nucking Futs
join:2007-01-22
Homewood, IL

jay608

Member

Re: Back-up plans ?? - switch from Blackberry

NS.. Nice that the owner can't get or send e-mails and then gets pissed when the IT dept says sorry can't help you. this is out of our hands. Do you want your Pocket PC back?
Foxbat121
join:2001-04-25
Ashburn, VA

Foxbat121

Member

Re: Back-up plans ?? - switch from Blackberry

Using WM5 Smartphone right now synced directly to company exchange server. However, not sure if it is coincidence or not, my company's OWA and apparently Mobile access went out 5PM yesterday as well until 2AM this morning. We support both WM5 and BB. Maybe the BB enterprise server running on the same server as OWA and mobile access server.

The Beer
I Love It When A Plan Comes Together
Premium Member
join:2001-07-24
Lincoln, NE

2 edits

The Beer to bigddybn

Premium Member

to bigddybn
One of my problems with my Q was it did not download entire messages, so I would have to click on "Download Entire Message" blackberry does that as you read.

I went back to Blackberry on Sunday.

Worth the extra $20 a month.

EDIT: BTW I was not down.

Mine did die.

And now it's working again

Darkflight
Stitch 626 LTE
join:2001-03-03
Rowland Heights, CA

Darkflight

Member

Re: Back-up plans ?? - switch from Blackberry

Appears to be intermittent. From what I hear, either you have all your messages, or you have none of them... hopefully they can get their queue of fun cleared up asap.

chanur
Premium Member
join:2001-02-26
Colorado Springs, CO

chanur

Premium Member

Re: Back-up plans ?? - switch from Blackberry

said by Darkflight:

Appears to be intermittent. From what I hear, either you have all your messages, or you have none of them... hopefully they can get their queue of fun cleared up asap.
My personal experience says this is unfortunately not true. Additionally, I have gotten some duplicates as well, and others trickling in 8 - 12 hours later. The whole thing is messy.
Foxbat121
join:2001-04-25
Ashburn, VA

Foxbat121 to The Beer

Member

to The Beer
said by The Beer:

One of my problems with my Q was it did not download entire messages, so I would have to click on "Download Entire Message"
That's a simple setting in the email setup. You can chose head only, first 0.5k, 1K ... or download all. The default is first 0.5k IIRC.

WM6, IIRC, adds download while you read functionality.
openbox9
Premium Member
join:2004-01-26
71144

openbox9 to bigddybn

Premium Member

to bigddybn
When MS gives me the security capabilities that RIM's BES does, then I'll agree with your assessment. Until then, additional products are required in the mix to equate to RIM's service.

The Beer
I Love It When A Plan Comes Together
Premium Member
join:2001-07-24
Lincoln, NE

The Beer

Premium Member

Re: Back-up plans ?? - switch from Blackberry

MS does have a handheld wipe like Blackberry now.
openbox9
Premium Member
join:2004-01-26
71144

openbox9

Premium Member

Re: Back-up plans ?? - switch from Blackberry

It's a lot more than remote wipe. My research is a couple of months old and this is off of the top of my head, but:

- Password complexity and enforcement
- Security policy implementation/enforcement for devices
- Wiping the whole device (including SD storage)

I'm sure there are others, but I don't have access to my research results.
Foxbat121
join:2001-04-25
Ashburn, VA

Foxbat121

Member

Re: Back-up plans ?? - switch from Blackberry

Your first two concerns are already implemented in current push email and WM5 devices. That is you can, actually most deployment should always do so, push down a security policy in order to let the device to sync with server.

As for wipe out SD card, I think it is left out in WM5 and WM6.0 implements SD card encryption instead.
openbox9
Premium Member
join:2004-01-26
71144

openbox9

Premium Member

Re: Back-up plans ?? - switch from Blackberry

Push email and and sync are there, but from what I recall, the security policy and enforcement to the level that I needed was not. Things might have changed in the last few months and maybe when WM6 rolls, even more will change, but when I looked, Exchange and WM5 didn't have what I needed and BlackBerry and BES did.

SD card encryption is a step in the right direction. This can currently be accomplished with 3rd party apps like Apriva and Trust Digital, but out of the box, Microsoft didn't provide a good solution.

NY Tel
Premium Member
join:2004-04-09
Smithtown, NY

NY Tel

Premium Member

Internet on Blackberry

Oddly enough, I now realize that when RIM has an outage, you also lose Internet access but my Verizon Text messaging and phone worked since it uses a different data connection.

nklb
Premium Member
join:2000-11-17
Ann Arbor, MI

nklb

Premium Member

So that's what happened

I had noticed my blackberry said it was on a "GRS" network rather than the usual "GPRS", now I know why.
mike54117
join:2005-04-02
Nazareth, PA

mike54117

Member

Re: So that's what happened

(it was a lowercase gprs btw)

I dont completely understand why people are making such a hugh deal of this. Okay, there was an outage. It was from sunday 8 to monday 8. Mostly during a Sunday night. I have a BB and thought it was pretty inpressive how it was only down for that little. When i had a sidekick the data service went out for 4 days. Now that is ridiculous. I understand it is not the best thing for businesses not to have BB's working, but im sure if they were working it would of no affected most of the people in that time frame anways. Outages happen, if email is that big of a deal shouldnt you also have a laptop to do it?

batterup
I Can Not Tell A Lie.
Premium Member
join:2003-02-06
Netcong, NJ

batterup

Premium Member

TPC did it.

Blame it on The Phone Company. There now don't you feel better?
jervin123
join:2005-04-14
Philadelphia, PA

jervin123

Member

Re: TPC did it.

that just causes more frustration
ricep5
Premium Member
join:2000-08-07
Jacksonville, FL

ricep5

Premium Member

Communication error

I noticed my BES logs showed the outage, but it was a communication stopage, otherwise I would have gotten an email forward error from BES.

This would indicate that Blackberry turned off the gateways (or someone did) so the BES router would hold outgoing mail in queue rather than outright lose it in transit.

After what everyone had to go through to get BES going with the DST changes, perhaps they had to eat some of their own dogfood.

There are alternatives like Motorola Good Link;

»www.good.com

JustHavingFun
@net1plus.com

JustHavingFun

Anon

BlackBerry vs MotoQ (Win Mobile 5)

I have both a BlackBerry and MotoQ. I must say that the BlackBerry blows Mobile 5 away. I added Good to the MotoQ and the functionality lifted to just about par with the BB, but the modem in the MotoQ doesn't seem to hold on to the connection as nice as the BB when you're on the edge of the network.

But on topic....RIM went down at about 8:15 Eastern and we dug into our two BES to see what was going on. RIM never put anything on their website as far as I know. They did update their telephone recording sometime on the 17th.

For those that has asked about "why didn't they have a backup" I'm not sure if you are talking about a data backup or not, but you figure even using snapshot technology there is a ton of data to backup and that takes time. Figure 8 million users...even at 25% utilization that could likely be 2 million messages a minute (for discussion sake)...now consider that during a restore you postpone inbound (every message has to come in before it goes out - at least from RIMs perspective). The amount of backlog increases during the restore. So at some point the delays just stack and you have to start throttling one way or the other.

I'm sure they are wildly clustered with multiple data stores, so the above is probably not very accurate, but in theory its a starting point.

Let's face it, it was probably that the firmware upgrade they did on their LinkSys Cable/DSL Router failed and Staples was closed so they had trouble getting a new one.

Enjoy.