dslreports logo
site
spacer

spacer
 
   
spc
story category
Massive BlackBerry Blackout
by Karl Bode 08:58AM Wednesday Apr 18 2007
Users in our forums note that the Research In Motion Blackberry network suffered a massive outage late yesterday. According to some news reports, the company's infrastructure failed around 8 pm Tuesday and was down until about 7 am Wednesday. While the network is back up, the company says they're holding back much of the backlogged data to help control network strain.

view:
topics flat nest 

manfmmd
Premium
join:2003-01-14
Earth, TX

Database Failure...

from what I hear...

It happens.

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
kudos:7

Re: Database Failure...

Yeah database servers fail. The question I would raise is how exactly did it fail and why wasn't there a backup? You'd think that if your entire business was reliant on that data you would want to guarantee high availablity no matter what...
--
Go Colts

kapil
The Kapil

join:2000-04-26
Chicago, IL

...Aaaah

....that's why my BB has been so quiet since yesterday.
caco
Premium
join:2005-03-10
Whittier, AK

Re: ...Aaaah

I got bombarded with email around 730 this morning. I was like WTF. This explains it.
--
www.seabee.org
dariena

join:2005-04-24
Germantown, MD

Not back up all the way....

Unfortunately only 75% of my org is back up, and the most important execs are not.

FFH
Premium
join:2002-03-03
Tavistock NJ
kudos:5

Back-up plans ?? - switch from Blackberry

RIM officials recommend all who depend on their BlackBerry as a major way of communication should make some back-up plans in case more problems occur Wednesday morning.
And one of the back-up plans their users should consider is evaluating alternatives to RIM. There are plenty of competitors out there both in hardware and in networks and their users should maybe look elsewhere.
--
--
My BLOG
My Web Page
dariena

join:2005-04-24
Germantown, MD

Re: Back-up plans ?? - switch from Blackberry

Unfortunately none of the competing products work quite as smoothly...
nokiatech

join:2000-10-18
Stuart, FL

Re: Back-up plans ?? - switch from Blackberry

My WM5 PDA phone synchs directly to exchange via Direct Push. How much smoother can you get?

jay608
Going Nucking Futs

join:2007-01-22
Chicago, IL

Re: Back-up plans ?? - switch from Blackberry

NS.. Nice that the owner can't get or send e-mails and then gets pissed when the IT dept says sorry can't help you. this is out of our hands. Do you want your Pocket PC back?
Foxbat121

join:2001-04-25
Herndon, VA

Re: Back-up plans ?? - switch from Blackberry

Using WM5 Smartphone right now synced directly to company exchange server. However, not sure if it is coincidence or not, my company's OWA and apparently Mobile access went out 5PM yesterday as well until 2AM this morning. We support both WM5 and BB. Maybe the BB enterprise server running on the same server as OWA and mobile access server.

The Beer
I Love It When A Plan Comes Together
Premium
join:2001-07-24
Atlantic, IA

2 edits
One of my problems with my Q was it did not download entire messages, so I would have to click on "Download Entire Message" blackberry does that as you read.

I went back to Blackberry on Sunday.

Worth the extra $20 a month.

EDIT: BTW I was not down.

Mine did die.

And now it's working again

Darkflight
Stitch 626 LTE

join:2001-03-03
Rowland Heights, CA

Re: Back-up plans ?? - switch from Blackberry

Appears to be intermittent. From what I hear, either you have all your messages, or you have none of them... hopefully they can get their queue of fun cleared up asap.
--
Darkflight- PCS Vision- Way of the future. STOP THE RIAA before they ruin another 12 year old's FUTURE!!!

chanur
Premium
join:2001-02-26
Colorado Springs, CO

Re: Back-up plans ?? - switch from Blackberry

said by Darkflight:

Appears to be intermittent. From what I hear, either you have all your messages, or you have none of them... hopefully they can get their queue of fun cleared up asap.
My personal experience says this is unfortunately not true. Additionally, I have gotten some duplicates as well, and others trickling in 8 - 12 hours later. The whole thing is messy.
Foxbat121

join:2001-04-25
Herndon, VA
said by The Beer:

One of my problems with my Q was it did not download entire messages, so I would have to click on "Download Entire Message"
That's a simple setting in the email setup. You can chose head only, first 0.5k, 1K ... or download all. The default is first 0.5k IIRC.

WM6, IIRC, adds download while you read functionality.
openbox9
Premium
join:2004-01-26
Germany
kudos:2
When MS gives me the security capabilities that RIM's BES does, then I'll agree with your assessment. Until then, additional products are required in the mix to equate to RIM's service.

The Beer
I Love It When A Plan Comes Together
Premium
join:2001-07-24
Atlantic, IA

Re: Back-up plans ?? - switch from Blackberry

MS does have a handheld wipe like Blackberry now.
openbox9
Premium
join:2004-01-26
Germany
kudos:2

Re: Back-up plans ?? - switch from Blackberry

It's a lot more than remote wipe. My research is a couple of months old and this is off of the top of my head, but:

- Password complexity and enforcement
- Security policy implementation/enforcement for devices
- Wiping the whole device (including SD storage)

I'm sure there are others, but I don't have access to my research results.
Foxbat121

join:2001-04-25
Herndon, VA

Re: Back-up plans ?? - switch from Blackberry

Your first two concerns are already implemented in current push email and WM5 devices. That is you can, actually most deployment should always do so, push down a security policy in order to let the device to sync with server.

As for wipe out SD card, I think it is left out in WM5 and WM6.0 implements SD card encryption instead.
openbox9
Premium
join:2004-01-26
Germany
kudos:2

Re: Back-up plans ?? - switch from Blackberry

Push email and and sync are there, but from what I recall, the security policy and enforcement to the level that I needed was not. Things might have changed in the last few months and maybe when WM6 rolls, even more will change, but when I looked, Exchange and WM5 didn't have what I needed and BlackBerry and BES did.

SD card encryption is a step in the right direction. This can currently be accomplished with 3rd party apps like Apriva and Trust Digital, but out of the box, Microsoft didn't provide a good solution.

NY Tel
Premium
join:2004-04-09
Smithtown, NY
kudos:3

Internet on Blackberry

Oddly enough, I now realize that when RIM has an outage, you also lose Internet access but my Verizon Text messaging and phone worked since it uses a different data connection.

nklb
Premium
join:2000-11-17
Ann Arbor, MI
kudos:2

So that's what happened

I had noticed my blackberry said it was on a "GRS" network rather than the usual "GPRS", now I know why.
--
for all your Linux questions
mike5411

join:2005-04-02
Nazareth, PA

Re: So that's what happened

(it was a lowercase gprs btw)

I dont completely understand why people are making such a hugh deal of this. Okay, there was an outage. It was from sunday 8 to monday 8. Mostly during a Sunday night. I have a BB and thought it was pretty inpressive how it was only down for that little. When i had a sidekick the data service went out for 4 days. Now that is ridiculous. I understand it is not the best thing for businesses not to have BB's working, but im sure if they were working it would of no affected most of the people in that time frame anways. Outages happen, if email is that big of a deal shouldnt you also have a laptop to do it?

batterup
I Can Not Tell A Lie.
Premium
join:2003-02-06
Netcong, NJ

TPC did it.

Blame it on The Phone Company. There now don't you feel better?
jervin123

join:2005-04-14
Philadelphia, PA

Re: TPC did it.

that just causes more frustration
ricep5
Premium
join:2000-08-07
Jacksonville, FL

Communication error

I noticed my BES logs showed the outage, but it was a communication stopage, otherwise I would have gotten an email forward error from BES.

This would indicate that Blackberry turned off the gateways (or someone did) so the BES router would hold outgoing mail in queue rather than outright lose it in transit.

After what everyone had to go through to get BES going with the DST changes, perhaps they had to eat some of their own dogfood.

There are alternatives like Motorola Good Link;

»www.good.com

JustHavingFun

@net1plus.com

BlackBerry vs MotoQ (Win Mobile 5)

I have both a BlackBerry and MotoQ. I must say that the BlackBerry blows Mobile 5 away. I added Good to the MotoQ and the functionality lifted to just about par with the BB, but the modem in the MotoQ doesn't seem to hold on to the connection as nice as the BB when you're on the edge of the network.

But on topic....RIM went down at about 8:15 Eastern and we dug into our two BES to see what was going on. RIM never put anything on their website as far as I know. They did update their telephone recording sometime on the 17th.

For those that has asked about "why didn't they have a backup" I'm not sure if you are talking about a data backup or not, but you figure even using snapshot technology there is a ton of data to backup and that takes time. Figure 8 million users...even at 25% utilization that could likely be 2 million messages a minute (for discussion sake)...now consider that during a restore you postpone inbound (every message has to come in before it goes out - at least from RIMs perspective). The amount of backlog increases during the restore. So at some point the delays just stack and you have to start throttling one way or the other.

I'm sure they are wildly clustered with multiple data stores, so the above is probably not very accurate, but in theory its a starting point.

Let's face it, it was probably that the firmware upgrade they did on their LinkSys Cable/DSL Router failed and Staples was closed so they had trouble getting a new one.

Enjoy.