dslreports logo
 story category
T-Mobile Struggles With Flooding
Bothell, Washington data center swamped, support offline
Severe flooding in the Pacific Northwest has swamped a T-Mobile data center in Bothell, Washington, impacting customer support, activiations, and some website functions. Several users have written in directing our attention to stories at both the Boy Genius Report and Data Center Knowledge. An employee posts his account:
quote:
The report is correct - The main data center is flooded right now, so we only have email and our coverage/outage map. No accounts can be accessed due to the systems being out, so this can and will affect short codes such as 646, 225, the website and just about every system. It started around 7:30pm EST and was still happening when I left around 2:00 am EST. They were hoping to at least get read only systems online in the next few hours, so that we can at least do a little for the customers. Hopefully it will be back soon - Any techies have a clue how long it takes to dry out servers and get them back online - That should give a better idea on timelines.
Calling T-Mobile's customer care line (1-800-937-8997) results in a recorded message telling you that the company can't access your account information.
view:
topics flat nest 
page: 1 · 2 · next

rbrussell82
join:2003-04-19
Elgin, IL

rbrussell82

Member

No Backup Servers?

They don't have any backup servers to atleast run some of their systems? I would think they would have a backup for the systems the call centers and stores use. I would think those backups should be fairly easy to implement once disaster strikes.
mikenolan7
Premium Member
join:2005-06-07
Torrance, CA

mikenolan7

Premium Member

Flooding

They've got to be kidding, right? Dry out servers? What's a main data center doing less than several stories up in a flood zone? I grew up in a place with floods like that, when the water recedes everything is filled with mud. Servers under that water are boat anchors.

battleop
join:2005-09-28
00000

battleop

Member

Re: Flooding

I was thinking that they don't actually mean the datacenter it's self is under water. I figured that the surrounding utilities are under water. Transformers, gas mains, fiber slicks and remote terminals. A lot of sites use natural gas for their gen sets but flooding could disrupt the flow of natural gas. The gas company may have to turn off the gas because of fire hazards and you can't have an unlimited supply of fuel to run the gen sets. Then I found this.

"The progress: as of 6:00 am this morning, power was restored and the long process of rebooting the systems has began again. Proof of the progress can be seen in the main corporate website, t-mobile.com, being live once more. As more applications come online, retailers can expect to see their Samson, Watson, and iCam systems functioning normally. The earliest estimates put these all being corrected around 15:00 PST, however it may be up to 48 hours. Thankfully, though some IT staff did have to relocate during the flood, no one was injured from within the company, and additional staff have already been sent in to facilitate with bringing it all back online."

CJ145
join:2002-12-17

1 edit

CJ145

Member

Time to restore servers

quote:
Any techies have a clue how long it takes to dry out servers and get them back online
If the servers are actually wet say bye to most of them. Even if they cut power and somehow all the electronics survived it takes days to dry out a building (never mind to air dry a computer case... it takes days). If HDD's are submerged they are most likely toast (water gets inside em).

hamburglar
join:2002-04-29
united state

hamburglar

Member

Amazing

WOW, no DR plan? A global company like that and they don't have replicated data centers and remote terminal capability. Someone is going to lose their job over this.

spamd
Premium Member
join:2001-04-22
Cherry Valley, IL

spamd

Premium Member

Re: Amazing

said by hamburglar:

WOW, no DR plan? A global company like that and they don't have replicated data centers and remote terminal capability. Someone is going to lose their job over this.
It's not a matter of someone losing their job. I'll bet the Network/Server or Backup Admins are screaming for this type of recovery over at TMO. It's just that upper management like the CIO simply thinks it's a waste of money and or the share holders thinks it's also a waste.

hamburglar
join:2002-04-29
united state

hamburglar

Member

Re: Amazing

Good point, but this is 2007 after all.
moonpuppy (banned)
join:2000-08-21
Glen Burnie, MD

moonpuppy (banned)

Member

Re: Amazing

said by hamburglar:

Good point, but this is 2007 after all.
Yeah, so?

Mr. IT guy meet Mr. Accountant.

MysticGogeta
The Robot Devil
Premium Member
join:2005-03-14
Katy, TX

MysticGogeta

Premium Member

Re: Amazing

Mr. Accountant meet door I can't immagine how much money they will lose because of this.

spamd
Premium Member
join:2001-04-22
Cherry Valley, IL

spamd

Premium Member

Where's the redundancy???

Wikipedia deffinition.
»en.wikipedia.org/wiki/Redundancy

Again more and more BIG MONEY business still not getting the hint. If your eggs are all in one basket, how are you going to explain outage issues to millions of customers.

The masses: Hey, TMO wheres my info!!

T-mobile: Well it's under water.. Please give us a few days to dry out please thank you.

Big business is going to be the down fall of America,, oh wait it all ready is!
Now where's my money green greed t-shirt.

==
dsless
join:2001-05-16
Pittsburgh, PA

dsless

Member

DR?

I guess T-Mobil doesn't have a DR plan. Let them answer that one to the stock holders. Could the users consider this a breech of contract?

Trinijoy
Premium Member
join:2005-09-12
Brick, NJ

Trinijoy

Premium Member

I love you people lol!

No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break.

hamburglar
join:2002-04-29
united state

hamburglar

Member

Re: I love you people lol!

For a company this size, there is no excuse. End of story.
soccerguy9
join:2004-06-28
Seattle, WA

1 edit

soccerguy9

Member

Re: I love you people lol!

I'm actually going to jump in on this...the Seattle area essentially got hit with three days of hurricane strength winds, with the associated rain and flooding, thousands out of power, whole towns cut off, people dead, freeways under water, people can't get to work, and you all seem to want to bitch about how T-Mobile's servers were down for a few hours.

I mean, get real. In a perfect world with unlimited IT dollars, yes they could have had a real time mirror or back up server, and I bet after this they will. But at the end of the day, it was not a really big deal. Phone service and internet stayed up, just customer service was down.

And no, I'm not a T-Mobile employee. Just a Seattle resident who is willing to cut a local company a little slack after the area has been repeatedly bitch-slapped by Mother Nature.
mikenolan7
Premium Member
join:2005-06-07
Torrance, CA

mikenolan7 to Trinijoy

Premium Member

to Trinijoy
I bet they have backups for their billing systems, just not for their customer's data.

spamd
Premium Member
join:2001-04-22
Cherry Valley, IL

spamd to Trinijoy

Premium Member

to Trinijoy
said by Trinijoy:

No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break.
Yeah, that's a crazy idea. Redundant offsite/hotsite recovery.
Just tell that to these guys.
»www.availability.sungard.com/

Again if your company has revenues in the billions there is no reason why you can't afford hotsite recovery...

hamburglar
join:2002-04-29
united state

hamburglar

Member

Re: I love you people lol!

You can do it a LOT cheaper than sungard, using virtualization.

Sircolby450
join:2005-11-26

Sircolby450 to Trinijoy

Member

to Trinijoy
said by Trinijoy:

No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break.
Whenever you are talking about this much important data, yes I would expect no less than that. If they don't have a backup, then that is just ignorance on their part.
mikenolan7
Premium Member
join:2005-06-07
Torrance, CA

mikenolan7 to Trinijoy

Premium Member

to Trinijoy
Prediction: customer data will be lost. Everyone will still get their bills right on time. The servers with the billing data are either on higher floors, or somewhere else, and have backups. The customer data isn't important, the cash flow is.

The frustration you see here is not about unavoidable events. It is about this attitude that the megacorps display again and again.
hottboiinnc4
ME
join:2003-10-15
Cleveland, OH

hottboiinnc4

Member

Re: I love you people lol!

I wouldnt count on that. I have TM and my bill is all messed up right now. I have a new Flex account and yesturday it said i was out of minutes and i couldnt use my phone since my flex bucket was empty. today i could use my phone for a few minutes. now all of a sudden its saying that i'm out of minutes and i need to put more on my account---even for myFav5

The system is actually down and its going to be a mess to fix.

battleop
join:2005-09-28
00000

battleop to Trinijoy

Member

to Trinijoy
People come to BBR for one reason. To bitch. How many of the ones bitching in this thread are even TMobil customers? Probably none. They are bitching for the sake of bitching. Bitching about ISPs, Cable Companies, and Telcos is the popular thing to do at BBR.
hottboiinnc4
ME
join:2003-10-15
Cleveland, OH

hottboiinnc4

Member

Re: I love you people lol!

I'm a customer and i'm not bitching. This can happen to anyone really.

I'm just surprised that the Data center didnt have mirrors to another states where they have other call centers or they don't have a backup of some type over at their Parent company.

But if the issue affects my ability to use my phone after they just charged my visa card i will be bitching to them once they open back up.

Sircolby450
join:2005-11-26

Sircolby450 to battleop

Member

to battleop
said by battleop:

People come to BBR for one reason. To bitch. How many of the ones bitching in this thread are even TMobil customers? Probably none. They are bitching for the sake of bitching. Bitching about ISPs, Cable Companies, and Telcos is the popular thing to do at BBR.
Then you have the people that bitch about the people that bitch. *cough*

CConverse
join:2006-01-31
Syracuse, NY

CConverse

Member

Re: I love you people lol!

LOL

danoNabq
@comcast.net

danoNabq to Trinijoy

Anon

to Trinijoy
I agree - company of this size, has no excuse for not having a solid DR plan in place. In fact, I worked with a really large insurance company in that area, and are they having this problem? Nope, they have a remote DR site...and they are smaller, customer wise, than t-Mo, or DTAG as a whole.

Im wondering why i am a t-mo customer now....
69742511 (banned)
join:2003-08-09
Costa Mesa, CA

69742511 (banned)

Member

All I can say is...

Thank God I had my refund check mailed out from them last Wednesday 11/28. I don't have to deal with T-Mobile anymore ... or this flooding mess.

hairspring
join:2007-11-23
Oakville, ON

1 recommendation

hairspring

Member

Where is the Disaster Recovery Path?

Surely T-Mobile can afford standby servers for web apps and some replication technology for their databases. This is truly inexcusable.
sunny8294
Shqipe
join:2001-03-15
Localhost ;)

sunny8294

Member

Re: Where is the Disaster Recovery Path?

should I start praying to have their billing servers to be all under water I overused 300 mins. this month!!! Please god, where are you?
hottboiinnc4
ME
join:2003-10-15
Cleveland, OH

hottboiinnc4

Member

Re: Where is the Disaster Recovery Path?

I think its a good chance for that or there systems are really messed up. It claims I'm out of minute on my Flex account but still allows me to make calls LoL. With no money in my "overage plan". Just keeps giving me a recorded greeting saying to add money soon and thats it. yesterday it wouldnt let me call anyone out side of myFavs. This is very interesting

For anyone trying to get ahold of them the only way to get ahold of TM is to call 611 from your handset though. Their 800-TMobile number is not accepting calls.
brianiscool
join:2000-08-16
Tampa, FL

1 edit

brianiscool

Member

hah

Your balance $492.52 . Now $0.00 Thanks T-mobile
sunny8294
Shqipe
join:2001-03-15
Localhost ;)

sunny8294

Member

Re: hah

Not sure about tmobile but i'll sure be a happy man!

jgkolt
Premium Member
join:2004-02-21
Avon, OH

jgkolt

Premium Member

Truly inexcuseable but look at this Not through our eyes

Now I would like to start off by saying thayt they should in fact have a fail over system in place, maybe even a backup oversees that could take over.

BUT

My phone service worked fine on the UMA (voip) and tmobile network side. that is what matters for now.

They had the site up and running from a read only image within hours and it happened when most people are sleeping.

The whole area was flooded and they were told to evacuate YET THEY still got it up and running.

If they have a system down they may or may not have another ocmputer to load the data on. If the system is all setup and togather and they may need to pull all the data back from tape. THAT WILL TAKE A WHILE. Now it woul dbe nice is they had a sans array where they could pull the backup up quickly and restore the server but if that is in the mudd then they cant restore it.

It would still be nice to see that a server in another part of the country could failover. Obviously their major product, the phone server, is setup that way to be redundant as i didnt have an outage.

sapo
Cruising Down Memory Lane
Premium Member
join:2002-09-16
Sacramento, CA

sapo

Premium Member

Where is my duplicate post?

Oh surely I can't just continue one thread instead of reposting the same idea over and over again.
reelbigfish
join:2002-06-06
Boston, MA

reelbigfish

Member

T-Mobile isn't the only company....

You'd be surprised to see what some companies data centers are like. People pick some horrible places and/or buildings for data centers.

TMobile
@optonline.net

TMobile

Anon

Systems up and running

T-mobile systems are back up and running. They were fixed around 2:30PM Eastern Standard time. The building got flooded and the power was knocked out but the servers werent damaged.

a customer
@comcast.net

a customer

Anon

Re: Systems up and running

Not so fast. I added a new phone number to my account a few days back. They were not able to setup my voice mail due to the outage. They told me to call 611 in a couple of hours and they would set it up. Two days later they are still unable to setup my voice mail and in addition to that my number "is not showing up in the system" as of an hour ago. The funny bit is that after giving them my Social Security number all that they can tell me is that I owe them money.
hottboiinnc4
ME
join:2003-10-15
Cleveland, OH

hottboiinnc4

Member

Re: Systems up and running

I've been trying to get into my account since the night it happened and a good share of the systems are still down and they admit they are if you talk to someone that knows what they're talking about there. They'll even tell you that the Data Center was built on a flood plain and is in the basement.
otis_sh
join:2007-05-04
Ann Arbor, MI

1 edit

otis_sh

Member

Back Up

Just added money to my T-mobile prepaid account at approx 7:30PM or so. All seems to be working fine.

Not bad, considering the weather and all.
mAlfunkti0n
join:2003-12-16
Loveland, OH

mAlfunkti0n

Member

I left em ...

awhile back. I couldn't drive down a major interstate here in Cincinnati without dropping calls left and right. T-Mobile didn't care what experience I had, so screw em, I hope they bankrupt.

This kind of stuff is insane, there seems to have been NO planning for any type of disaster, but I doubt they care.

quetwo
That VoIP Guy
Premium Member
join:2004-09-04
East Lansing, MI

quetwo

Premium Member

DR Plans

Lots of places do have backup, and DR plans, they just don't have LIVE and Redundant systems. I bet none of their data got lost, they just didn't have connectivity to their data while the servers were down.

Why does everybody always freak out (they lost all the data!)? It might be the power (or other) situation is just causing them to be cautious about turning the systems back online.
mAlfunkti0n
join:2003-12-16
Loveland, OH

mAlfunkti0n

Member

Re: DR Plans

When you are a company the size of T-Mobile, you NEED redundant systems. No one being able to call customer support doesn't seem to be a big issue to you? You apparently have never worked a CS job then.
hottboiinnc4
ME
join:2003-10-15
Cleveland, OH

hottboiinnc4

Member

Re: DR Plans

You could and still can call Customer Service you have to use your cell phone and dial 611 their 800-Tmobile number wasnt working and the automated system was down for the most part.

pcdebb
birdbrain
Premium Member
join:2000-12-03
Brandon, FL
ARRIS DG1670

1 edit

pcdebb

Premium Member

Hmmmm

In the building I work in, one of their collections call centers occupies a few floors. a bunch of them were sent home today because their computer system was doa with no eta. that may also explain some wierd things with my service? and at this very moment the site is VERY slow. been waiting for a minute

La Luna
Fly With The Angels My Beloved Son Chris
Premium Member
join:2001-07-12
New Port Richey, FL

1 edit

La Luna

Premium Member

All fixed already?

Apparently, they had some kind of "backup" plan after all.

Back to our regularly scheduled programming, nothing to see here....

Get a Clue
@comcast.net

Get a Clue

Anon

Flooding

In response to your charming responses there is nothing perfect in this world disaster happens. T-mobile was there for the Hurricane Victims and there phones worked. Not to mention T-Mobile customers still could use there phone and all there features so no delay in service. Not all phone carriers have 24 hour customer service like Alltel and a few others. Yes Tmobile does have plans in place but even the best plans may not always work. But why put these type of trashy responses that is not right
page: 1 · 2 · next