|
No Backup Servers?They don't have any backup servers to atleast run some of their systems? I would think they would have a backup for the systems the call centers and stores use. I would think those backups should be fairly easy to implement once disaster strikes. | |
|
mikenolan7 Premium Member join:2005-06-07 Torrance, CA |
FloodingThey've got to be kidding, right? Dry out servers? What's a main data center doing less than several stories up in a flood zone? I grew up in a place with floods like that, when the water recedes everything is filled with mud. Servers under that water are boat anchors. | |
|
| |
Re: FloodingI was thinking that they don't actually mean the datacenter it's self is under water. I figured that the surrounding utilities are under water. Transformers, gas mains, fiber slicks and remote terminals. A lot of sites use natural gas for their gen sets but flooding could disrupt the flow of natural gas. The gas company may have to turn off the gas because of fire hazards and you can't have an unlimited supply of fuel to run the gen sets. Then I found this.
"The progress: as of 6:00 am this morning, power was restored and the long process of rebooting the systems has began again. Proof of the progress can be seen in the main corporate website, t-mobile.com, being live once more. As more applications come online, retailers can expect to see their Samson, Watson, and iCam systems functioning normally. The earliest estimates put these all being corrected around 15:00 PST, however it may be up to 48 hours. Thankfully, though some IT staff did have to relocate during the flood, no one was injured from within the company, and additional staff have already been sent in to facilitate with bringing it all back online." | |
|
1 edit |
CJ145
Member
2007-Dec-4 4:51 pm
Time to restore serversquote: Any techies have a clue how long it takes to dry out servers and get them back online
If the servers are actually wet say bye to most of them. Even if they cut power and somehow all the electronics survived it takes days to dry out a building (never mind to air dry a computer case... it takes days). If HDD's are submerged they are most likely toast (water gets inside em). | |
|
|
AmazingWOW, no DR plan? A global company like that and they don't have replicated data centers and remote terminal capability. Someone is going to lose their job over this. | |
|
| spamd Premium Member join:2001-04-22 Cherry Valley, IL |
spamd
Premium Member
2007-Dec-4 5:06 pm
Re: Amazingsaid by hamburglar:WOW, no DR plan? A global company like that and they don't have replicated data centers and remote terminal capability. Someone is going to lose their job over this. It's not a matter of someone losing their job. I'll bet the Network/Server or Backup Admins are screaming for this type of recovery over at TMO. It's just that upper management like the CIO simply thinks it's a waste of money and or the share holders thinks it's also a waste. | |
|
| | |
Re: AmazingGood point, but this is 2007 after all. | |
|
| | | moonpuppy (banned) join:2000-08-21 Glen Burnie, MD |
Re: Amazingsaid by hamburglar:Good point, but this is 2007 after all. Yeah, so? Mr. IT guy meet Mr. Accountant. | |
|
| | | | MysticGogetaThe Robot Devil Premium Member join:2005-03-14 Katy, TX |
Re: AmazingMr. Accountant meet door I can't immagine how much money they will lose because of this. | |
|
spamd Premium Member join:2001-04-22 Cherry Valley, IL |
spamd
Premium Member
2007-Dec-4 4:53 pm
Where's the redundancy???Wikipedia deffinition. » en.wikipedia.org/wiki/RedundancyAgain more and more BIG MONEY business still not getting the hint. If your eggs are all in one basket, how are you going to explain outage issues to millions of customers. The masses: Hey, TMO wheres my info!! T-mobile: Well it's under water.. Please give us a few days to dry out please thank you. Big business is going to be the down fall of America,, oh wait it all ready is! Now where's my money green greed t-shirt. == | |
|
dsless join:2001-05-16 Pittsburgh, PA |
dsless
Member
2007-Dec-4 4:53 pm
DR?I guess T-Mobil doesn't have a DR plan. Let them answer that one to the stock holders. Could the users consider this a breech of contract? | |
|
Trinijoy Premium Member join:2005-09-12 Brick, NJ |
Trinijoy
Premium Member
2007-Dec-4 4:53 pm
I love you people lol!No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break. | |
|
| |
Re: I love you people lol!For a company this size, there is no excuse. End of story. | |
|
| | 1 edit |
Re: I love you people lol!I'm actually going to jump in on this...the Seattle area essentially got hit with three days of hurricane strength winds, with the associated rain and flooding, thousands out of power, whole towns cut off, people dead, freeways under water, people can't get to work, and you all seem to want to bitch about how T-Mobile's servers were down for a few hours.
I mean, get real. In a perfect world with unlimited IT dollars, yes they could have had a real time mirror or back up server, and I bet after this they will. But at the end of the day, it was not a really big deal. Phone service and internet stayed up, just customer service was down.
And no, I'm not a T-Mobile employee. Just a Seattle resident who is willing to cut a local company a little slack after the area has been repeatedly bitch-slapped by Mother Nature. | |
|
| mikenolan7 Premium Member join:2005-06-07 Torrance, CA |
to Trinijoy
I bet they have backups for their billing systems, just not for their customer's data. | |
|
| spamd Premium Member join:2001-04-22 Cherry Valley, IL |
to Trinijoy
said by Trinijoy:No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break. Yeah, that's a crazy idea. Redundant offsite/hotsite recovery. Just tell that to these guys. » www.availability.sungard.com/Again if your company has revenues in the billions there is no reason why you can't afford hotsite recovery... | |
|
| | |
Re: I love you people lol!You can do it a LOT cheaper than sungard, using virtualization. | |
|
| |
to Trinijoy
said by Trinijoy:No matter how bad the situation is with anything, it never cease to amaze me that people like the posts above expect perfection and always a "never fail, always works" backup situation. Give me a break. Whenever you are talking about this much important data, yes I would expect no less than that. If they don't have a backup, then that is just ignorance on their part. | |
|
| mikenolan7 Premium Member join:2005-06-07 Torrance, CA |
to Trinijoy
Prediction: customer data will be lost. Everyone will still get their bills right on time. The servers with the billing data are either on higher floors, or somewhere else, and have backups. The customer data isn't important, the cash flow is.
The frustration you see here is not about unavoidable events. It is about this attitude that the megacorps display again and again. | |
|
| | |
Re: I love you people lol!I wouldnt count on that. I have TM and my bill is all messed up right now. I have a new Flex account and yesturday it said i was out of minutes and i couldnt use my phone since my flex bucket was empty. today i could use my phone for a few minutes. now all of a sudden its saying that i'm out of minutes and i need to put more on my account---even for myFav5
The system is actually down and its going to be a mess to fix. | |
|
| |
to Trinijoy
People come to BBR for one reason. To bitch. How many of the ones bitching in this thread are even TMobil customers? Probably none. They are bitching for the sake of bitching. Bitching about ISPs, Cable Companies, and Telcos is the popular thing to do at BBR. | |
|
| | |
Re: I love you people lol!I'm a customer and i'm not bitching. This can happen to anyone really.
I'm just surprised that the Data center didnt have mirrors to another states where they have other call centers or they don't have a backup of some type over at their Parent company.
But if the issue affects my ability to use my phone after they just charged my visa card i will be bitching to them once they open back up. | |
|
| | |
to battleop
said by battleop:People come to BBR for one reason. To bitch. How many of the ones bitching in this thread are even TMobil customers? Probably none. They are bitching for the sake of bitching. Bitching about ISPs, Cable Companies, and Telcos is the popular thing to do at BBR. Then you have the people that bitch about the people that bitch. *cough* | |
|
| | | |
Re: I love you people lol!LOL | |
|
| |
danoNabq to Trinijoy
Anon
2007-Dec-4 7:31 pm
to Trinijoy
I agree - company of this size, has no excuse for not having a solid DR plan in place. In fact, I worked with a really large insurance company in that area, and are they having this problem? Nope, they have a remote DR site...and they are smaller, customer wise, than t-Mo, or DTAG as a whole.
Im wondering why i am a t-mo customer now.... | |
|
69742511 (banned) join:2003-08-09 Costa Mesa, CA |
69742511 (banned)
Member
2007-Dec-4 4:58 pm
All I can say is...Thank God I had my refund check mailed out from them last Wednesday 11/28. I don't have to deal with T-Mobile anymore ... or this flooding mess. | |
|
1 recommendation |
Where is the Disaster Recovery Path?Surely T-Mobile can afford standby servers for web apps and some replication technology for their databases. This is truly inexcusable. | |
|
| |
Re: Where is the Disaster Recovery Path?should I start praying to have their billing servers to be all under water I overused 300 mins. this month!!! Please god, where are you? | |
|
| | |
Re: Where is the Disaster Recovery Path?I think its a good chance for that or there systems are really messed up. It claims I'm out of minute on my Flex account but still allows me to make calls LoL. With no money in my "overage plan". Just keeps giving me a recorded greeting saying to add money soon and thats it. yesterday it wouldnt let me call anyone out side of myFavs. This is very interesting
For anyone trying to get ahold of them the only way to get ahold of TM is to call 611 from your handset though. Their 800-TMobile number is not accepting calls. | |
|
1 edit |
hahYour balance $492.52 . Now $0.00 Thanks T-mobile | |
|
| |
Re: hahNot sure about tmobile but i'll sure be a happy man! | |
|
jgkolt Premium Member join:2004-02-21 Avon, OH |
jgkolt
Premium Member
2007-Dec-4 5:40 pm
Truly inexcuseable but look at this Not through our eyesNow I would like to start off by saying thayt they should in fact have a fail over system in place, maybe even a backup oversees that could take over.
BUT
My phone service worked fine on the UMA (voip) and tmobile network side. that is what matters for now.
They had the site up and running from a read only image within hours and it happened when most people are sleeping.
The whole area was flooded and they were told to evacuate YET THEY still got it up and running.
If they have a system down they may or may not have another ocmputer to load the data on. If the system is all setup and togather and they may need to pull all the data back from tape. THAT WILL TAKE A WHILE. Now it woul dbe nice is they had a sans array where they could pull the backup up quickly and restore the server but if that is in the mudd then they cant restore it.
It would still be nice to see that a server in another part of the country could failover. Obviously their major product, the phone server, is setup that way to be redundant as i didnt have an outage. | |
|
sapoCruising Down Memory Lane Premium Member join:2002-09-16 Sacramento, CA |
sapo
Premium Member
2007-Dec-4 5:47 pm
Where is my duplicate post?Oh surely I can't just continue one thread instead of reposting the same idea over and over again. | |
|
|
T-Mobile isn't the only company....You'd be surprised to see what some companies data centers are like. People pick some horrible places and/or buildings for data centers. | |
|
|
TMobile
Anon
2007-Dec-4 7:27 pm
Systems up and runningT-mobile systems are back up and running. They were fixed around 2:30PM Eastern Standard time. The building got flooded and the power was knocked out but the servers werent damaged. | |
|
| |
a customer
Anon
2007-Dec-5 6:36 pm
Re: Systems up and running Not so fast. I added a new phone number to my account a few days back. They were not able to setup my voice mail due to the outage. They told me to call 611 in a couple of hours and they would set it up. Two days later they are still unable to setup my voice mail and in addition to that my number "is not showing up in the system" as of an hour ago. The funny bit is that after giving them my Social Security number all that they can tell me is that I owe them money. | |
|
| | |
Re: Systems up and runningI've been trying to get into my account since the night it happened and a good share of the systems are still down and they admit they are if you talk to someone that knows what they're talking about there. They'll even tell you that the Data Center was built on a flood plain and is in the basement. | |
|
otis_sh join:2007-05-04 Ann Arbor, MI 1 edit |
Back UpJust added money to my T-mobile prepaid account at approx 7:30PM or so. All seems to be working fine.
Not bad, considering the weather and all. | |
|
|
I left em ...awhile back. I couldn't drive down a major interstate here in Cincinnati without dropping calls left and right. T-Mobile didn't care what experience I had, so screw em, I hope they bankrupt.
This kind of stuff is insane, there seems to have been NO planning for any type of disaster, but I doubt they care. | |
|
quetwoThat VoIP Guy Premium Member join:2004-09-04 East Lansing, MI |
quetwo
Premium Member
2007-Dec-4 7:37 pm
DR PlansLots of places do have backup, and DR plans, they just don't have LIVE and Redundant systems. I bet none of their data got lost, they just didn't have connectivity to their data while the servers were down.
Why does everybody always freak out (they lost all the data!)? It might be the power (or other) situation is just causing them to be cautious about turning the systems back online. | |
|
| |
Re: DR PlansWhen you are a company the size of T-Mobile, you NEED redundant systems. No one being able to call customer support doesn't seem to be a big issue to you? You apparently have never worked a CS job then. | |
|
| | |
Re: DR PlansYou could and still can call Customer Service you have to use your cell phone and dial 611 their 800-Tmobile number wasnt working and the automated system was down for the most part. | |
|
pcdebbbirdbrain Premium Member join:2000-12-03 Brandon, FL ARRIS DG1670
1 edit |
pcdebb
Premium Member
2007-Dec-4 8:16 pm
HmmmmIn the building I work in, one of their collections call centers occupies a few floors. a bunch of them were sent home today because their computer system was doa with no eta. that may also explain some wierd things with my service? and at this very moment the site is VERY slow. been waiting for a minute | |
|
La LunaFly With The Angels My Beloved Son Chris Premium Member join:2001-07-12 New Port Richey, FL 1 edit |
La Luna
Premium Member
2007-Dec-4 8:32 pm
All fixed already?Apparently, they had some kind of "backup" plan after all.
Back to our regularly scheduled programming, nothing to see here.... | |
|
|
Get a Clue
Anon
2007-Dec-4 9:44 pm
FloodingIn response to your charming responses there is nothing perfect in this world disaster happens. T-mobile was there for the Hurricane Victims and there phones worked. Not to mention T-Mobile customers still could use there phone and all there features so no delay in service. Not all phone carriers have 24 hour customer service like Alltel and a few others. Yes Tmobile does have plans in place but even the best plans may not always work. But why put these type of trashy responses that is not right | |
|
|
|