Search:  

 
 
   News
newer
story category How Not To Run a Hosting Provider
Server move brings chaos for Hostway customers
(old news - 12:45PM Thursday Aug 02 2007)
tags: business · trouble
Tipped by Jafo232 See Profile
Our webhosting forum users say that what was supposed to be a routine scheduled migration of 3,700 Hostway servers from Miami to Tampa went terribly wrong last week. The original e-mail to customers promised the outage would be less than a day:
The 12-15 hour outage will take place beginning this Friday July 27th at 8 PM EDT. The outage time will be incremental for customers. Therefore, if your server is taken off line at 8 PM EDT, you should expect your server to be back on line between 8 AM and 11 AM on Saturday July 28th. The estimated completion of the project is July 28th at 7 PM EDT. If your server is still not responding at this time, please call technical support at 888.846.xxxx.
Instead, many customers were offline well into this week, and some are still having problems. Impacted users say the company has completely failed to communicate with customers during the outage. You can track the carnage in this multi-day thread over at the ValueWeb (now part of Hostway) official forum.

Related:
  1. Fairpoint Accused Of Faking Network Readiness Test
  2. Bankruptcy Judge Slams Charter Founder Allen
  3. AT&T's 'Blogger Guy' Faces Public Backlash
  4. CallCentric Suffers National VoIP Outage
  5. Slate Wants You To Pay More For iPhone Data
  6. Sidekick Users Get Outage AND Total Data Loss
  7. Unions Blame Verizon For Fairpoint Disaster
  8. CRTC Blocks Canada's WIND Wireless Network
Forums » How Not To Run a Hosting Provider
view: topics flat text 
Post a:

ninjatutle
Premium

join:2006-01-02
San Ramon, CA

>

Haha

nklb
Premium
join:2000-11-17
Ann Arbor, MI
clubs:

Migrations are difficult

When you have a diverse userbase, a migration of this sort is incredibly hectic.

When it's just your own services running on a machine, it's often easy enough to setup some sort of replication and then just take the primary offline and you're done, but not in a situation like this.

Periodic status updates, however, would have been a good idea.
--
for all your Linux questions

Jafo232
You Can't Spell Democrat Without Rat.
Premium
join:2002-10-17
Boonville, NY
·RoadRunner Cable

Re: Migrations are difficult

said by nklb See Profile :

When you have a diverse userbase, a migration of this sort is incredibly hectic.
I think it would have been better had they moved the servers incrementally, say in batches every weekend. At least after the first attempt, they would have learned what not to do on the second run..

Many of these were dedicated servers that people lease and then resell space on. There were many thousands of people affected by it, including myself.
--
Custom PHP/Perl Development. Vbulletin And Wordpress Mods Too!

stevek1949
We're not in Kansas anymore

join:2002-11-13
Virginia Beach, VA
·Verizon FIOS
·Verizon Online DSL

Blame the Benjamins

If the bean counters had a hand in the move (which they usually do), the lease on the old facility was probably up on the 1st of Aug, so they waited until the last weekend to move everything. Almost always a bad idea. Also the cost of renting a truck to move the servers, plus the labor on a weekend is at a premium.

I bet dollars to donuts that the almighty dollar had a part in this fiasco.

Jafo232
You Can't Spell Democrat Without Rat.
Premium
join:2002-10-17
Boonville, NY

Eweek has a good article

They point it out in a little more detail:

»www.eweek.com/article2/0,1759,2165290,00.asp

vaxvms
ferroequine fan
Premium
join:2005-03-01
Worcester, MA
·Charter Pipeline

Ignoring the users. Again

I was (past tense) a Hostway customer. I went thru an almost exact situation years ago. A scheduled 12 hour outage was still out 18 hours later. 12 hours after the scheduled back-in-operation time Hostway posted a notice saying Oops there was a problem and everything is working now. But it wasn't. Hostway was quiet for another 12 to 18 hours before posting a note saying There's a problem. It'll be fixed really soon. Soon=days.
The one time I needed to use tech support it was a nightmare. The technical skills required to be in support was knowing how to use an on/off switch.
--
Ferroequinologist

wdoa

join:2001-10-16
Spencer, MA
·Verizon Online DSL

deja vu

This reminds me of what happened a few years ago, when a small host I was a customer of got purchased by Interland. They promised a minimum of downtime, but it ended up being days. Apparently they just disconnected the servers in midwest, loaded them up on trucks and drove to Atlanta. After a few days the sites came back up, but they were so slow they were unusable. There tech people were unable to resolve issues and I think eventually all of the customers of the purchased web host moved elsewhere.

n1zuk
My wood is stacked
Premium
join:2001-10-24
South Burlington, VT
·Future Nine Corpor..
·ViaTalk
·Comcast

Re: deja vu

If you buy a hosting company, isn't the most important thing you are buyiny is those customers? It would of been just as easy (easier) to just purchase a few more servers. Seems they would try a little harder.

Reminds me of the JetBlue advertisement: If it wasn't for our customers, we would just be a bunch of planes flying TV sets around the country...
--
New to Forum Life? Click here and learn.

hse
High Speed Edition
Premium
join:2003-10-25
ON, Canada

3700 servers

Moving 3700 must be quite a job. Not something I'd like to be involved with.

natter

join:2000-12-18
Littleton, CO

nightmares...

I run a data center and can't imagine moving all those servers in a few days. The earlier poster was correct, it should have been phased over many weekends. I might have nightmares about this story tonight....

Maxeh
Woot?
Premium
join:2002-12-23
Chicago, IL
clubs:
·DSL EXTREME

Re: nightmares...

When we did our datacenter move into a larger building we spread it out over a 2 month period, only doing work nights/weekends during off peak times and only at the convienience of the customer. these guys seriously dropped the ball
--
Woot?

battleop

join:2005-09-28
00000

There is some truth to the saying...

You get what you pay for. I know it sucks, and it's no excuse but if your website is really as important as some claim they should invest in redundancy of their own.

sporkme
drop the crantini and move it, sister
Premium,MVM
join:2000-07-01
Morristown, NJ
·Optimum Online

Funny Karl!

a routine scheduled migration of 3,700 Hostway servers from Miami to Tampa
There is no such thing as a "routine" move of 3700 servers across Alligator Alley.

You'd think if they were moving, they'd not move from one hurricane-prone area to another...

Someone there must have been quite an optimist. I can't even imagine unracking and packing 3700 servers into trucks in 12 hours, much less trucking them across and then up the state, unpacking and re-racking. Wow.

computerman0

join:2006-07-19
Houston, TX

In house hosting for the win.

Well when I say in house I mean in my house. But yeah dumb plan all around. phasing and having moved a small batch of test servers would have been a good way to start and establish a good protocol to follow. no reason for this kind of stupidity.

P.S. these are the kind of people in IT who should never be allowed to touch a computer.

tuaris
You Clicked on the Apple

join:2001-10-19
Naples, FL

The hardest part is email

The hardest part of moving accounts is the email.

I once worked for a company that was in the middle of a migration, their where over 11,000 email accounts on one server that had to be moved to a dual server configuration.

It took 6 months to complete all the testing for an automatic migration using a perl script.
--
»www.pacyworld.com
Forums » How Not To Run a Hosting Provider


Tuesday, 10-Nov 11:43:48 Terms of Use | Privacy Policy | Hosting by www.nac.net - DSL,Hosting & Co-lo | feedback | contact
over 10 years online! © 1999-2009 dslreports.com.