dslreports logo
spacer

spacer
 
   
spc
story category
Navisite Outage
Weekend server move goes badly...
by Karl Bode 01:39PM Tuesday Nov 06 2007
Customers of hosting provider Navisite have been offline since Saturday. The company recently acquired Alabanza, and over the weekend scheduled the migration of much of their equipment. Like a similar nightmare for customers of Hostway last August, the routine maintenance has turned into a massive outage. The company has been posting updates on the outage to their website here.
quote:
A full complement of engineering resources have been committed to completing the overall migration and have been working around the clock to resolve overall service issues and to resolve specific problems. . . we regret the circumstances and apologize for significant inconvenience and remain positive that the long-term decision to migrate will ultimately be in our collective best interests.
Cynthia Brumfield of IP Democracy is one of many thousands of customers impacted by the outage. She called the company to inform them that she'd be heading to the center to retrieve servers and data, but was hung up on. She's heading to Andover armed with a video camera.

view:
topics flat nest 

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Never had this problem

When I take one of my web servers down I tell my clients that it will be down and I dunno when it'll be back up.

I also do this for free in most cases; pay me with chocolate chip cookies.

And I still host their gaming servers. Good source of free cookies!
--
"I have measured out my life with coffee spoons..." - T.S Eliot
Check Out the Tech Bench »johnball.wordpress.com/tech-benc ··· h-bench/
Ma blog: »www.johndball.com

n2jtx

join:2001-01-13
Glen Head, NY

Want To Bet?

Cynthia Brumfield of IP Democracy is one of many thousands of customers impacted by the outage. She called the company to inform them that she'd be heading to the center to retrieve servers and data, but was hung up on. She's heading to Andover armed with a video camera.
Want to bet they have her arrested for trespassing?
--
I support the right to keep and arm bears.

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Re: Want To Bet?

said by n2jtx:

Cynthia Brumfield of IP Democracy is one of many thousands of customers impacted by the outage. She called the company to inform them that she'd be heading to the center to retrieve servers and data, but was hung up on. She's heading to Andover armed with a video camera.
Want to bet they have her arrested for trespassing?
I'm not dumb, but who is this lady?
--
"I have measured out my life with coffee spoons..." - T.S Eliot
Check Out the Tech Bench »johnball.wordpress.com/tech-benc ··· h-bench/
Ma blog: »www.johndball.com
66466388

join:2006-11-22

Re: Want To Bet?

Cynthia Brumfield of IP Democracy

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Re: Want To Bet?

said by 66466388:

Cynthia Brumfield of IP Democracy
Thanks Captain Obvious, but what is she all about? What does she do?

Karl Bode
News Guy
join:2000-03-02
kudos:44

Re: Want To Bet?

She's just a cable industry analyst who has worked and a number of providers in the industry. She's been tracking Navisite's piss poor customer service over at her blog.

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Re: Want To Bet?

said by Karl Bode:

She's just a cable industry analyst who has worked and a number of providers in the industry. She's been tracking Navisite's piss poor customer service over at her blog.
Thanks Karl Bode See Profile
--
"I have measured out my life with coffee spoons..." - T.S Eliot
Check Out the Tech Bench »johnball.wordpress.com/tech-benc ··· h-bench/
Ma blog: »www.johndball.com
Alliancenet

join:2005-04-30
Canada eh?

Re: Want To Bet?

Cynthia Brumfield should learn to maintain offsite daily backups of her data instead of blaming all her problems on Navisite. What would she do if her servers were hacked or experienced catastrophic hardware failure?

If her data was as important as she claims perhaps there would be redundant infrastructure at another datacenter?

Although I feel for the Navisite clients still offline, perhaps they should consider re-engineering any critical services they maintain.

-T
axus

join:2001-06-18
Washington, DC
Reviews:
·Comcast

Re: Want To Bet?

You pay a hosting provider to do that stuff for you. They aren't doing what she paid them for, so she's taking her data to someone else who will do what they're paid to do. If things aren't up by the time she gets there, she's justified in doing it right?

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Re: Want To Bet?

said by 66466388:

you're welcome Genius Boy,

maybe you should have used the Google, instead of waiting to be spoonfed ...

its even a verb now, example of usage:
"in trying to find out who Cynthia Brumfield is, i googled the name"
Just attempting to promote conversation; that which is lost when everything is searched for online.
--
"I have measured out my life with coffee spoons..." - T.S Eliot
Check Out the Tech Bench »johnball.wordpress.com/tech-benc ··· h-bench/
Ma blog: »www.johndball.com
66466388

join:2006-11-22

Re: Want To Bet?

maybe you should hang out with real people
not spending time "promoting conversation" ... ONLINE.

we dont live in "the matrix" ...yet

exocet_cm
Free at last, free at last
Premium
join:2003-03-23
kudos:3

Re: Want To Bet?

I hang out with real people all the time, its my job. I asked because
•I didn't know about this person
•I would like to converse with people on this forum

Next time, try to say something constructive.

This conversation is over.
--
"I have measured out my life with coffee spoons..." - T.S Eliot
Check Out the Tech Bench »johnball.wordpress.com/tech-benc ··· h-bench/
Ma blog: »www.johndball.com
66466388

join:2006-11-22

Re: Want To Bet?

ditto

Squeezers

@charter.com

Re: Want To Bet?

I know former Navisite employees. And what is truly funny is they laid off everyone but the low paid Google masters and high-overturn India employees.

This is so funny, I'm sending this on to the former Navi employees I know. They will get a laugh out this.
rantou

join:2002-06-04
Wylie, TX

From my experiences with Navisite...

This is quite an amusing thing to read. I can't believe that a company which decorates its datacenter in Dallas with (fake) plants is actually having so many problems...

Wait... I forgot.. this place doesn't have 24 hour operations technicians! They instead have a rent-a-cop and an electrician (if they're lucky) on site at night. At least the rent-a-cop and the hand-scanner will keep somebody from cutting through their walls (ala CI Host).

But seriously, a company like Navisite that doesn't man their DCs 24 hours with at least 2 NOC technicians shouldn't have attempted this.

sporkme
drop the crantini and move it, sister
Premium,MVM
join:2000-07-01
Morristown, NJ

Re: From my experiences with Navisite...

said by rantou:

But seriously, a company like Navisite that doesn't man their DCs 24 hours with at least 2 NOC technicians shouldn't have attempted this.
That is odd. The company that hosts DSLR, NAC.net, has always had someone on site 24/7, generally more than one person (they have two locations separated by a few miles in NJ). They are pretty big, but not a huge operation... How do they do it when they are relatively small compared to these uber-hosters?

Perhaps the "$5/month" hosting craze has lowered expecations...
rantou

join:2002-06-04
Wylie, TX

Re: From my experiences with Navisite...

I have all of my stuff at the practically no-name facility in Dallas, Colo4Dallas, and it is all hand-scan, keychain-wand, and it is manned by an L3 technician (usually not the best, but at least somebody who can do changes including BGP changes) overnight. I have been a lot more impressed with their facilities than any of the others in the Dallas Infomart because while the Infomart has all the big name clients, multiple fiber routes, multiple power routes, etc., Colo4Dallas has been able to achieve the same things, but they are also in a single-tenant building which they own, and it's not like it's anything fancy on the outside -- it's definitely pretty impressive inside.

simplykristi
Cancer Sucks
Premium
join:2001-11-28
Blue Springs, MO
I pay $29.95 for webhosting with HostCaters. My webhost is fantastic and unfortunately she got caught up in this fiasco. I will never change providers because of the fantastic customer service I get from my webhost. In the six-plus years I've been with HostCaters, this is the longest outage we've ever had.

Kristi
--
My Photo Gallery: »www.simplykristi.smugmug.com/

LBDSL
Lightning Bolt

join:2002-01-07
Auburn Hills, MI

Re: From my experiences with Navisite...

We have had a a server with Alabanza since 1999, and yes this is the longest outage. Alabanza has had great uptime, but this new ownership has changed a lot.
--
Lightning Bolt Technologies

LBDSL
Lightning Bolt

join:2002-01-07
Auburn Hills, MI
said by rantou:

This is quite an amusing thing to read. I can't believe that a company which decorates its datacenter in Dallas with (fake) plants is actually having so many problems...
Are you sure you are thinking of the right company? Navisite's NOC is in Andover, MA, at least the one where all the Alabanza servers where moved to.
--
Lightning Bolt Technologies
rantou

join:2002-06-04
Wylie, TX

Re: From my experiences with Navisite...

Sure thing. They have datacenters all over the country for colocation.

See:
»dallascolocation.navisite.com/to ··· allas-TX

And also look at the bottom right picture you can click on -- this is a view from the front door entry into the colocation space. Straight from there, if you walk toward that yellowish post on the left (just after the last cabinet) there is a planter right in front if it (Note the gray box in front of it). That is where people would stuff their trash while they were in there. I found a childrens toy, looked like it came from a kids meal of some kind, as well as some paper scraps (don't worry, no identifiable information on them).

But plants don't belong in a data center. Even if they're fake they don't belong there. Otherwise I would say it was a decent facility and it wasn't manned 24/7. The guy that was working nights when I was in there doing some contract work was an electrician by trade, employed by Navisite, and he had a number to their on-call technician, but that technician was in VA and had to be woken up at just after midnight, leading me to believe, at least at this time 2 years ago, they did not have 24/7 network operations, and especially not in Dallas since this guy that was working there barely knew how to pull up a web browser.

mbartlett

@quantumhydraulic.com

Navisite

The sites went down like they never existed- mail gone, everything gone. No warning, no notice, then continued obfuscation about their efforts and how long people would be affected.

Now they're acting like they don't know how many sites are affected, and trying to minimize the number. However, when they were bragging about acquiring Alabanza and Jupiter Hosting they proudly proclaimed 165,000 sites at Alabanza.
Here's a link to their press release:

»www.jupiterhosting.com/news/arti ··· ing.html

Better look at it quickly before they realize it's up and knock it off the Internet- they are excellent at taking pages down.

S_engineer
Premium
join:2007-05-16
Chicago, IL

Re: Navisite

Well surely they were smart enough to have a contingency.....

They're buying time while they consult with their attorneys....thanks Cindy
--
Where have the adults gone?
amungus
Premium
join:2004-11-26
America

sigh...

Like so many things in life...
if you fail to plan, you plan to fail...

If I were a customer, I'd already be with another hosting provider.

FFH5
Premium
join:2002-03-03
Tavistock NJ
kudos:5

Re: sigh...

said by amungus:

Like so many things in life...
if you fail to plan, you plan to fail...

If I were a customer, I'd already be with another hosting provider.
You might and I might switch. But many customers are with companies like this because they are CHEAP. And they stay CHEAP because the service they provide is minimal. And they don't have the staff on hand to deal with emergencies.

So, some customers may switch, but I'd bet they go to another provider just as bad because they don't want to pay any more than they already do.
--
Internet News
My BLOG
My Web Page

LBDSL
Lightning Bolt

join:2002-01-07
Auburn Hills, MI

Re: sigh...

Alabanza was not a "cheap" host, in fact most people who had/have servers there were paying thousands of Dollars a month, for Dedicated managed Servers.
--
Lightning Bolt Technologies
amungus
Premium
join:2004-11-26
America
Reviews:
·Cox HSI
·KCH Cable

Re: sigh...

hmm...
I won't get into the "cheap" argument here because I have no idea what they charged for what services.

If I was paying that kind of dough for a dedicated server, I'd be uploading my data right now to another host of ANY type.

Even a little text saying "we'll be back" would be better than nothing at all at this point.

LBDSL
Lightning Bolt

join:2002-01-07
Auburn Hills, MI

1 edit

Re: sigh...

I agree 100%, many host that are affected, have huge database driven sites, like DSLReports. If the owner can get to the data, there is no way to upload it to a new server.

We have backup's of all our sites hosted on all servers here at our office, so we are not in this position. However some host don't have local back-ups of thier whole servers. Stupid in my opionion, but that is a whole different conversation.

It would take me hours to cover the complete cluster that has been involved with this. Navisite dropped the ball, plain and simple, and many hosting companies that depend on thier servers are going to end up out of business or very close to it, due to lack of planning on Navisite's end.

I was talking with one host via a confrence bridge yesterday, and they were talking about one client they had who was running for some office, with the Election today, having that site down all weekend, and even today, may have hurt his ellection chances.

There was a reason people paid 3-4 times more for servers at Alabanza verses other cheaper data centers. But the new ownership, and management has shown/proven, the high price doesn't mean anything anymore.
--
Lightning Bolt Technologies

RARPSL

join:1999-12-08
Suffern, NY

Bad Migration Plan

What is so hard about the concept of keeping the Alabanza Equipment up-and-running during the migration? You set up the new hosts at the Navisite site, restore back-up images of the sites to Navisite hosts, set up the Navisite DNS servers, and then update both the Alabanza and Navisite DNS Servers to point at the Navisite versions of the hosts. After the TTL on the A Records expire, you can shut down the Alabanza hosts. A final Incremental backup of the Alabanza host (or a stopping of FTP Update activity) insures that the Navisite back-up created host is in sync with the live Alabanza version.

LBDSL
Lightning Bolt

join:2002-01-07
Auburn Hills, MI

If anyone wants inside info

Feel free to PM me, I have some info, as we do have one server with them.
--
Lightning Bolt Technologies

RCB

@atx.net

More than Just Alabanza

They also host mail for ATX/Broadview Network, who is my provider. My corporate email has been down sice Saturday!
Windogg

join:2002-07-24
Cambridge, MA

May head on over

I'm about a 20 minute drive from Andover, MA. I might just head over with a camera to snap pics of the woman with the camera getting tazed.

All kidding aside, Navisite seems to be playing the "How Not To Handle A Crisis" rulebook down to the letter. I know migrations can go wrong real fast but having a good communication and contingincy plan goes a long way toward maintaining good relationships. About the only thing they need to do to complete the charade is burn the place down.
unixwolf

join:2007-05-04
Flower Mound, TX

1 edit

Re: May head on over

Don't taze me Bro!!! shhhhreeeeeeeeeeeeeeeeeekkk!!!!

Albert V

@comcast.net

NaviSite Failure will cost them their stock (NAVI)

NaviSite is now Aproaching 100+ hours of downtime, considering that the data center move was to start at 10:00PM EST on Friday 11/2. It is obvious that many Alabanza and NaviSite clients are already out of business and those who have been lucky enough to have their services restored look forward to answer hundreds perhaps even thousands of complaints from their clients, most of which will be cancellations for sure. We alone have lost of 100+ clients due to this fiasco!

Has anyone yet been approached by a class action lawyer?

Heres the Offical WebhostingTalk post:
»www.webhostingtalk.com/showthrea ··· t=646844

simplykristi
Cancer Sucks
Premium
join:2001-11-28
Blue Springs, MO

My Website Was Down

I was one of the people impacted by this. My admin kept apologizing to all of her customers. I felt really bad for her. My site was down from sometime Sat. evening until yesterday morning. My mailserver for my domain is all messed up.

Kristi
--
My Photo Gallery: »www.simplykristi.smugmug.com/
vicorjh
Premium
join:2007-06-24
Arlington, MA

DDOS

From reading customer comments, it seems Navisite was attempting to claim DDOS attacks at one point during the outage...

Gimme a break. Navisite, you guys are really building up the customers trust...

ftthz
If love can kill hate can also save

join:2005-10-17

geez....

Think this is one of the worst down times I've seen ... hopefully people will be able to move on and that they did not loose too much and at the 3 day point have to wonder if its time to move on

John2020

@myvzw.com

The Great - The Bad and the Ugly

The Great:
I have about nearly 500 domains on 3 servers involved in this fiasco. I was with Alabanza since 1998.

Alabanza was a great company. They were expensive - not cheap as some have implied here. I gave them nearly 2000 PM for my 3 managed servers.

But, our servers were almost never down and if we were, it was almost always an upstream cause. The only significant outage I can remember in all those years involved about 6 hours down time several years ago caused by an underground electrical fire in which all power to Baltimore was cut for the safety of firefighters and Alabanza took good care of us and took steps to make sure it did not happen again.

The Bad:
Enter Navisite.... A company that bears little resemblance to the Alabanza I knew for so many years. Tom Cunningham must be crushed by what the new owners have done to the reputation he built for many years.

The Ugly:
Coming Soon at a hosting company in Andover, MA. I can only imagine what is headed their way.

Stamper

@metrocast.net

website down

My website is on their servers. I have had it since 1998 with no issues to date. I don't have any crucial data on the site - it is mostly an informational site for a retail business, but when customers call and ask of you are out of business because your website is down, you have to wonder how much business you are losing. I would love to hear about any class-action. What really got me going today was an "update" from one of the senior VPs who said that the people who are up on the new servers have noticed better service - which is reportedly "one of the reasons that they did this migration." I sent back an email and said that "I wouldn't know as my site has been down since saturday." I recommended that they concentrate their efforts on getting people back online instead of touting how wonderful the new servers are. No response, as I am sure you can imagine. We've made WBZ1030 AM news radio. Thier tech editor said that they "cheaped out" and tried to do this without leasing servers, transferring the sites, moving the physical servers, then transferring the information from the leased servers back to the moved ones. From what he said, they can't do it this way, so I am thinking that my site will never return. Time to look for a new host provider.
L00ker

join:2007-01-18

I worked there at one point...

For about a year as they outsourced all their NOC engineers/tech's to india, And their indian counterparts are COMPLETE morons... As an example a customer had a single webserver setup in their DC in Andover MA who had 2 websites on it. An alert (via their in house monitoring software) came in as "ICMP Ping failure (insert servername etc here" which is when the india NOC notified me "xyz customers server trapped a ICMP ping failure, we cannot RDP into it or connect to it" I thought "yeah well no ping=no rdp but whatever I will check it out" so AS I AM WALKING OUT TO THE SERVER the same "tech" called me to tell me (much to his surprise) that the web pages hosted are not loading so he opened up another ticket being so intelligent... When I asked him why on earth would he open a ticket up for a webserver not loading pages when it wasn't responding to pings or RDP sessions, and all he had to say is "I had no way of knowing if the two issues were related"... These are the people managing (mangling really) the Navisite infrastructure, I am AMAZED that they are still in business, secondly they have government systems they maintain for companies like Siemens Inc. that are remotely being managed by non-us citizens in india, which is COMPLETELY illegal... They are about as ethical as Dell...

minthost

@ssa.gov

Re: I worked there at one point...

The project Manager for the NaviSite/Alabanza datacenter migration is Jay Keating (jay.keating@navisite.com), You can thank him for this total mess!
posted by: Jay Keating is your man! | November 08, 2007 12:28PM

navishiite

@rr.com

Re: I worked there at one point...

FYI Jay Keatings email is now offline... so don't bother trying to email him at the navisite address.

Oh well... guess we'll just have to call in the Clowns! Oh I forgot... they're already all at Navisite in Andover, MD!
robroth

join:2005-04-16
Newburyport, MA

wow

My servers are colocated at Navisite in Andover. These are servers I own and are in a dedicated cabinet. I pay Navisite a boatload of $$ for this but now I'm glad I have my own hardware and do my own backups. I cannot imagine what you guys are going through. If this had happened to my servers, I would be out of business. Our software is used by Fortune 50 companies from 6am-9pm by hundreds of people all day long. If my websites went down, these companies would literally have hundreds of employees hanging around with absolutely no work to do. We'd almost certainly be out of business and our investors out millions. I had no idea about this fiasco until today, but it explains why my casual emails to Navisite personnel have gone unanswered.

ellives

@comcast.net

Navisite

Certainly Navisite made mistakes but I agree with those criticizing Cynthia Brumfield. She seems to be primarily out there drumming up exposure for herself. If she was so smart she would have her own backups as others have posted out there. Remember if you're the decision maker, the buck stops with you, even if you "pay people to do this for you."