site Search:


 
   
story category
Google Suffers 'Embarrasing' Outage
Though it only impacted 14% of users, company says...
by Karl Bode Thursday 14-May-2009 tags: bandwidth · trouble · Google
Tipped by wifi4milez See Profile
Some of you might have noticed that Google today suffered through some outages impacting not only GMail, but Google news. While some might credit this to the coming bandwidth apocalypse, Google posted a brief explanation to their blog stating that the outage only impacted about 14% of users, while redefining vague with their explanation. "An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam," says Google SVP of Operations, Urs Hoelzle. "We've been working hard to make our services ultrafast and "always on," so it's especially embarrassing when a glitch like this one happens," Hoelzle says.

view: topics flat text 
Post a:
page: 1 · 2

major marco
Res Firma Mitescere Nescit
Premium
join:2003-02-13
Stepford, CA

The Sky Has Fallen...

While some might credit this to the coming bandwidth apocalypse...
Yeah, the outage was all because of the ::LOOMING:: bandwidth apocalypse. The only people who will swallow that are clueless whores representatives and the corporate lobbyists who own them.

jwalk
jwalking

join:2008-02-04
Old Orchard Beach, ME

Re: The Sky Has Fallen...

agreed.....and...the bandwidth apocalypse makes me smile.
--
* don't fear the penguin & don't spit in the wind *

S_engineer
Premium
join:2007-05-16
Chicago, IL

Re: The Sky Has Fallen...

said by jwalk:

agreed.....and...the bandwidth apocalypse makes me smile.
if only we had listened to Craig Moffet...
bn1221

join:2009-04-29
Cortland, NY
Google was out for me yesterday and HP was out today.

Well, Google worked over my VZ DSL but not my MetroE. BGP issues?
iansltx

join:2007-02-19
Golden, CO
kudos:2

Re: The Sky Has Fallen...

Appeared to be. Qwest connections seemed to be hit pretty hard.
phills_suck

join:2004-10-11
Burlington, NJ

Me!

I was one of those 14%.

IowaMan
Premium
join:2008-08-21
Grinnell, IA
Reviews:
·Iowa Telecom
·Mediacom

Re: Me!

Me Too Here is where the traffic died
Tracing route to www.l.google.com [72.14.213.99]
over a maximum of 30 hops:

1 20 ms 7 ms 9 ms 12-215-25-254.client.mchsi.com [12.215.25.254]
2 15 ms 12 ms 11 ms 12-215-4-22.client.mchsi.com [12.215.4.22]
3 63 ms 60 ms 64 ms cr2.cgcil.ip.att.net [12.122.132.98]
4 75 ms 59 ms 63 ms cr2.st6wa.ip.att.net [12.122.31.130]
5 79 ms 77 ms 75 ms 12.122.146.157
6 * * * Request timed out.
7 * * * Request timed out.
8 274 ms * * 216.239.46.208
9 * * * Request timed out.
10 * * * Request timed out.
11 282 ms 283 ms * pv-in-f99.google.com [72.14.213.99]
12 * 381 ms * pv-in-f99.google.com [72.14.213.99]
13 381 ms * 268 ms pv-in-f99.google.com [72.14.213.99]

Trace complete.

Parogadi
What? Stop Looking At Me Like That
Premium
join:2003-03-31
Racine, WI

Re: Me!

Likewise, I had done a 100 ping to google test and got 66% packet loss, though the ping was within ms spec of normal.

Tzale
Proud Libertarian Conservative
Premium
join:2004-01-06
NYC Metro
said by phills_suck:

I was one of those 14%.
Me too.

cork1958
Cork
Premium
join:2000-02-26
said by phills_suck:

I was one of those 14%.
Me too.

Personally,
I think Google stinks anyway. Is no big thing when Google goes down for me.
--
The Firefox alternative.
»www.mozilla.org/projects/seamonkey/

RolteC
0h

join:2001-05-20
Fresh Meadows, NY
kudos:1

No wonder why I couldn't access Google / Gmail

hmm I see now why.
quatrix
Premium
join:2005-02-11
Davie, FL
kudos:2

Re: No wonder why I couldn't access Google / Gmail

Thanks for the insight.
cchhat01
The Guru

join:2001-05-01
Elmhurst, NY

Re: No wonder why I couldn't access Google / Gmail

said by quatrix:

Thanks for the insight.
I cracked up here! Seriously you guys make me laugh!
--
Chirag's Website

v35_pilot
Whoops, there goes another AMU
Premium
join:2005-12-12
Fayetteville, NY
said by RolteC:

hmm I see now why.
Yoda uses Gmail and posts to DSLR. Cool.

Uncle Paul

join:2003-02-04
USA
kudos:1

I wonder

I wonder if China enjoyed what they pulled off the wire today...

Matt
All noise, no signal.
Premium
join:2003-07-20
Jamestown, NC
kudos:12

Re: I wonder

said by Uncle Paul:

I wonder if China enjoyed what they pulled off the wire today...
Wow, what a racist comment. Is China all of Asia now?

The effected users were being routed through Japan. See this post: »Re: YouTube Down?

Sir Meowmix III

@205.255.240.x

Re: I wonder

No, just the biggest land-mass there...

morbo
Complete Your Transaction

join:2002-01-22
00000
Reviews:
·Charter

1 edit
said by Matt See ProfileWow, what a racist comment. Is China all of Asia now?


no, but it is one of two acceptable answers based on population density in Asia: China and India. the Maldives would have been an unacceptable answer.

major marco
Res Firma Mitescere Nescit
Premium
join:2003-02-13
Stepford, CA

Re: I wonder

said by morbo:

no, but it is one of two acceptable answers based on population density in Asia: China and India. the Maldives would have been an unacceptable answer.
LMAO - Like anyone else here even know what the Maldives are.

zalternate

join:2007-02-22
freedom land
Remember when Japan was called the Orient?

Matt
All noise, no signal.
Premium
join:2003-07-20
Jamestown, NC
kudos:12

More than one outage

They had more than one outage then. Some folks were being directed through Asia but me and a ton of others had Level 3 routing issues in Atlanta and Washington at the peering points.

»YouTube Down?

Karl Bode
News Guy
join:2000-03-02
kudos:29

Re: More than one outage

For about three hours, Google News wasn't available from anywhere as far as I can tell, so yeah...

ThrowDemsOut
If you can't convince 'em, confuse 'em
Premium
join:2002-03-03
Mullica Hill, NJ
kudos:4

Cause was IPV6 conversion causing network routing problems

»news.cnet.com/8301-1023_3-102411···1_3-0-20
Google this morning attempted to make changes to key Internet routing numbers--known as autonomous system numbers--as part of its ongoing transition from an older networking standard to a newer one called IPv6. An unknown "bug" inside Google's network involving some sort of hardware failure or glitch prevented Internet service providers from finding Google's new ASNs on the Internet--effectively sealing it off from many customers, he said.

AT&T or Verizon--appeared to be disproportionately hurt because large ISPs "peer" with Google, or interconnect their networks with Google's networks in order to improve speed and reduce bandwith costs, Alperovitch said.
They can use that old saying: "Temporary disruption; but permanent improvements".

Logan 5
Enjoying the Cataclysm
Premium,MVM
join:2001-05-25
Austin, TX
kudos:7

Apocalypse "now"?

It was the RIAA, Global Warming and Mark Cuban all working in concert to punish those evil "Googlealians"....
firewire9999

join:2004-07-11
Livonia, MI

Re: Apocalypse "now"?

Hmmmm you forgot the following to blame:

George W. Bush
Carl Rove
Dick Cheney
Republican Party (in general)

Logan 5
Enjoying the Cataclysm
Premium,MVM
join:2001-05-25
Austin, TX
kudos:7
Wait... can't forget the Illuminati as well as the cast of the smash Broadway play "GitMo on Ice" either....

Any others we can blame for this?

Doctor Four
My other vehicle is a TARDIS
Premium
join:2000-09-05
Dallas, TX

Re: Apocalypse "now"?

said by Logan 5:

Wait... can't forget the Illuminati as well as the cast of the smash Broadway play "GitMo on Ice" either....

Any others we can blame for this?

What about Rupert Murdoch?
--
"The trouble with computers, of course, is that they are very sophisticated idiots." - Doctor Who (from Robot)

C0deZer0
Oc'D To Rhythm And Police
Premium
join:2001-10-03
Davenport, FL
said by Logan 5:

Any others we can blame for this?
Yes.

Blame Canada.

»www.youtube.com/watch?v=nYSYipouABI

HarleyDude

join:2002-01-04
Kennett Square, PA

Re: Apocalypse "now"?

I don't care who you are, THATS FUNNY!

iam x
Sungazer
Premium
join:2005-02-23
said by Logan 5:

Any others we can blame for this?

Sure, the Rothschild's.
They are responsible for everything.
patcat88

join:2002-04-05
Jamaica, NY
kudos:1
said by Logan 5:

Wait... can't forget the Illuminati as well as the cast of the smash Broadway play "GitMo on Ice" either....

Any others we can blame for this?

Forgot about the Freemasons.
staticfive

join:2006-08-01
Santa Rosa, CA

Re: Apocalypse "now"?

Wow, did I miss something?? When did BBR become a soap box for right-wing pundits?

HarleyDude

join:2002-01-04
Kennett Square, PA

You all got it wrong

I has to be Global Warming..
zenafu

join:2007-06-12
Brooklyn, NY

Mo Knows All

"You know what I blame this on the breakdown of? Society."

-Mo Syzlak
ke4pym
Premium
join:2004-07-24
Charlotte, NC
Reviews:
·Verizon Broadban..
·VOIPo
·RoadRunner Cable
·Northland Cable ..
·Packet8

Never down ... HAH!

You know, Google Co. sure has had their share of outages over the past 6-8 months.

Seems like it started when one of their GSA reps was in our building touting how we should dump our in house Exchange platform (and Office) for their always on, always available system (yes, yes, I know, not quite the same thing ... but still..).

insomniac84

join:2002-01-03
Schererville, IN

What outage?

No problems at all in KC.
Stojko
Premium
join:2007-10-20
St John's NL

Re: What outage?

said by insomniac84:

No problems at all in KC.
No problems here either.

rob_in_chatt
Premium
join:2004-09-17
Chattanooga, TN

what outage?

no problems here in Chattanooga TN with google or GMail. it has been up and running all day as far as i can tell.

AnonDOG

@kaballero.com

No part of this is vague...

"An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam," says Google SVP or Operations, Urs Hoelzle.

No part of that quote is vague, unless you wanted IP routes and addresses. It says that they screwed up a route and sent some (probably a lot) of web traffic through Asia (during the day time in the Western world) and that choked a pipe in Asia...

So, I dunno, no part of that is vague to me... but what do I know, I'm not a journalist.
Maze

join:2006-02-02
Brooklyn, NY

1 edit

Include NYC

Me and my friend had no gmail here in Brooklyn / Manhattan. Both of us though were routed to a temporary unavailable errors and what to look for. But that did not fix the issue as google was having it. Like to add google.com was not resolving instantly but reported by my friend.

NY Tel
Premium
join:2004-04-09
Smithtown, NY
kudos:3

Google Suffers 'Embarrasing' Outage

I couldn't get to it from my work pc but was able to send and receive GMAIL from my Blackberry.

CharterandATT

@charter.com

Some up some down

Google was down earlier for my AT&T DSL, so I switched to my Charter connection and it worked fine, near Chattanooga, TN.
ShellMMG

join:2009-04-16
Grass Lake, MI

Talk about bad timing...

I've been having a signal war with Alltel/Verizon/Sprint (Alltel can't use Sprint towers to roam anymore...and I was affected), I'd just installed a Cradlepoint router and was working out the kinks...and Google goes down. I was tearing my hair out, thinking I'd screwed up yet ANOTHER setting, and it was all Google.

There ain't enough aspirin on the planet to erase THAT headache.

Michail
Premium
join:2000-08-02
Boynton Beach, FL
kudos:1

Saw it too

I was one of the 14% also.

Usually if google is down I just assume the entire Internet is. But no, OMG, MSN was there!

AnonDOG

@kaballero.net

Re: Saw it too

said by Michail:

I was one of the 14% also.

Usually if google is down I just assume the entire Internet is. But no, OMG, MSN was there!
Ayup, that sucks when it happens. Thankfully, Microsoft is gradually loosing ground to better programmers, better systems, and better software.

It will take time, but eventually they will be relegated to a place in the market which they deserve and have earned on their merits. It will always be a significant place, but it will not be the dominant place.
VerizonCynic

join:2006-10-25
Lakewood, CA

nostradamus sez....

Google was down totally for fios here in socal from 730 am to about 10am

look everyone claims 99.99% uptime. they dont like to say .o1% downtime. But you gotta squeeze that in somewhere. If humans touch it there is no such thing as 100% perfect.

Google is free and makes Yahoo and MS look like 1997 internet. (the pre-cambrian period)

We just need to have redundancy and backups built in. That is why I have two other email providers other than google. Just like verizon or earthlink or anyone else, they do tend to go down.

What is funny is when these "hiccups" (f-ups) come along now in any system (like oh say the financial system) we are all on their ass in 30 seconds with email, bloggs and now tweets. As soon as I saw google was down I said oh man will they have hell to pay today. Someone will face a firing squad.

plain old me

@comcast.net

outsourcing

Rerouted over Asia - coincidence? Maybe, maybe not?
It seems that's what you get from outsourcing to Asia.
You get people writing code, whose only code writing experience is that of copying their homework assignments from the guy next to them in school.
Somebody probably ended up hard-coding Asian routing information into some piece of software.
And since QA is either outsourced too or just skipped entirely these days, no one catches the mistake until after implementation...
ditka_b
Premium
join:2001-10-05
Barrington, IL

Google

Didn't pay their bill lol

Hpower
Roflmao

join:2000-06-08
Glendale, CA

Re: Google

lol that's what I thought too. "WTH SEND THE BILL ASAP!!!"

Saturday, 11-Feb 21:29:57 Terms of Use & Privacy | feedback | contact | Hosting by nac.net - DSL,Hosting & Co-lo
over 12.5 years online! © 1999-2012 dslreports.com.