how-to block ads
topics flat nest
|Comments on news posted 2013-01-22 12:18:33: AT&T customers in our forums note that AT&T has been experiencing a service outage across numerous states. .. |
Pompano Beach, FL
Re: Can this really be a single-point failure? Most of this is just speculation. (but I'm a fairly good at this.).
I suspect it's a u-verse boxen verses AT&T redundant centralized DHCP servers issue.
1. Since AT&T is required(for DCMA purposes) to keep track(log) of each users IP address at any given point in time. It usually requires a disk write or something, (which is going to be somewhat slow).
2. At&t now has hit ~6 million uverse, (my guess), boxes in the field, which is a significant number.. and when they loose their IP address, they hit on the DHCP server.. (Just a guess ..every ten seconds or so ..in order to shorten the downtime/reboot after a localized power failure.. )
3. Sometime on Sunday evening or Monday morning.. AT&T's redundant DHCP servers died, fell over/crashed and lost track of everything. They restarted(or where replaced by inferior backups), but they lost track of all prior DHCP assignments..
4. The ensuing mayhem.. DHCP servers and connecting data links are being hit by over 100,000 DHCP requests per second.
Net result High packet loss in the data links, techniques to confirm prior IP assignments randomly failing, duplicate IP assignments being handed out, just no way to handle that type of volume. (You really need ssd's and other fancy stuff.)
Result all sorts of mayhem for the customers.. ..
P.S.. Judging by the posts I've seen.. I'm betting that this has affected at least 30% to 50% of u-verse customer base. Not the 1% At&t claims.. (You really don't expect At&t to tell the truth??)
Why didn't this occur before. As long as the DHCP servers we up and data links to Texas remained intact as the network grew.. It was just small incremental chunks of traffic hitting the DHCP's at the same time. Which they could handle.
Now.. they got a Centralized hit, with millions of Boxes down.
Kappow.. That's a knock out hit, At&t's overall network/server design simply can't handle that type of massive numbers of requests from their u-verse boxes.
Disparate Service Providers, No Bundling I use a different company or service provider for basically every telecommunication service I have, even though it's more expensive than bundling all from one provider. There are two benefits to this. One is I like to choose the provider for the best product for the service I want in my area. For example, I have an AT&T land line because I want to have service if there's a power outage or an Internet outage. I have AT&T U-verse for Internet because I was led to believe by Consumer Reports that they have a more consistent mps during peak hours than Comcast; I have Comcast for TV because they have the best picture quality, at least compared to AT&T U-verse which I had for a year previously; although with U-verse I got a lot more for less money regarding HD programming, the picture would pixelate or scan jerkily whenever there were was action or a sweeping shot and faces would look watercolor-ish, especially when they moved even slowly. AT&T always tells you that it's a fiber optic cable system for their television, but that's true in most places only to the neighborhood hub which might be 1000 feet away and from there it goes through copper wires to your house. Whether the problem is because of that or their inherent transmission scheme that over compresses the signal, I could not get an explanation or a remedy when I had their service. Also, the U-verse TV remote control is rather unpredictable in it's operation as far as whether it registers or how long it takes to execute the command, but that was the lesser problem, as towards the end of the year I was getting somewhat adjusted to it. Although Comcast has a much richer movie like HD TV picture, as a company they are not exemplary either, having lured me back with more HD programming (maybe three quarters of what U-verse had) and then taking about half of the HD premium movie channels away a year later to devote the HD space to useless (to me) lower tier programs while still charging me the same high rate--I felt it was a bait and switch. I investigated satellite TV but being in South Florida I didn't want to take the chance of periodic stretches of interruption due to storms. Finally, I use Verizon for wireless because, again according to Consumer Reports, and other people I've spoken to, it's the best service in this area. The secondary benefit of not putting all my eggs in one basket is that if there is an outage in one particular service provider it doesn't affect the others.
Re: Disparate Service Providers, No Bundling I was just reporting what Consumer Reports said about those particular services. I don't believe they are right about many things, and in many fields seem to know nothing: taste tests, for example, or nutrition. Their product evaluations often contradict themselves and their overall rating doesn't seem to correlate with their individual feature ratings half the time and they offer no explanation for the higher or lower rating. I put more stock in their membership surveys of satisfaction with a service and that's what they reported for Verizon (not just customer service, but phone service). I don't base my decisions on Consumer Reports; I research elsewhere and I test and evaluate the actual product as best I can. Somethings like an Internet provider I can't do ahead of time, so I try to ask neighbors and others in the area. (Consumer Reports' comments about AT&T U-verse mps was based on their own limited testing.) I've noticed when they review televisions they do tend to give the best brands the highest ratings, but they don't really break it down as to what is better about certain pictures over others. I often wondered why they don't review audio equipment. To be honest if there's any field that needs standardized evaluation that's it, but to some degree personal preferences come into play, which is why I think it's odd that they purport to select the best coffee or olive oil. Also, to really test audio equipment and adhere to their policy of buying everything retail, for high-end audio that would mean spending as much as they do on cars, so it's just easier I guess to say everything sounds the same, which basically means they are not very discriminating, not a good thing for a products rating magazine.