verizongone
@verizon.net | reply to jsanders
Re: [trouble] 914 dsl down again
3/12 AGAIN
»914 at it again? |
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY
 ·Verizon Online DSL
| reply to jsanders
Seemingly rolled back to a steady state
As annoying as this is, the good news seems to be that once reported, the WP CO team now knows what to do to restore service quickly.
No if we only had a way to communicate to that team this issue when experienced  -- Level 1 techs clearly can't do that, and seem satisifed not doing what it takes to get the information into the hands of the right people right away. |
|
Pathfinder
Dazed Confused
Premium
join:2000-03-26
Mount Vernon, NY | reply to jsanders
Back to normal?
 |
|
raresticks
join:2007-07-31
White Plains, NY
| reply to jsanders
Normal line rates now being shown.
The takeaway here is that if enough pressure is brought quickly enough they WILL respond.
If you are SURE it is an outage let the call center know that you are calling to report a problem not to do troubleshooting. My experience has been if the escalation team gets enough calls like this they will take some action. If the escalation team does not receive those calls because the subscribers don't press the level 1 support to escalate, NOTHING will be logged properly and the issue continues until they do some sort of internal correlation which will be arbitrary at best. |
|
raresticks
join:2007-07-31
White Plains, NY
| reply to jsanders
Re: [trouble] 914 dsl down again
Yes, you are correct. My bad. I was reading the service history too fast as my head was spinning due to fact that I received a totally contradictory report from VZ no more than an hour ago (see below).
What Is It NOW??? VERIZON DOWN AGAIN IN 914
For all those impacted, please be advised that the official word is that they cannot (will not) even begin looking into the problem until Monday morning.
Why? 'There are no technicians or engineers on shift on Sunday. Only people to answer the phone at the CO.' Keep in mind this is coming from a major telecommunications company.
In my mind this translates into yes, we do have folks on call but we don't WANT to call them as we don't consider the problem worthy enough and besides we would have to pay them OT.
LAUGHABLE.... |
|
Pathfinder
Dazed Confused
Premium
join:2000-03-26
Mount Vernon, NY | reply to raresticks
Actually the outage on 3/15 was fixed Sunday 3/16 at @ 10:30 PM. |
|
raresticks
join:2007-07-31
White Plains, NY | reply to jsanders
Don't buy it.
The outage on 3/15-3/17 was reported on a 3/15 (a Saturday). Purportedly they were working on it continuously. In the end, service was restored only on Monday 3/17. |
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY | reply to jsanders
Supervisor just told me that this has been recognized as an official outage in White Plains, NY (tell me something I don't know) and that engineers are working on it. No ETTR (of course). |
|
raresticks
join:2007-07-31
White Plains, NY
| reply to jsanders
RE: What is it NOW???
Just got of the phone with them. I was told there are not technicians or engineers onsite on a Sunday. As such they will not even START looking at the problem until tomorrow morning. 'The CO only has people to answer the phone today'.
This translates into 'we have senior people on call' but we don't want to call them as the problem is not worthy enough and we don't want to pay them OT.
|
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY | reply to blave
Yes, it's died again.
Logging my tech support request right now.
How can the VZ WP CO team keep f%$king this up |
|
nathan42100
join:2008-03-16
New Rochelle, NY | reply to jsanders
Yes, having trouble. The mobile speed test won't even finish a 50k test. Took me ~15 minutes just to get here from the home page. seems very eratic. Sometimes the page loads in >30seconds, sometimes it never finishes. |
|
raresticks
join:2007-07-31
White Plains, NY
| reply to blave
WHAT IS IT NOW????
914 down again, no question. Line rate here 176/722.
More than likely problem is NOT the local CO but rather a regional office where the routers are housed. The local CO from what I understand has nothing but passive DSL gear.
Looks like same issue as 3/16 debacle and 4/2 trainwreck! |
|
blave
join:2008-03-15 | reply to jsanders
Anyone having troubles again on April 6?
 |
|
Pathfinder
Dazed Confused
Premium
join:2000-03-26
Mount Vernon, NY | reply to altermatt
How is Gracie? Don't see her on here anymore. |
|
altermatt
Premium
join:2004-01-22
White Plains, NY
·Verizon Online DSL
| reply to BriGuy89
Excellent points, BriGuy89. My friend (and ex-DSLR MVP) Gracie does just that kind of corporate culture consulting and has said almost the same thing as she's struggled with her own connection, also in the 914 area. After the first of this series of outages, she was given an email addy of a district manager by a VZ insider she knows, and did email her, with no response.
From your description, it would actually be quite easy for the pattern identification to take place, since the detection and decision would be centered in the CO---all it requires, as you've said, is for the CO team to be alerted by the reps to each report of an outage that affects their area, and when it reached a magic number (which should be relatively low), investigate further. Just stupid to shoot themselves in the foot by ignroing this valuable first-line alert system: their users.
--
The truth of a thing is the feel of it, not the think of it. -- Stanley Kubrick |
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY
·Verizon Online DSL
| reply to altermatt
said by altermatt  :Verizon NEEDS to teach its tech support reps that when there's a sudden report from a user (and especially from a number of users) that seems unrelated to any changes in the user's computer, they should log it more carefully, and the logs made available more widely, so that if a pattern emerges (even 5 or 10 users calling from one area with identical symptoms) should tip you off that something MAY be going on) they will see it a lot sooner. The problem is the VZ CSRs have no ability to make a "pattern" pronouncement -- or even a suggestion. The VZ CO team has that responsibility. CSRs have no process for this, and are very hesitant to step on others' toes. CSRs are trained to let the VZ CO team make that determination. They are allowed to focus solely on Tier 1 triage only.
VZ CSRs have no ability to call and talk to anyone on the VZ CO team to alert them to possible emerging patterns. The only tool available to them to communicate with the CO team is via online chat.
More careful logging and such will make no difference if that detailed information is not presented to and considered seriously and carefully by the VZ CO team, which by all appearances has quite a cavalier attitude and doesn't want to be told how to do its job.
So as much as there's a clear lack of tools and processes, there's also likely a big cultural/political element to this as well -- hence my opinion that nothing's going to change much until someone in top VZ management exerts pressure from the top down. |
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY
·Verizon Online DSL
| reply to jsanders
As I've noodled about this outage and the other one a few weeks back -- and the VZ CO team's adamant denial of an issue until 24 hours passed -- I had the thought that this may not be a physical issue.
This is probably more likely the result of a misconfiguration. It would explain the "rapid" recurrence of the same exact issue in the same area.
Misconfigurations can be hard to detect. They reinforce denial (i.e., "we just updated the configuration files, so that can't be a problem"). And it doesn't matter which IT research firm you go with; they all concur that a significant majority -- numbers vary; I've seen ranges from 60 percent to 85 percent -- of network problems result from changes.
But we'll probably never know. |
|
altermatt
Premium
join:2004-01-22
White Plains, NY
·Verizon Online DSL
| reply to jsanders
Back up here: 
A few points from this series of outages:
1. Verizon NEEDS to teach its tech support reps that when there's a sudden report from a user (and especially from a number of users) that seems unrelated to any changes in the user's computer, they should log it more carefully, and the logs made available more widely, so that if a pattern emerges (even 5 or 10 users calling from one area with identical symptoms) should tip you off that something MAY be going on) they will see it a lot sooner. Both in the case of the first recent outage and this one, it took over a day for them to notice it, and even then:
2. Verizon NEEDS to start communicating more honestly and effectively with their users. Insisting there's no problem, refusing to post anything on the status page, etc. does NOT protect you from angry users---we KNOW there's an outage, we're living it. This stonewalling only fuels the irritation.
3. Premium support is just that: a PAID service and not available to regular users. Despite the helpful post here mentioning it, most of the time, these MUCH improved reps will have to refer you back to regular support. They provide a very nice level of service, but frankly---that's the kind of service I'd expect from regular tech support. Just acknowledging the problem, and reporting it, instead of jumping us through hoops.
Side note: would be interested to know if anyone else noticed spam during these outages when you usually don't get spam. I've been very pleased with VZ's spam filters and rarely see more than one a day, but during all three outages, a slew came in, mostly early on.
--
The truth of a thing is the feel of it, not the think of it. -- Stanley Kubrick |
|
BriGuy89
Let's Go Orange
join:2001-05-08
Yorktown Heights, NY
·Verizon Online DSL
| reply to Pathfinder
said by Pathfinder :I am really amazed that with monitoring equipment they aren't able to see this type of outage before 100 customers have to call in. History shows that outbound Internet performance problems are the Achilles' Heel of the VZ CO team in this area. I know of five instances in the last eight years that this exact performance-related problem has occurred, and the pattern is exactly the same: deny it is a network issue; refuse to correlate trouble tickets to see if a broader problem pattern exists; dispatch a technician to eliminate the user, line, and local CO as a potential source of the problem; eventually admit that an outbound Internet router was severely compromised but not completely fried, so it got missed as a problem by VZ monitoring systems. (Admittedly, many automated management systems have a very difficult time accurately and quickly picking up performance-related problems in compromised but not failed devices, such as routers.)
Clearly, this team lacks the tools to automatically identify this kind of performance problem. They likely lack the capital ($ and/or political) to make significantly improving the ability to identify this type of problem a priority. I sincerely hope that it is not that this CO team lacks perspective and has not learned from history.
But I also grudgingly admit this probably isn't the last time this type of problem will happen. |
|
