| [OT] network troubleshooting "tunnel vision" How do you guys avoid tunnel vision when troubleshooting a network issue?
For example, just a few days ago, there was a STP loop in our network. There are two switches that are connected to each other. Switch A is showing the mac address is on the port that's connected to switch B, and switch B is showing the mac address on the port that's connected to Switch A's port.
However, during the troubleshooting, I did not see that the mac address is bouncing "back and forth" in between the ports. Instead, I thought that the issue was caused by the private VLAN configuration on the port (the port is on isolated VLAN).
This cause a lot of frustrations on my side. How do you guys manage it?
Thanks. |
|
|
|
| you can eliminate tunnel vision by following the step by step process of elimination from Layer 1 to Layer 7... most can run up and down the list in their heads but long hours may sometimes interfere. Those situations you write down your steps and ask a colleague what's missing... 
K.I.S.S. IT should be modo for any network issues... |
|
aryobaPremium,MVM
join:2002-08-22
kudos:3 | reply to daaron
Re: [OT] network troubleshooting "tunnel vision" said by daaron :just a few days ago, there was a STP loop in our network. There are two switches that are connected to each other. Switch A is showing the mac address is on the port that's connected to switch B, and switch B is showing the mac address on the port that's connected to Switch A's port.
However, during the troubleshooting, I did not see that the mac address is bouncing "back and forth" in between the ports. Instead, I thought that the issue was caused by the private VLAN configuration on the port (the port is on isolated VLAN).
This cause a lot of frustrations on my side. How do you guys manage it?
Thanks.
Short answer: the network needs to be redesigned in proper way.
Long answer: setup Layer-3 switches as closed as possible to hosts; dual-homing Layer-2 switches; don't rely on Layer-2 network design much; switch virtualization should be considered. |
|
| reply to daaron
Nothing replaces experience. You learn from mistakes and trial and error. If you can retain what you learn as you go along you most of it will become second nature.
Pick a starting point, usually where it works, and work your way toward where it doesn't work. As Da Geek Kid said, start at layer 1 and work toward layer 7 though most of the time you know where the problem is by the time you get to layer 3.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company. |
|
| reply to daaron
1) follow a standard troubleshooting plan / steps
2) ask the stupid questions first
3) Document! Document! Document!
4) get advice / a second opinion wherever possible.
5) insert comment about ASS, U and ME -- all too often I find ppl take the "oh, it MUST be because of this, therefore..."
Myself included...
6) corollary of this is the comment about a tree in the forest -- keep the details straight but don't forget the overall
picture.
My 00000010bits.
Regards |
|
| reply to daaron
Just have 2 lists of information and reevaluate whenever a new piece of information hits the equasion.
What is a fact and what is an assumption. Do not draw conclusions, only do so when you have managed to resolve the issue and have all relevant facts. |
|
 TomS_Git-r-donePremium,MVM
join:2002-07-19
London, UK
kudos:4 | reply to daaron
Dont listen to the manager. Start your own investigation.
I had a case of my manager asking me to look at something particular with a customer, and it was totally unrelated.
Managers should stick to managing. Theres a reason they hire techs .. to do the technical stuff.  |
|
aryobaPremium,MVM
join:2002-08-22
kudos:3 | said by TomS_:Dont listen to the manager. Start your own investigation.
I had a case of my manager asking me to look at something particular with a customer, and it was totally unrelated.
Managers should stick to managing. Theres a reason they hire techs .. to do the technical stuff.
From my experience, the manager of technical departments has to be technical and understand the practical part of the technical environment. No manager is to be the manager in the first place if the person does not understand technical practical day-to-day part or is not technical at all.
Another perspective is to be trustful of and rely on their own senior or most technical person in the department. This means trust the judgement, back up the decision, and delegate the work details fully. If such does not happen, then either the manager or the senior technical person should not be within the same department/company.
Note that the manager does not have to be super technical knowing ins and outs of day-to-day practical technical details since that would be the job of the senior technical person. The manager however has to be technical enough to appreciate the day-to-day practical technical details and appreciate the senior technical person judgement/decision in order to talk to upper management of any issues.
|
|
| ah, how nice to see the mirage....
I guess they don't lie when they say grass is greener on the other side... |
|
TomS_Git-r-donePremium,MVM
join:2002-07-19
London, UK
kudos:4 | reply to aryoba
In a perfect world, every manager would know about the field they are managing. Doesnt always happen though. |
|
aryobaPremium,MVM
join:2002-08-22
kudos:3 | said by TomS_:In a perfect world, every manager would know about the field they are managing. Doesnt always happen though.
I agree unfortunately
Most of my previous companies had manager that either was not technical appropriate or was not managing at all. That was one of the reason why those companies are now behind me |
|
