TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| reply to TSI Marc
Re: TekSavvy Down
as a side note, if you were connected to ERX's 1 through 5, you would have stayed online however could not have routed out. These sessions do not need to restart their modems if they had not disconnected.
--
TSI Marc - TekSavvy Solutions Inc. |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| We have an environment monitor that we setup in the Peer1 cage. Looking through the logs it appears as though temperature had been rising over the last 7 days as well as air flow sensors were tripping in the 24 hours prior to the problem. Temperature just before the power problem was a whopping 84f. Temperature jut when things came back online was 69f and it's now back to 78f.
Based on this, it would appear to me that the root of the problem is A/C related or at least that A/C was a big part of the problem. We will be looking into moving some gear out of there once we get there. We'll give Peer1 another call to see what they have to say about it all.
--
TSI Marc - TekSavvy Solutions Inc. |
|
justen
join:2006-02-21
Montreal, QC | reply to TSI Marc
Stuck at "Timeout waiting for PADO packets" every little bit for a while now. I wonder how long until the RADIUS server(s) can cope?
justen |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| you're likely being denied by the DSLAM at this point. Turn off your equipment for 5 - 10 minutes and try again after. If you try too many times it'll start thinking that you're hammering the network and block you.
--
TSI Marc - TekSavvy Solutions Inc. |
|
exseven
Premium,VIP
join:2003-05-23
Beamsville, ON
1 edit | reply to TSI Marc
Wow that sucks guys. Are all your routing links back up to TorIX and Peer1? |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
1 edit | everything appears to back up at this point.
EDIT: yes, it sure does suck :\
--
TSI Marc - TekSavvy Solutions Inc. |
|
AkFubar
Resistance is Futile
join:2005-02-28
Toronto Can. | reply to TSI Marc
Was the fire actually in TSI's suite? |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | yes. in fact, the last maintenance we did, we removed a bunch of servers from a rack that would have been just beside the fire as I've been told. I haven't seen it for myself yet though.
--
TSI Marc - TekSavvy Solutions Inc. |
|
AkFubar
Resistance is Futile
join:2005-02-28
Toronto Can. | wow... fortunate no equipment was damaged |
|
51019512
join:2009-05-19
151 Front St | reply to TSI Marc
I got one line to come up on 07. 2nd line keeps getting PADO Timeouts. |
|
TSI Martin
Bilingual Tech Support
Premium
join:2006-02-23
Chatham, ON
1 edit | reply to 51019512
Re: TekSavvy Down
My own connection
Status 2 of 2 Connected
MLPPP Multi-link
Connection Uptime 0 days, 01:12:05 |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| reply to AkFubar
said by AkFubar  :wow... fortunate no equipment was damaged yeah, when i first heard about a fire. I was assuming the worst. I was freaking out pretty good. A bunch of people called and offered help if we needed it. That part was really nice to know we had options. They also kept us updated, a few were onsite and giving us updates. By the time I posted here, it seemed to be fairly obvious that no plane had crashed into the building and that we should be fairly safe. Our cage is completely at the opposite side of the room. I was also wondering with all the changes we had made, if it was somehow possible that our equipment could have started the fire. very weird. but I wrote that off quickly.. everything still shinny it's all so new.
anyway, I'm sure relieved at this point. it could easily have been worse. Even the time of night helped since the same thing during the day would have been much worse of a problem.
--
TSI Marc - TekSavvy Solutions Inc. |
|
justen
join:2006-02-21
Montreal, QC
 ·TekSavvy Solutions..
1 edit | reply to TSI Marc
Well, I'm back. Partly. I used the downtime to call in and do the official MLPPP switch, so I'm now authenticating using copperdsl.ca instead of dslinternet.ca. Kind of impressive that, even given the disaster scenario, I got through to a rep in about one second. I guess if you're going to have your data centre catch on fire, it's best to have it happen early Sunday morning?
justen
(p.s. I, too, am only getting one link at the moment. Still getting "Timeout waiting for PADO packets" on the other link. Bummer. Going to leave the second modem off for another ten minute cycle, see if that helps.) |
|
HeadSpinning
join:2005-05-29
Windsor, ON
| reply to TSI Marc
said by TSI Marc :you're likely being denied by the DSLAM at this point. Turn off your equipment for 5 - 10 minutes and try again after. If you try too many times it'll start thinking that you're hammering the network and block you. The DSLAMs operate at Layer 2, so technically,won't specifically allow or deny anything. The denial would be occurring further upstream at Bell's ERX. |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| reply to justen
yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.
for the auth, we did just fix a problem with one of the radius. maybe that will help.
--
TSI Marc - TekSavvy Solutions Inc. |
|
justen
join:2006-02-21
Montreal, QC
·TekSavvy Solutions..
| said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch. 
justen |
|
51019512
join:2009-05-19
151 Front St | reply to TSI Marc
Is there anyway this could all be blame on poor rick and his bad luck with computers? |
|
R0CKY
TSI Rocky
Premium,VIP
join:2005-05-19
Chatham, ON
| reply to justen
said by justen :said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch. justen It has been a long night for all.... heading to eat some breakie with a few of them right now!
--
TSI Rocky - TekSavvy Solutions Inc.
Authorized TSI employee ( »TekSavvy FAQ »Official support in the forum )
|
|
51019512
join:2009-05-19
151 Front St
| reply to justen
said by justen :said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch. justen As TSI employees come into work a onsite nurse connects them via IV to a tube system that runs in the office so that caffeine can be pumped into the techs and reps so they can keep going without the need to stop for a coffee break. They tried to push food through the tube but it didn't work. |
|
