TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
2 edits | TekSavvy Down (EDIT: Now back online) EDIT: we are back online now. the fire is out, literally!
We are down.
The latest news I have is that a battery blew up in our Peer1 colo space and that a fire had started. The situation appears to be under control, I'm told that they are trying to bypass the UPS' and go to direct power. Hopefully this is the case and we can get things online again. We are making our way there where we have help waiting to help if needed.
Sorry for the inconvenience.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
 |
 |
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down Just got word that we should start seeing some power shortly.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| |
pnjunction
Teksavvy Premium
Premium
join:2008-01-24
Toronto, ON
1 edit | So at least all of the fancy new equipment didn't go up in flames?  | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| Re: TekSavvy Down said by pnjunction  :So at least all of the fancy new equipment didn't go up in flames? The gear is fine. Everything is fine except there's no power. I don't have an ETA, but they are working to get the power online ASAP.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| |
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down New ETA, should be another hour as per Peer1's website... so roughly 6AM.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
follaz
join:2008-12-08 | Well. At least we get to know what's going n through this lovely community. Get well Teksavvy | |
|
HKPolice
join:2002-08-09
Scarborough, ON | Battery blew?? Why wasn't it changed regularly  | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down *shrug*
whatever it is. It's not good.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| |
| 
EG
Ho Ho Ho
Premium
join:2006-11-18
Union, NJ
| said by HKPolice :Battery blew?? Why wasn't it changed regularly Why assume a lack of maintenance right away ??
Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? | |
|
| | 
DKS
Damn Kidney Stones
Premium,ExMod 2002
join:2001-03-22
Owen Sound, ON
clubs:
 ·Bell Sympatico
| Re: TekSavvy Down said by EG :said by HKPolice :Battery blew?? Why wasn't it changed regularly Why assume a lack of maintenance right away ?? Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? There is anecdotal information that there has been a smell of burning electricals at peer 1 for a long time.
--
Need-based health care not greed-based health care. | |
|
| | TheMG
join:2007-09-04
Edmonton, AB
·TELUS
| Meanwhile in Alberta not even a hiccup on my connection.
said by EG :Why assume a lack of maintenance right away ?? Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? Is that just rumors or an actual fact from an official source that the batteries were at fault?
Anyways, I do believe the batteries for these large UPS units are lead-acid batteries (just like in your car). Those do not burst into flames when overcharged, they start spitting acid and hydrogen gas. Though with the amount of ventilation, it's unlikely hydrogen could have accumulated enough to burn/explode.
More likely there was an internal fault in the actual UPS. At the power levels we're dealing with here (several hundred kVAs) we're talking serious arcs, arc flash, and vaporized metal.
Anyways, I've seen the insides of such a UPS in person. Scary stuff, even just the sheer size of the bus bars!
Another fact many people don't know: the batteries are not normally in the same cabinet as the UPS. There's a separate cabinet for the batteries alone, which can often be in a totally separate room from the UPS itself. | |
|
| | | 
Guspaz
Guspaz
Premium,MVM
join:2001-11-05
Montreal, QC | Re: TekSavvy Down I believe that pretty much all UPS batteries are lead acid. In fact, my consumer grade APC unit is lead acid; all of APC's stuff is. | |
|
| | | | Davesnothere
join:2009-06-15
| Battery Types It's my understanding that they are a variant of Lead-Acid called a 'Gel Cell' battery and are supposed to be completely 'maintenance free' (cannot top up the electrolyte as it is not a free flowing liquid), unlike some lead-acid car batteries which were also known as 'maintenance free', though Could be topped up if you pried off some covers believing you needed to.
Those car batteries are now are called 'Low Maintenance', due to changes in / enforcement of consumer laws, I believe. | |
|
| | | | | 
jfmezei
Premium
join:2007-01-03
Beaconsfield, QC
·ELECTRONICBOX
1 edit | Re: Battery Types Many moons/centuries ago, I visited the Bell CO on Belmont street in Montreal. They still have crossbar equipment feeding McGill university back then. (this was one step ahead from the first generation rotary switches, and the first to support touch tone). And they still had one rack of rotary (step by step) switches, but they had just been made offline and not yet removed).
They had a large portion of a floor dedicated to batteries. They had semi transparent plastic enclosure, and you could see bubbles flowing up between the metal plates inside. Pretty neat sight.
When I worked for a bank, I did see their turbine generators on the top floor, but I don't recall their batteries. (that would have been about 10 years after visiting the Belmont one). When I visited the Whitehorse CO in 1989 they had more modern equipment there. (and got to meet some of the operators, some of whome I spoke t later on when making calling card calls | |
|
| | |
| | | Davesnothere
join:2009-06-15 | Re: TekSavvy Down Interesting.
I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out. | |
|
| | | | HeadSpinning
join:2005-05-29
Windsor, ON
| Re: TekSavvy Down said by Davesnothere :Interesting. I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out. Pregnant batteries in APC UPSs is a common problem.
I had always supsected they're running the float voltage too high. A few years ago, I was at a lunch seminar and was seated at a table with one of the founders of APC, and he basically confirmed (without actually saying it...) that some units have a problem with float voltage regulation, that might cause battery failure. | |
|
dpkform
join:2003-04-13
Toronto, ON
| no wonder I suddenly woke in the night and immediately tried to go online. I must have sensed a disturbance in the Teksavvy "force" as if a million connections, suddenly cried out and then vanished......
the evil empire wanted to force me under their control - I have had to tether to my Bell Mobility Mobile.........
do I go back to sleep, or should I look for my tin hat? | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down LOL, yes, the force made my cell phone freak out too
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| | follaz
join:2008-12-08 | Re: TekSavvy Down So are we all stealing our neighbor's wireless connection now? :P | |
|
| | MissM
join:2007-12-13
Toronto, ON | Hehe Rocky's TWitter freaked me out. I'm posting from my cell phone! I think it was B(H)ell trying to sabotage TSI!  | |
|
| | 
Jesse Hastie
@terago.net
| Well my cell rang off the hook as well, for other things.. mainly our Ontario MPLS network going up and down.. tried to logged into my VPN to check router logs and was like.. hmmmmmm
thank god for a Rogers stick... LOL.. 4
Knew something major happened when the teksavvy.com site was down.. | |
|
xxgg
join:2004-01-15 | it's back up now! | |
|
| follaz
join:2008-12-08 | Re: TekSavvy Down Yaaaay. I can breath again | |
|
Nonteksavvyuser
@rogers.com | There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these.
Sorry bout your luck | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| Re: TekSavvy Down said by Nonteksavvyuser :There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these. Sorry bout your luck The redundant equipment is there, just 2/3 setup. The last changes for the upgrades we've been doing were to deal with exactly these types of problems. Had this have happened a month from now we would have remained online for the most part.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| |
fkkejrbn
@rogers.com
| Re: TekSavvy Down said by TSI Marc :said by Nonteksavvyuser :There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these. Sorry bout your luck The redundant equipment is there, just 2/3 setup. The last changes for the upgrades we've been doing were to deal with exactly these types of problems. Had this have happened a month from now we would have remained online for the most part. Is the redundant equipment in the same building? What happens if 151 burns down?? | |
|
| | | 
R0CKY
TSI Rocky
Premium,VIP
join:2005-05-19
Chatham, ON | Re: TekSavvy Down If 151 Front burns down, you pretty much have a national telecommunications issue. | |
|
| | | |
jfgkhdufu
@rogers.com | Re: TekSavvy Down True, but wouldn't it make more sense to have the redundant equipment in a different location?? | |
|
| | | HeadSpinning
join:2005-05-29
Windsor, ON
| said by fkkejrbn :
Is the redundant equipment in the same building? What happens if 151 burns down??
You're more likely to have a major power issue or other problem in the building. Although possible with all that diesel on site, a large fire isn't that likely.
What is more likely is a small fire, causing the power (including backup generators and UPSs) to be shut down for safety.
Remember 220 Simcoe's fire years ago? | |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| We're on our way to Toronto to make sure everything is back online. Gabe and I are connected via our cellphones and tweaking things while in transit. Everything appears to be back online, still checking things though.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| as a side note, if you were connected to ERX's 1 through 5, you would have stayed online however could not have routed out. These sessions do not need to restart their modems if they had not disconnected.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
|
See 6 replies to this post |
|
AkFubar
Resistance is Futile
join:2005-02-28
Toronto Can. | Was the fire actually in TSI's suite? | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down yes. in fact, the last maintenance we did, we removed a bunch of servers from a rack that would have been just beside the fire as I've been told. I haven't seen it for myself yet though.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| |
AkFubar
Resistance is Futile
join:2005-02-28
Toronto Can. | Re: TekSavvy Down wow... fortunate no equipment was damaged | |
|
| | |
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| Re: TekSavvy Down said by AkFubar :wow... fortunate no equipment was damaged yeah, when i first heard about a fire. I was assuming the worst. I was freaking out pretty good. A bunch of people called and offered help if we needed it. That part was really nice to know we had options. They also kept us updated, a few were onsite and giving us updates. By the time I posted here, it seemed to be fairly obvious that no plane had crashed into the building and that we should be fairly safe. Our cage is completely at the opposite side of the room. I was also wondering with all the changes we had made, if it was somehow possible that our equipment could have started the fire. very weird. but I wrote that off quickly.. everything still shinny it's all so new.
anyway, I'm sure relieved at this point. it could easily have been worse. Even the time of night helped since the same thing during the day would have been much worse of a problem.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| | | | 
51019512
join:2009-05-19
151 Front St | Re: TekSavvy Down Is there anyway this could all be blame on poor rick and his bad luck with computers? | |
|
51019512
join:2009-05-19
151 Front St | I got one line to come up on 07. 2nd line keeps getting PADO Timeouts. | |
|
| |
| 
TSI Martin
Bilingual Tech Support
Premium
join:2006-02-23
Chatham, ON
1 edit | Re: TekSavvy Down My own connection
Status 2 of 2 Connected
MLPPP Multi-link
Connection Uptime 0 days, 01:12:05 | |
|
justen
join:2006-02-21
Montreal, QC
·TekSavvy Solutions..
1 edit | Well, I'm back. Partly. I used the downtime to call in and do the official MLPPP switch, so I'm now authenticating using copperdsl.ca instead of dslinternet.ca. Kind of impressive that, even given the disaster scenario, I got through to a rep in about one second. I guess if you're going to have your data centre catch on fire, it's best to have it happen early Sunday morning?
justen
(p.s. I, too, am only getting one link at the moment. Still getting "Timeout waiting for PADO packets" on the other link. Bummer. Going to leave the second modem off for another ten minute cycle, see if that helps.) | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
| Re: TekSavvy Down yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.
for the auth, we did just fix a problem with one of the radius. maybe that will help.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
| | justen
join:2006-02-21
Montreal, QC
·TekSavvy Solutions..
| Re: TekSavvy Down said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch.
justen | |
|
| | |
R0CKY
TSI Rocky
Premium,VIP
join:2005-05-19
Chatham, ON
| Re: TekSavvy Down said by justen :said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch. justen It has been a long night for all.... heading to eat some breakie with a few of them right now!
--
TSI Rocky - TekSavvy Solutions Inc.
Authorized TSI employee ( »TekSavvy FAQ »Official support in the forum )
| |
|
| | |
51019512
join:2009-05-19
151 Front St
| said by justen :said by TSI Marc :yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch. I hope you're providing extra coffee and other caffeinated drinks to said bunch. justen As TSI employees come into work a onsite nurse connects them via IV to a tube system that runs in the office so that caffeine can be pumped into the techs and reps so they can keep going without the need to stop for a coffee break. They tried to push food through the tube but it didn't work. | |
|
twizlar
I dont think so.
Premium
join:2003-12-24
Brantford, ON | Glad all the equipment is ok, that would have sucked majorly. | |
|
Kareeser
hm?
Premium
join:2006-07-18
Hamilton, ON | For Teksavvy, a 5 hour downtime is pretty damn amazing.
Well done! | |
|
davidl
join:2008-07-11
Vaudreuil-Dorion, QC | Connection is back!
Thanks to all at TekSavvy (and Peer1?) for getting on top of this so quickly on a Sunday morning | |
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | We're in the suite now. We can see where the fire was. everything appears to be online.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
justen
join:2006-02-21
Montreal, QC
·TekSavvy Solutions..
| Still having issues.
I'm currently running into this:
Or, alternatively, this:
If I swap the copper between the two modems, I can eventually get one or the other up, but so far I've not been able to re-establish multiple links. Even after multiple ten minute (or more) breaks for the errant (throttled?) modem, I'm still getting:
Any ideas?
justen | |
|
|
See 10 replies to this post |
|
pnjunction
Teksavvy Premium
Premium
join:2008-01-24
Toronto, ON | Is there where I complain that TSI should have redundant systems on the moon or at the bottom of the ocean? | |
|
|
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down I agree that multiple locations would be even better. for now, this was the best possible plan.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
|
sdfjsjdh
@rogers.com | how about just out of the down town core?? | |
|
| |
TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON | Re: TekSavvy Down this was already a massive upgrade, had we had it completed, it would have saved us from this problem. I think we're on the right line of thinking overall.
--
TSI Marc - TekSavvy Solutions Inc. | |
|
Trisomy21
join:2006-04-27
Kingston, ON | I bet Bell tried to Sabotage you. | |
|
|
See 9 replies to this post |
|
|
|
|
|
