Optical big iron
Ive been lucky lately to participate in a staging project for a DWDM line system. So I have a couple of pictures of the progress so far (1 week in.)
Typically this gear is used in long haul scenarios, however, this will be used for a high capacity metro system.
The configuration is a ring between 4 sites/nodes in a large European city. Each node has two degrees, facing two other nodes within the same city, and all together waves can be switched between and though any site to create any required connectivity.
We start with a photo of a rack containing two chassis/shelves, one (top) will house some muxponders, while the bottom shelf contains the amplifiers, WSS, and other associated modules for the line system (or degree.)
The next photo is a close up of the degree components with the shelf housing some of the node components on top, and some passive splitters and another module at the bottom (more on this later.)
And then we go a bit closer to the degree shelf. From left to right we have:
* Node controller card (top) and DC input filter (bottom) for A feed
* Pre-amp (if I recall correctly), amplifies the incoming optical signal
* Post-amp/booster, amplifies the outgoing optical signal
* Wave tracker (top) and blank (bottom)
* Redundant node controller (top) and DC input filter (bottom) for B feed
The black cables connected to the node controller form a ring between all shelves that make up the node. Although there is none connected to the redundant controller, for staging this is not necessary, but it will be installed in the production configuration. At the very top with the red cable is a panel that contains ports that give ethernet and serial access to the device for management.
Next is a close up of the splitter shelves. This is where the individual optical channels will be presented, and where the muxponders will connect. This particular system can do 88 channels (2 splitters with 44 channels each) of 100Gbit each (with current technology) - yeah, thats 8.8Tbit/sec total capacity. Each degree has its own set of splitter shelves, allowing the same wavelength to be used in two directions from a node for maximum capacity. The small module at the bottom is an interleaver. IIRC the transmit direction from each splitter shelf is fed in to this little box and combined in to a single output which is fed into the WSS, which can then selectively switch individual wavelengths in any direction.
Correction: the interleaver actually works in the opposite direction to what I stated above. All of the waves to be dropped at a terminal are fed through one port on the WSS, which feeds them in to the interleaver, which breaks them into their red and blue channel groups. From there they feed in to the input on their respective splitter shelf, which uses a prism to split out each individual wave and present it at its appropriate port.
Next is a photo of a shelf that will house muxponders. Theres not a lot to see as we havent installed any muxponders yet, but in total for this node there will be 4 muxponder shelves. The muxponder cards are 3 slots wide, and with 16 slots available that gives 5 muxponders in the current configuration. At 100Gbit (10x10) each, thats 500Gbit per shelf. Newer cards will be 2 slots wide, allowing for 800Gbit per shelf. Scary amounts of bandwidth!
And finally an overview of this entire node in staging (there are another 3 behind me.) Its a bit messy, but the primary goal of this staging is to test the hardware, upgrade software, configure, find any faulty components and replace them, and get it prepared to install in the production sites. The staging will probably take 3 weeks in total, and installation, along with all production commissioning and testing will take a further 3-4. The staging just makes it easier to get the system prepared, since all the nodes are in the same room and we can roll between them on our chairs rather than driving between sites. It also means less time needs to be spent overall in cold and noisy telco facilities.
Still to do, with an obvious lack of it, is fibre up the cards, and also run some inventory cables between the degree shelves and interleaver and splitter shelves, and configure and do some initial testing.
This is some really interesting and fun stuff to play with. Pity I dont get to do it in my every day job. :-(
Toms River, NJ
magnificent! thanks for sharing.
|reply to TomS_ | said by TomS_:
Scary amounts of bandwidth!
Here's my question... any contention / oversubscription in the gear itself?
None. Its all line rate.
The muxponders dont talk amongst themselves. All (de)multiplexing is done on the card, with the 100G signal transmitted and received via an on-card line port.
So no contention or oversubscription to worry about on the backplane (which isnt even used in this configuration except for system/management traffic.)
|reply to TomS_ |
That's just sick
Niiiiiiiice! I'd love to see more pics of this stuff fibered up if it's possible.
Silly question, as I've never even seen one of these rigs, let alone used them...
What are this big bastard cables going from the controller cards? (The big silver metal sheath ones)
They must weigh a ton on the connectors themselves?
They are the DC power cables, actually plugged in to the DC input filters. Those cards and the node controllers are half a slot in height, same as the wave tracker card.
Bit of weight to them from all the copper, but not really that heavy at all.
Should be able to get a pic next week of it fibred up, and of one of the muxponders.
|reply to TomS_ |
The equipment looks more like high end kitchen appliances than networking goodies. Not used to the silver, I guess.
|reply to TomS_ |
Gotta love those Alcatels. I snapped this at work earlier today. We have a few of them.
|reply to TomS_ |
Just HOW much bandwidth is that set up running JoeHemi
? I want my own private plug in.
|reply to TomS_ |
Wow. Now that is some REAL big iron right there.
|reply to HELLFIRE |
Problem with plugging in to this kind of kit is that it only gets you to the other end of the wavelength. You need to plug in to a router somewhere.
This is transmission kit, not routing kit.
edit: Although I do see some 100G and 40G muxponders in there (top left and middle shelves along with others.)
|reply to TomS_ |
I've been eyeing Ireland to move to, if I could work on stuff like that it would make my decision even easier. Spent 3 weeks there few months ago and loved every minute of it.
|reply to TomS_ |
How do you find the ALU stuff? I've put in a ton of Nortel/Cienna WSS gear, but don't have much experience with other vendors...
Nice work, btw
Optical stuff is not my day job, but so far it seems nice. Build quality is definitely up there. Got a couple of stubborn cards that dont seen to want to work though, and the guys I am working with tell me they had similar issues with a previous install they did.
It seems quite amazing to me that in this day, with alllll of the experience there must be designing and building this kind of equipment, there are still stupid issues and bugs. Its the one industry that never seems to learn from the past? lol
Ive done a bit of installation work of some optical systems and a little bit of operational maintenance on existing systems in my current job, but nothing with kit on this kind of scale.
This will probably just be a one off thing that I'll try and take in as much as I can from, then back to the real world.
It seems quite amazing to me that in this day, with alllll of the experience there must be designing and building this kind of equipment, there are still stupid issues and bugs.
That's just it. There isn't a lot of experience in these areas. The equipment is extremely expensive, so it doesn't find it's way into so many installs. And where it does land, it's installed in a very precise manner, specific to that one job. If you're the traveling engineer, it's unlikely you'll see the same installation twice. (unless you're working at the same company/site again.) And if it's like all the toys I've worked with over the years, you aren't working with them constantly and forget all the finer details of how to work with them.
It's not like Cisco IOS... been around almost unchanged for 20 years. It's found on an unimaginable number of devices that millions of people have used, openly documented, and continue to touch on a daily basis. When you have a problem, one of the first 3 google results for the error message you've gotten will have the solution.
If that ain't the truth.
There is a lot of carrier hardware out there that is like that.
The only people that know anything about it are the guys at the factory, the handfull of field engineers that work for the factory and the guys that are buried deep in the end user company that never see the light of day.
None of those groups talk to anyone outside that circle. The stuff is complex enough that most people wouldn't understand anyways.
"Above all, I would teach him to tell the truth. Truth-telling, I have found, is the key to responsible citizenship. The thousands of criminals I have seen in 40 years of law enforcement have had one thing in common: Every single one was a liar."
WiFiguruTo infinity... and beyondPremium
|reply to TomS_ |
Wow. Nice gear!
Surprised I don't see any Infinera gear posted on here. I may have to take a couple pictures.
|reply to TomS_ |
So a little more info and some pictures.
First is a block diagram of the connections between the components, illustrating how the system is "wired" up.
On the left we have our fibre pair coming in from another site. On the inwards direction this is fed into a pre-amp where the signal is amplified before being sent in to the WSS. A connection from here to the wave tracker (which is essentially a very basic OSA) allows us to see the power level of all wavelengths that are coming in to this amplifier.
On an extremely long span there may also be a Raman amplifier installed before the pre-amp. In terms of span length, 100km and longer is possible between amplifiers.
The WSS can work a lot of magic. If we want to drop a wave to terminate it at this site, it can be passed out of the DROP port at the bottom which feeds in to the ITLU (interleaver.) The ITLU splits the incoming wavelengths in to odd and even and feeds them into the appropriate splitter where a prism splits each wavelength out on to its appropriate port on the front of the panel.
If we want we can also allow the wavelength to express through this node, in which case it will pass to the THRU OUT port on the right and in to the adjacent degree and out on to the line towards the next site. The THRU ports are used specifically in a 2 degree node that may be mid span where you might need to drop some waves. In a situation where you have a multi-degree node, i.e. 3 or more degrees, you wouldn't use the THRU ports, but there are other ports that allow meshing of all of the degrees to allow waves to be switched individually as required.
In the other direction, waves are fed in to the top splitters which feed in to the WSS, which can switch them in various directions. One of which may be the SIG OUT direction, in which case the signals are sent through the post-amp and out on to the fibre to the adjacent site. We also have a feed in to the wave tracker here, so we can see the power levels of all outgoing channels. The equipment is quite good in that we can see the power levels of signals and channels in various stages through the system for troubleshooting etc.
As I understand it there is one limitation, in that you cant pick up a wave on one degree and send it through the THRU port to go out on to the line via another degree. Each degree should have its own set of splitters which will accept waves to be sent out in that direction.
With this diagram, if you combine two of these together, linked via the THRU ports, you have a 2 degree node, something that could sit mid-span on a long haul route, or in this case, in a ring.
Next is a picture of the degree wired up. I took this earlier in the week, have been busy with testing and commissioning.
A couple of notes about the picture and optics in general:
The two amps on the left have a couple of small patch leads at the top. These are actually the OSC (Optical Supervisory Channel) connections that link this node to the node in the opposing site connected via this degree. The OSC connection is an STM-1 in this case (but can also be 100BaseFX), and OSPF is enabled on. This means if there is a particular failure mode affecting management connectivity directly to this node, traffic can be routed via another node and over the OSC so you dont lose connectivity to the node.
There is also a very tiny loop on the post-amp, this is the "DCM" in this case. I am not sure of the particular reason why a "proper" DCM module is not used, the guys that designed it I guess figured that for a metro system it wasnt necessary. Im not an optical guy, so Im not even going to try and understand the mechanics behind it...
DCM is Dispersion Compensation Module, and basically it works like this. Different wavelengths travel at different speeds through fibre. If you transmit a pulse of light a the same time on 3 different wavelengths, then over distance what you would see is that one of them arrives before another, followed closely behind by the 3rd, and the pulses will have "stretched". Consider that high frequency pulses of light that start stretching, blurring, in to each other are not good for recovery of the original signal that was transmitted. Dispersion compensation corrects these effects so that by the time the signal is received its "almost brand new." Some very long fibre spans, e.g. submarine cables, use different types of fibre spliced together to counteract the effects en route, rather than using (but possibly in conjunction with) DCMs (which are basically just a long loop of fibre in a small module/box/cartridge) on either side. Simple explanation, but hopefully suffices (someone correct me if Im wrong!)
And in terms of amplification, it is not possible to simply amplify a signal infinitely. Different signals, usually governed by speed, need to be regenerated every so often. A 1Gbit signal can be amplified and transported much further than a 10Gbit signal, which will go further than 40 or 100Gbit. Each time a signal is amplified, noise is introduced which over time will begin to reduce the optical SNR, and once SNR gets too low it starts to become difficult to determine what the original signal was, and you end up with errors. Think of it like trying to listen to someone speak in a room that is gradually filling with people.
And finally, the black cable coming out of the top of the WSS unit is the inventory cable connecting to the ITLU at the bottom. An inventory cable then runs from the ITLU to the top splitter shelf, and then a 3rd from that splitter shelf to the bottom one. I havent asked exactly what the inventory connection does, or what kind of things happen over it, but I do know that it allows the system to "see" those particular devices, so I am guessing that there are some smarts involved with it.
And youre not seeing things, but none of the cards were plugged all the way in. When I took this photo (and the ones before) we were still in the early stages of initial commissioning, which involved upgrading the software on the first controller card, and then allowing the software to be upgraded on all the other shelves controller cards before we went and complicated things with lots of other modules.
Its a very precise step by step process, its not a plug and play environment!
Zzzzzz... whu.. huh? Sorry, tl;dr TomS_
Safe to say, it's a big layer 1 device with a bunch of internal gear to split and amplify light however you want, am I right?
|reply to TomS_ |
Now that some serious capacity.