how-to block ads
80.0 Network Design
You might not have the best network yet, however you have done a lot of work to keep things going. So far, the business network is up and running. But then you realize that network performance is not as it was used to be. Therefore you wonder, what happen? What cause the network to seem to go "slow"? And perhaps more important question is how you should improve the network performance.
Following discussion might remind yourself of your network with its current problem.
»Advice on Networking.
As solving any issue, you have to collect all necessary info to fully understand what happen on your network. In terms of network performance, you could have the following metrics in general.
* Line quality
* Network Device environment
* Network Design
* Bandwidth utilization
* CPU and memory utilization
* Software/Application performance
A first step of improving network performance is to set a baseline of how the network should perform based on metrics like the above. With that in mind, there should be a QA (Quality Assurance) or staging phase before anything is implemented into a production network. To accommodate potential "real" network slowness incidents such as delays and jitters, such should be introduced into the staging phase in order to have better understanding of how software or application's behavior is during network issues.
Once a baseline is set, you can assess to find out if there are illegitimate traffic or unexpected behavior in place or if you need to upgrade hardware, software, or services to meet your network requirement.
The term "line" here applies to all of the following
* Wiring and cabling between network devices (LAN)
* Circuit wiring (WAN)
LAN Wiring and Cabling
Let's say you have two network devices such as two servers. These servers are connected to each other using some kind of cable. In LAN technology, typically cables used are some Category 3 (RJ-11 analog telephone or POTS), Category 5/5E (RJ-45 10/100 Ethernet cable), or Category 6 (RJ 10/100/1000 Ethernet cable or LC/SC fiber cable). For best performance of connection between the two servers, the cable used should be Category 6 or fiber since Category 6 or fiber supports wider bandwidth.
When connection between these two servers go through wall or go between building, then typically there are patch panel and inside wiring (wiring within the wall) that are involved. You need to make sure that these patch panels and inside wiring support the same specification as the cable you use to connect the server and the patch panel. As example; if you use Category 6 cable to connect the server and the patch panel, then cables between patch panels and inside wiring also must match or exceed Category 6 specification. To make sure that end-to-end cable specification matches the Category 6 requirement, it is highly recommended to use special network gear such as Fluke meter to test the physical cable specifications.
»[Other] Cisco 6509 Possible Distance Issue
Beside using cable and wire to provide connections between the two servers, it is possible to use wireless technology. When such wireless technology is in place, you also need to make sure that the wireless connection support Category 6 specification. Since wireless connection is in general more prone to disruption than wired connection, there are more things involved. Check out the following FAQ for more info.
»Wireless Networking Forum FAQ
When connection between the two servers go over dispersed geographic location, then typically there are some WAN circuits involved such as DSL, Cable, T1/E1, ISDN, and the likes. Similarly, you have to make sure that these WAN circuits are up to their specification without degradations. Depending on how your network is setup, typically there are some kind of modem or router that provide WAN circuit connection. You might want to check the circuit quality by looking the modem or router reports on the line health. Check out the following FAQ for illustration.
»Cisco Forum FAQ »Technical Aspects in xDSL/Cable Internet connection
»Cisco Forum FAQ »Circuit Commission and Troubleshooting
Network Device Environment
As any machines, network devices generate heat. Therefore air flow is one of the key to keep network devices in healthy condition. One factor to keep good air flow is to make sure intake air is cool and exhaust air is hot. You also need to know how exactly your network device air flow works to make sure cool air is in and hot air is out.
Network devices also need to operate within certain temperature. Typically room temperature is preferable, however the device manual or official technical support is your best source. Certain room coolant like AC system is recommended to provide steady temperature.
Just like any electronic devices, network devices require power source. In addition, most network devices require power source with steady voltage and current. Such steady voltage and current coming from power source is called clean power source. Therefore network devices should never receive dirty power where the voltage and current are fluctuating. Typically you need some kind of UPS (Uninterrupted Power Supply) to ensure your network devices receive clean power source at all times.
Regularly maintain physical room where your network devices are is critical. The devices should never have dust bunnies, unknown gooey stuff, dirt, water, or anything that should not be there; since these kind of things are blocking the devices to have good air flow, to work within certain temperature, and to receive clean power source. Therefore typically the room where the network devices are should be well-insulated, dust free, and well-maintained.
Check out the following FAQ for more info
»Cisco Forum FAQ »Network Design Tips
»Cisco Forum FAQ »Tips in connecting hosts to switches
»Catalyst 3500 Series Dropping Like Flies?
Here is a list of common issues related to poorly design which can severely degrade network performance.
* Low reliability of network gears
* Poor cabling and physical interconnectivity
* Improper routing and switching design/implementation
* Poor concept and implementation of security zones
* Running untested/unbenchmarked software
* Incorrect software or operating system setup
Hire senior engineers to professionally review network design and setup is a good start to make decision moving forward. The review mechanism may involve vendors and telco providers in addition to lengthy and grueling process.
Bandwidth Utilization and Latency
There is a myth that in order to speed things up over network, simply upgrade the bandwidth as resolution. This statement is inaccurate since network bandwidth is only part of potential problem. The network latency instead is most likely be more of a concern rather than the bandwidth size.
As illustration, let's say your application is most likely using 2 Mbps bandwidth every time. If you currently have 10 Mbps bandwidth, upgrading the bandwidth from 10 Mbps to 100 Mbps with the intention to solve slowness issue may not be the best solution since 2 Mbps is still 20 percent bandwidth utilization of 10 Mbps pipe, which is well below used.
Your operating system, software, and all applications run over your network devices require certain bandwidth and/or latency threshold to work as expected. Consult vendor of the operating system, software, and all applications to find out how much bandwidth or latency threshold these systems require to normally operate, since the vendor should be the best source. Keep in mind that this step is essential to obtain baseline of how the systems should operate.
In addition, there should be a regular monitoring of how the bandwidth there is in your network is consumed. If the bandwidth utilization is high, find out if this utilization is caused by legitimate traffic and not caused by some virus or errors on the operating system, software, or any applications run over your network devices. When there is a baseline of how the systems should operate, your job of finding legitimate traffic will be easier.
When the bandwidth utilization is high and is caused by legitimate traffic, then upgrade the bandwidth is highly recommended to ensure the smooth operation of the system.
CPU and Memory Utilization
Your network devices are pretty much computers. Just like any computers, there are CPU (computer's brain power) and memory to process all traffic pass through the devices. Typically by running more and more software and applications, there will be a need for more brain power (CPU power) and larger memory.
Regular check of these CPU and memory utilization will suggest the current and possible future utilization. Similar to the bandwidth utilization, upgrading CPU and memory is highly recommended to ensure the smooth operation of the system when CPU and memory utilization is high and is caused by legitimate processes.
As mentioned briefly, there should be a baseline of all software and application run over the network. This baseline include the inner working of the software to find out what network to expect to ensure smooth operation of the software. Typical inner working within the baseline is how client-server relationship should occur within certain time period. In addition there should be a baseline how much bandwidth, CPU, and memory the software need to ensure the smooth operation.
Note that the software vendor might not be able to provide a precise baseline since networks are different from one organization to another. Therefore you might need some kind of software performance monitoring like OPNET and Cascade to find out exactly what happen during the client-server relationship and if those events happen as expected or not. The use of such software performance monitoring will also help you to establish baseline of your own network, which should be more precise than general baseline provided by the software vendor.
More info on performance monitoring
Check out following FAQ for more info on bandwidth, CPU, memory utilization monitoring and software/application performance monitoring.
»Cisco Forum FAQ »Automatic Network Health Monitoring and Reporting System: An Introduction
»Cisco switch into non-cisco switch, unknown initial lag
»Cisco Forum FAQ »Quick and Easy Subnetting on Routing, Switching and Network Design Relationship
In network design, there are several basic considerations such as scalability and reliability. Reliability is about stable and seamless communication between hosts. Scalability is about how a network growth is anticipated properly with minimal change.
To have scalable and reliable network design, all Layer 1 to Layer 3 good side aspects should be met. Following are certain factors that affect network scalability and reliability, from Layer 1 to Layer 3.
* Don't run cables on floor
* Don't wrap network cables and power cords into one bundle
* Don't use too-long or too-short cables
* Cables should run over (the ceiling), under (the tile), or inside (the wall)
* Have cable slacks between devices for easy and proper cable work space
Network Cable Choice
Use fiber cable around your building for best performance, scalability, and reliability. When you have multiple devices that employ various cable types (i.e. coax, Category 3, Category 5); you might want to have some kind of multiplexer to multiplex all of those various cable types into a single fiber cable.
When you do have to use Category 5 cables to interconnect devices, then it is suggested to use Category 6 instead of using Category 5 or 5E cables. Category 6 cables are more reliable and are flexible to any network environment.
When you have multiple network devices that need to be in the same room, it is then suggested to have dedicated racks for them. In addition, the rack itself should also be mountable to the floor and/or to the wall for steady standing position.
The assumption is that the network devices should also be rack mountable. If the network devices are not rack mountable, the devices then should be wall or desk mountable.
Between AC and DC Power
Most common network devices are probably AC-powered. By nature, AC-powered devices consume more energy and produce more heat than DC-powered devices. One big no-no on any network devices are having too-high temperature environment, which will shut down (or even melt down) the devices.
Whenever possible, use DC-powered network devices to reduce power bill. This is true especially when you need cooling system to cool down your network devices.
UPS (Uninterruptable Power Supply), Electrical Wiring, and Power Drop
Network devices are quite sensitive to "dirty power" that might affect resources such as CPU and memory chips. With UPS, the network devices would receive cleaner power and constant power supply with its battery backup.
Keep in mind that having UPS itself might not be sufficient. You may also need to verify your building electrical wiring and power drop from your Utility company.
As illustration, verify that the ground prong on your three-prong outlet is wired properly to the Earth. This way, you are sure you have proper power supply end to end.
To support a lot of users, some people like to span a large Layer-2 network. The network is usually looking like at least one of the following:
* Employ three or more switches in daisy-chain connection
* Employ more Layer-2 switches rather than Layer-3 switches or routers
* One VLAN is spread throughout at least almost the entire organization
* One VLAN covers large area
* Use subnet size larger than /24 on one VLAN
* Assign multiple subnets under the same Layer-3 interface
As illustration, the network setup could look like the following
»Cisco Forum FAQ »Resilient Network Tips for Small Businesses
When the Layer-2 network is a broadcast network, then there will be something called ARP broadcast. This ARP broadcast is used to establish Layer-2 communication among hosts within the same Layer-2 network. As any other traffic, ARP broadcast traffic consume bandwidth and network device resources such as CPU and memory.
The more hosts reside within the Layer-2 network, then there will be more ARP broadcast traffic take place. The larger ARP broadcast traffic take place, the less bandwidth and network device resource available for the actual communication between two hosts within the same Layer-2 network. As a note, a large ARP broadcast traffic is sometime referred as broadcast storm.
One way to reduce broadcast storm effect is to partition a Layer-2 network into smaller multiple Layer-3 networks. Usually it is the best approach to use Layer-3 switches to partition such Layer-2 network since you can keep the Layer-2 switching speed even though you are using Layer-3 routing to route traffic among the Layer-3 networks.
The Layer-2 network partition should also consider native VLAN partition (usually VLAN 1). When there are multiple Layer-2 switches, you don't really need to let native VLAN (or any VLAN) to spread across the entire network. You could just segment VLAN 1 into smaller multiple VLAN 1 networks and use Layer-3 switching to interconnect them. This way, any behavior change on native VLAN (due to maintenance or DOS attack) would not bog down the entire network.
Between Flat and Segmented Networks
In some small companies, a lot of time the network is setup to accommodate all hosts (i.e. PC, server, router, firewall, etc) within the same subnet. Some networks implement 172.16.0.0/16 subnet to accommodate all of those hosts. In this 172.16.0.0/16 subnet implementation; all PC, server, router, firewall, and all other hosts are having the same 172.16.xx.xx IP addresses; sharing the same default gateway. This network setup is called Flat Network.
Once the network is growing, there might be a need to migrate to Segmented Network. By having Segmented Network, the network is segmented into smaller networks or subnets where each subnet accommodate specific hosts based on functionality. As illustration, Segmented Network could have Management network (network management IP subnet within 172.16.0.0/24 subnet), Infrastructure network (where all routers, firewalls, and switches reside within 172.16.1.0/24 subnet), Server network (where all servers reside within 172.16.2.0/24 subnet), and User network (where all PC reside within 172.16.3.0/24). Each segmented network has its own default gateway and probably has its own physical switch or router.
When at least one of the following conditions meets, usually it is already time to migrate from Flat Network to Segmented Network.
* A lot of Broadcast Storm in place
* Some users are permitted to access servers and some other users don't
* Some users are only permitted Internet access without access to any internal servers or PC
* Public-accessible servers are not permitted to access internal not-for-public servers
* There is a need to setup firewall between users and servers for security purposes
You can check out the following FAQ for more info on Flat vs. Segmented Networks
»Cisco Forum FAQ »Running Out Of IP Addresses due to 'flat network' design
With Segmented Network, routing between networks and the Internet becomes significant. Keep in mind that a good Segmented Network design is in general is based on carefully-assigned subnet and good routing design.
Check out the following FAQ for sample configuration of Segmented Network
»Cisco Forum FAQ »Should I use Layer-3 switch or router?
Subnet Assignment and Contiguous Network
Let's review the following network design
The network design represents the network setup of an entire organization network. Router 1 is managing three networks. Similarly Router 2 is managing one network and Router 3 is managing two networks. All of these networks are broadcast networks.
There is also another broadcast network within the four routers themselves. Network between Router 4 and Firewall is point-to-point network. Network between the Internet (ISP) and Firewall could be any network type (point-to-point, broadcast, or non-broadcast).
To provide connectivity within the organization, a 192.168.0.0/24 subnet is used. Let's say for now that this subnet must be sufficient to support the entire network within the organization.
Let's say we have the following host number to support within each network
1st network: 30 hosts
2nd network: 20 hosts
3rd network: 10 hosts
4th network: 5 hosts
5th network: 3 hosts
Server farm: 12 hosts
Each network would be independent network. There will be IP routing to provide network interconnection and Internet access.
By subnet calculation, we have the following subnet size
1st network: /27 subnet to cover 30 hosts
Between Router 4 and Firewall
Host # : 2
Host # : 4
Let's look at Router 1. Since there are multiple networks behind the Router 1, it is a good idea to have supernet on the router to represent all the networks behind it. Similar concept applies to Router 3 that have multiple networks as well.
To supernet, the smaller networks should be in consecutive order such as 192.168.0.0/27 and 192.168.0.32/27 for 1st and 2nd networks respectively.
Let's review supernet consideration at Router 1. There are following networks behind the router
1st network: 192.168.0.0/27 (192.168.0.1 - 192.168.0.30)
2nd network: 192.168.0.32/27 (192.168.0.33 - 192.168.0.62)
3rd network: 192.168.0.64/28 (192.168.0.65 - 192.168.0.78)
As mentioned, you can supernet both /27 networks to be a single /26 network which is 192.168.0.0/26. With this supernet choice, Router 1 represents the following networks to the rest of the organization.
192.168.0.0/26 (192.168.0.1 - 192.168.0.62)
192.168.0.64/28 (192.168.0.65 - 192.168.0.78)
Let's say you want Router 1 to represent only a single network to the rest of organization. You could choose to supernet all networks behind Router 1 into a single /25 network, which would be 192.168.0.0/25 (192.168.0.1 - 192.168.0.126).
When you do this /25 network supernet, then you have to make sure that the following subnets are only behind the Router 1.
192.168.0.80/28 (192.168.0.81 - 192.168.0.94)
192.168.0.96/27 (192.168.0.97 - 192.168.0.126)
If one of these networks are not behind the Router 1 and there is 192.168.0.0/25 supernet on Router 1, the one network could become unreachable from the rest of the organization; which could lead to unreliable network.
From scalability perspective, you need to forecast if the above assigned subnets are sufficient to support all the three networks. If the 3rd network host number will grow to 30 within 2 years let's say, then it is probably a good idea to assign /27 network instead of /28 network to the 3rd network to anticipate the growth. This way, the 3rd network subnet assignment would be good until at least 2 years in the future.
Static and Dynamic Routing
Let's review the following network design
The 192.168.1.0/24, 192.168.2.0/24, and 192.168.3.0/24 are broadcast networks. 192.168.0.0/28 is point-to-point network.
192.168.1.0/24 is behind Router 1. 192.168.2.0/24 is behind Router 2. 192.168.3.0/24 is behind Router 3.
192.168.0.1 and 192.168.0.6 are Router 3. 192.168.0.2 is Router 1. 192.168.0.5 is Router 2.
To access the Internet from 192.168.1.0/24, 192.168.2.0/24, and 192.168.3.0/24 networks; the outbound traffic must pass Router 4. Only Router 3 has direct connection to Router 4. To access the Internet from 192.168.1.0/24 and 192.168.2.0/24 networks, the outbound traffic must pass Router 3.
Note that static routing should be sufficient to provide reliable connection within the entire network. Followings are the static routes.
Use 192.168.0.1 (Router 3) to reach any network outside 192.168.1.0/24
Use 192.168.0.6 (Router 3) to reach any network outside 192.168.2.0/24
Use 192.168.0.2 (Router 1) to reach 192.168.1.0/24
Use 192.168.0.5 (Router 2) to reach 192.168.2.0/24
Use 192.168.0.13 (Router 4) to reach any network outside 192.168.0.0/22
Use 192.168.0.14 (Router 3) to reach 192.168.0.0/22
Use the ISP device IP address to reach the Internet (any network outside 192.168.0.0/22)
Let's say that Router 1 and Router 2 now have direct connection as follows
where 192.168.0.9 is Router 1 and 192.168.0.10 is Router 2.
Let's look at the Router 1. From Router 1, it is possible to reach Router 3 directly or indirectly via Router 2. Similarly from Router 3, it is possible to reach Router 2 directly or indirectly via Router 1.
With this new connection, static routes no longer reliable choice. It is suggested that dynamic routing is used on Router 1 to 3. Router 4 may just keep using static route to the ISP device and to the Router 3.
When all of Router 1 - Router 3, Router 1 - Router 2, Router 2 - Router 3 connections have equivalent bandwidth and have equivalent traffic load; you may consider to use RIP. In real network, it is unlikely to have equivalent traffic load across the three connections. Therefore it is then suggested that Router 1 to Router 3 are to run OSPF, IS-IS, or EIGRP.
In Cisco routers, following is the EIGRP sample configuration
router eigrp 10
network 192.168.1.0 255.255.255.0
network 192.168.0.0 255.255.255.252
network 192.168.0.8 255.255.255.252
router eigrp 10
network 192.168.2.0 255.255.255.0
network 192.168.0.4 255.255.255.252
network 192.168.0.8 255.255.255.252
router eigrp 10
network 192.168.3.0 255.255.255.0
network 192.168.0.0 255.255.255.248
The static routes
ip route 0.0.0.0 0.0.0.0 192.168.0.13
ip route 192.168.0.0 255.255.252.0 192.168.0.14
ip route 0.0.0.0 0.0.0.0 [ISP DEVICE IP ADDRESS]
»[Config] HSRP Config
Check out the following FAQ for insights
»Cisco Forum FAQ »Various Network Design using Routers, Layer-3 Switches, and more
When you start building the network, there may only be few PC and one server. Then there are times when more machines are added such as printers, more servers, and more servers. At first, all machines might connect to one same switch, while the switch connects to one Internet router.
More and more machines added, then having only one switch can no longer sufficient. One option is to add another switch which will then daisy chained or interconnected to the existing switch. Example of this situation are Cisco Catalyst 2560 or 3560 fixed switches.
Another option is place a new switch with more ports, perhaps in form of switch module. Example of this situation is Cisco Catalyst 4503 modular switch.
Advantage of having multiple fixed switches in place is that you could possibly let existing switch untouched and simply hook up a cable to the new switch from the existing switch. On the other hand, advantages of having one modular switch in place are that you only need one switch to manage, less cable running (comparing to daisy chaining multiple switches), and higher switch port density.
Larger growth anticipates multiple modular switches in place where one or two modular switches are on one side of building while another set of modular switches are on another side of building. Each set of modular switches connect directly to LAN machines such as servers, PC, and printers. Switches (either modular or fixed type) dedicated to only connect to LAN machines are called access switch.
To provide connectivity to routers and firewalls, all access switches usually connect to a set of switches, acting as traffic hubs to other building, Internet, and perhaps WAN. This hub switches (either modular or fixed type) dedicated to only connect to access switches, routers, and firewalls (or simply non-LAN machines) are called core switches.
As you may imagine, the access switches are simply providing Layer-2 connectivity to LAN machines while the core switches are providing Layer-2/3 connectivity to LAN machines and are providing Layer-3 connectivity to non-LAN machines. Depending on network requirements, access switches can also be designed to provide Layer-2/3 connectivity to LAN machines while the core switches provide only Layer-3 connectivity.
With these in mind, following considerations in LAN design are in order.
* Fixed and modular switches from perspective of port flexibility, cable management, port density, and switch management
* Dedicating switches as either access switches or core switches to simplify network management
»Server TO core
FYI, your company may not run out IP address just yet. There might be something improper with the company's current network design. The following thread might give you a fresh insight.
»running out of IP addresses
Basically the sample company incorporate "flat network". I refer "flat network" as using the very same /24 subnet for all departments. As illustration, your company uses 10.10.10.0/24 subnet for all departments; where 10.10.10.2 can be in Sales, 10.10.10.5 can be in Accounting, 10.10.10.9 can be in IT, and so on.
As the thread shows, such usage of "flat network" is not suggested. When your company grows as the sample company, then you need to break up your company network into smaller logical network. As example, set 10.10.0.0/24 only for Sales; set 10.10.1.0/24 only for Accounting; and so on. Also as the thread shows, you need either a layer-3 switch or a router with a layer-2 switch that is capable to do inter-VLAN routing and trunking.
Check out the following FAQ for sample implementation
»Cisco Forum FAQ »Should I use Layer-3 switch or router?
»Cisco Forum FAQ »Setting Up Private Site-To-Site Connections
* Use private subnets whenever feasible
* If most of branches or departments within your organization already use let's say 192.168.x.x subnet, then keep using it. You may have renumber the 3rd octet to avoid overlap.
* Avoid discontiguous networks at all costs
* Assign separate subnets for infrastructure interconnections, servers, workstations, routers, switches, firewalls, IDS/IPS devices, UPS, and all other network devices
* When assigning subnets to network devices, plan network growth room for at least the next 5 years
* When there would be only two devices directly connected to each other, assign /30 or /31 subnet
* Avoid running dynamic routing protocols unless there are multiple exits with the equivalent administrative distances, connection technology, or bandwidth
* Should you decide to use OSPF, then keep in mind that you don't really need to have multiple areas unless you have a good reason to have so
* Consider resilient disaster recovery which might require layer-2 and layer-3 redundancies
* Do not "force" to have load balance over multiple links due to possible side effects of asymmetric routing
1. Hub and Spoke
* There is one site (probably the main or corporate office) that has direct connection to all other sites; called "Hub"
* All other sites (usually remote offices or branches) only has a single connection to the Hub; called "Spoke"
* Hub-to-Spoke communication use the direct connection
* Spoke-to-Spoke communication must go through the Hub as "intermediate hop"
* Connection to external network (i.e. the Internet) only exists at the Hub
* Communication between Spoke and external network must go through the Hub
* Since from Spoke perspective, traffic must go through Hub to reach other sites or external network; a single static route as default gateway pointing to Hub should be sufficient to cover all communication type
* From Hub perspective, traffic must go through each dedicated connection to reach specific Spoke or external network; a single static route as default gateway pointing to the external network (i.e. the ISP) and several static routes to reach Spokes should be sufficient to cover all communication type
* No need to run dynamic routing
* To have more resilient connection, bonded circuits (i.e. bonded T1/E1 circuits) between Hub and Spokes can be considered. Other consideration is to have redundant circuits between Hub and Spokes that are served by multiple ISP
* Hub network device should be the most powerful one compared to the Spoke network device since Hub must support traffic from all Spokes and the external network where the Spoke only support traffic within itself
* Should there future need to have backup connection beyond bonded circuit, refer to the next setup
2. Full Mesh
* There is probably no single main or corporate office as the Hub
* All sites have direct connection to all other sites
* All sites might have direct connection to external network (i.e. the Internet)
* Multiple path to reach the same site from one site perspective might exist
* When there are multiple path to reach the same site, running dynamic routing protocol (i.e. OSPF or EIGRP) is highly recommended to have resilient or optimal connection
* All sites should have equivalent network device specification and circuit bandwidth to maintain predictable network behavior
3. Partially Mesh
* There are probably at least two main sites (two "Hubs" that have direct connection to all other sites ("Spokes")
* All other sites (the Spokes) have one direct connection to 1st main site and another direct connection to 2nd main site
* Connection from Spoke to 1st main site is probably the preferred (primary) connection
* Connection from Spoke to 2nd main site is probably the alternate (backup) connection
* Connection between two main sites is probably using the most reliable and feasible connection, that could be in a form of single or multiple redundant connections
* Multiple path to reach other site from one site perspective exists
* Connection to external network (i.e. the Internet) might only exist at main site
* As illustration, the Spoke primary connection is in a form of Frame Relay circuit where the backup connection is in a form of ISDN (dialup) or broadband circuit; and connection between two main sites is in a form of bonded T1/E1 circuits
* Since there are multiple path to reach the same site, running dynamic routing protocol (i.e. OSPF or EIGRP) is highly suggested to provide resilient and optimal connection
Network Topology Variations
* Hub and Spoke with one primary and one backup connections between Hub and all Spokes
* Each Spoke has direct connection to reach the external network (i.e. the Internet) without go through Hub
Running Dynamic Routing Protocol in Primary-Backup Connection Scenario
1. Both Primary and Backup Circuits are always up on flat rate from billing perspective
* At remote site, dynamic routing protocol run over both primary and backup circuit
* Primary circuit from the routing protocol perspective should have lower cost compared to the backup circuit
* When there are multiple main sites (the Hub and Spoke with multiple Hub), route from Spoke to Main Hub might be preferrable over route from Spoke to Secondary Hub
2. Only Primary Circuit is always up on flat rate from billing perspective
* To avoid the backup circuit goes up due to the Hello mechanism, no dynamic routing run over backup circuit; just static routes with higher administrative distance or metric than the dynamic routing protocol's
* When primary circuit terminates at different equipment than the backup circuit, there might be a need to redistribute the static route that run over backup circuit to the dynamic routing protocol domain with the purpose of introducing known alternate path
For illustrations, check out following threads:
»Pix 520 with multiple ISP connections?
»[Config] ISDN Multisite dialup using OSPF cost issue
1. Group them in one VLAN as possible
Identify which computers that most of the time access particular servers. Then group all of those computers and the associated servers in one VLAN as possible.
2. Setup dedicated VLAN for servers only when necessary
When you have users from multiple departments accessing one specific server, then put this particular server in dedicated VLAN. Do not put regular non-server computer into this VLAN due to security vulnerability.
3. "Minimize Distance" between computers
Whenever possible; put all hosts (computers, servers, printers, etc.) in one single switch. When all of the hosts are spread into multiple switches, shorten the distance between switches. Especially for high traffic or critical application, use the largest bandwidth possible to "minimize distance" between hosts.
We use a switch with 24 ports of 10/100 Mbps speed and 2 ports of 10/100/1000 Mbps speed. Should we use the Gigabit port to connect to servers and use Fast Ethernet ports as trunk?
When you have hosts spread into multiple switches, then you should use the Gigabit-capable port to connect to servers and 10 Gigabit-capable port as trunk (of course whenever possible).
Keep in mind that as general idea, should always use the fastest port available as trunk when there are hosts spread into multiple switches. Do not, as example, use Gigabit port for server connections and Fast Ethernet port as trunk in this multiple switches situation since the traffic across switches are bogged down at the trunk (the trunk causes "bottleneck"). It is then better to use the Fast Ethernet port for server and the Gigabit port for trunk.
Should I setup the network as following?
Of course you can setup such design. However keep in mind that such design is a bad one. Why? Each link between one switch and another is single point of failure. If the link between S3 and S4 breaks; S5 cannot reach Router, S1, and S2. The same sense applies to link between S2 and S3, link between S1 and S2, and link between Router and S1.
Now to solve the failure link possibility, let me ask you this. How resilient do you wish your network be? How do you like your network design between the followings:
So far, the network resiliency issue discussed is just from the internal network (layer-2) point of view. There are also other issues to think of. The followings are just examples.
1. Redundant Routers
In case one router fails (either hardware or software failure).
2. Multiple ISP
When one ISP goes down or has bottleneck traffic issues, there is still another to load balance. You might want to have the 2nd ISP to use different telco lines (different local loop) that connects your site to theirs.
3. Having UPS (Uninterruptable Power Supply)
In case of power outage, there is still time to make necessary file access, save, or backups.
For further info of network resiliency, check out the following FAQ
»Cisco Forum FAQ »Network Design Tips
»Cisco Forum FAQ »Redundant Link Graceful Internet Load Balance/Failover
»Cisco Forum FAQ »Redundant Link Graceful Internet Load Balance/Failover
Prepending Your AS to setup automatic BGP failover mechanism on remote router via iBGP
One of the BGP feature is AS prepending, done by utilizing the BGP AS_PATH attribute. Let's say your organization edge routers are running BGP with your ISP. There are multiple links connecting your edge routers to ISP routers. One link has larger bandwidth than another.
You would then prefer that incoming traffic from your ISP to your organization via your edge routers would consider link with larger bandwidth as primary path over another link. In other words, other link would be the secondary or backup link in case the primary link goes down.
By prepending your AS on the edge routers' secondary link and no AS prepend on the primary link, your ISP would see that it takes longer path to reach your organization via the secondary link over the primary link. Therefore your ISP would prefer to use the primary link for incoming traffic from the Internet to your organization.
Following is a sample configuration
R1 hostname R1 ! interface Loopback0 description Required for Stable iBGP Peering ip address 192.168.12.1 255.255.255.255 ! interface Serial0 description ISP A ip address 192.168.31.2 255.255.255.252 ! interface Ethernet1 description LAN ip address 192.168.21.1 255.255.255.0 ! router bgp 100 no synchronization network 192.168.21.0 neighbor 192.168.12.2 remote-as 100 neighbor 192.168.12.2 update-source Loopback0 neighbor 192.168.12.2 next-hop-self neighbor 192.168.31.1 remote-as 300 neighbor 192.168.31.1 route-map AS-300_out out neighbor 192.168.31.1 filter-list 2 in neighbor 192.168.31.1 filter-list 1 out no auto-summary ! ip route 192.168.12.2 255.255.255.255 192.168.21.2 ! ip as-path access-list 1 permit ^$ ip as-path access-list 2 permit ^300_[0-9]*$ ! ip prefix-list AS-300_Primary description BGP Primary ip prefix-list AS-300_Primary seq 5 permit 192.168.21.0/24 ! route-map AS-300_out permit 10 match ip address prefix-list AS-300_Primary set local-preference 150 ! end
R2 hostname R2 ! interface Loopback0 description Required for Stable iBGP Peering ip address 192.168.12.2 255.255.255.255 ! interface Serial0 description ISP B ip address 192.168.42.2 255.255.255.252 ! interface Ethernet1 description LAN ip address 192.168.21.2 255.255.255.0 ! router bgp 100 no synchronization network 192.168.21.0 neighbor 192.168.12.1 remote-as 100 neighbor 192.168.12.1 update-source Loopback0 neighbor 192.168.12.1 next-hop-self neighbor 192.168.42.1 remote-as 400 neighbor 192.168.42.1 route-map AS-400_out out neighbor 192.168.42.1 filter-list 2 in neighbor 192.168.42.1 filter-list 1 out no auto-summary ! ip route 192.168.12.1 255.255.255.255 192.168.21.1 ! ip as-path access-list 1 permit ^$ ip as-path access-list 2 permit ^400_[0-9]*$ ! ip prefix-list AS-400_Backup_1 description BGP Backup #1 ip prefix-list AS-400_Backup_1 seq 5 permit 192.168.21.0/24 ! route-map AS-400_out permit 10 match ip address prefix-list AS-400_Backup_1 set as-path prepend 100 ! end
In a real network, prepending your AS number on R2 more than once might be required to achieve the desired result. In addition, R1 and R2 might be running HSRP or similar redundancy technique. Check out the following link of HSRP-BGP Combination sample configuration.
How to Use HSRP to Provide Redundancy in a Multihomed BGP Network
In BGP theory, there are several options to setup automatic failover mechanism. Some of them are AS prepending, MED, and BGP Community utilization.
When you manage the entire BGP network (that have multiple AS numbers within a single network administration), then you can use any techniques to control the failover mechanism. This applies usually when you run internal BGP network within your organization.
Managing external BGP network with multiple providers (ISP) would be different story. Not all ISP honor MED. AS prepend on your own AS number (or the ISP AS number) may not affect the inbound traffic from the Internet where transit provider still see particular ISP as the most actractive connection.
Following threads are some discussions
»[HELP] Multi homed with one BGP and one non BGP link
»[HELP] BGP with same ASN from two different locations
»BGP Multihoming default-originate only setup
»[HELP] BGP at 2 location
»[HELP] BGP over MPLS
»[Config] Redistributing EIGRP into BGP - best path
»Routing - is there something blocking this?
»Route distribution for dual WAN
BGP Community utilization should be the best approach to control inbound and outbound traffic over multiple AS (including over multiple providers in external BGP network). When your ISPs provide decent BGP Community string, you will have a good automatic failover mechanism.
In addition, Cisco provide BGP Conditional Subnet Advertisement feature that might be useful as workaround when MED, AS prepend, and BGP Community do not fulfill your expectation. However this feature looks like still in a stage phase since there is still unresolved Cisco Bug ID relating to "uncooperative" subnet advertisement problem.
Additional Sample Configurations
»Cisco Forum FAQ »Various Network Design using Routers, Layer-3 Switches, and more
»[Config] Unable to ping through OSPF neighbor router
Feedback received on this FAQ entry:
»[Config] BGP and OSPF redistribution
There are multiple networks that like to share the same medium. This medium in general sense can be anything, ranging from the same physical network devices to same Internet connection or same circuits. The requirement is that each network cannot access or see each other.
A traditional approach of this situation is to implement ACL to filter out traffic. With ACL approach however, all the networks share the same routing table at some point. Another downside is when each network uses private IP address scheme which can be used by anybody for any purpose that may conflict with other network. In some environment, the ACL approach might be security risk in addition to potential operation problem.
Another approach is to implement VRF (Virtual or VPN Routing and Forwarding). With VRF approach, each network has its own routing table. Since each network has its own routing table, no ACL is necessary and any network can use any IP address scheme including private ones without conflicting other networks.
Check out the following link for more info
RFC 4381: Analysis of the Security of BGP/MPLS IP Virtual Private Networks (VPNs)
»MPLS with a Single Router?
In some VRF implementation, usually LDP (Link Distribution Protocol) and MP-BGP (Multiprotocol BGP) are used to forward traffic of all networks over one network transparently. These LDP and MP-BGP usage is typical implementation of MPLS network.
Check out the following links for more info on LDP and MP-BGP
RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)
RFC 3031: Multiprotocol Label Switching Architecture
RFC 3036: LDP Specification
RFC 2105: Cisco Systems' Tag Switching Architecture Overview (TDP) - Cisco's version of LDP
For some simple networks, something called VRF lite can be used. When VRF lite is implemented, LDP is not used although at some point MP-BGP may be used.
With either LDP or non-LDP implementation, any routing protocols can be used to interconnect places. VRF works with connected networks, static routes, and dynamic routing protocols (RIP, OSPF, EIGRP, BGP).
Check out the following link for more info on VRF-based routing protocols
RFC 4577: OSPF as the Provider/Customer Edge Protocol for BGP/MPLS IP Virtual Private Networks (VPNs)
Where MPLS/VRF fits
»How to maintain VLAN Tags across Routers
»[Config] 4500 inter-vlan routing
»Cisco Forum FAQ »Multiple networks share same Internet lines or same devices transparently
»Cisco Forum FAQ »Separate Internet: Dedicate T1/E1 for server, dedicate DSL/Cable for LAN
»Cisco Forum FAQ »Configure DMZ on routers
ISP-offered MPLS Solution
Since its birth, MPLS technology has been exciting to ISP since they could cut costs tremendeously by sharing physical infrastructure compared to good-old Frame Relay technology where ISP has to build unique physical infrastructure for each network. ISP MPLS technology implementation comes up with various names, however they can be categorized by either Layer-3 or Layer-2 solution.
The Layer-3 solution is typical MPLS where ISP manages PE and P routers. In some extends, ISP may offer to manage the CE routers as managed MPLS solution so that MPLS BGP mechanism is transparent to their customers.
The Layer-2 solution is more modern approach where the ISP may call it as Metro Ethernet or Point-to-Point solution. In some extends, ISP may actually use point-to-point fiber infrastructure to offer the Point-to-Point solution compared to "simulated" point-to-point infrastructure.
For those more advanced customers, ISP also offer VPLS-based solution or some may call it CSC (Carrier Supporting Carrier). With this solution, the customer manages its own PE or may even manage its own P routers (the Layer-3 side) while the ISP manage the Layer-2 side. By managing its own PE or even P routers, customer has total control of the MPLS Cloud from assigning its own VRF (MPLS Labels) to Traffic Engineering so that the customer can create custom-made MPLS network for specific network need such as unique Layer-3 MPLS network for voice and data without a need of managing the Layer-2 side.
At customer site, typically ISP implement at least T1/E1 circuits as part of the MPLS solution they offer to customers. For larger bandwidth demand, ISP may implement DS-3, OCx, or DWDM circuits. Depending on the requirement, ISP may bring in their own equipments to install at customer site such as PE or CE router, or some Ethernet-based switch for those Ethernet-handoff solution. With any telco circuit implementation, good old multiplexer (Muxes) or simple Smartjack boxes for T1 circuits are also part of the MPLS solution which may or may not be managed by the MPLS provider. Depending on the customer area availability, there may or may not be fiber drop at customer site as part of the MPLS solution.
With any MPLS solution, prospective customer should review how ISP actually use their fiber infrastructure in offering the MPLS solution. Some ISP may or may not have direct physical fiber connectivity between areas. Some ISP may offer more expensive solution with lower latency. Further, prospective customer should review their internal network design of how should their network design look like with MPLS solution come to play in order to verify reliability, scalability, and top-notch network performance.
Various MPLS Topics
»Anyone experience with L2TPv3?
TE (Traffic Engineering)
»[bgp] multisite multihome via mpls and bgp
MTU on MPLS
»L2TPv3 MTU problem/question.
MPLS QoS: DiffServ and IntServ (Differentiated Services and Integrated Services)
»IP QOS(quality of service) over MPLS usingIntegrated Service
»Packets are not being marked - QOS over MPLS
»Regional routing in WAN cloud?
[CCNA] Configuring Default-network EIGRP
»Cisco Forum FAQ »BGP Design
Some discussions regarding typical setup of hosting environment, which are applicable to public or private use. Some examples of the setup application is in public web hosting environment at data center location. Following is list of illustrations.
»[HELP] BGP with same ASN from two different locations
»[CCNA] BGP between two sites with same ASN no & same IP range
[H/W] Equipment suggestion
»Speed between Switches & IDF MDF