Disclaimer

This FAQ is written as a guide or reference, rather than a requirement. Your situation may or may not fit the description though an effort has been made to cover most situations.

Network Setup

The following is considered typical network setup for home and small businesses.

Equipments to use
* Routers, firewalls, switches
* Category 5/5e/6 patch cable for wired connection
* Servers, workstations (PC)

1. Router
* In most cases, you need to do IP routing between your ISP (the Internet) and your network
* With that in mind then you need a router that at least has two Layer-3 (routing) interfaces, one is facing the ISP and another facing your LAN
* Depends on the router model, one interface that is facing your LAN is Ethernet interface and another interface that is facing your ISP could be Ethernet or non-Ethernet interface
* Non-Ethernet interface could be T1/E1 (Serial), ISDN, and DSL
* When the router has Non-Ethernet interface, then the router might have integrated modem
* When you have T1/E1, DSL, or cable Internet; you could use dual-Ethernet interface router when there is supporting external modem with Ethernet port
* When the router has multiple Ethernet ports (i.e. dual-Ethernet router), verify if any of those ports are capable as Layer-3 (routing) interface
* When the router has integrated switch, then the all switch ports are considered one Layer-3 (routing) interface that will be facing your LAN
* The router might need to do NAT/PAT between your internal private subnet and the IP address provided by the ISP
* Typically routers don't do OSI Layer 5-7 inspection and/or filtering (i.e. SPAM email filter, anti virus, anti malware). You might need a firewall specifically for these.

2. Firewall
* In most cases, you need to do IP routing between your ISP (the Internet) and your network
* In addition, you also need to have firewall for some Internet security
* With that in mind then you need a firewall that at least has two Layer-3 (routing) interfaces, one is facing the ISP and another facing your LAN
* Usually the firewall interfaces are Ethernet only without integrated modem
* You need to have an external modem or external integrated modem/router to connect the firewall to your ISP assuming no integrated modem exists
* When the firewall has multiple Ethernet ports, verify if any of those ports are capable as Layer-3 (routing) interface
* When the firewall has integrated switch, then the all switch ports are considered one Layer-3 (routing) interface that will be facing your LAN
* The firewall might need to do NAT/PAT between your internal private subnet and the IP address provided by the ISP
* If you need OSI Layer 5-7 inspection and/or filtering (i.e. SPAM email filter, anti virus, anti malware), the firewall may offer such with additional license or fee

3. Switch
* Most home or small business network use Layer-2 switch
* With Layer-2 switch, all ports are considered one Layer-3 (routing) interface
* Layer-2 switch does not do routing; only switching or bridging
* You still need to do routing between your ISP (the Internet) and your LAN; hence you still need either a router or a firewall
* You will connect the switch to the router or firewall LAN interface
* When the router or firewall has integrated switch, then you probably need a crossover Category 5/5e/6 patch cable instead of the straight-through type when connecting the switch to the router/firewall

4. Servers and Workstations
* You will connect servers and workstations to the switch ports
* When the workstations need to receive IP address automatically, then you may need to set the router or firewall as DHCP server and the workstations as DHCP client
* Servers need to have static IP address; refer to the server operating system on how to set static IP address

A router/firewall/switch combo might fit to most home users while businesses might need a dedicated equipment of each. In the example of getting certain Internet connectivity from your local ISP, such router/firewall/switch combo may be provided as part of the service so then you may not need additional equipment except some PC, servers, and mobile devices.

Choosing ISP

Whenever possible, choose ISP that has reliable connection to backbone network. Note that the ISP does not need to be the Tier-1 class (such as AT&T or Verizon), especially when your area is only served by Tier-3 class ISP. As long as the ISP has such reliable connection, you should be in good shape for the most of time.

For business-level Internet connectivity, you may need to ask following questions to find out how reliable your ISP connection to backbone network.
* What kind of circuit does the ISP have to the backbone network? OC-X (OC-3, OC-12, or higher)? SONET ring? DWDM?
* How many transit provider does the ISP connect to? Three should be the "standard"
* Who are the transit providers? Are they Tier-1 class providers? Something like AT&T, Verizon, Level 3, Cogent, or Sprint should be sufficient.

Choosing Connection Type to ISP

Typical connection for home or small businesses is broadband; either Cable Internet or xDSL. As comparison, here are some of enterprise-level types.

1. T1/E1, Point-To-Point (Dedicated Leased Line such as Ethernet or dedicated fiber), or Frame Relay
2. ISDN
3. Wireless (Microwave, 4G)

Since these connectivity types originally come from enterprise world, they are considered "top of the line" for home or small businesses. One obvious difference is a term called "circuit" with associated circuit ID, showing such dedicated form. Another is the standard SLA (Service Level Agreement) which includes 4-hour guaranteed response time which does not present on broadband connectivity kind. In most cases, these circuit kinds have higher reliability grade than the broadband; hence require "top dollar" fee compared to the broadband.

Choosing connection type to ISP depend on how critical your Internet applications are. If you or your organizations require constant, stable, and reliable Internet connection 24/7, then the first two circuit kind should be the choice. If you or your organization can tolerate some down time (no Internet connection for some time), then the last two choices should be sufficient.

Another consideration is whether you plan max out the connectivity 24/7 or so, which broadband provider will not permit. No such permission issue comes in with the enterprise-type connectivity.

Note that Wireless solution can or cannot be equal to T1/E1 or even DS-3 circuits, depending on your area. Key in Wireless solution is LOS (Line of Sight) and distance. If the distance between your location and ISP is closed enough and there is a clear LOS between the two locations, then Wireless can be cheap and robust solution.

Hosting Network Devices

As mentioned earlier, the T1/E1, the point-to-point or Ethernet, Frame Relay, and ISDN circuits are "top of the line" services which require "top dollar" fee compared to the broadband connection. When your intention of bring in Internet connectivity is to host your own server to be accessible from the Internet, those "top of the line" should be a top consideration. You may be able to go for lesser service level such as 100 Mbps Business-Level cable Internet however no SLA in place and no guaranteed bandwidth either.

Another consideration is to have (managed) hosting solution. Typically companies that provide hosting solution have at least DS-3 or larger bandwidth connection to ISP and the Internet. When you have your network devices (i.e. servers) hosted within these companies' data center, you don't have to worry about costs of bring in your own circuits. In addition, you don't have to worry about power consumption and cooling system your network devices need since all of these are included and managed by the hosting companies. All you need to worry is to make sure your applications run on those servers are behaving as they should.

Between T1/E1, DSL, and Cable Internet

Let's say you have following choices of ISP connection speed (bandwidth)
1. A 1.5 Mbps full T1 circuit
2. A 1.5 Mbps ADSL over POTS (phone line)
3. A 3 Mbps Cable Internet
4. Fiber-to-home situation

For home users or small businesses, the third choice looks most attractive since it usually offers more bandwidth with the lowest cost. Keep in mind that broadband connection (including Cable Internet) has minimal or no SLA compared to the T1 circuit.

In addition, a lot of time Cable Internet provider has some kind of bandwidth limit. The 3 Mbps bandwidth or speed is most likely the burstable speed and may not reflect the actual speed. If you or your organization constantly use up the 3 Mbps speed, the Cable Internet provider might give you or your organization penalties like charge extra fee or might reduce the speed without your consent or knowing.

Unlike Cable Internet, there is no such penalty on ADSL connection. In most cases, the speed connection is constant. When you have both T1 and ADSL from the same provider, you or your organization might be able to have some kind of Internet connection load balance or failover mechanism.

Side Note:
Check out following FAQ for more info on load balance or failover mechanism
»Cisco Forum FAQ »Redundant Link Graceful Internet Load Balance/Failover

Note that ADSL (and other xDSL technology) speed depends on the distance between your site and the ISP. The closer your site to the ISP, the more bandwidth or higher speed available to you. Specifically with xDSL connection that ride over POTS, there might be some electromagnetic interference factors you also need to consider.

Regardless of penalty existence, another factor to consider in Broadband solution is oversubscribing. Oversubscribing is a practice of overselling or over-providing bandwidth with a chance of burstable speed. If your area has too many (let's say) 50 Mbps subscribers while the ISP presence in the area is only capable to provide 50 Mbps to some of the subscribers at a given time, then two things happen; one, your ISP is implementing oversubscribing practices; and two, you may not have the expected 50 Mbps at a time.

A new approach in some area is fiber-to-home solution where fiber cable termination is directly at residential and small office locations. This means in general that the local loop (the cable between your ISP and your location) is brought in by fiber cable where there are these fiber cable strands in your location's building. If your area is fortunate enough to have such fiber termination, this solution is probably the cheapest and robust one. When your ISP offer such fiber-to-your-house solution, make sure that you can see those fiber cable strands or whether the ISP can extend their fiber cable to your building.

Choosing Connection Speed/Bandwidth

How fast should your connection be? Is 1.5Mbps connection fast enough? Should I choose the 6Mbps speed instead of 1.5Mbps speed?

Choosing connection speed should be based on your application performance. Locate your critical Internet applications that will take the ISP connection bandwidth the most. These applications vary between home users or small businesses. As illustration, the applications could be simple Internet browsing, email, online gaming, voice or video over the Internet, and web hosting.

Once you locate the applications, the next step is to find out what the most appropriate speed for such applications considering their workload. When you are unsure what the most appropriate speed is, the application customer support should be the first to contact.

If you are still unable to find out the most appropriate speed afterward, then the next consideration factor is your financial budget. When your budget is limited, then you should pick the least expensive connection (which also mean the slowest connection). Should you need faster connection in the future, you could always consider upgrading the speed.

Internet speed expectation

The notion of 1 Gbit/sec Internet bandwidth does not equal to 1 Gbit/sec download speed is still incompatible with today's reality for various reasons before we even get to the technology in the home. The value of "big phat pipes" is not of how fast an individual device can download data - although a single device will get as much as it can possibly take - but is of the number of devices that can have as much as they can take. Several factors to consider are the individual device's various elements such as harddrive read/write speed, NIC port speed, physical connectivity (i.e. cable or wireless quality) between the device and the ISP, video cards, memory, CPU, software to use, and operating system settings.

There are too many misunderstanding with people using smart phones and tablets for speed testing as example where 1 Gbps wireless speed is expected when the wireless network is a wireless N which is in theory only capable to support 300 Mbps. By the same token, tome expectation has to be managed but it is unrealistic expectation to get 200 Mbps on the wires when the ISP bandwidth is 1 Gbps.

Discussion
»A 10Gbps residential Internet bandwidth - but for how much?

Choosing Internet gateway device

The most common Internet gateway devices for home or small businesses are routers and firewall. Routers are usually preferable since they fit to most Internet connection environment compared to firewall. However firewall could be the choice when you or your organization only require default gateway route to your ISP and no plan of having T1/E1, Point-To-Point, Frame Relay, or ISDN circuit to your ISP.

Whichever device you choose, you should choose device that can provide at least decent security features or protections. In addition, business grade device is recommended since they are more reliable than the consumer grade.

In Cisco world, routers for home or small businesses are 800 series or higher. As to firewall choices, they should be ASA 5500 series or PIX Firewall.

Choosing Modem

As mentioned, you have a choice to use either external or internal (integrated) modem. When you have a broadband Internet such as ADSL and Cable Internet, typically you need to have an external modem. Should you prefer to use internal modem that is integrated into the Internet gateway device, make sure that the modem is compatible to your ISP connection.

In case that you use external modem, you need to verify if the modem is "just" a modem (dumb modem) or if the modem is an integrated modem/router. A simple dumb modem typically need no special configuration. You can just connect the modem into your Internet gateway device. If the modem is an integrated modem/router, then you need to confirm further issues like bridge/route mode, NAT/PAT active, and so on.

Connecting Your Network to ISP

Most home and small business users simply need a default route using ISP as the Internet gateway. Your network device (either firewall or router) will have to maintain the route to the ISP in order to keep Internet connectivity alive, by keeping such route into the device's routing table. This default route can be established dynamically between your network device and the ISP using DHCP and PPP (PPPoE or PPPoA). Some ISP may say static route will be deployed instead. Either way, this route will make sure your network has Internet connection.

Connecting Router or Firewall To Your ISP

Followings are most common network scenarios for each ISP connection type

1. T1/E1, Point-To-Point, or Frame Relay
* use a router (or router-like firewall) with either internal or external DSU
* receive static IP address with specific subnet mask from ISP
* the ISP static IP address may be a public IP address (Internet routable) or may be a private IP address (non-Internet routable)
* may or may not receive ISP DNS IP address

2. DSL
* use a router or firewall with either internal or external DSL modem
* When using a Cisco router with internal DSL modem, there might be a need to have interface BVI1 activated and to set VPI/VCI value for the ATM interface
* When there is no internal DSL modem, you should not need BVI interface
* receive either static or dynamic IP address with specific subnet mask from ISP
* the ISP IP address is a public IP address (Internet routable)
* ISP assign the IP address by either PPP (PPPoE or PPPoA), DHCP, or static
* may or may not receive ISP DNS IP address

2.1 When ISP uses PPP
* When you use Cisco router as the ISP gateway, there is a need to have interface Dialer1 activated
* You need to tie the WAN port interface with the interface Dialer1
* Under the interface Dialer1, there is a need to have either "ip address x.x.x.x y.y.y.y" (statically assigned) or "ip address negotiated" (dynamically assigned)

2.2. When ISP uses DHCP or static
* When using a Cisco router with internal DSL modem, there might be a need to have either "ip address x.x.x.x y.y.y.y" (statically assigned) or "ip address negotiated" (dynamically assigned) under the interface BVI1
* You might be required to set specific MAC address under the interface BVI1
* When you do use interface BVI1, you need to tie the WAN port interface with the interface BVI1
* When the router has no internal DSL modem, then the IP address assignment (either statically or dynamically) should be under the ISP-facing Ethernet interface
* Should you need to set specific MAC address and there is no internal DSL modem, the MAC address should be under the ISP-facing Ethernet interface

3. Cable Internet and Fiber-to-home connectivity
* use a router or firewall with either internal or external cable modem
* receive either static or dynamic IP address with specific subnet mask from ISP
* the ISP IP address is a public IP address (Internet routable)
* You might be required to set specific MAC address under the WAN port interface (interface cable0 or Ethernet interface)
* may or may not receive ISP DNS IP address

4. ISDN
* use a router with either internal or external ISDN modem
* receive either static or dynamic IP address with specific subnet mask from ISP
* the ISP IP address is a public IP address (Internet routable)
* may or may not receive ISP DNS IP address
* since ISDN uses PPP, also check on part of "2.1 When ISP uses PPP"

Find out your suitable WAN connection type

Usually you already know that your LAN is Ethernet environment. But do you know what WAN environment you would have? Is it T1/E1, DSL, PPPoE, PPPoA, DHCP, or what?

The only people who know what your WAN environment would be is your ISP. Please consult with your ISP representative regarding the connection type. Usually when you are a new customer, your ISP would provide you necessary info of how to connect your LAN to the Internet; either by mail, email, or phone.

Keep in mind that the ISP provided info might not be as technical or unclear. Here is a suggestion. Document all info provided here in this FAQ. You then consult the WAN connection type with your ISP representative. Ask the representative to find out which WAN connection type provided here that would match.

Some key words you need to discuss with your ISP representative are followings:
* Physical (Layer 1) connection: T1/E1, Ethernet (fiber), ISDN, DSL, Cable Internet
* Modem existence: external or internal modem
* Layer 2 connection: PPPoA, PPPoE, DHCP, Static IP addresses
* IP Address Assignment: Which IP address must be the gateway; which should be host
* NAT/PAT: Is it possible to use gateway (router) IP address as the only Public IP address servicing multiple LAN machines to go out to the Internet using PAT?
* DNS IP addresses: Which are they? How do you use them on your system?

If your representative is not technical enough, ask to speak with one of their technical person. This way, you would be sure you have necessary info on how to connect your LAN to the Internet.

As an insight, following is some technical aspect description of DSL and Cable Internet
»Cisco Forum FAQ »Technical Aspects in xDSL/Cable Internet connection

Preparing Yourself before discussing with ISP representative

Before contacting your ISP, you need to understand your system you plan to use. This system is including your Internet gateway (router or firewall), servers, workstations, and all other hosts. Familiarize yourself with the router or firewall innerworking and features, as well as the operating system of your workstations, servers, and all other hosts. The key technology to familiarize with is how to setup network using DHCP, PPP (PPPoA/PPPoE), and static IP addresses on your system.

As to the router and firewall, it is suggested that you to be comfortable around various WAN connection type and deployment. Review router and firewall sample configuration of all WAN connection type; from DHCP, PPP, to static IP address. Even though your ISP would be using DHCP and not PPP for example, it is a good idea to be familiar on both to understand similarities and differences between the two technologies.

Check out the following FAQ for further info regarding DHCP, PPP, dynamic, and static IP address
»Cisco Forum FAQ »Between DHCP, PPP, Dynamic, and Static IP Address

Following is the sample configuration list of specific WAN connection type for further review. The sample configuration covers most common WAN connection type such as T1/E1, cable Internet, DSL, external and internal modem, PPPoA, PPPoE, DHCP, and Static IP. It also cover multiple platforms; from routers of various model to PIX Firewall or ASA.
Various PPPoE/PPPoA/DHCP/Static Sample Configuration with Cisco

Most of all sample configurations are written in CLI (Command Line Interface) and not in a Web GUI. In case you are not familiar with CLI, following FAQ is showing CLI introduction.
»Cisco Forum FAQ »Straight-forward way to configure Cisco router: Introduction to CLI
»Cisco Forum FAQ »Straight-forward way to configure Cisco PIX Firewall/ASA: Introduction to CLI

By reviewing all of your system innerworking in advance, you are better prepared; which would make the ISP WAN connection type and deployment discussion process with their representative go smoother.

Deployment Process

When you are ready to do the actual deployment, you can check out the following FAQ for insights
»Cisco Forum FAQ »Quick and Easy Subnetting on Routing, Switching and Network Design Relationship
»Cisco Forum FAQ »Choosing Gateway IP Address for a network
»Cisco Forum FAQ »NAT, PAT, Port Forward, Internet and Server Access: Introduction and Practices
»Cisco Forum FAQ »Network Design Tips
»Cisco Forum FAQ »Setting Up Private Site-To-Site Connections

Prerequisite reading:
»Cisco Forum FAQ »Things to expect when setup network for home or small business

When you decide to have Broadband Internet access using xDSL (i.e. ADSL, SDSL) and Cable Internet, you most likely will deal with following aspects

* To use either a router or a firewall as the Internet gateway
* Layer-1: using either internal or external modem; Category 5/6 cable extension
* Layer-2: PPP (PPPoA, PPPoE); MAC address for DHCP
* Layer-3: auto-negotiate or static WAN IP address

Following are some details.

PPP

When you are using xDSL, ISDN, or T1/E1 circuits, you probably will be dealing with PPP technology. In a nutshell, PPP is Layer-2 technology providing connectivity to remote user (PPP client) to server (PPP server) using specific username and password. In this case, the PPP client is your Internet gateway (either router or firewall) and the PPP server is the ISP.

Typically you need a router as the PPP client. Specifically with PPPoE, you could use a firewall. However for PPPoA or legacy PPP, you need a router.

DHCP

When you are using either xDSL or Cable Internet, you probably will be dealing with DHCP technology. In a nutshell, DHCP is a mechanism that providing IP address and subnet mask dynamically to specific machine that needs one. In this case, the machine is your Internet gateway (either router or firewall) which will be the DHCP client and the DHCP server is at the ISP network.

Typically you could use either router or firewall as DHCP client. Unlike PPP that uses username and password to connect, DHCP process might require certain MAC address to connect to the ISP.

Following FAQ is some info on PPP, DHCP, and Static IP address assignment
»Cisco Forum FAQ »Between DHCP, PPP, Dynamic, and Static IP Address

Between Internal Modem and External Modem Usage

When you use an external modem, your Internet gateway might receive Ethernet hand-off. This is applicable when you use a firewall or a router without integrated modem. From practical perspective, you then only need to configure the Layer-2/Layer-3 aspect on the Internet gateway. For PPP, in general you only need to configure the username, password, and authentication method. For DHCP, in general you only need to verify that your Internet gateway MAC address is on the ISP database.

There are some things you need to confirm whether you use external or internal (integrated) modem. Some examples are your ISP DSL signaling type, bridge mode configuration, and VPI/VCI value settings when you use xDSL service. Fortunately, you may not need to worry about this when you use the "ISP-approved" external modem since those settings are pre-configured. Note that the keyword is "may".

When you use a router with integrated DSL modem for xDSL service, your integrated modem/router may not be the "ISP-approved" xDSL equipment. Note that even though the router is not "ISP-approved", doesn't necessarily mean that the router won't work. In any case (either using integrated modem or external modem; "ISP-approved" or "ISP-non-approved"), you need to verify the Layer-1/Layer-2/Layer-3 aspects. As illustration, you need verify things like DSL signaling and ATM VPI/VCI value in addition of the username, password, and authentication method. Whatever technology your ISP uses (DSL, Cable, or else), you need to make sure their setup matches yours to make things work. Check out the following FAQ for more info.

»Cisco Forum FAQ »Generic PPPoA/PPPoE/RFC1483 Bridging/RFC1483 Routing Guide

One good thing about using integrated modem within a router is that you can see Layer-1/Layer-2/Layer-3 aspects on one device which is the router itself. When you use an external modem, then you need to confirm two device configurations which are the external modem and the router.

Understanding DSL Technology

Connecting Cisco Routers to Service Provider DSL Networks

Cisco DSL router DSL/ATM command output descriptions

ADSL Sample Technology - show dsl interface atm command output
ATM IMA Sample Technology - show controllers atm

Some Deployment and/or Troubleshooting Insights

Following are some discussions of troubleshooting Layer-1/Layer-2/Layer-3 issues
»Trouble with ADSL connection after a weird router reset 877
»[HELP] How to read dsl interface
»What do these sh dsl int atm0/0 - atm0/1 mean ??
»2800 series routers
»[HELP] Cisco 1721 and WIC-1ADSL Slow, 320Kbit
»Frequent disconnects with 1801
»[HELP] Cisco 857W and Qwest
»[HELP] cisoc 3640 nm-1fe-2w + wic1-adsl speed problem.
»[Config] Fun with Cisco 1720 WIC-1ADSL, WIC-1ENET and Cisco PIX
»Cisco 8x7 CRCs on logical interface only
»[HELP] Help with CISCO 1801 router

For more info on Layer-1 xDSL troubleshooting, you can always visit the DSL forum FAQ such as this
»SBC DSL FAQ

For more info on Layer-1 Cable Internet troubleshooting, you can always visit the Cable Internet forum FAQ such as this
Cable Modem General Info
Cable Modem Troubleshooting

Suggested prerequisite reading
»Cisco Forum FAQ »Things to expect when setup network for home or small business

When you are using ISP to connect to the Internet, most likely you will be dealing with DHCP, PPP, dynamic, or static IP address assignment (whether you are aware or unaware of it).

Let's say you have to configure Cisco router Ethernet 0 interface to have specific IP address. Then the following is illustration on how to configure the IP address.

1. Assign IP address by DHCP

interface Ethernet0
ip address dhcp

2. Assign IP address by PPP

interface Ethernet0
ip address negotiated

3. Assign IP address statically

interface Ethernet0
ip address xx.xx.xx.xx yy.yy.yy.yy

where xx.xx.xx.xx is the IP address and yy.yy.yy.yy is the subnet mask

In early days; DHCP and PPP were used to dynamically assign IP address to hosts. However with additional features, it is technically possible to assign "static IP address" via DHCP and PPP. By referring to specific MAC address of a host, the host is always receiving the same IP address via DHCP. By referring to specific username and password, a host is also always receiving the same IP address via PPP.

Why would your ISP use DHCP or PPP to "statically assign" IP addresses to their customer and not use the traditional way of statically assign IP addresses? Probably it is simpler from their network administration point of view. Whatever the reason is, you have to choose the most appropriate way to assign your ISP IP address and experience with the tip and tricks when you need to access the Internet using your ISP.

Assign Your Internet Gateway's IP Address

In term of configuring your Internet gateway's IP address, you need to consult with your ISP as to how exactly they assign IP address to your device.

When your ISP says the IP address would be assigned dynamically, you need to confirm the followings

* if they use DHCP or PPP (or PPPoE/PPPoA) technology to assign the IP address
* if they use PPP, confirm the username and password for the PPP authentication process
* if they use DHCP, confirm if the ISP lock down your IP address with specific MAC address
* if the IP address is always the same everytime or constantly changing
* assuming the IP address is changing, how frequent the change takes place and which event will trigger the change

When your ISP says the IP address would be static, you need to confirm the followings

* if they use DHCP or PPP technology to assign the IP address
* if the IP address might change
* assuming the IP address is changing, how frequent the change takes place and which event will trigger the change

Important Note:

Make sure that when you discuss this with your ISP representative, the representative is the technical person who knows what he or she is talking about. You don't want to get misinformed since you might not be able to access the Internet when you don't have the correct info.

Static IP without DHCP or PPP

If your ISP says "No DHCP, No PPP. It is static"; then it might mean that you have to statically configure your Internet gateway device with your assigned IP address. On Cisco router, you should then use the "ip address xx.xx.xx.xx yy.yy.yy.yy" command.

Check out this forum's FAQ for specific sample configuration of Cisco router with statically-assigned IP address
»Cisco Forum FAQ »How can I configure broadband router with cable/dsl using static IP address

Static IP with DHCP

When your ISP uses DCHP to "statically assign" your Internet gateway device, then from device perspective it is still DHCP (still somewhat dynamic IP address with "sticky IP" approach). To configure your Cisco router, you then still need to use the "ip address dhcp" command under the ISP-facing interface.

Check out this forum's FAQ for specific sample configuration of Cisco router as DHCP client.
»Cisco Forum FAQ »Configure router as DHCP client using external modem

Dynamic IP with DHCP

From DHCP client perspective, there is no difference between "static" and dynamic IP address assignment. As mentioned, "statically assigned" DHCP-based IP address is still dynamic process. Therefore you can use the same above FAQ for specific sample configuration of Cisco router as DHCP client when you only have dynamic IP address from your ISP.

As a note, the difference between DHCP-based static and dynamic IP address is probably the ISP requirement to lock down your Internet gateway device MAC address to a specific IP address. Although it is possible that the ISP administer MAC address lock down for both dynamic and static IP account customers due to network management simplicity. Check out the following thread for insight.

»[help] 851W and ISP DHCP

Dynamic IP with PPP

In general, your ISP usually supplies username and password for the PPP authentication process. Once your Internet gateway device successfully establishes PPP connection with your ISP (pass the Layer-2 process), then your device will deal with the IP address assignment issue (the Layer-3 process).

Under normal PPP-IP network environment, dynamic IP address assignment will require the "ip address negotiated" command under the ISP-facing interface on Cisco equipments. With static IP address, you need to use the "ip address xx.xx.xx.xx yy.yy.yy.yy" assignment on Cisco router. However there might be exception for certain ISP. If you have a static IP with PPP, read the next discussion.

Check out this forum's FAQ for specific sample configuration of Cisco router as PPP client
»Cisco Forum FAQ »Quick Guide of Configuring Cisco router for PPPoE using external modem

Static IP with PPP

When your ISP uses PPP to "statically assign" your Internet gateway device, then you may experience some unusual situation. To configure a Cisco router, you need to use the "ip address xx.xx.xx.xx yy.yy.yy.yy" command under the ISP-facing interface in normal static IP address environment. However for some ISP, you need to use the "ip address negotiated" command under the ISP-facing interface.

If you are in this situation, then you might try to use the 1st approach (the "ip address xx.xx.xx.xx yy.yy.yy.yy" command) and see if you are able to host public servers or establish VPN IPSec tunnel with remote end. If your public server is inaccessible from the Internet or you are unable to establish VPN tunnel, then try the 2nd approach ("ip address negotiated" command) and see if it makes any difference. When the 2nd approach works, then the 2nd approach is considered the most appropriate way to assign IP address to your ISP-facing interface.

Like the DHCP, static and dynamic IP address assignment in PPP-IP environment is using similar configuration. Therefore you can refer to the previous sample configuration of Cisco router as PPP client in static IP address assignment.

Additional Sample Configurations

For more sample configurations, check out the following FAQ
Various PPPoA/PPPoE/DHCP/Static Sample Configuration with Cisco

Communications between internal sites within the same organization is preferred to be delivered in a form of secure or private connection, which ride over some circuit. The circuit could be in the form of dedicated circuit or broadband circuit such as DSL and Cable Internet.

Dedicated Circuit

Dedicated Circuit is a circuit to provide private dedicated connection between two sites or more. In other word, no other organization will use this circuit since it is dedicated to only one organization among its all sites end to end.

Following is the most common dedicated circuit type
1. T1/E1, DS-3
2. ISDN
3. Frame Relay
4. Fiber: OC-X, Metro Ethernet, SONET Ring, DWDM
5. Microwave (wireless)

To have this circuit, usually organization contact its preferred ISP to setup one. The organization could choose to use the ISP network as "intermediate network" between organization sites, or choose to have direct connection between sites bypassing ISP network.

Using T1/E1 circuit for such direct connection for example, the circuit would be some type of leased line; point to point between two sites. When there are more sites to connect, usually organization would use the ISP network at some point to reduce cost and to be more manageable.

This kind of connection technology is considered "top of the line" since it is the most reliable connection (at least for most of the time) compared to broadband connection such as DSL and Cable Internet. This nature requires the organization to pay premium maintenance cost compared to the broadband connection.

Wireless

In some situations, using wireless technology (i.e. microwaves) to provide private site-to-site connection is a good approach. Typically following are the situations that make wireless deployment is a "no-brainer" solution.

• Distance between all sites are pretty closed to each other

• Line of sight (LOS) between antennas are not blocked. In other words; neither trees, hills, mountains, nor buildings are between sites

• You need "unlimited" bandwidth with limited time and budget constraints to deploy

• "Little service abruption" is acceptable

Check out the following for more insights.
»Wireless Networking Forum FAQ
»Carrier Grade ~1Gbps solutions
»Best Design

VPN (Virtual Private Network)

With today's virtual communication technology, one organization could use some form of VPN (Virtual Private Network) to provide private and secure site-to-site connection. Using VPN, connection between two locations could ride over public network (i.e. The Internet) while keep maintaining secure or private connection. This is done by creating logical or virtual connection between the locations that ride over any physical circuit.

There are several technology to set such connection
1. HTTPS/SSL
2. IPSec
3. MPLS

Following is the breakdown.

HTTPS/SSL-based Approach

One factor that contributes to decisions of setting up private or secure connection for internal communications is depending on the application, such as the file transfer and email. Let's say your organization uses web-based email or any web-based application accessible using your Internet browser (such as Internet Explorer, Netscape, or Mozilla) for site inter-communication. When this is the case, then one way of setting up private connection is to utilize HTTPS/SSL-based connection over the Internet.

HTTPS/SSL-based connection is basically HTTP (web) communication that can ride over any connection, including the Internet (public network) via any ISP while still maintain secure and private environment. By utilizing this HTTPS/SSL-based technology approach, any organization sites only need basic Internet connection without require special network setup.

Note that HTTPS/SSL-based network over the Internet only works when all necessary applications within the organizations are web-based applications. Some applications cannot be accessed simply by using Internet browser. For example, you cannot use Internet Explorer (as the Internet browser) to map share drives in Active Directory Microsoft network.

When remote users need to access these applications, then in general HTTPS/SSL-based approach will not work. To make it work, there would be a need to have network-layer connection technology approach (by go lower to OSI Layer 1 to 3) to setup such secure or private connections.

Using network-layer connection technology approach, any application (web or non-web based) will work since this approach is more general and not depended by specific application types.

IPSec Approach

Both IPSec and HTTPS/SSL technology are VPN connection. They both create encrypted data connection ("tunnel") between two sites. The difference is that HTTPS/SSL is web (OSI Layer 7) approach and IPSec is network (OSI Layer 3) approach.

As mentioned, IPSec VPN is capable of supporting web or non-web applications since it is using network-layer connection technology approach. Example of non-web application is accessing data in Microsoft Active Directory network share drives.

Note:

Both IPSec and HTTPS/SSL VPN technology is also applicable to remote users connecting to office temporary as following description. Within an organization, there is probably at least one employee that is always "on the run" and need to access work remotely from anywhere. Sometime this type of employee is called "road warrior". There are also other type of employees that need to access work remotely from home, hotels, or any place from time to time.

The nature of such connection need is temporary access, where access is available only when it is needed. When the access is not needed anymore, the access could be closed or removed. For this nature of remote access, either IPSec or HTTPS/SSL VPN should be a good choice to provide private and secure connection to office/sites; since these VPN technology create "temporary tunnel" between the office and remote users or sites to provide necessary data passing between the locations. When there are no more data passing, the tunnel will be removed.

On implementation, the employees (remote users) could go to the nearest Internet cafe or could use public wireless network to establish IPSec tunnel or HTTPS/SSL to office for work; assuming the employees have necessary tools or equipments.

Between Broadband and Dedicated Circuit

For most small organizations, broadband connection such as DSL and Cable Internet are preferred instead of having dedicated point-to-point circuit due to financial constraint. To provide the private and secure site-to-site connection, such organizations would utilize HTTPS/SSL, IPSec, or both technology.

As illustration, there is a small organization that has two sites. One site has DSL and another has Cable Internet connection. To provide a private an secure site-to-site connection, the organization has a choice to deploy T1 circuit to connect the two sites. Another choice is to deploy IPSec VPN tunnel between sites where each site utilizes the existing broadband connection.

Since the T1 circuit is "more expensive" than the DSL or Cable Internet, the organization then chooses to deploy the second choice. Keep in mind that DSL and cable Internet have lower SLA compared to the dedicated circuit. When the broadband connection is down, the ISP response time will be longer than the dedicated circuit ISP response time.

In addition, these VPN technology could be down "by itself" without obvious reason. Using dedicated circuit, in general the connection is more stable.

MPLS

MPLS is OSI Layer-2/3 VPN approach which is using dedicated point-to-point circuit between organization site to its ISP. Unlike the previous Dedicated Circuit network, MPLS will use the ISP public network that ride over ISP IP-based network devices without deal with the customer IP information. In other word, MPLS approach is somewhat between the Dedicated Circuit approach and IPSec VPN approach.

Generally speaking, ISP network will handle the VPN aspect and use the ISP public network securely and privately; which will be transparent to the organization (the ISP customer) sites. Using MPLS, site-to-site connection is pretty much like the previous dedicated site-to-site connection between sites from the organization perspective.

Network-Layer Site-to-Site Connection Approach

The network-layer site-to-site connection approach refers to IPSec VPN, Dedicated Circuit, and MPLS technology. As mentioned, this network-layer approach is needed to provide connection to the remote sites for any application type including non-web-based applications.

The next discussion will relate to considerations of having such site-to-site connection. Note that these considerations apply to site-to-site connection and do not apply to road-warrior-to-site connection.

Network Topology

When there are only two sites to communicate, the site-to-site connection setup should be just a straight point-to-point. When there are more sites to communicate, there are further considerations to review. One of the consideration is the network topology. Most common site-to-site network topology setup for three sites or more as follows
1. Full Mesh
2. Hub and Spoke
3. Partially Mesh

Full Mesh

With Full Mesh connection, each site has dedicated connection to each other site as follows:


Typical organization that employ this connection is organization that has small number of branches or sites with relatively low data throughput.

When the organization has dedicated point-to-point circuits, then there will be (let's say) multiple dedicated T1 connections between sites. Reviewing illustration above, there will be three T1 from one site to others; which make the total of six T1 circuits. When the organization had dedicated VPN tunnels, there will be a total of six tunnels which each site has three tunnels to others.

Since each site has dedicated connection to each other, there will be no single point of failure. If one site is down, other sites still have connections within themselves.

However this kind of setup is considered high cost to manage when number of sites grow and/or larger data throughput are pushed down. With more sites, there will be more dedicated connections to each additional sites.

With dedicated circuits, then there will be more circuits to setup at each site which may be financially prohibitive. With VPN tunnels, then there will be more tunnels to setup which may consume too much VPN device resources such as CPU and memory.

Hub and Spoke

With Hub and Spoke connection, each site will only have a single connection to one central site. This central site then has multiple connections to each other site as follows


Site A to D are called "spoke" and Site Z is called "hub". Note that some people refer this setup as "star topology".

Usually medium to large organizations have this setup. The hub is usually the corporate office and the spokes are branches, smaller offices, or remote offices.

When the organization uses dedicated circuits, there is only a single circuit needed to connect any other sites. With VPN tunnels, the VPN device resources are not consumed much compared to the Full Mesh setup.

The down side is that there is a single point of failure at Site Z (the central site). When this site is down, then all other sites lose connections.

Partially Mesh

Reviewing the two previous setup, you may wonder which the feasible setup that has no single point of failure but not cost prohibitive. The answer is probably the Partially Mesh setup.

With Partially Mesh setup, there will be not much existing connections like Full Mesh; and no single point of failure like Hub and Spoke. Following is illustration.


The Site Y and Site Z are the "hubs". Site A to F are "spokes" to both Site Y and Site Z.

This setup is the preferred one on medium to large organizations. The both hubs are usually two large offices. The spokes are branches, smaller offices, or remote offices.

IP Routing

With either Point-to-Point, Hub and Spoke, Full Mesh, or Partially Mesh network setup; IP routing should be used to interconnect all sites. With this in mind, each site has its own subnet. Router will be used to interconnect sites.

Specifically for IPSec VPN, you could consider to have the router to terminate the VPN tunnel. You could also consider using dedicated VPN box such as firewall or VPN concentrator to provide the VPN tunnel; and use router only to interconnect sites.

Combination of Point-to-Point and Partially Mesh

As mentioned, traditional connection between two sites is just a single point-to-point. However it is possible to have redundant (multiple) point-to-point connection between two sites to provide automatic failover and/or load balance mechanism; where each connection has its own circuit on each site.

Following is the illustration. Let's say there are two sites that have two redundant point-to-point connections between each other. One site has a dedicated point-to-point T1 circuit to the other site and DSL connection. Another site has the other end of dedicated point-to-point T1 circuit and Cable Internet connection. Between the DSL on one site and Cable Internet on the other site, there is a IPSec VPN tunnel connecting the two sites as alternate path of the T1.

With such automatic failover and/or load balance mechanism in mind, following setup could be in place as well.

• Redundant connections between two Hubs in Partially Mesh network

• Redundant connections between one Hub and one Spoke

When there are redundant connections, it means there are multiple path between two sites. Note that with Full Mesh and Partially Mesh network, there are also multiple path between two sites. For such multiple path, dynamic IP routing should be deployed to optimize connections. In addition, packet-based or destination-based load balancing could be considered as well. With hub and spoke setup, static routing should be sufficient.

Starting to Design the Network

When you start designing the network, several aspects come into play

• Circuit choice

• IP address or subnet to use

• Routing protocol to provide connection

Typical network design for site-to-site connection from circuit choice perspective are following

• Dedicated circuit between sites; either uses private point-to-point, frame relay, or MPLS

• Dedicated circuit between sites as primary connection and IPSec VPN tunnel between sites as alternate connection

• IPSec VPN tunnel between sites

For small organizations, it is probably preferable to have full-mesh site-to-site VPN using broadband connection (DSL or Cable Internet) at each location. For simplicity, it is suggested to use the same ISP to provide the broadband connection at all sites. As illustration, all sites could be using Cisco ASA 5505 with 30MBps Cable Internet connection to have the full-mesh site-to-site VPN.

When you choose to have partially mesh or hub and spoke setup (either the circuit or VPN), make sure that the hub has large bandwidth and powerful network device to handle data throughput from other sites. As illustration, the hub could be using Cisco 3825 router with DS-3 circuit where spokes could be using Cisco 1841 router with 1.5MBps DSL connection to have hub-and-spoke site-to-site VPN.

Note:
For more info on Cisco equipment performance, check out the following FAQ
»Cisco Forum FAQ »Cisco Equipment Performance (per pps and Mbps)

Following is illustration. Let's say you decide to use the second choice where there are dedicated circuits between sites as primary connection and IPSec VPN tunnel over the Internet between sites as alternate connection. To start designing the network, you may start to question yourself these and go from there.

• Do you need dedicated equipment for Internet gateway and another for private site-to-site connection?

• Which is the suitable routing protocol to set dedicated circuit as primary connection and to set IPSec VPN tunnel as alternate connection?

• Is there possibility of site-to-site interconnectivity without going over IPSec VPN tunnel eventhough the connection goes over the Internet?

• Which IP address or subnet to use, Private or Public IP address?

• Will there be a NAT/PAT process in place?

• How much budget to spend to cover everything (equipments, circuits, infrastructures, etc.)

• How much of connection downtime you can tolerate

• How much of data throughput travel across each connection

• How long it takes to test the new network setup

• How immediate you need to have "live" network

Next discussions will view other important aspects.

Network Device Choice

When the organization chooses to use dedicated circuits to have private site-to-site connections, usually the network device would be either router or layer-3 switch where the WAN port would match the circuit specification.

Let's say the circuit would be Frame Relay and the organization selects Cisco router for all sites as the network device. You would use the router WAN port to connect to the Frame Relay circuit. This WAN port should be something like WIC T1 or E1 for internal DSU/CSU or WIC 1T for external DSU/CSU.

If the circuit is Gigabit Ethernet for example, then the network device could be a router or layer-3 switch. In Cisco world, the router could be something like 2821 model; and the layer-3 switch could be something like Catalyst 3750 switch.

When VPN connection is selected to provide the private site-to-site connection, there are also multiple network device alternatives such as router, layer-3 switch, firewall, and VPN concentrator. For small businesses, typical choices are firewall and router. In Cisco world, the firewall is ASA 5500 series and the router is 800 series or higher.

Whichever network device chosen, it is suggested to have the same brand for all of them. When you decide to use Cisco equipments let's say, then all sites should also use Cisco as the network device peer. In theory, multi-vendor equipments are inter-operate-able. However in practice, there are sometime unexpected behaviors when establishing connections between multi-vendor equipments. With single-vendor equipment, network behaviors are more predictable and controllable, leads to more stable network.

Another aspect of having the same-vendor equipments throughout the organization is network administration simplification. Network administrators could concentrate to only single brand/vendor to administer. You don't have to deal with multi vendors when it comes to the network device technical or customer support. You might even receive discounts when you have device large volume number from the same single vendor.

Note:
To guide you in choosing the proper Cisco equipment, check out the following FAQ
»Cisco Forum FAQ »Which Cisco solution is right for my situation?

Internal and External Connections

All the site interconnections such as file transfer between sites are considered internal connection. External connection is a connection to an outside world, such as connection to server located at the Internet or at external site; or Internet browsing.

For internal connections, the traffic should take the private connection. For external connections, there are multiple choices to consider. One way is to go directly out off the site to the external site. Another way is to go through other internal site before going out to the external site.

Let's review the following situation. Let's say one remote office need to have the updated Microsoft Windows patches. To retrieve the patches, there are several choices. One is to go directly out to the Internet, access the Microsoft sites, and download patches. Another way is to go to central office where the central office run a server that provide updated patches.

For small organizations, usually the preferred way for the remote office to receive the patches is by going directly out to the Internet to retrieve patches. However some situations require the remote office to access the central office's server to retrieve patches.

Should the organization have this second situation, there would probably a need to configure remote office network device to direct traffic to the central office's server for remote office upgrade patch need; and block any attempt from remote office to access the Internet directly to retrieve patches. With this situation, the network is considered more secure since the traffic is more controllable.

Remote Site and Internet Access

In some situations require remote office to access central office before accessing external sites. However situation such as Internet browsing could not require central office access from remote office perspective. The remote office could just go out to the Internet for Internet browsing.

A good side of accessing the Internet directly without going through central office is that the central office bandwidth is not bogged down by the remote office's Internet traffic. The central office bandwidth then can be conserved for strictly internal access such as file sharing.

The down side of this approach is that the central office probably has no or minimum control of remote office's Internet access activities. Without such control, there is possible security risk or improper use of Internet access such downloading illegal software or virus/worm attack without the central office approval. Therefore for larger organizations, all traffic from remote offices including Internet access must go through central offices for data traffic management, including traffic policing at all sites. Note that from network security and network management perspective, traffic policing at all sites might be considered necessary even though it could create network administrative burden.

Keep in mind that it is possible to have the same level of control of remote office Internet access activities as the central offices when those remote offices have their own local Internet connection. With this kind of setup, the organization then has to control multiple Internet connection that are spread among multiple sites (both central and remote offices). Any type of control that take place in central offices must take place in remote offices as well. This is also a common practice for larger organizations. Note that this kind of remote office control might mean additional investment on each remote office to duplicate or to mimic central office.

Whichever the preferred setup, the network administrator should consider the trade offs between the two setup choices. For small business, direct Internet access from remote offices could be the preferred choice. When the organization is concerned more on the network security, then the organization might consider the second setup choice.

IPSec VPN and Internet (External Connection) Access

Let's say an organization permit their remote offices to go out to the Internet directly without going through central office. Typically there would be two separate connections at the remote office. One is to serve the internal access and another is to serve the Internet access.

Specifically for organizations that use IPSec VPN connections to serve the site inter-communication, there should be some kind of split tunneling to provide the separate connections between the Internet access and internal access. For Internet access, typically PAT (Port Address Translation) is used to bridge Private Subnet used in internal network (LAN) and the Internet. Using PAT; application traffic that use the most common IP protocol such as TCP, UDP (and ICMP) from local LAN are PAT-ed to the Public IP address.

Let's review the IPSec VPN tunnel setup requirement. IPSec tunnel would use IP Protocol 50 (ESP) or 51 (AH) to setup the VPN tunnel. Unlike TCP and UDP, ESP and AH have no concept of port numbers; hence in theory, these security protocols cannot be PAT-ed.

Should the organization permit remote offices to go out to the Internet directly and the organization deploys VPN tunnel to serve internal access; then each site should have at least two Public IP addresses. One IP address would serve the Internet access (to be PAT-ed as many as needed) and another IP address would be reserved for the VPN peer to other sites (or for any IP protocols that are un-PAT-able).

For small business, it is probably preferable to have each site having those two Public IP addresses assigned to the same gateway (or peer) network device, which then the traffic will ride over the same circuit. For medium or large business that quite large number of sites, each Public IP address could reside at different network device and could ride over different circuit.

Name Resolution

In sharing files between sites, the organization might use DNS server to resolve name to IP addresses. When the organization deploys Microsoft network, then there might also be WINS server in addition to the DNS server.

Let's say the organization permit remote office to go out to the Internet directly without going through the central office. The preferred way is to have the remote office to use the local ISP DNS server to reach the Internet sites. For internal access, the remote office uses internal DNS server to reach internal servers. The unwanted setup is to have the remote office to use the central office's internal DNS server to access the Internet since it will bog down the central office's bandwidth.

To have the preferred way, there are alternatives to setup the DNS/WINS servers at remote offices. One way is to setup local DNS/WINS server at each remote site. With this setup, any traffic (internal or external traffic) from remote office will use the local DNS/WINS server. The central office's DNS/WINS servers will be used only if the traffic are internal. When the traffic are external, only ISP DNS server will be used. The external traffic from remote office will never go through the central office. The down side is that this setup is probably cost prohibitive, not to mention network administration prohibitive.

Another way to setup is to assign multiple DNS/WINS IP addresses at remote site hosts. Assign both central office's DNS/WINS servers and also assign the remote site's local ISP DNS IP addresses to all remote site hosts. In addition, there might be a need to create traffic filtering on the remote office's network device to allow name resolving traffic to use central office's DNS/WINS server only when the traffic are internal; and to block attempted central office's DNS/WINS server access for external traffic. Similarly, there would be traffic filtering to allow name resolving traffic to use the local ISP DNS IP address only when the traffic are external. With this setup, there should be no need to deploy DNS/WINS servers at each remote site to provide name resolving and still be able to avoid central office bandwidth bogged down by the remote office's external traffic.

Real Network Illustration

Check out the following threads for illustration

»IPsec help 1811
»[HELP] BGP Failover to IPSEC
»How to Loadshare between a E1 LInka nd Ebgp(MPLS) Link
»[Config] LAN Extension L2 config help

Deployment Process

Check out the following FAQ for following topics in network design

1. Between Hub and Spoke, Full Mesh, and Partially Mesh
»Cisco Forum FAQ »Tips in Designing Network on Hub-and-Spoke, Full-Mesh, or Partially-Mesh setup

2. IPSec VPN
»Cisco Forum FAQ »Between GRE/IPSEC and IPSEC VPN tunnels
»Cisco Forum FAQ »Various Site-to-Site IPSec VPN: Cisco, Juniper, Checkpoint, Sonicwall, Zywall
»Cisco Forum FAQ »Private Routing over VPN: NAT/PAT, GRE, IPSec Sample Configurations

Suggested pre-reading
»Cisco Forum FAQ »Straight-forward way to configure Cisco router: Introduction to CLI

CCNA level Cisco Commands and Descriptions

Following is a list of commands that are applicable to most IOS-based equipments such as routers and switches. Check out the following links for full commands.

IOS Commands 12.4 version on Routers
IOS and Catalyst OS Commands on 6500 series Switches
IOS Commands 12.2 version on 4500 series Switches
IOS Commands 12.2 version on 3560 series Switches

? Gives you a help screen

0.0.0.0 255.255.255.255 A wildcard command; same as the any command

access-class Applies a standard IP access list to a VTY line

access-list Creates a list of tests to filter the networks 9

any Specifies any host or any network; same as the 0.0.0.0 255.255.255.255 command

Backspace Deletes a single character

bandwidth Sets the bandwidth on a serial interface

banner Creates a banner for users who log into the router

cdp enable Turns on CDP on an individual interface

cdp holdtime Changes the holdtime of CDP packets

cdp run Turns on CDP on a router

cdp timer Changes the CDP update timer

clear counters Clears the statistics from an interface

clear line Clears a connection connected via Telnet to your router

clear mac-address-table Clears the filter table created dynamically by the switch

clock rate Provides clocking on a serial DCE interface

config memory Copies the startup-config to running-config

config network Copies a configuration stored on a TFTP host to running-config

config terminal Puts you in global configuration mode and changes the running-config

config-register Tells the router how to boot and to change the configuration register setting

copy flash tftp Copies a file from flash memory to a TFTP host

copy run start Short for copy running-config startup-config; places a configuration into NVRAM

copy run tftp Copies the running-config file to a TFTP host

copy tftp flash Copies a file from a TFTP host to flash memory

copy tftp run Copies a configuration from a TFTP host to the running-config file

Ctrl+A Moves your cursor to the beginning of the line

Ctrl+D Deletes a single character

Ctrl+E Moves your cursor to the end of the line

Ctrl+F Moves forward one character

Ctrl+R Redisplays a line

Ctrl+Shift+6, then X (keyboard combination) Returns you to the originating router when you telnet to numerous routers

Ctrl+U Erases a line

Ctrl+W Erases a word

Ctrl+Z Ends configuration mode and returns to EXEC

debug dialer Shows you the call setup and teardown procedures

debug frame-relay lmi Shows the lmi exchanges between the router and the Frame Relay switch

debug ip igrp events Provides a summary of the IGRP routing information running on the network

debug ip igrp transactions Shows message requests from neighbor routers asking for an update and the broadcasts sent from your router to that neighbor router

debug ip rip Sends console messages displaying information about RIP packets being sent and received on a router interface

debug ipx Shows the RIP and SAP information as it passes through the router

debug isdn q921 Shows layer-2 processes

debug isdn q931 Shows layer-3 processes

delete nvram Deletes the contents of NVRAM on a 1900 switch

delete vtp Deletes VTP configurations from a switch

description Sets a description on an interface

dialer idle-timeout number Tells the BRI line when to drop if no interesting traffic is found

dialer list number protocol protocol permit/deny Specifies interesting traffic for a DDR link

dialer load-threshold number inbound/outbound/either Sets the parameters that describe when the second BRI comes up on an ISDN link

dialer map protocol address name hostname number Used instead of a dialer string to provide more security in an ISDN network

dialer string Sets the phone number to dial for a BRI interface

disable Takes you from privileged mode back to user mode

disconnect Disconnects a connection to a remote router from the originating router

duplex Sets the duplex of an interface

enable Puts you into privileged mode

enable password Sets the unencrypted enable password

enable password level 1 Sets the user mode password

enable password level 15 Sets the enable mode password

enable secret Sets the encrypted enable secret password. Supersedes the enable password if set

encapsulation Sets the frame type used on an interface

encapsulation frame-relay Changes the encapsulation to Frame Relay on a serial link

encapsulation frame-relay ietf Sets the encapsulation type to the Internet Engineering Task Force (IETF); connects Cisco routers to off-brand routers

encapsulation hdlc Restores the default encapsulation of HDLC on a serial link

encapsulation isl 2 Sets ISL routing for VLAN

encapsulation ppp Changes the encapsulation on a serial link to PPP

erase startup Deletes the startup-config

erase startup-config Deletes the contents of NVRAM on a router

Esc+B Moves back one word

Esc+F Moves forward one word

exec-timeout Sets the timeout in seconds and minutes for the console connection

exit Disconnects a connection to a remote router via Telnet

frame-relay interface-dlci Configures the PVC address on a serial interface or subinterface

frame-relay lmi-type Configures the LMI type on a serial link

frame-relay map protocol address Creates a static mapping for use with a Frame Relay network

Host Specifies a single host address

hostname Sets the name of a router or a switch

int e0.10 Creates a subinterface

int f0/0.1 Creates a subinterface

interface Puts you in interface configuration mode; also used with show commands

interface e0/5 Configures Ethernet interface

interface ethernet 0/1 Configures interface e0/1

interface f0/26 Configures Fast Ethernet interface 26

interface fastethernet 0/0 Puts you in interface configuration mode for a Fast Ethernet port; also used with show commands

interface fastethernet 0/0.1 Creates a subinterface

interface fastethernet 0/26 Configures interface f0/26

interface s0.16 multipoint Creates a multipoint subinterface on a serial link that can be used with Frame Relay networks

interface s0.16 point-to-point Creates a point-to-point subinterface on a serial link that can be used with Frame Relay

interface serial 5 Puts you in configuration mode for interface serial 5 and can be used for show commands

ip access-group Applies an IP access list to an interface

ip address Sets an IP address on an interface or a switch

ip classless A global configuration command used to tell a router to forward packets to a default route when the destination network is not in the routing table

ip default-gateway Sets the default gateway of the switch

ip domain-lookup Turns on DNS lookup (which is on by default)

ip domain-name Appends a domain name to a DNS lookup

ip host Creates a host table on a router

ip name-server Sets the IP address of up to six DNS servers

IP route Creates static and default routes on a router

ipx access-group Applies an IPX access list to an interface

ipx input-sap-filter Applies an inbound IPX SAP filter to an interface

ipx network Assigns an IPX network number to an interface

ipx output-sap-filter Applies an outbound IPX SAP filter to an interface

ipx ping A Packet Internet Groper used to test IPX packet on an internetwork

ipx routing Turns on IPX routing

isdn spid1 Sets the number that identifies the first DS0 to the ISDN switch

isdn spid2 Sets the number that identifies the second DS0 to the ISDN switch

isdn switch-type Sets the type of ISDN switch that the router will communicate with; can be set at interface level or global configuration mode

K Used at the startup of the 1900 switch and puts the switch into CLI mode

line Puts you in configuration mode to change or set your user mode passwords

line aux Puts you in the auxiliary interface configuration mode

line console 0 Puts you in console configuration mode

line vty Puts you in VTY (Telnet) interface configuration mode

logging synchronous Stops console messages from overwriting your command-line input

logout Logs you out of your console session

mac-address-table permanent Makes a permanent MAC address entry in the filter database

mac-address-table restricted static Sets a restricted address in the MAC filter database to allow only the configured interfaces to communicate with the restricted address

media-type Sets the hardware media type on an interface

network Tells the routing protocol what network to advertise

no cdp enable Turns off CDP on an individual interface

no cdp run Turns off CDP completely on a router

no inverse-arp Turns off the dynamic IARP used with Frame Relay; static mappings must be configured

no ip domain-lookup Turns off DNS lookup

no ip host Removes a hostname from a host table

No IP route Removes a static or default route

no shutdown Turns on an interface

o/r 0x2142 Changes a 2501 to boot without using the contents of NVRAM

ping Tests IP connectivity to a remote device

port secure max-mac-count Allows only the configured amount of devices to attach and work on an interface

ppp authentication chap Tells PPP to use CHAP authentication

ppp authentication pap Tells PPP to use PAP authentication

router igrp as Turns on IP IGRP routing on a router

router rip Puts you in router rip configuration mode

secondary Adds a second IPX network on the same physical interface

Service password-encryption Encrypts the user mode and enable password

show access-list Shows all the access lists configured on the router

show access-list 110 Shows only access list 110

show cdp Displays the CDP timer and holdtime frequencies

show cdp entry * Same as show cdp neighbor detail, but does not work on a 1900 switch

show cdp interface Shows the individual interfaces enabled with CDP

show cdp neighbor Shows the directly connected neighbors and the details about them

show cdp neighbor detail Shows the IP address and IOS version and type, and includes all of the information from the show cdp neighbor command

show cdp traffic Shows the CDP packets sent and received on a device and any errors

Show controllers s 0 Shows the DTE or DCE status of an interface

show dialer Shows the number of times the dialer string has been reached, the idle-timeout values of each B channel, the length of call, and the name of the router to which the interface is connected

show flash Shows the files in flash memory

show frame-relay lmi Shows the LMI type on a serial interface

show frame-relay map Shows the static and dynamic Network layer-to-PVC mappings

show frame-relay pvc Shows the configured PVCs and DLCI numbers configured on a router

show history Shows you the last 10 commands entered by default

show hosts Shows the contents of the host table

show int f0/26 Shows the statistics of f0/26

show inter e0/1 Shows the statistics of interface ethernet 0/1

show interface s0 Shows the statistics of interface serial 0

show ip Shows the IP configuration of the switch

show ip access-list Shows only the IP access lists

show ip interface Shows which interfaces have IP access lists applied

show ip protocols Shows the routing protocols and timers associated with each routing protocol configured on a router

show ip route Displays the IP routing table

show ipx access-list Shows the IPX access lists configured on a router

show ipx interface Shows the RIP and SAP information being sent and received on an individual interface; also shows the IPX address of the interface

show ipx route Shows the IPX routing table

show ipx servers Shows the SAP table on a Cisco router

show ipx traffic Shows the RIP and SAP information sent and received on a Cisco router

show isdn active Shows the number called and whether a call is in progress

show isdn status Shows if your SPIDs are valid and if you are connected and communicating with the provider's switch

show mac-address-table Shows the filter table created dynamically by the switch

show protocols Shows the routed protocols and network addresses configured on each interface

show run Short for show running-config; shows the configuration currently running on the router

show sessions Shows your connections via Telnet to remote devices

show snmp Gives you the router's serial number as the "chassis" output

show start Short for show startup-config; shows the backup configuration stored in NVRAM

show terminal Shows you your configured history size

show trunk A Shows the trunking status of port 26

show trunk B Shows the trunking status of port 27

show version a show hardware command equivalent, which gives the IOS information of the switch, as well as the uptime and base Ethernet address

show vlan Shows all configured VLANs App.

show vlan-membership Shows all port VLAN assignments

show vtp Shows the VTP configuration of a switch

shutdown Puts an interface in administratively down mode

Tab Finishes typing a command for you

telnet Connects, views, and runs programs on a remote device

terminal history size Changes your history size from the default of 10 up to 256

trace Tests a connection to a remote device and shows the path it took through the internetwork to find the remote device

traffic-share balanced Tells the IGRP routing protocol to share links inversely proportional to the metrics

traffic-share min Tells the IGRP routing process to use routes that have only minimum costs

trunk auto Sets the port to auto trunking mode

trunk on Sets a port to permanent trunking mode

username name password password Creates usernames and passwords for authentication on a Cisco router

variance Controls the load balancing between the best metric and the worst acceptable metric

vlan 2 name Sales Creates a VLAN 2 named Sales

vlan-membership static 2 Assigns a static VLAN to a port

vtp client Sets the switch to be a VTP client

vtp domain Sets the domain name for the VTP configuration

vtp password Sets a password on the VTP domain

vtp pruning enable Makes the switch a pruning switch

vtp server Sets the switch to be a VTP server

Nexus NX-OS
Cisco Nexus 1000V Command Reference, Release 4.0(4)SV1(3)

IOS (routers, Catalyst switches)
Cisco IOS Configuration Fundamentals Command Reference

ASA OS 8.2 and later version
Cisco ASA Series Command Reference, S Commands

Quick Result Commands

1. Reset an IOS-based device (routers, switches) back to factory default



2. Get system info

show version

3. Get license info

show license

4. Get route table

Routers, Switches: show ip route
PIX/ASA Firewalls : show route

5. Get MAC address table

Depending on image flavors, the command is either show mac-address table or show mac address-table

6. Get active protocols

IOS (routers, layer-3 switches)
IP Protocols : show ip protocols
General protocols: show protocols

Cisco website has Feature Navigator tool which allows you to search several different ways such as

1) Image name as displayed in a "sh ver"
(example: System image file is "flash:/c3550-i5q3l2-mz.121-14.EA1a.bin")

2) by Platform
(example: 2500, 802, UBR905)

3) Serial number as displayed in a "sh diag"

4) Or by IOS major release

In order to access the tool from Cisco you need a CCO account. CCO accounts are free and simple to set up. You can register for a CCO here.

Note:
Starting with the new line of routers and switches, there is no longer these kind of confusion since there is only one IOS image file to deal with which is a Universal IOS image file. A lot of features have been included in the basic IOS image. However for those who needs advanced services such as OSPF, BGP, EIGRP (part of Advanced IP Services) or IPSec (part of Advanced Security), note that you need advanced license in order to use.

BBR has this link.
"These are the distance estimates we got from these providers, for your address."

We also have this link to the BBR site distance charts.
"CLECs and ILECs work from distance estimates or actual checks before accepting an order. Here is what we know of the distance limits they work by."

Bonded xDSL

»Bonded ADSL

CRC

»[Config] "crc 16"

DS-1/T1/E1

»[HELP] Async3 interface on cisco 890 lineup
»[OT] Why is my internet slow?
»Troubleshooting a T1 circuit
»[HELP] Virus could be attaching router
»New t1 connection: Standard CPE Configuration???
»T1 over a short run of Cat3 cable?
»What is at the other end of a T1
»E1\T1 circuit - serial interface errors
»[Config] Alarm light on 2621 T1 WIC
»Flapping PTP T1 on Cisco 1721's
»1841 service-module T1 timeslots 1-12
»Cisco 1751 with t1 card

DS-3/E-3

»DS3 comming, what do I need?
»DS3 Point To Point
»VPN & Frags

Cisco documentations

Troubleshooting Serial Lines
Understanding the show controllers e1 Command
Blue and Yellow T1 Alarm Troubleshooting
T1 Layer 1 Troubleshooting
T1 Alarm Troubleshooting
T1 Alarm Troubleshooting Flowchart
Troubleshooting Line Problems and Errors on DS-3 and E3 ATM Interfaces

ATM

»What exactly is an atm interface?

DSL/ATM Provider

»[HELP] DSL project need help with ATM switch interface?

OC-3

»Why I'm not Pinging using Alternating Pattern 0x5555?

Metro Ethernet

»Fiber GBIC Question

Data Center/Cross Connect/Carrier Hotel

»[HELP] ASR 9k troubleshooting input errors and crcs on a point to point CTC

PDF file
Practical Introduction to Optical Network Engineering

Discussion
»DWDM fiber rings
»Optical big iron
»[Config] Cisco Switch to Switch Fiber Connectivity Issue

Ever wonder how your ISP actually announce your subnet via BGP to the Internet? Are there any summarizations in place or is your subnet announced as it is? Using the following Looking Glass access, you can find that out.

The routers in these links reside in public Internet. Therefore the BGP view should reflect your ISP BGP subnet announcement policy.

BGP IPv4/IPv6 Looking Glass Servers - BGP Route Servers
World Route Servers

Some discussion
»[HELP] Anything to look out for when setting up a new WAP?
»Is a WLC worth it?

Sample Configuration
»Cisco Forum FAQ »Wireless Router Sample Configuration

Further info
»Wireless Networking Forum FAQ

»[CCNA] Sanitizing a 3005 Concentrator...

Introduction

In typical "real-life" network, there should be some kind of automatic network health monitoring and reporting system. The idea of having such system is to have some network health management and report to provide at least general idea of how network health state is at.

One aspect of network health management is the monitoring part. With automated system, one can receive automatic alert of general network health such as up/down connection, bandwidth utilization, network device status and utilization. Those automatic alert can be in form of either email, SMS/text, or flashy display on your PC monitor should the system detects issues. Such automatic alert is helpful when there are too many network devices to manage or there are no luxury to manually monitor network health in real time.

As illustration, let's say there is a T1/E1 circuit that is crucial to business requirement, either as Internet circuit, private link, or the like. To ensure smooth business transaction over this circuit, the circuit's bandwidth utilization should never reach above 80%. You as network administrator would like to know if and when the circuit bandwidth utilization is "too high" without spending time manually watch the circuit utilization.

With automated network health monitoring system, you can set the system to send you "yellow" (warning) alert when the circuit bandwidth utilization reaches 50% and to send you "red" (crucial) alert when the circuit bandwidth utilization reaches 80%.

As mentioned, the automatic alert can be in form of either email, SMS/text, or flashy display on your PC monitor should the system detects issues. Therefore you could be physically away from the circuit doing other things yet you don't miss the moment of when the circuit is "over-utilized".

Knowing immediately whether bandwidth currently over-utilized is great to know especially when users complaining of slow access, either slow Internet access (if the circuit is Internet circuit) or slow access to private server (if the line is private line). By knowing such info immediately, your job as network administration will be less troublesome since you already have the valid cause of such slow access (latency issue).

The next aspect of network health management is the reporting part. With automated system, one can receive report of network usage within certain time range. This report type varies, which can include how often a circuit is up/down, how much bandwidth utilization is on certain circuit or connection, how much memory and CPU utilization is on certain network device, and how slow/fast certain application or software response is; depending on the automated system feature.

As illustration, let's say you like to know how a circuit bandwidth utilization looks like since last month. With automated network health reporting system, you can set the system to send circuit bandwidth utilization report starting from last month to today.

The monthly report typically shows some bar graph with daily use of the circuit bandwidth. On the report you may see that on Day 1, the circuit bandwidth is used up to 40%. On Day 15 let's say, the report may say that the circuit bandwidth is used up to 80%.

With this kind of report, it will be useful to track the circuit bandwidth utilization level. When the circuit bandwidth is too often over-utilized (too often of 80% utilization let's say), then further action might be in order. Such action could be an investigation of what kind of traffic using the bandwidth and if those traffic are either legitimate or illegitimate. Another action could be considering of upgrading the bandwidth to larger one.

How the Automatic Network Health System looks like

The automatic network health monitoring and reporting system itself is a software installed in some server (typically either Unix, Windows, vendor-specific, or proprietary server). The software will communicate with the network devices to be monitored in some kind of protocol, which will be explained later. The network devices that can be monitored vary; typically routers, switches, firewall, server, printer, and wireless access point.

Automatic Network Health System mechanism

Most common monitoring system deal with IP-based network devices, meaning any devices that can have IP address. There are some monitoring system that deal with non-IP-based devices. This non-IP-based devices are typically legacy or "old-school" devices such as analog PBX or phones and legacy DAX in telco environment.

The IP-based monitoring system as mentioned communicate with the monitored network devices use some kind of protocol (IP protocol). Most common protocol used are ICMP, TCP-, or UDP-based protocol. Example of TCP- or UDP-based protocol used is Syslog, SNMP, and Netflow (Cisco specific).

Note that more advanced IP-based monitoring system can also monitor using higher level protocol like HTTP and SQL databases. In addition, this kind of software or application monitoring system can also detect and monitor IM (Instant Messaging) protocol and even peer-to-peer protocol such as Kaaza and eDonkey. This software or application monitoring system is typically deployed when specific software or application performance is crucial to business requirement.

There are a lot of software out there that do the IP-based monitoring, from the "free" version to "premium-pay" version. Following are some of technology key words on how the software is designed.

* Syslog
* ICMP (Internet Control Message Protocol)
* SNMP (Simple Network Management Protocol)
* Netflow (Cisco specific)
* Software/Application performance monitoring: HTTP, HTTPS, SQL databases, IM, peer-to-peer protocols

Syslog

Typical business-grade network devices (i.e. routers, firewalls, switches from major vendor such as Cisco and Juniper) should be able to generate some kind of logs due to some event or incident such as up/down interface, routing updates, and configuration changes. This kind of logs in general are in the form of syslog messages. By default, these syslog messages are stored within the devices themselves.

When you have an automatic health monitoring system, the system should have a syslog server which collects all syslog messages generated by all network devices. To have this, following are the general idea.

* Install a syslog server
* Configure the server to receive and to store syslog messages from your network devices
* Configure your network devices to send syslog messages to syslog server

Note that you should be able to check syslog messages on the network devices themselves. However those devices are not designed to store syslog messages for a long time. Usually after a short period of time, the logs are deleted. Using a syslog server, you can store syslog messages much longer period (typically for 1 to 3 months) and even can back up the messages to other media such as tape backup.

ICMP (Internet Control Message Protocol)

In a lot of time, you may need to see if certain circuit or Internet connection is up or down. One simplest and common way to find out is to ping the Internet gateway (your ISP equipment) or pretty much any device that is at the other side of the circuit. This ping mechanism is based on assumption of receiving ICMP echo reply from the device you monitor in certain time frame as a response of ICMP echo your monitoring system is sending. If in certain time the ICMP echo reply is not received, the other end device or the connection could be safely assumed to be either down or busy.

Most network devices by default should be ping-able. By ping-able means that the device will send ICMP echo reply as a response to the ICMP echo it receives. Note that certain firewall however by default will not be ping-able. Should you choose to monitor network devices by ICMP, verify if the devices response to ping.

SNMP (Simple Network Management Protocol)

In some cases, having a syslog server to collect syslog messages are insufficient. One case is that syslog messages don't provide more specific info regarding specific events or devices such as device CPU or memory utilization, bandwidth utilization, and device temperature. This is something that SNMP does provide.

SNMP is another essential part of your automatic health monitoring system. Similarly to Syslog, a SNMP server collects SNMP traps from SNMP clients. These SNMP clients could be any IP-based network devices such as routers, firewalls, switches, printers, and production servers (i.e. web or mail). As mentioned; up/down interface, CPU and memory utilization, port or bandwidth utilization, temperatures, and low on laser printer toner are just little things SNMP traps from specific devices can represent those device health condition. Depending on the network device feature, you may be able to configure the device to generate limited choice or large choice of SNMP traps.

Once SNMP server receives all of those SNMP traps, the server can generate reports on those specific conditions. If you like to see CPU and memory utilization on specific SNMP clients within certain time range for instance, you can pull a report regarding those. You can do similar task for switch port utilization.

Further, you can link your SNMP server to your mail server. This way you (or just anybody within your company) can receive mail alert when specific condition take place such as device temperature hits 80 degree Fahrenheit, CPU or memory utilization of a device hits 80% or more, and down devices.

Typically only business-grade network devices support SNMP. This support means that the device will generate SNMP traps and is capable to send those SNMP traps to certain SNMP server. Should you decide to monitor the network device condition by SNMP, verify such SNMP trap you look for (i.e. up/down interface, CPU and memory utilization, port or bandwidth utilization, temperatures) is supported on the device.

Cisco Netflow

Specifically for bandwidth utilization, SNMP report only tells how much specific port or connection is utilized (i.e. 10% or 90% utilized). However the report does not tell you which traffic are utilizing the bandwidth.

When your network devices are Cisco that can provide Netflow reports, you can utilize Netflow to provide such specific details. In a nutshell, the Netflow reports show which traffic are utilizing the bandwidth from perspectives of source and destination IP address, TCP or UDP port, and how many IP packets are going through. For instance, your internal user (let's say 10.0.10.254 IP address) accesses your internal webserver (let's say 10.0.0.2 on TCP port 80) and www.yahoo.com on the Internet using 80% of available bandwidth.

More info on Cisco Netflow is available here

Software/Application Performance

A lot of time, network or Internet slowness is caused by software or application run on server or PC. This software or application could be mail (SMTP), web (HTTP, HTTPS, SSL, TLS), FTP, SQL databases, or even peer-to-peer applications such as Kaaza and eDonkey. Beside monitoring the network, monitoring the software and/or application performance is highly recommended as these software and/or application can be written incorrectly by the developers, causing poor performance.

There are many monitoring system you can choose as the software or application performance monitoring. Some of them are OPNET and Ixia. By using OPNET for example, you can find out exactly what happen during the client-server relationship on some software or application and if those events of client-server relationship happen as expected or not. The monitoring result should give you ideas of what happen and if the events you see may cause performance problem.

Another example of application monitoring system is CSM (Cisco Security Manager) which can detect the use of IM (Instant Messaging) protocol such as Yahoo! or AOL IM; in addition to peer-to-peer protocol detection such as Kaaza and eDonkey based on Cisco network appliance reporting such as ASA firewall and IPS module. In some organization or company, the use of such application is forbidden, especially peer-to-peer application since such application can use up available bandwidth to the point where no more available bandwidth can be used for business-legitimate application.

More info on CSM can be found here

To monitor HTTP and HTTPS connectivity status, an HTTP Watch application can also be considered.

Software To Choose as Automatic Monitoring and Reporting System

Note that you don't have to use the mentioned monitoring system. Those mentioned monitoring system are just picked as illustration (although they are proven to work and helpful on real-life production network). As a rule of thumb, any monitoring system should do as long as they are able to serve your need.

There are many software that can do Syslog, ICMP, SNMP, and Netflow collection and report as mentioned. A lot of companies like to use Solarwind or Whatsup products. Some companies like to use CiscoWorks, which may include CSM.

There are free ICMP and SNMP software that are widely used such MRTG and Cacti. One popular free Syslog software is Kiwi Syslog.

As mentioned, basically any software that you think work should do. Typically the "premium-pay" software is preferred when you have a large or complex networks, or you like details or thorough reports.

Related Topic
»Cisco Forum FAQ »Improving Small Business network performance

Discussion

»Free Simulated Network Congestion Tool
»Network Monitoring
»[OT] Network Test tool http/Sql/Mapi/SIP, etc
»Program for monitoring DSL statistics
»Show Ip flow Top-talkers detail
»[Config] NetFlow on ASA

»Cisco WAAS real life results
»[OT] Help Testing my WAN Accelerator

NTP services
Objective: setup NTP server for all network devices

Meinberg NTP Software Download

Discussion
»Recommendations for Netflow software
»A Work Smarter way to map MAC>IP>hostname on Catalyst/Nexus

Introduction

Non-Official Cisco Support
Introduction - How PIX Operates and the CLI.
Basic PIX configuration
Slightly Advanced PIX Configuration
TCP, UDP, NAT and PAT as the PIX sees it
Access Control Lists and Content Filtering
Object Grouping

Official Cisco Support
Using PIX Firewall
Cisco Security Appliance Command Line Configuration Guide, Version 7.0

Security Level as Stateful Firewall feature foundation

Cisco ASA/PIX Firewall is designed as stateful firewall. From Cisco implementation perspective, there is a concept of Security Level as foundation of all stateful firewall features.

In basic firewall concept, there are three security zones. The first zone is Untrusted network where Cisco implements as Outside network. The second zone is Trusted network where Cisco implements as Inside network. The third zone is DMZ network where Cisco also implements as DMZ network.

A firewall in its nature is designed as perimeter guarding traffic flow between zones. With the concept of Security Level, the Untrusted (Outside) network has the lowest level of trust where Cisco by default assign the trust level as 0 (zero). Consequently the Trusted (Inside) network has the highest level of trust where Cisco by default assign the security level of 100. Since DMZ network is considered somewhat trusted and untrusted, Cisco by default assign (typically) even number between 0 and 100.

Based on associated Security Level; you may notice that the higher a network level is, the more trusted a network is. In other words, Inside network is more trusted or more secure that DMZ network and DMZ network is more trusted or more secure than Outside network. When you put Cisco ASA/PIX Firewall as your Internet gateway or Internet firewall for example, the Outside network is the Internet, the Inside network is your internal network, and the DMZ network is your publicly-accessible web or email server.

If you like to go further, you may segment your internal network further by putting a dedicated firewall between your internal servers and users' PC where the Inside network is where the internal servers are and the Outside network is where the users' PC are. When you consider to use only one firewall for all, then you may want to create multiple DMZ networks where the Outside network (Security Level 0) is the Internet, Inside network (Security Level 100) is the internal servers, DMZ 1 network (i.e. Security Level 1) is the publicly-accessible web or email server, DMZ 2 network (i.e. Security Level 4) is a guest wireless network, DMZ 3 network (i.e. Security Level 6) is the user's PC, and so on and so forth.

Additionally, any incoming traffic from lower Security Level to higher Security Level is by default denied. When you have publicly-accessible web or email server let's say on your DMZ network, then you have to permit certain incoming traffic from the lower Security Level (the Internet or Outside) network to enter higher Security Level network which is the DMZ by using either nat command or static command. You can also control how many incoming permitted sessions for further protection.

How Cisco ASA/PIX Firewall Treats TCP-based traffic differently than ICMP-based traffic

In general, typical traffic initiated from Inside network (Trusted) to Outside network (Untrusted) are permitted. These typical traffic are mostly protocols TCP and UDP based traffic. In OS 6.x or earlier, the fixup command controls these traffic types by default, while the inspect command instead controls the traffic in OS 7.0 and later.

Protocol ICMP on the other hand is considered not part of typical traffic, in addition to different behavior compared to protocols TCP and UDP. In order to make the ICMP traffic flow go successfully through the firewall from Inside to Outside, you have to create permit rule for incoming ICMP echo reply packets from least trusted network as a response of ICMP echo packets issued by a machine within more trusted network.

For TCP-based traffic, by default all returning TCP traffic coming from least trusted network as a response of TCP packet initiated by a machine within more trusted network are permitted. Therefore you don't need to create rules to permit such returning TCP traffic.

The reason of no need to create rules to permit such returning TCP traffic is that the firewall understands the concept of 3-way TCP handshake. Every time there is outbound TCP-based traffic initiated from more trusted network to less trusted network is inspected and stored in connectivity table (the show conn command reveals such table). When the firewall sees matching TCP packet coming from less trusted network toward the more trusted network as part of the 3-way handshake, the firewall permits those returning traffic. In PIX/ASA firewall implementation, the fixup and inspect commands ensure these situations take place.

ICMP-based traffic however has different properties. Since there is no concept of 3-way handshake in ICMP, each ICMP traffic is treated as one-way traffic. Therefore you have to permit any necessary incoming ICMP traffic from less trusted network towards more trusted network when you plan to use something like ICMP ping or traceroute from more trusted network to less trusted network.

Discussion
»[HELP] ASSA 5520 ASDM 7.1 denied traffic
»[Config] Connectivity issues behind my ASA 5505

TCP Transaction Protection

Of those TCP traffic, all incoming TCP traffic are inspected by Cisco ASA/PIX Firewall to make sure that there will be a 3-way handshake per TCP mechanism to complete TCP transaction. The firewall will drop any incomplete TCP transaction for protection from possible TCP-based attack.

As example, the firewall keeps TCP session as part of the TCP 3-way handshake protection mechanism where there is some kind of hold timer. The firewall expects to receive responses from server within the hold timer interval, which the timer will expire. At the time the firewall does not receive the server response when the timer expires, the firewall drops any related TCP session and also drops "late" server response.

Another example is having the firewall drops TCP packets when the TCP client keeps sending TCP synchronization (SYN) packet or sending TCP acknowledge (ACK) packet without sending TCP SYN packet first. In this situation, the firewall drops the TCP SYN and TCP ACK accordingly.

There is also a TCP Initial Sequence Number (ISN) randomization protection feature which by default randomizing TCP sequence number to negotiate between client and server in order to provide TCP Sequence Prediction Attacks protection.

One optional feature is setting maximum number of simultaneous TCP and UDP connections through the firewall for the entire subnet. The default is 0, which means unlimited connections and the firewall lets the server determine the number.

Another optional feature is specifying the maximum number of embryonic connections per host. An embryonic connection is a connection request that has not finished the necessary handshake between source and destination. Set a small value for slower systems, and a higher value for faster systems. The default is 0, which means unlimited embryonic connections.

The embryonic connection limit lets you prevent a type of attack where processes are started without being completed. When the embryonic limit is surpassed, the TCP intercept feature intercepts TCP SYN packets from clients to servers on a higher security level. The software establishes a connection with the client on behalf of the destination server, and if successful, establishes the connection with the server on behalf of the client and combines the two half-connections together transparently. Thus, connection attempts from unreachable hosts never reach the server. The PIX firewall and ASA accomplish TCP intercept functionality using SYN cookies.

TCP/UDP Application-Specific Protocol Protection

By default, the PIX Firewall and ASA provide TCP/UDP application-specific protection of the following protocols.


Various Cisco ASA/PIX Firewall Features

1. SSH and Telnet as firewall management access

You can only use SSH for the firewall management access when you are sitting in non-Inside network. By default you can use either telnet or SSH for the firewall management access when you are sitting in Inside network.

2. NAT

In the PIX or ASA OS version prior 8.3, by default there is NAT in place for traffic between zones. In these earlier OS versions, you typically use the nat 0 command to eliminate NAT for traffic between zones. You could also use static command with the same IP subnet of pre- and post- NAT process. Further, there is a rule called NAT Order of Operation in earlier OS version to make sure that the NAT-related business is in order.

NAT Concept on PIX Firewall running OS version 6.3 or later and ASA running OS version prior 8.3

Introduction to NAT Operation

In network environment where there is a private network that is not (and should not) be visible directly from Outside network should be made invisible to the Outside network. PIX Firewall and ASA were originally designed to provide such invisibility and do NAT by default for traffic across security zones such as between Inside and Outside network.

When the Outside network access is needed from more trusted network, you need to NAT the outbound traffic by using nat command. If the traffic is just outbound where connections are initiated from more trusted network to less trusted network, then the nat command should be associated with a global command.

For inbound traffic where connections are initiated from less trusted network to more trusted network, the static command is needed to accommodate the NAT process. With the static command, the traffic flow between the less and more trusted network is established both way; meaning that the Outside network (less trusted network) can initiate traffic to the Inside network (more trusted network) at anytime and vice versa. There is no need to create specific nat command to accommodate the traffic flow.

In regards of the static command use, you have a choice to either use the same or different IP address/subnet between the less and more trusted network. Following is list of possibilities where you want to use different IP address/subnet appearing on the less trusted network.

1. The private network (residing at the more trusted network) uses IP scheme that is not routable at the less trusted network; i.e. Internet access from LAN using private network of 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 (non-Internet-routeable subnets per RFC 1918).

2. The less trusted network is unable to do routing. In this case, the more trusted network uses NAT-ed IP address that is within the less trusted network IP subnet

3. There is conflicting IP scheme between less and more trusted network. In this case, the more trusted network uses NAT-ed IP address that is within the less trusted network IP scheme. Furthermore, you need to NAT the inbound traffic from less to more trusted network using NAT-ed IP address that is within the more trusted network IP scheme.

When none of the above situation meets, you should use the same IP address/subnet between less and more trusted network. Note that just because you use the same IP address/subnet between less and more trusted network, it does not mean that there will be security risk on the more trusted network since the PIX Firewall or ASA provides sufficient stateful security feature as mentioned at earlier discussion.

Different Types of NAT

1. Dynamic PAT

Commands to use: nat, global
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is not needed

Example 1.1
nat (inside) 1 192.168.1.0 255.255.255.0
global (outside) 1 203.43.45.93

Description:
Any hosts within Inside IP subnet of 192.168.1.0/24 will be PAT-ed into 203.43.45.93 when there is outbound traffic from Inside to Outside network

Example 1.2
nat (outside) 1 203.43.45.0 255.255.255.0
global (inside) 1 192.168.1.93

Description:
Any hosts within Outside IP subnet of 203.43.45.0/24 will be PAT-ed into 192.168.1.93 when there is inbound traffic from Outside to Inside network

2. Static PAT

Commands to use: static
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is needed

Example 2.1
static (inside,outside) tcp 203.43.45.93 80 192.168.45.93 80 netmask 255.255.255.255

Description:
Host 192.168.45.93 will be PAT-ed to 203.43.45.93 when there is outbound traffic initiated from 192.168.45.93 (within the Inside network) using TCP port 80 as source TCP port to the Outside network. Similarly, any IP address within Outside network will access 203.43.45.93 using TCP port 80 as destination TCP port in order to access 192.168.45.93 on TCP port 80

Example 2.2
static (outside,inside) tcp 192.168.45.93 80 203.43.45.93 80 netmask 255.255.255.255

Description:
Host 203.43.45.93 will be PAT-ed to 192.168.45.93 when there is inbound traffic initiated from 203.43.45.93 (within the Outside network) using TCP port 80 as source TCP port to the Inside network. Similarly, any IP address within Inside network will access 192.168.45.93 using TCP port 80 as destination TCP port in order to access 203.43.45.93 on TCP port 80

3. Static NAT of single IP address

Commands to use: static
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is needed. Furthermore, the command uses the entire IP protocols and ports within the provided IP address.

Example 3.1
static (inside,outside) 203.43.45.93 192.168.45.93 netmask 255.255.255.255

Description:
Host 192.168.45.93 will be NAT-ed to 203.43.45.93 when there is outbound traffic initiated from 192.168.45.93 (within the Inside network) using any IP protocol (including ESP, TCP, and UDP) to the Outside network. Similarly, any IP address within Outside network will access 203.43.45.93 using any IP protocol in order to access 192.168.45.93.

Note:
This static statement may seem as security risk since you are opening the IP address to any incoming IP protocol from less to more trusted network. Such risk is mitigated when there is access-list controlling inbound traffic to open necessary IP protocol and ports (i.e. just open inbound TCP port 80 and 443 where others are denied).

Example 3.2
static (outside,inside) 192.168.45.93 203.43.45.93 netmask 255.255.255.255

Description:
Host 203.43.45.93 will be PAT-ed to 192.168.45.93 when there is inbound traffic initiated from 203.43.45.93 (within the Outside network) using any IP protocol (including ESP, TCP, and UDP) to the Inside network. Similarly, any IP address within Inside network will access 192.168.45.93 using any IP protocol in order to access 203.43.45.93.

4. Static NAT of entire IP subnet

Commands to use: static
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is needed. Furthermore, the command uses the entire IP protocols and ports within the provided IP address.

Example 4.1
static (inside,outside) 203.43.45.0 192.168.45.0 netmask 255.255.255.0

Description:
Any hosts within 192.168.45.0/24 will be NAT-ed to 203.43.45.0/24 when there is outbound traffic initiated from 192.168.45.0/24 (within the Inside network) using any IP protocol (including ESP, TCP, and UDP) to the Outside network. Similarly, any IP address within Outside network will access 203.43.45.0/24 using any IP protocol in order to access 192.168.45.0/24.

Using IP subnet static NAT indicates the following static NAT in place

As you can see, the last octet will be the same while only the first three octets are different between the Outside and the Inside IP addresses.

Note:
The command is useful when you need to NAT the entire subnet without the requirement of creating multiple static command of each pair of Outside-Inside IP addresses. You can simply create static NAT for the entire subnet instead.

Example 4.2
static (outside,inside) 192.168.45.0 203.43.45.0 netmask 255.255.255.0

Description:
Any hosts within 203.43.45.0/24 will be NAT-ed to 192.168.45.0/24 when there is outbound traffic initiated from the Inside network using any IP protocol (including ESP, TCP, and UDP) to the Outside network. Similarly, IP address within 203.43.45.0/24 Outside network will access the any IP addresses within Inside network as 192.168.45.0/24 using any IP protocol.

5. Static NAT of entire IP subnet and keep the same IP scheme between less and more trusted network

Command to use: access-list, nat 0, and/or static
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is needed. Furthermore, the command uses the entire IP protocols and ports within the provided IP address. All of these processes take place while keeping the same IP scheme between less and more trusted network.

Example 5.1 - NAT exemption

access-list nonat_inside-outside permit ip 192.168.45.0 255.255.255.0 192.168.1.0 255.255.255.0
nat (inside) 0 access-list nonat_inside-outside

Description:
Any hosts within 192.168.45.0/24 will appear as themselves when there is outbound traffic initiated from 192.168.45.0/24 (within the Inside network) using any IP protocol (including ESP, TCP, and UDP) to the Outside network of 192.168.1.0/24. Similarly, any IP address within Outside network of 192.168.1.0/24 will access 192.168.45.0/24 using any IP protocol directly.

Example 5.2

static (inside,outside) 192.168.45.0 192.168.45.0 netmask 255.255.255.0

Description:
Any hosts within 192.168.45.0/24 will appear as themselves when there is outbound traffic initiated from 192.168.45.0/24 (within the Inside network) using any IP protocol (including ESP, TCP, and UDP) to any Outside network IP address. Similarly, any IP address within Outside network will access 192.168.45.0/24 using any IP protocol directly.

Example 5.3 - Identity NAT

nat (inside) 0 192.168.45.0 255.255.255.0
static (inside, outside) 192.168.45.0 192.168.45.0 netmask 255.255.255.0

Description:
The behavior is similar as Examples 5.1 and 5.2. This configuration is less popular since it seems more complex than it has to.

6. Static NAT Policy

Command to use: access-list and static
Objective: to allow outbound traffic from more trusted network to less trusted network where inbound traffic is needed. Furthermore, the command uses the entire IP protocols and ports within the provided IP address. All of these processes take place while keeping the same IP scheme between less and more trusted network.

Example 6.1

access-list nat1_inside-outside permit ip 192.168.45.0 255.255.255.0 23.54.6.0 255.255.255.0
access-list nat2_inside-outside permit ip 192.168.45.0 255.255.255.0 23.54.7.0 255.255.255.0
nat (inside) 1 0.0.0.0
static (inside,outside) 23.54.6.254 access-list nat1_inside-outside
static (inside,outside) 23.54.7.254 access-list nat2_inside-outside
global (outside) 1 203.43.45.32

Description:
Any 192.168.45.x within Inside network will be statically NAT as 23.54.6.254 when 192.168.45.x access 23.54.6.x that resides at Outside network. Similarly, any 192.168.45.x within Inside network will be statically NAT as 23.54.7.254 when 192.168.45.x access 23.54.7.x that resides at Outside network. When 192.168.45.x access any other IP addresses at Outside network beside 23.54.6.x and 23.54.7.x, the 192.168.45.x will be dynamically PAT-ed as 203.43.45.32.

NAT Implementation Illustration

For the sake of illustration, we assume the following

Outside network: any IP subnet
DMZ 1 network: 192.168.0.0/24, 192.168.1.0/24
DMZ 2 network: 192.168.2.0/24, 192.168.3.0/24
Inside network: 192.168.32.0/24, 192.168.33.0/24, 192.168.45.0/24

Example 1

access-list nonat permit ip 192.168.32.0 255.255.255.0 192.168.1.0 255.255.255.0
nat (inside) 0 access-list nonat
nat (inside) 1 192.168.32.0 255.255.255.0
global (outside) 1 203.45.32.84

Description:
When any IP address within 192.168.32.0/24 access the 192.168.1.0/24, the 192.168.32.x appears as themselves. If the 192.168.32.x access anything else that is at Outside network, there will be dynamic PAT to use 203.45.32.84 IP address to appear on the Outside network.

Further, any machine within 192.168.1.0/24 can access 192.168.32.0/24 as themselves. In other words, 192.168.32.0/24 appears as themselves in the 192.168.1.0/24 presence and vice versa.

The 192.168.33.x cannot access anything beyond Inside network. Similarly, the 192.168.0.x cannot access anything beyond DMZ 1 network. Anything at Outside and DMZ 2 cannot access anything at DMZ 1 and 192.168.33.x Inside network.

Example 2

access-list nonat permit ip 192.168.32.0 255.255.255.0 192.168.0.0 255.255.255.0
nat (inside) 0 access-list nonat
nat (inside) 1 0.0.0.0
nat (dmz1) 2 192.168.0.0 255.255.255.0
global (dmz2) 1 192.168.2.254
global (outside) 2 204.54.65.231
static (inside,outside) 192.168.32.0 192.168.32.0 netmask 255.255.254.0

Description:
The 192.168.0.x and 192.168.32.x can see each other as themselves. Any IP address within Inside network (including those that are not 192.168.32.x or 192.168.33.x if any such as 192.168.45.x) is able to access 192.168.2.x and 192.168.3.x using PAT-ed IP address of 192.168.2.254. Both 192.168.32.x and 192.168.33.x will appear as themselves when they are accessing Outside network. Any 192.168.0.x will appear as 204.54.65.231 to access Outside network.

Example 3

access-list nonat permit ip 192.168.32.0 255.255.254.0 192.168.0.0 255.255.254.0
access-list nonat permit ip 192.168.45.0 255.255.255.0 192.168.0.0 255.255.254.0
access-list nat1_inside-dmz2 permit ip 192.168.32.0 255.255.254.0 192.168.2.0 255.255.255.0
access-list nat1_inside-dmz2 permit ip 192.168.45.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list nat2_inside-dmz2 permit ip 192.168.32.0 255.255.254.0 192.168.3.0 255.255.255.0
access-list nat2_inside-dmz2 permit ip 192.168.45.0 255.255.255.0 192.168.3.0 255.255.255.0
access-list nat_inside-outside permit ip 192.168.32.0 255.255.254.0 any
access-list nat_inside-outside permit ip 192.168.45.0 255.255.255.0 any
nat (inside) 0 access-list nonat
nat (inside) 1 access-list nat1_inside-dmz2
nat (inside) 2 access-list nat2_inside-dmz2
nat (inside) 3 access-list nat_inside-outside
global (dmz2) 1 192.168.2.254
global (dmz2) 2 192.168.3.254
global (outside) 3 204.54.65.231-204.54.65.253
global (outside) 3 204.54.65.254
static (dmz1,outside) 204.54.64.0 192.168.0.0 netmask 255.255.255.0

Description:
The 192.168.32.x, 192.168.33.x, and 192.168.45.x appear as themselves when they access 192.168.0.x, 192.168.1.x and vice versa. The 192.168.32.x, 192.168.33.x, and 192.168.45.x appear as 192.168.2.254 when they access 192.168.2.x and appear as 192.168.3.254 when they access 192.168.3.x.

The 192.168.0.x appear as 204.54.64.x when they access Outside network. Similarly, Outside network access 204.54.64.x in order to access 192.168.0.x.

The 192.168.32.x, 192.168.33.x, and 192.168.45.x on the Inside network appear as any available IP address within range of 204.54.65.231 and 204.54.65.253 when those Inside networks access Outside network. Such range is called NAT pool where there will be dynamic one-one NAT relationship between 192.168.32.x, 192.168.33.x, 192.168.45.x on the Inside network and any available IP address within range of 204.54.65.231 and 204.54.65.253. When all IP addresses within the NAT pool are used up, the 204.54.65.254 will be used as last resort (as dynamic PAT instead of dynamic NAT).

Note:
For illustration, please check out all sample configuration using Cisco ASA/PIX Firewall in this Cisco Forum FAQ to better understand how Cisco firewall implementation look like.

Traffic Flow Across Security Zones

1. Default Behavior and Ways To Tweak

As a firewall, PIX Firewall and ASA by default expect to have traffic flow comes from one security zone to another. Any routing traffic that comes from one security zone and bounce back to the same security zone (called hair pinning) is denied. Another default behavior is to block traffic flow between security zones with equal security level.

In regards of traffic flow coming from one security zone to another, following is default behavior
* Initiated from Less-Trusted zone to More-Trusted zone, traffic is denied
* Initiated from More-Trusted zone to Less-Trusted zone, traffic is permitted
* Initiated from one security zone to another with equal security level, traffic is denied
* Initiated from one security zone and bounce back (hair pinning), traffic is denied

To adjust the above default behavior, following is the list of choices that applies for PIX Firewall and ASA running OS version 6.3 and later

* Implement nat 0 or static command in addition to implement access-group command tied with specific access-list command to allow initiating traffic from Less-Trusted zone to More-Trusted zone
* Implement access-group command tied with specific access-list command to restrict initiating traffic from More-Trusted zone to Less-Trusted zone

When the PIX Firewall or ASA runs OS version 7.0 or later, following is a list of choices to adjust various default behaviors
* Implement same-security-traffic permit command to allow initiating traffic from one security zone to another with equal security level. The same command is used to also allow hair-pinning traffic
* Transform the Layer-3 firewall default behavior into Layer-2 firewall using firewall transparent command to avoid the firewall participating in routing
* Transform the single physical firewall into multiple virtual firewall using mode command to allow Active/Active or Active/Standby traffic flow separating routing table between each virtual firewall

2. Traffic Flow Order of Operation

For those traffic flow initiating from Less-Trusted to More-Trusted network, here is what Cisco devices including PIX Firewall and ASA expect
* Incoming traffic hits IP address as seen in the IP scheme of the Less-Trusted network. If there is NAT in place, then the incoming traffic hits the NAT-ed IP address.
* Cisco devices check incoming traffic to see if there is a match within the access-list. When there is a match; Cisco devices stop searching, treat the traffic per the rule, and exit. When there is no match, by default Cisco devices deny traffic
* If static command is in place to manage the NAT/PAT-ed IP addresses, Cisco devices translate IP address accordingly and forward the traffic based on the routing table

Since PIX Firewall and ASA are firewall, by design the firewall does traffic inspection before forwarding traffic based on the routing table as mentioned in early discussion. Any traffic that do not pass the inspection will be dropped and will not be forwarded.

What Is New On ASA (Or PIX OS 7.2 and above) Compared To PIX Firewall Running PIX OS 6.3?

Note:
* PIX Firewall 500 series only support PIX OS up to 8.0(4) version. The ASA 5500 series support beyond OS 8.0(4) with possible DRAM/Flash upgrade
* There is no known "real" differences between PIX OS 7.x and ASA OS 7.x from software perspective

For further info, check out the following official Cisco online documentation links for specific OS version features.

Features

Legacy OS 6.3(5)
http://www.cisco.com/en/US/docs/security/pix/pix63/release/notes/pixrn635.html

OS 7.0(1)
http://www.cisco.com/en/US/docs/security/pix/pix70/release/notes/pix_70rn.html#wp169795

OS 7.0(4)
http://www.cisco.com/en/US/docs/security/pix/pix70/release/notes/pix704rn.html#wp213502

OS 7.0(5)
http://www.cisco.com/en/US/docs/security/pix/pix70/release/notes/pix705rn.html#wp213502

OS 7.2(1)
http://www.cisco.com/en/US/docs/security/pix/pix72/release/notes/pixrn72.html#wp185529

OS 7.2(2)
http://www.cisco.com/en/US/docs/security/pix/pix72/release/notes/pixrn722.html#wp191103

OS 7.2(3)
http://www.cisco.com/en/US/docs/security/pix/pix72/release/notes/pixrn723.html#wp213761

OS 8.0
http://www.cisco.com/en/US/docs/security/pix/pix80/release/notes/pixrn80.html#wp191103

OS 8.0(3)
http://www.cisco.com/en/US/docs/security/pix/pix80/release/notes/prn803.html#wp191103

OS 8.0(4)
http://www.cisco.com/en/US/docs/security/pix/pix80/release/notes/pixrn804.html#wp191103

OS 8.1
http://www.cisco.com/en/US/docs/security/asa/asa81/release/notes/asarn81.html#wp229690

Enable/Disable Communication on OS 7.0 image and newer

1. Troubleshooting on OS 7.0 image and newer

Establish and Troubleshoot Connectivity through PIX/ASA
Packet/Traffic Troubleshooting

2. Sample Configuration on OS 7.0 image and newer

ASA/PIX EIGRP Routing Support

Backup/Failover Routing

Single Firewall Partitioned Into Multiple Independent Firewalls: Introduction to Multiple Context

Active/Active PIX/ASA Stateful Redundancy

Active/Standby PIX/ASA Stateful Redundancy

Transparent (Layer-2) Firewall

QoS

ASA As SSL Server
SSL VPN Client (SVC) on ASA with ASDM Configuration Example
Clientless SSL VPN (WebVPN) on ASA Configuration Example
Thin-Client SSL VPN (WebVPN) on ASA with ASDM Configuration Example

Block or Restrict the Instant Messaging (IM) Traffic

URL Filtering

New Features and Deprecated Commands Starting at OS version 8.3

You may notice that PIX Firewall appliances are unable to run latest OS version. PIX 501 can only run up to OS version 6.3(5) while PIX 515E and larger appliances can only run up to OS version 8.0. You need ASA 5500 series appliance to run newer OS version than 8.0.

Cisco ASA 5500 Migration Guide for Version 8.3

Discussion of OS version 9.1
»ASA 5520 Fan Question

Default Behaviors Starting at OS version 8.3

One default behavior with this newer OS version is that no NAT is in place. Further, static and nat commands have been consolidated into a newer format of nat command. In addition, there is also a new format of object-group command to incorporate the new nat format command.

NAT and PAT Statement Use on the Cisco Secure ASA Firewall Configuration Example

Following is an illustration.

Cisco ASA NAT Cheat Sheet

Example 1

The first part of object command is to define a network as part of the obj-Inside-Network network object. You may notice two things that
1. The object-group command is now replaced by an object command
2. The network-object command is now replaced by a subnet command

The second part of the object command is to define how the subnets as source subnets to work with nat command. This second part basically saying all subnets defined in obj-Inside-Network network object will be coming from inside interface and will be natted into outside interface IP address dynamically (PAT) when there are traffic initiated from such subnets from inside interface towards outside interface.

Example 2

Regular Static NAT

Pre-8.3 NAT

8.3 NAT

Regular Static PAT

Pre-8.3 NAT

8.3 NAT

Static Policy NAT

Pre-8.3 NAT

8.3 NAT

Example 3

Regular Dynamic PAT

Pre-8.3 NAT

8.3 NAT

Regular Dynamic PAT

Pre-8.3 NAT

8.3 NAT

Regular Dynamic PAT of All Internal Subnets

Pre-8.3 NAT

8.3 NAT

Dynamic Policy NAT

Pre-8.3 NAT

8.3 NAT

Policy Dynamic NAT (with multiple ACEs)

Pre-8.3 NAT

8.3 NAT

Outside NAT

Pre-8.3 NAT

8.3 NAT

NAT and Interface PAT together

Pre-8.3 NAT

8.3 NAT

NAT and Interface PAT with additional PAT together

Pre-8.3 NAT

8.3 NAT

Twice NAT with both source IP, Dest IP and Source port, Dest port change.

On the inside:
Source IP: 10.30.97.129
Dest IP: 10.30.97.200
Source port: 5300
Dest port: any port

On the outside:
Source IP: Interface IP
Dest IP: 172.16.1.10
Source port: 5300
Dest port: 1022

8.3 NAT

Static NAT for a Range of Ports

8.3 NAT

Not Possible - Need to write multiple Statements or perform a Static one-to-one NAT

Original Ports: 10000 - 10010
Translated ports: 20000 - 20010


NAT Order of Operation behavior starting OS 8.3

Here they are starting from higher priority to lower, as displayed in show nat command.

Section 1. Manual NAT policies
Section 2. Auto NAT policies
Section 3. Manual NAT policies

Here are some descriptions, as displayed in show running nat command.

Manual NAT policies (Section 1)

The NAT commands with source parameter are part of this section. Illustration is as follows.

nat (Inside,any) source dynamic [SOURCE DEFINED NETWORK OBJECT real] [SOURCE DEFINED NETWORK OBJECT shown as] destination static [DEST DEFINED NETWORK OBJECT real] [DEST DEFINED NETWORK OBJECT shown as]

nat (Outside,any) source dynamic [SOURCE DEFINED NETWORK OBJECT real] [SOURCE DEFINED NETWORK OBJECT shown as] destination static [DEST DEFINED NETWORK OBJECT real] [DEST DEFINED NETWORK OBJECT shown as]

nat (Inside,Outside) source static [SOURCE DEFINED NETWORK OBJECT real] [SOURCE DEFINED NETWORK OBJECT shown as] destination static [DEST DEFINED NETWORK OBJECT real] [DEST DEFINED NETWORK OBJECT shown as]

Auto NAT policies (Section 2)

The NAT commands with combination of object network and nat commands are part of this section. Illustration is as follows.

object network [OBJECT NETWORK NAME defining the real IP address]
nat (Inside,Outside) static [shown-as IP address]

Manual NAT policies (Section 3)

The NAT commands with after-auto parameter are part of this section. Illustration is as follows.

nat (Inside,Outside) after-auto source dynamic [OBJECT NETWORK NAME defining the real IP address] interface

OS and Hardware Compatibility

Thinking of upgrading to certain OS version? Check the following link to verify your hardware compatibility.
Cisco ASA Compatibility

Hair Pinning feature

OS version 8.3 or later

nat (outside,outside)
ASA: 8.3 "Nat U-turn" Example - RA VPN Client traffic

Application Inspection

Cisco documentation
Configuring Special Actions for Application Inspections (Inspection Policy Map)

New Features starting OS version 9.2

Finally ASA firewall supports BGP among other things as noted below.
Release Notes for the Cisco ASA Series, 9.2(x)

Licenses

For those who are eager to get their hands on ASA or PIX Firewall, they need to consider the license factor. With either ASA or PIX Firewall, you should get the one with Unlimited Inside Hosts instead of 10 or 50 Inside Hosts. For PIX Firewall, one with Unrestricted license has more features compared to one with Restricted license; while one with the Failover license can only work as backup firewall of the Unrestricted license. For ASA, one with Security Plus license supports more features similarly. Both Inside Hosts number and license type that firewall carries can be verified through the show version.

Upgrading from lower license to higher license may cost you dearly where at that point, getting a new firewall with higher license may cost you less compared to upgrade your existing firewall to have higher license.

You can check out the following discussion for some illustration.
»[HELP] Upgrade ASA 5505 License

Failover in ASA

When stateful failover is enabled, the Active unit continually passes per-connection state information to the Standby unit. After a failover occurs, the same connection information is available at the new active unit. Supported end-user applications are not required to reconnect to keep the same communication session.

The state information passed to the Standby unit includes these:

* The NAT translation table
* The TCP connection states
* The UDP connection states
* The ARP table
* The Layer 2 bridge table (when it runs in the transparent firewall mode)
* The HTTP connection states (if HTTP replication is enabled)
* The ISAKMP and IPSec SA table
* The GTP PDP connection database

The information that is not passed to the Standby unit when stateful failover is enabled includes these:

* The HTTP connection table (unless HTTP replication is enabled)
* The user authentication (uauth) table
* The routing tables
* State information for security service modules

If failover occurs within an active Cisco IP SoftPhone session let's say, the call remains active because the call session state information is replicated to the standby unit. When the call is terminated, the IP SoftPhone client loses connection with the Call Manager. This occurs because there is no session information for the CTIQBE hang-up message on the standby unit. When the IP SoftPhone client does not receive a response back from the Call Manager within a certain time period, it considers the Call Manager unreachable and unregisters itself.

After all this replication happens the ASA assume the active ip address and send a gratuitous arp to the devices on the network so they can update their ARP entries.

A better solution to implement failover perhaps is to use a virtual mac address, similar to Cisco HSRP mechanism. If you will use the Virtual mac address for failover then the ARP entries will not get changed and there will be no timeout anywhere on the network. If you are not using the virtual mac address and failover occurs, the arp entries will be changed; and when the new device takes over the active state then it will send the gratuitous arp.

Redundant and Port Channel Interfaces

Redundant Interface

A logical redundant interface is a pair of one active and one standby physical interface. When the active interface fails, the standby interface becomes active.

The firewall will remove all interface settings when adding the physical interface to a redundant group.

The logical redundant interface will take the MAC address of the first interface added to the group, because this will also become the active interface. This MAC address is not changed with the member interface failures, but changes when you swap the order of the physical interfaces added to the pair; optionally, a vMAC can be configured for the redundant interface. With redundant interfaces, the nameif, security-level, and IP address configuration is done at the logical interface level. This feature is not preemptive.

Etherchannel (Port Channel)

In Etherchannel configuration, ASA supports both active and passive modes; where active initiates the LACP negotiation, and passive expects to receive LACP negotiations. The logical portchannel interface will take the MAC address of the lowest number interface from the group; optionally, a vMAC can be configured for the etherchannel interface.

Sample Configuration


interface Ethernet0/0
no nameif
no security-level
no ip address

interface Ethernet0/2
no nameif
no security-level
no ip address

interface Redundant1
member-interface Ethernet0/0
member-interface Ethernet0/2
nameif OUTSIDE
security-level 0
ip address 136.1.34.17 255.255.255.0

interface Ethernet0/1
channel-group 1 mode passive
no nameif
no security-level
no ip address

interface Ethernet0/3
channel-group 1 mode passive
no nameif
no security-level
no ip address

interface Port-channel1
lacp max-bundle 2 port-channel load-balance src-dst-ip-port
nameif INSIDE
security-level 100
ip address 136.1.93.17 255.255.255.0



Check ARP table on router side to confirm that MAC address fo the first ASA interfaces added to the group show up here.

Switch configurations to support these features.

Redundant interface configs:

Ether-channel config:


The Woes of Using an ASA as a Default Gateway

Consider the following network setup, where the ASA firewall runs OS version 8.2.



Requirements:

* Hosts Must Use 10.1.1.100 as their Gateway
* ASA Must Direct Traffic Destined to 100.1.1.0/24 to 10.1.1.200
* ASA Must Perform PAT (NAT Overload) for traffic going to the Internet
* No STATICs or ACL for inbound traffic

At first glance, this seems really simple. We've all done this with routers, so we just need the ASA equivalent of ip route 100.1.1.0 255.255.255.0 10.1.1.200. Right? That should be easy. Let's just go ahead and configure up our ASA for PAT and then add the static route.


interface ethernet0/1
nameif inside
security-level 100
ip address 10.1.1.100 255.255.255.0
!
interface ethernet0/0
nameif outside
security-level 0
ip address 66.49.27.1 255.255.255.0
!
global (outside) 1 interface
nat (inside) 1 0.0.0.0 0.0.0.0
!
route outside 0.0.0.0 0.0.0.0 66.49.27.153
route inside 100.1.1.0 255.255.255.0 10.1.1.200


Well that was easy, but does it work? When you try to ping something on 100.1.1.0/24 from a host that is using the ASA as a default gateway, you will find that it fails. You've seen that before, right? Remember, we always have to turn on ICMP inspection on ASA's. The shortcut for that is:


fixup protocol icmp


Great, but it still doesn't work. Then it hits you. The ASA is a product that came from the PIX firewall. The PIX, if you recall, will never forward a packet out the same interface it was received on. But the ASA was supposed to allow this with a strange command.


same-security-traffic permit intra-interface


That's still not too bad, if that was all you actually had to do. Unfortunately it still doesn't seem to work. Maybe we better take a look at how the ASA is processing our ICMP echo. We can issue a command like packet-tracer input inside icmp 10.1.1.50 0 0 100.1.1.20. That will show us all of the steps that the ASA goes through when processing the packet. What you will find is that the ASA is actually trying to do NAT and there is a missing global statement for the inside interface. But we don't really want to do NAT for traffic to or from our third party network. So we need to write a NAT exemption rule and test our connectivity once again.


access-list NONAT_inside-outside extended permit ip any 100.1.1.0 255.255.255.0
nat (inside) 0 access-list NONAT_inside-outside


Now let's ping something in 100.1.1.0/24 from one of our hosts again. Success! It should be working at this point, but we're not done yet. Try using the TCP protocol to reach something at 100.1.1.0/24. If you look at this in Wireshark, you'll probably see something like SYN, SYN-ACK, ACK, RST or SYN, SYN-ACK, ACK, ACT (Retrans), ACK (Retrans). What is going on? The ASA is actually trying to create a session for the TCP connection. It is actually inspecting the TCP traffic. Since the router delivers the second part of the three-way handshake directly to the host, the ASA never sees the "SYN-ACK". Therefore, the ASA doesn't believe the three-way handshake has occurred and does not allow the third packet. Well that just sucks. The ASA is trying to create a traffic session that doesn't even go through the appliance. What to do?

ASA OS 8.2 introduced a feature called TCP State Bypass. That allows the ASA to pass traffic without validating the TCP state. The configuration of that uses the modular policy framework (MPF).


access-list STATEBYPASS extended permit ip any 100.1.1.0 255.255.255.0
!
class-map STATEBYPASS
match access-list STATEBYPASS
policy-map STATEBYPASS
class STATEBYPASS
set connection advanced-options tcp-state-bypass
!
service-policy STATEBYPASS interface inside


Now a test using TCP from one of the hosts to something on 100.1.1.0/24 should succeed. What else should we do? Anytime I am doing anything strange with NAT on the ASA, I disable proxy-arp. This case shouldn't require it, but I have had cases where the ASA responds to ARPs that it shouldn't and it's really hard to track down. So for good measure, I would add the following command.


sysopt noproxyarp inside


The final configuration looks something like the following:


interface ethernet0/1
nameif inside
security-level 100
ip address 10.1.1.100 255.255.255.0
!
interface ethernet0/0
nameif outside
security-level 0
ip address 66.49.27.1 255.255.255.0
!
nat (inside) 0 access-list NONAT_inside-outside
nat (inside) 1 0.0.0.0 0.0.0.0
!
global (outside) 1 interface
!
route outside 0.0.0.0 0.0.0.0 66.49.27.153
route inside 100.1.1.0 255.255.255.0 10.1.1.200
!
access-list NONAT_inside-outside extended permit ip any 100.1.1.0 255.255.255.0
!
access-list STATEBYPASS extended permit ip any 100.1.1.0 255.255.255.0
!
same-security-interface permit intra-interface
!
class-map STATEBYPASS
match access-list STATEBYPASS
policy-map STATEBYPASS
class STATEBYPASS
set connection advanced-options tcp-state-bypass
!
service-policy STATEBYPASS interface inside
!
sysopt noproxyarp inside


In conclusion, the ASA is not a router. The network setup is also not recommended. A best practice in this case is to not having ASA as default gateway, rather using a router or Layer-3 switch. Any traffic destined to the 100.1.1.0/24 (or anything internal) should not reach the ASA to avoid confusion.

Different approach would be having the router serving the 100.1.1.0/24 to terminate at one of the ASA's DMZ interface. In other words, there would be one DMZ interface dedicated for reaching 100.1.1.0/24 in addition to the existing inside and outside networks. The router's LAN interface is no longer part of the 10.1.1.0/24, rather to be part of the DMZ LAN interface. The 10.1.1.0/24 remains part of inside network, still having the 10.1.1.100 ASA inside interface IP address as default gateway.

This solution is recommended when the 100.1.1.0/24 is managed-services network, part of business partner network, or has lower security level or less trusted network compared to inside network. There would be a need to assign new network to be the DMZ LAN interface part of, which ought to be a simple process.

As mentioned, this illustration assumes OS version 8.2 or later. If you are using PIX or ASA running older OS version, features described might not be available. Regardless going through the exercise of making the ASA behave like a router can help you understand some of the logic and processing order of the firewall appliance.

Single vs. Multiple Context

»ASA LLQ

Discussions

[Config] Crossing internal networks
[Config] Which route will be used....?
[Config] Question about a pix 506e
Firewalls
Cisco ASA latest version VPN issue

Cisco website
Cisco ASA 5500 Series Configuration Guide using the CLI, 8.4 and 8.6 - Managing Feature Licenses

Discussion
»New DMZ int with ASA5505 / Base Lic..
»Cisco ASA 3rd edition

Introduction

Unlike traditional Catalyst switches running IOS, Nexus switches run NX-OS. There are some similarity between IOS and NX-OS. Both NX-OS and newer IOS products are Linux based. Some new features and commands are introduced in NX-OS.

Discussion
Linux and NX-OS

In regards of CLI commands, there are several new commands on Nexus NX-OS image. There are also old commands you find on regular IOS image, and there are modified command compared to the regular IOS. Legacy command such as write memory is not supported anymore, therefore you have to get used to the copy running-config startup-config command.

A nice feature in Nexus switch is that you don't have to exit configuration mode to type in any non-configuration commands. You don't type in the do command when you are on configuration mode to type in any non-configuration commands. You simply type in the non-configuration commands directly whether you are on regular enable mode or configuration mode, similar to PIX Firewall or ASA.

All switch ports in Nexus switches only support 1 Gbps and 10 Gbps speed. Interestingly, these gigabit ports do not show as GigabitEthernet ports or TenGigabitEthernet ports on switch configurations. Instead the ports show as Ethernet interfaces. To find out which speed the ports are acting current, you can simply issue the good old show interface status or simply show interface command.

Along with new commands and features, there are several new concept and technology in place. One new technology found in Nexus switch is FEX (Fabric Extender). Typically you use this FEX technology when you have Nexus 2000 and Nexus 5000 interconnectivity.

This FEX technology is similar to the Catalyst 3750 stacking technology where switch configuration within the same "stack" is visible through just one switch. Similar to Catalyst 3750 stack switch configuration, the Nexus 5000 shows as the "module 1" and the Nexus 2000 shows as the "module 2".

Unlike Catalyst 3750 stack switch, the Nexus do not use stack cable. The switch port to interconnect the two Nexus switches are SFP slot. In order to interconnect the two Nexus switches, the switch ports are configured as FEX ports instead of regular trunk or access ports.

To start using this FEX feature, you have to activate FEX on the Nexus 5000. As you will see, you have to activate telnet and tacacs+ should your network need to use those as well. In other words, there are some features that you have to active when you plan to use the features as part of your Nexus switch network topology.

Further, you have to define how the Nexus 2000 port number should look like. If let's say you configure the FEX port as FEX 101, then all Nexus 2000 switch port will show as interface Ethernet 101 (the "module 2") while the Nexus 5000 switch port show as the regular interface Ethernet 1 (the "module 1").

Note that there is no console port on Nexus 2000. There is console port however on Nexus 5000. Therefore you need to use the FEX technology to interconnect Nexus 2000 and Nexus 5000 in order to have console access to Nexus 2000.

When you need to use the management port on the Nexus 5000 (and also Supervisor 6E of Catalyst 4500 series), make sure you have at least some familiarity with VRF (VPN Routing and Forwarding) technology since these management ports are using involving VRF.

You can't disable the VRF or make the management (mgmt) interface as part of default VRF or global routing table since such action is not supported. The idea of having management port in different routing table is to separate management network and production network, in addition to integrate VRF into Nexus switch platform and new Catalyst 4500 Supervisor Engines.

You will notice that there is a little difference in VRF command implementation between traditional IOS and NX-OS. You can also put in subnet mask in CIDR format, since Nexus platform saves any IP address info in CIDR format.

Unlike traditional Catalyst switches that come with default Layer-2/3 VLAN 1, Nexus 5000 switches only come with default Layer-2 VLAN 1. If you are considering of using non-management switch port as your customized management port, it might not work. Note that Nexus 5000 and 2000 switches are designed as Layer-2 switches originally. The Layer-2 switch design means that you can't create Layer-3 VLAN on Nexus switches as management VLAN (i.e. SVI VLAN interfaces 1, 50, or else) like you usually expect in traditional Catalyst switches. You can't convert any non-management switch port as routing port either. In other words, there is no choice but to use the mgmt port and get used to VRF environment when you are not used to it yet.

After certain NX-OS releases, the Nexus 5000 switches are now Layer-3 capable though the 2000 model remains Layer-2 switch. You may need to upgrade the NX-OS image and/or upgrade the license on the 5000 model in order to support this Layer-3 functionality.

Some management command like backing up your Nexus configuration to TFTP server (copy running-config tftp: command) is also considering VRF. With copy running-config tftp: command, you will be asked if the TFTP server is located within the default VRF or else (like the management VRF).

Understanding the Nexus 5000

The Nexus 5000 is for those of us migrating and needing to protect investment in 100 M and 1 Gbps ports. It allows Top of Rack consolidation of cabling. Thats the distributed switch model mentioned above. Its a way to buy equipment that may be used in other ways going forward, but that supports your current tangle of 1 Gbps connections.

Bear in mind there are some other uses, which may make up more of the N5K use going forwards. Right now the Nexus 5000 provides a way to do Fiber Channel over Ethernet (FCoE) or Data Center Bridging (DCB). So you can lose the server HBA and use one (or two, for redundant) 10 G connections to carry both network and SAN traffic up to the N5K. That requires the special 10 G NIC, a Converged Network Adapter or CNA.

The current approach is for you to then split out the data and SAN traffic to go to Ethernet or SAN switches (or FC-attached storage). In the future, your traffic may be all FCoE until reaching a device where the FC device is attached (or perhaps with FCoE that handles management plus SAN traffic?).Thats a pretty straight-forward use.

Cisco white paper on Unified Access Layer
Unified Access Layer with Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders Solution Overview

You can also configure your FCoE configured N5K to do Network Port Virtualization, or NPV. This is a per-switch choice, you use either Fabric or NPV mode. When the switch is in NPV mode, it does not acquire a domain ID (a limited resource). Instead, it relays SAN traffic to the core switch, in effect extending the core switch. The N5K looks like a host to the fabric. This helps the fabric scale better, and makes the N5K transparent (nearly invisible) as far as the fabric. There's a theme here: fewer boxes to configure, thats a Good Thing!

The complementary NPIV (Network Port Identifier Virtualization) feature supports multiple servers on one interface, e.g. separate WWNs (SAN identifiers) per VM on a VMware virtual server host. This is highly attractive for security (SAN LUN masking and zoning). Click here for details . Note that certain Cisco MDS switches also perform NPV and NPIV, and that NPIV has been standardized.

For those doing blade servers and VMware, the Nexus 1000v virtual switch allows aggregation onto UCS chassis 10 Gbps links instead of many separate 1 Gbps links. The VN-Link feature allows internal logical (1000v) or (future) external physical tracking on a per-VM (virtual machine) basis. I currently understand physical VN-Link as a tag on the media from a VN-Link capable NIC or driver, tied to logical apparatus to have configuration track VN-Link virtual interfaces on the N5K. The reason to do this: offload the 1000v processing to external hardware.

VN-Link reference

Cisco UCS Integrated Infrastructure
Cisco VN-Link: Virtualization-Aware Networking
Overview of VN-Link in Cisco UCS

This is "Network Interface Virtualization"

The Nexus 5000 (N5K), as well as the Nexus 7000 (N7K) both support Virtual Port Channel. Think 6500 VSS but the two "brains" (control planes) stay active. Or think PortChannel (LACP) that terminates in two separate switches, with the other end of it none the wiser. There is a fairly tight vPC limit on the N5K right now.

There are also some gotchas and design situations to avoid, e.g. mixing non-vPC and vPC VLANs using the same vPC link between switches.That is, if you have VLANs that aren't doing vPC PortChannel uplinks, you'll want a separate link between the distribution switches the uplinks go to. Similarly in some L3 FHRP (HSRP, VRRP, GLBP) routing situations. The issue is traffic that comes up the "wrong" side and goes across the vPC peer link cannot be forward out a vPC member link on the other component of the vPC pair, which might happen in certain not-too-rare failure situations.

Understanding the Nexus 2000

There are three fabric extender ("fex") devices available, typically for Top of Rack ("ToR") use. Use two (and two N5K's) for redundancy in each rack. See also:

Cisco Nexus 2000 Series Fabric Extenders Data Sheet

The Nexus 2148 and 2248 are discussed below, under Gotchas. There is also the 2232 PP, which is 32 10 Gbps Fiber Channel over Etherent (FCoE) ports (SFP+) and 8 10 G Ethernet / FCoE uplinks (SFP+). That's 4:1 oversubscribed, which isn't bad for current server throughput and loading. If you want less oversubscription, you don't have to use all the ports (or you can arrange things your way with port pinning, I assume). If you want 1:1 oversubscription ("wire speed"), you'd probably run fiber right into the N5K, unless you want to use the N2K as a costly 1:1 10 G ToR copper to fiber conversion box.

Note those 10 G ports are FCoE ports. Right now, the N5K is the only Cisco switch I'm aware of doing FCoE. The Nexus 2232 does so as an extension of the N5K.

Note that NPV and NPIV are basically some proxying in the N5K, so the N2K should just act as server-facing FCoE ports for those functions.

Discussion
Understanding FEX

Gotchas and Tips

The Nexus 5000 is by default Layer 2 only. That means any server VLAN to VLAN traffic between two different VLANs will need to be routed by another box, probably your Core and / or Aggregation Nexus 7000. You'll want some darn big pipes from the N5K to the N7K, at least until the N5K can do local routing.

The Nexus 2000 does no local switching. Traffic from one port on a 2K to another on the same 2K goes via the N5K. There should be enough bandwidth. Thats why the Nexus 2000 is referred to as a fabric extender, not a switch.

The Nexus 2148 T is a Gigabit-only blade, 48 ports of Gig (not 10/100/1000) with up to 4 x 10 G fabric connections. Use the new 2248 TP if you need 100/1000 capability (the data sheet does NOT list 10 Mbps).

You'll probably want to use PortChannel (LACP) for the fabric connections. Otherwise, you're pinning ports to uplinks, and if the uplink fails, your ports pinned to it don't work; probably like a module failure in a 6500. You can now do the PortChannel to two N5K's running Virtual Port Channel (vPC). See the above link for some pictures.

If you attach to the fabric extender (fex, N2K), you can issue show platform software redwood command. The sts, rate and loss keywords are particularly interesting. The former shows a diagram, the latter show rates and oversubscription drops (or so it appears). I like being able to see internal oversubscription drops without relying on external SNMP tools; which usually show rates over relatively long periods of time, like 5 or more minutes, rather than milliseconds.

Putting a N5K into NPV mode reboots the switch and flushes its configuration. Be careful!

Designing with the Nexus 5000 and/or 2000

I've got a couple of customers where the N5K/N2K have seemed appropriate. I thought I'd briefly mention a couple of things that I noticed in trying to design using the boxes; maybe fairly obvious, maybe a gotcha. I'd like to think the first story is a nice illustration of how the N5K/N2K lets you do something you couldn't do before!

Case Study 1

The first customer situation is a site where various servers are in DMZ's of various security levels. Instead of moving the servers to a physically separate data center server zone, as appears to have been originally intended (big Nortel switches from a few years back), they extended the various DMZ VLANs to the various physical server zones using small Cisco switches with optical uplinks. That gear (especially the Nortel switches) is getting rather old, and it's time to replace it.

For that, the N5K/N2K looks perfect. We can put one or a pair of N5K's in to replace the big Nortel "DMZ overlay core" switches, and put N5K's out in the server zones (rows or multi-row areas of racks). For redundancy, we can double everything up. Right now one can make that work in a basic way, and it sounds like Cisco will fairly soon have some nice VPC (Virtual Port Channel) features to minimize the amount of Spanning Tree in such a dual N5K/N2K design, using Multi-Chassis EtherChannel (aka VPC). Neat stuff!

The way I'm thinking of this is as a distributed or "horizontally smeared" 6500 switch (or switch pair). The N2K Fabric Extender (FEX) devices act like virtual blades. There's no Spanning Tree Protocol (STP) running up to the N5K (good), and no local switching (maybe not completely wonderful, but simple and unlikely to cause an STP loop). So the N5K/N2K design is like a 6500 with the Sup in one zone and the blades spread across others.

From that perspective, the 40 Gbps of uplinks per N2K FEX is roughly comparable to current 6500 backplane speeds. So the "smeared 6500" analogy holds up in that regard.

The sleeper in all this is that the 10 G optics aren't cheap. So doing say 10-12 zones of 40 G of uplink, times optics and possibly special multi-mode fiber (MMF) patch cords, adds say 12 x ($2000) of cost, or $24,000 total. Certainly not a show-stopper, but something to factor into your budget. If you're considering doing it with single-mode fiber (SMF), the cost is a bit higher. On the other hand, that sort of distributed Layer 2 switch is a large Spanning-Tree domain if you build it with prior technology.

Case Study 2

The second customer situation is a smaller shop, not that many servers but looking for a good Top of Rack (ToR) solution going forward. The former Data Center space is getting re-used (it was too blatantly empty?). And blade servers may eventually allow them to fit all the servers into one or two blade server enclosures in one rack. Right now were looking at something like 12 back-to-back racks of stuff, including switches.

For ToR, the 3560-E, 3750-E, 4900M, and N5K/N2K all come to mind. The alternative solution that comes to mind is a collapsed core pair of 6500s. The cabling would be messier, but the dual chassis approach would offer more growth potential, and a nice big backplane (fabric).

The 3560-E and 3750-E have a 20 G of uplink per chassis limitation, not shabby, not quite up to the 6500 capacity per blade. That's workable and not too limiting.

The issue is, what do you aggregate them into? A smaller 6500 chassis? In that case, the alternatives are 6500 pair by themselves, or 6500's (maybe smaller) plus some 3560-E's or other small ToR switches, at some extra cost.

Or the N5K/N2K, one might think. The N5K/N2K is Layer 2 only right now, so you need some way to route between the various server VLANs (gotcha!). Without Layer 3 availability, you still would need to connect the N5K/N2K's to something like 4900M's or 6500's, to get some pretty good Layer 3 switching performance between VLANs. Right now, that external connection is either a pretty solid bottleneck, or you burn a lot of ports doing 8 way or (future) 16 way EtherChannel off the N5K/N2K. Bzzzt! That starts feeling rather klugey.

Some Conclusions

• The N5K/N2K right now seems to fit in better with a Nexus 7000 behind it. And I'd much prefer local Layer 3 switching to maximize inter-VLAN switching performance.

• The initial set of Nexus line features are probably chosen for larger customers; standalone Layer 3 N5K/N2K being something more attractive to a smaller site. And smaller sites tend not to be early technology adopters.

• You can mitigate this to some extent by careful placement of servers in VLANs. On the other hand, my read on current Data Center design is that the explosive growth in numbers of servers and the need for flexibility have left "careful placement of servers" in the historical dust. Nobody's got the time anymore.

Sample Configurations

Check out the following FAQ for illustrations.

»Cisco Forum FAQ »Sample Configuration: Nexus 5000 and Nexus 2000 with FEX

Dual Connecting Nexus 2000s to Nexus 5000s

Loading Images Order Matters!

After previously dual-connecting one of the FEXes, we upgraded the NX-OS on the N5Ks. As I recall, we upgraded N5K-2 first, then N5K-1. This is non-optimal, if N5K-1 is the vPC primary as was the case.

When we updated N5K-2, as you might expect, N5K-2 downloaded a new image to its connected FEX. When we upgraded N5K-1, it also downloaded the same image to its connected FEX. This is the same FEX module, and each download of the image took the FEX offline for 15 minutes or so.

Cisco documents state that the NX-OS software by design will allow an upgraded dual-home FEX to interoperate with the vPC secondary switches running the original version of Cisco NX-OS while the primary switch is running the upgrade version. You will have to have some downtime to get the image loaded.

However, the documentation doesn't say anything about what happens when you first upgrade the secondary N5K of dual-home FEX. My recommendation is not to do it, you may need a second image download to the FEX.

Adding Uplink to Second N5K

All of the FEXes were supposed to be dual connected to both N5Ks. Due to timing constraints and fiber availability, some FEX modules were left single connected for a period of time. In this case, they had only been connected to N5K-2, the vPC secondary switch and were running the current NX-OS image.

Based on our experiences updating the image, we were not sure if connecting the uplink to the N5K-1 would bring the FEX down while N5K-1 reloaded the image. I was not able to verify from the Nexus documentation what would happen though Cisco documentation recommends connecting the primary first. However, we did find that when we brought up the never-previously connected link to the N5K-1, the FEX stayed on line.

Pre-provision the FEX

You can and should pre-provision FEX modules, for example:


This allows you to pre-load the VLANs, speed, duplex, description etc for the host interfaces before the FEX modules are connected. Note that you need to know what type of FEX you have for this command since the N2K-C2248T is different than the N2K-C2248TP-E-1GE, and is what you want when you have a model number N2K-C2248TP.

Good Handling of Improperly Connected FEX Modules

The NX-OS appears to handle cross-connected FEX modules appropriately. At one point, someone connected the second uplink for FEX 101 to the N5K interface configured as port-channel 102 (FEX 102 should have been placed there). However the NX-OS noticed the mismatch, knew that FEX 101 was mis-cabled, alerted and left the second N5Ks FEX offline, but did not shutdown the active FEX.

Some Discussions

»NX-OS

Suggested pre-reading
»Cisco Forum FAQ »What to expect from Cisco new product lines

New version of IOS upgrades Layer-2 switches to Layer-3

»[Config] Routing on Layer 2 Switch

New IOS flavors

»Cisco Forum FAQ »Linux-based Cisco routers and switches

Router/Switch License Concept Introduction

Those routers and switches that run with IOS image file called Universal IOS is considered the new router and switch platform. For routers that run IOS 15.x and switches that run 12.2 showing Universal on the IOS image file are part of this discussion. To verify whether your router or switch run Universal IOS, simply issue show version which display IOS image it runs.

Starting with this Universal IOS concept, Cisco introduces IOS licensing. By default, all new routers and switches only need single IOS image file for any purposes. Gone the days when you have to download Advanced IP Services IOS, Advanced Security IOS, or the like whenever you need the router or switch to do certain functionality. With this Universal IOS concept, you need to provide license to the routers or switches for those functionality that come with advanced services such as OSPF, BGP, EIGRP (part of Advanced IP Services); IPSec VPN (part of Advanced Security). For basic security such as capability to ssh into routers or switches, you may need UniversalK9 IOS image file which can be downloaded from Cisco website.

Depending on your situation, you may or may not be required to install licenses. Some routers and switches already come with appropriate license. If you recently buy a new router or switch, by default they will come with basic license which the license upgrade fee may cost you more in total (license upgrade fee + basic hardware price) compared to those that are already equipped with proper license.

License Installation

Here is a list of steps

1. Prepare the following

• Router/Switch UDI (Unique Device Identifier) which are the PID (the router/switch model) and the serial number. These info can be obtained from either show version, show diag, or in some cases the show running-config where the PID is the Product (FRU) Number (provided by show diag) or the model number (provided by show version while the Serial Number is the Chassis Serial Number (provided by show diag) or the System serial number (provided by show version). Another way is to simply issue show license udi which will show PID, SN, and the UDI (the union of PID and SN) in the same line.

• Product Authorization Key (PAK), which is obtained from any local Cisco reseller to provide entitlement of the license installation

• Internet access, active Cisco Smartnet contract, and active Cisco online account to request the license file

• TFTP Server to store the license file

2. Request License File online from Cisco

• Open up the following link http://www.cisco.com/web/go/license/index.html

• You will be asked to enter the appropriate PAK in addition to the router/switch PID and serial number

• You will also be asked to provide email address of which Cisco would email the license file to. By default, Cisco used the email address provided in the online account however you could change the email address into something else

• Verify all info and click submit

3. Getting and Downloading the License file

• Cisco would then email the appropriate license file to install. You may note that the license file only works for the device with matching PID and serial number

• The license file is sent as attachment, so make sure your mail firewall does not block any attachment file with .lic as file type

• The license file size is very small (around 2 KB) therefore the Cisco license file email should not cause issue with "attachment file size too big" error situation

• Once you receive the email, download and save the license file into appropriate folder/sub-directory and store it to some TFTP server

4. Installing the License file

• Log into the router/switch and issue copy tftp flash using the TFTP server IP address and the license file name as source IP address and file name respectively. Make sure there is enough room in the flash to store the license file.

• Issue license install flash:[ENTER THE LICENSE FILE NAME HERE]

• You will get message of either Successful or Failed. If you receive Failed message, make sure you read and understand the error message; and re-mediate. If you receive Successful message, proceed to the next step

• Issue show license to verify the newly-installed license

• To activate the license, reload the router/switch

5. Verification

• When the router or switch boots up, notice a statement saying License Level, License Type, and Next reload license Level where the License Type should say Permanent with the new License Level. In addition, the License Level and Next reload license Level should mention the same License Level.

• Should you miss such statement during router or switch boot up, a show version and show license should display a same or similar output

Features and Memory Requirements

Cisco IOS Release 15.0 Feature Sets and Memory Recommendations for Cisco 1900, 2900 and 3900 Series Routers

Discussion
»[Config] Reload Ios and Lincseing

Catalyst 6800 series switches

Cisco documentations
Cisco Catalyst 6800 Series Switch Models
Cisco Catalyst 6880-X Series Extensible Fixed Aggregation Switch Data Sheet

1. Compatibility

The 6807 and 6880 models both run sup2t with same 15sy train as in 6500. The 6807 supports 6900 series linecards with dfc4, same as 6500. The 6880 uses half-width cards built on same forwarding architecture.

On the other hand, the 6800ia is campus fex which is a unique offering on c6k that is homed on both c6500 and c6800 platforms using 6904 linecards (6807, 6500) or built-in card (6880).

Overall, these 6800 series switch models are the same platform in forwarding and code train. A 6880-x is cut from the same cloth as the c6k, has the same sup, the same guts, same operating system; quacking, ducks, and all.

2. Performances

In the 6880-x chassis, one can shove up to (80) 10gbe ports or up to (20) 40gbe ports. You won't get that same density in the classic c6500 line, especially when you compare it against similar sized chassis.

Additionally, the 6880-x can handle 2m ipv4 routes in tcam whereas the sup2t-xl only gives 1m ipv4 fib.

Further, no one buys a big honking box anymore. The reason Cisco settled on the 6807 was that a large majority (85%?) of customers use the 9-slot chassis (no one uses the 13s, they're dumb), and those chassis have between two and four cards (non-sup cards) inserted. Rarely does one build out a full 9-slot 6509 with all linecards full. Why pay the space premium if you don't ever use it?

This concept also applies in the Nexus space (look at the dense 7700s, 6000s, and the 9300/9500s) as well as in the carrier space (asr9904, 9006, 9010 are all small chassis).

3. Capacity Planning

Following is an illustration of boiling down to a pure "upgrade" game. Let's assume a moderately sized campus; say (15) user switches where each switch with a single 10g uplink to the 6880-x distro (distro in pairs), such that each access switch has 20gbe of uplink. Even if you plan for 30% growth *in the distro*, you are still only consuming about 5 additional 10gbe uplinks per chassis after the first year.

When you break it down, five additional ports means five additional access switches. If we play it conservative and assume maybe 3*48 ports in each idf, then you're adding (144*5) or 720 additional users! Needless to say that you could grow by thousands of users and still have density left on the 6880-x. The argument about unfettered and unrestricted growth at the aggregation/core layers just doesn't happen. You may use a few additional ports here and there but you don't consume ports like you do at the access layer.

A quick run on the cisco powercalc numbers against c6513e w sup2t (single) and decked out with 10gbe linecards compared to the c6880-x with all 10gbe linecards have a following result.

Cisco Power Calculator - Power Results: Catalyst 6500
Cisco Power Calculator - Power Results: Catalyst 6800

A c6880-x provides (80) 10gbe interfaces with a total power consumption of about 2000w (utilizing redundant 3kw a/c power supplies) and a total heat dissipation of about 7800btu/hr.

A c6513 provides (88) 10gbe interfaces with a total power consumption of almost 7800w and a total heat dissipation of about 30000btu/hr. To put that in perspective, you need about a ton of cooling for every 12000btu/hr of heat output.

Therefore for the same rack space, power, and heat; you can shove almost (4) 6880-x in the same footprint as a single c6513.

We'll forgo "movement of a few cards".

ASR 1000 series

An ASR 1001 is a nice box, capable of 2.5 Gbps esp -- unlockable to 5 Gbps. The router has been the next evolution (sort-of) to the 7200vxr chassis.

There has been a large discussion about the deeper dive into their architecture on c-nsp. Quite the read if you have the time to dig through gossamer-threads.

The biggest thing is likely to be the router ability to turn off stuffing of routes into TCAM. As illustration with the RP2, the router essentially takes upwards of 12 million routes into RAM as an BGP Route Reflector which makes the box very attractive for the Service Provider or large enterprise environment by having a few boxen sitting out there for BGP especially as VPNv4/VPNv6 aren't required to be in the forwarding path for best-path computations.

Another nifty feature is the vasi interfaces which is allowing a virtual interface pair to create a connection between vrfs for things like ACL, nat, ipsec that all while allowing label imposition towards the core. More info can be found here.

With things like mac-sec, otv, and voice-related features being added into the code trains, this router has been attractive to be phasing out 7200 VXR series.

Virtualization in phasing out HSRP, GLBP, VRRP approaches

New router and switch platform such as Nexus 7000 switch and ASR 9000 router comes with the following.

• vpc -- Nexus 7000 multichassis etherchannel. By default a pair of these provide dual-active paths, with dual active hsrp forwarding, breaks forwarding simplicity, as northbound path is determined by layer-2 hashing algorithm.

• mc-lag -- multichassis etherchannel generic term, though generally referred to on asr9000 series. mostly same as above, though forwarding is performed only by fhrp-active device. Traffic destined towards fhrp-standby is forwarded over iccp (inter-chassis) link. This solution is generally recommended to use lacp commands to create hot-standby lacp links facing the operarional secondary, thus creating config and operational overhead.

• nv -- network virtualization. In a pair of ASR 9000, such technology acts of taking two asr9000 chassis and of forming a single control plane (vss for asr9000). Similar to previous technology mentioned, this virtualization creates non-trivial forwarding design decisions, due to both chassis being "active" especially when port channels are in use.

Discussion

»[H/W] Cisco ISR1100-series, anyone heard of these?
»Catalyst 9000 - Anyone hear about / play with these?
»[H/W] Catalyst 9000
»[Config] Username and password on ASR 9000
»[Config] Reload Ios and Lincseing
»Router Porn for the Day (3925 show tech)
»Cisco 3925 web traffic slows to a crawl
»Cisco 2921 Throughput
»3750 series breakdown
»[H/W] 3750G Lockup
»[H/W] (Maybe late, but...) New Cisco Products
»Accidental Discovery - A Mistake to Possibly Learn From
»[Config] Cisco Cloud Services Router
»CPU Utilization - At what point should I be concerned?

Discussion

»6500 & ASA Packet Loss Tardedness
»EIGRP and Nexus 7k
»[Info] BGP and 512K routes
»Sampled netflow and Nexus 7k
»catalyst 6509
»[H/W] What sup for a 6500?
»6509 Simulator
»2000 users 6509 traffic policing
»[Config] QOS Questions
»[H/W] nexus 6k released....
»HSRP question
»[Other] HSRP VRRP or other option?

Cisco documentation

Cisco Catalyst 6500 VSS and Cisco Nexus 7000 vPC Interoperability and Best Practices White Paper

Nexus performances and its comparison to 6500 series

The Nexus line has a much higher backplane speed than any of Cisco's other products (perhaps other than the CRS) (N7k). The FEX are 4:1 oversubscribed, while the N5k and N7k are 1:1 line-rate for Gig. The N7k has a switch fabric of 15 Tbps. Note that this backplane number comes with some gotchas, mostly that it depends on the number of fabric modules, which each can handle 46 Gbps per slot (Fabric-1) or 110 Gbps (Fabric-2).

To delve, it is about the midplane design in the n7k. The 'fabric' speeds are much higher in this platform than c6k. Each 'fabric' module provides a finite throughput to each slot. Fab-1 are 46 gbps/slot, Fab-2 are 110 gbps/slot. Each linecard will round-robin the VOQ to each Fab as appropriate (i.e. there's no chance of deterministic flows or possible polarization).

A Sup 720 (which has a fabric of 720 Gbps) provides the capability of 40 gbps/slot in a c6509 or smaller chassis. However this isn't always an accurate level of throughput due to the potential for CFC (which requires central arbitration from the Sup). On the other hand everything in the n7k is DFC-based (or performs as if a DFC were present).

A Sup 2T provides up to 80 gbps/slot in all slots up to a 6513 model with assumptions that the switch requires DFC4-based cards for everything except the x6148. It is unclear of the way that the Cat 6k/Sup 2T operates with the 6148 blade bus card installed due to the fact that it is dubious to put that card in an Sup 2T installed system anyway. Such specification of Sup 2T on 6513 model offers an overall similar behavior to an n7k. Keep in mind that positioning of chassis varies greatly and its important to dive into the weeds, lest you design such specification at scale.

The Sup doesn't matter for the N7k, as most of the traffic is handled by the Fabric modules since only Layer 3 and things like VDCs are passed to the Sup. To clarify

(a) Sup 1/2/2e on n7k provides control-plane *only*. Once the control-plane bits settle (ARP, IGP, EGP, etc) converges, the fib is created and downloaded to the linecards. All switching (layer-2/layer-3) is all performed in hardware. There is no ability to punt to the Routing Processor as there is in c6k-land. Anything management plane is also handled by the sup.

(b) Sup does matter in n7k-land as each iteration provides different features (from a management only perspective). Sup 1 provides 3+1 vdc, Sup 2 provides 4+1 vdc, and Sup 2e provides 4+1 with an additional 4 vdc available with the incremental vdc license. From a control-plane perspective, licensing exists on the sup to permit different feature sets. Features of enterprise (routing), advanced (dcnm, vdcs), transport-services (otv, lisp), mpls, storage (fcoe) are all installed on sup to provide additional "unlocked" services to the linecards -- even though they can handle services natively -- the license unlocks the software hooks into the card.

The Catalyst 6500 series Sup 2T provides enablement for 80gbps/slot provided correct line card in use. However n7k did this day 1. With cards like the f1 and f2 combined with fab2 the n7k will blow this out of the water with upwards of 550gpbs/slot. once the M2 line cards are released (2hcy12/1hcy13 or so, I believe) the n7k will have a higher speed, robust and feature-rich l2/l3 services line card for datacenter/campus core use.

With the removal of wan cards (es/es+; flexwan, etc) from sup2t support, the only thing that the c6k does that the n7k doesn't at this point is vpls. I believe the roadmap for this is around 1hcy13 or so.

There are several people running n7k as split agg/pe roles (including some denizens that inhabit these forums). The n7k provides some intuitive troubleshooting features and is designed for a higher level of uptime and redundancy.

Given that you basically have to rip and replace all line cards within your existing c6k chassis to use sup2t, why not invest the capital in a next-gen platform that is at the beginning of its lifecycle rather than some aging dinosaur that Cisco keeps incrementing so as to not piss off the user base?

It is now a questionable reasoning of anyone who wants to put c6k in their datacenter as there may be *no point* to it except for wasting money and delaying the inevitable move to n7k a year down the road. Anyone who is refreshing their campus core with 10 gbe to the access should also highly consider n7k and in addition to remove the c6k off the mdf.

This isn't to discount what the c6k has been for 15 years, but perhaps it is time for c6k to die and go away.

Nexus 7000 features

* VDC
* Multiple Context

Some Descriptions

The target for the n7k is multi-tenancy or separation of roles and responsibilities within teams (test/dev, q/a, and prod) or so. however, some of that separation has been removed on the sup1 platform due to the requirements of things like otv and fcoe each requiring their unique VDC. Per Cisco best practice, it is best to keep user-land traffic off the default VDC. This really leaves you with three contexts to work with. There will be further analysis needed to see what Supervisor 2T can afford to the users in terms of contexts (as well as licensing, cost, limitations, etc).

In addition, VDC is supported *only* on the n7k platform. There is no support in the n5k, n5.5k, n4k, n3k, n1010, or the n1000v.

The 10 Gbps Ethernet Components

If you're looking for top-of-rack 10gbe, it is going to get expensive, regardless of what you're looking at. N5020 will allow 10gbe and either unify it with fcoe or allow for an additional fibre-channel expansion within the chassis. depending on your rack density and desired over-subscription, you may be able to run multiple racks per n5020, utilizing n2000 fabric extenders when 1gbe copper connectivity is needed.

At the aggregation layer the 6509-E chassis with the ws-x6708 blades will work. If you're looking at doing much queueing, you'll want to only run it in dedicated mode (4*10gbe on a 40gbps/slot fabric) to allow for the largest buffers etc. assuming you know what your traffic pattern looks like. However, if you want the highest 10gbe density, you're going to have to jump to the n7000-series since you'll hit 80gbps/slot and twice the number of dedicated interfaces.

Note that all of this is moot if you don't have the -e chassis. If you're looking to buy new, the n7000 has matured for a lot of applications and I'm sure that if you get in touch with your Cisco account manager or Cisco reseller, they may be able to offer you some deep discounts.

»[HELP] Small ISP oversubscription ration?

Basic Commands: A Quick Guide

show version
show inventory
show environment

show module
show redundancy status
show system resources
show feature
show boot
show role

show int counters errors

show run int
show run int eth 1/4-12
show int eth 1/4-12
show int brief
show int transceiver

show cdp neighbors
show cdp neighbors int e1/15 detail

int e1/4
beacon

Cool pipe options: grep, less, no-more, wc, sed, diff; which are basically UNIX/Linux commands.

show ip arp
show mac address-table

show vrf
show vrf default interface (per-interface listing)
show ip int brief vrf all

show int status module 2 | grep disabled

show log last 10

dir
where
pwd

Detach from any module:
~,

show spanning-tree vlan 5

password strength-check

ping 192.168.100.23 vrf management
ssh 192.168.100.23 vrf management
telnet 192.168.100.23 vrf management

switchport (L2)
no switchport (L3)

* 5000 Series Features

show feature
show feature | grep enabled
show license usage

dhcp-snooping - DHCP Snooping
fcoe - Fibre Channel over Ethernet (LICENSE REQUIRED)
fex - Fabric Extender
http-server - HTTP Server (for management)
interface-vlan, SVI (Switch Virtual Interface)
lacp - LACP, required for PortChannels
ldap - LDAP
lldp - Link Layer Discovery Protocol
niv - Network Interface Virtualization
private-vlan - PVLAN
privilege -
sshServer - SSH Server (for management)
tacacs - TACACS Authentication
telnetServer - Telnet Server (for management)
udld
vpc - Virtual PortChannel, aka MEC (Multichassis EtherChannel)
vtp - VLAN Trunking Protocol

* Licensing

www.cisco.com/web/go/license

show license host-id
show license usage

copy scp://jeremy@192.168.1.25/home/jeremy/cisco/foo.lic bootflash:foo.lic
install license bootflash:foo.lic

* Upgrade NX-OS (Nexus 5010, NX-OS 5.0(2)N2(1), non-disruptive)


* PortChannel (EtherChannel)



* Enable Jumbo Frames (Nexus 5010)



* VPC - Virtual PortChannel (aka MEC, Multichassis EtherChannel)

Yes, it's a feature so nice, it gets two acronyms, and sometimes a third, as some folks call Multichassis EtherChannel MCE.

But wait! Cisco has a newer, better technology called FabricPath. See Scale Data Centers with Cisco FabricPath and Migration from Virtual PortChannel to Cisco FabricPath for further info.

* Checkpoint

Note: If you use the default syntax (checkpoint foo), the checkpoint file is place in volatile memory, and is lost on reload!
You probably want to write the file to flash, as illustrated below:

Summary:



Full Example:



* FEX (4x 10Gb connection from 5010 to 2148T)

Note: The doc indicates there is another way to attach a FEX, without the port channel, using 'pinning max-links 4' and
directly associating each interface with a set of ports on the FEX (for example, on the 2148T, each 10Gb connection
would map to 12 1 Gb ports). This method makes each 10 Gb link a single point of failure, and should not be used.
Below is the preferred configuration:



Note: At this point, wait a minute or two for the FEX module to come online



* FCOE (5010, Two servers connecting to a storage array)



Note: Zones use the PORT WWN, not the NODE WWN. Pay attention to the output of show fcns database!



Descriptions

Checkpoints and rollbacks

Using the Cisco Nexus 7010, 5010 and 2148's has changed some of the habits I have traditionally used for the Cisco IOS command set. Some of the new Nexus commands have become second nature and I now miss them on IOS. Being able to use grep is one I really wish was incorporated into IOS. I am used to having it with the ASA platform and now with the Nexus platform - going back to IOS 12.x and not having it there is annoying.

A new command that is really useful on the Nexus platform is checkpoint. There are several things that are unique about checkpoints and how you can use them. First, checkpoints are primarily used for rollback situations. They allow you to make changes on the system and if required due to an error rollback to a known good configuration on the system. There are three rollback types.

• Atomic rollback is done when the configuration can be applied with NO errors

• Best Effort rollback will ignore errors and push the configuration onto the system

• Stop At First Failure will process the rollback request until it hits an error and then stops

The default rollback type is Atomic and this is likely the most common rollback method you would use on a production environment. I am not aware of many folks wanting to rollback to a "Stop At First Failure" or "Best Effort" scenario situation unless true desperation has kicked in. There might be a case of the order of rollback if you are using VDC's and moving physical resources from one VDC to the other in which case perhaps Best Effort might be useful.

Also of note, the rollback feature must be used per Virtual Device Context (VDC), in other words, you have to run the command in each VDC. This is expected behavior as each VDC is it's own NX-OS instance and you have to run all the same commands to get the desired behavior out of the NX-OS platform.

The command itself is very simple:


Example:
checkpoint cp-running-config-known-good-2010-03-22 description checkpoint of running config

There are some restrictions on the checkpoint name (max length 80 characters) and there are restrictions on the filename (max length of 75 characters and filename can't start with the word "system") but otherwise it is pretty straightforward process to get this going. I am using this on NX-OS version 4.3.1, earlier versions had more restrictions on file names and such so read the documentation if you are on an earlier release.

To see what the checkpoint command does you can use the show commands. To see all the checkpoints that are in a given VDC:

The checkpoint command basically keeps a small database of checkpoints to allow you to rollback to a specific one and calculates the differences between a current state or checkpoint and that checkpoint you want to move to. It will generate a rollback script when you use the rollback command. If you want to see the differences that are being generated you can do that too:


Example:
show diff rollback-patch running-config checkpoint cp-running-config-known-good-2010-03-22

To actually do a rollback:

Example:
rollback running-config checkpoint cp-running-config-known-good-2010-03-22 atomic

To see the status of rollbacks

You can also clear out the checkpoint history and files, use the command with caution.
clear checkpoint database

This is a VERY useful command to build into your scripts prior to pushing out production changes on gear. It allows you to have a well known state stored locally and be able to rollback to it quickly in case of problems in your scripts. Awesome!

Nexus Features

To turn a feature on within configuration mode, it is simply feature followed by name of the feature. For example:
feature interface-vlan (Allows to add an IP address on a vlan interface)
feature lacp (Port-Channel Mode)
feature vpc (Virtual Port Channel)
feature lldp (Similar to CDP but is not Cisco proprietary)
feature vtp (Vlan Trunking Protocol)
feature fex (Used when connecting Nexus 2ks)

DNS and Name Resolution

ip domain-lookup (turns on name resolution)
ip domain-name domain-name (DNS domain name, i.e. could be your active directory domain or real world domain name)
ip name-server x.x.x.x (x.x.x.x being the IP address of your DNS server. Repeat this command to add multiple DNS servers)

Access Lists

ip access-list access-list-name (creates an access list with a name)
10 remark Access-List-For-Remote-Access (creates a remark in position 10 of Access-List-For-Remote-Access for the current access list)
20 permit tcp x.x.x.x/24 any eq 22 (creates an access rule in position 20 to allow the network x.x.x.x/24 to any for SSH)
30 deny ip any any log (creates an access rule in position 30 to deny everything and log)
no 25 permit tcp y.y.y.y/24 any eq 22 (removes the access rule in position 25)

Spanning-Tree

spanning-tree mode rapid-pvst (Turns Rapid Per-Vlan Spanning Tree on. Other option is Multiple Spanning-Tree mode MST)
spanning-tree port type edge (Configured on the interface when connecting to end devices. This is essentially port-fast)
spanning-tree port type normal (Configured on the interface and/or port-channel. Used when uplinking to Non-Nexus switches. Does not use Bridge Assurance)
spanning-tree port type network (Configured on the interface and/or port-channel. Used when uplinking to Nexus switches. Uses Bridge Assurance)
spanning-tree port type network default (Makes spanning-tree type network default for all interfaces that do no use one of the above options)
spanning-tree vlan 1 priority 4096 (Makes this switch the root for vlan 1)

Fex Nexus 2000

fex 100 (Creates a FEX ID of 100. This is used to identify the Nexus2k. i.e. port 1 of the Nexus2k will be eth100/1/1)
pinning max-links 2 (Allow a maximum of 2 uplinks from the Nexus 2k FEX)
description Nexus2k-Level-1 (Creates a description of Nexus2k-Level-1)
interface ethernet1/1 (Enter into the interface you wish to connect the fex to)
switchport mode fex-fabric (Put the interface into fex-fabric mode)
fex associate 100 (Associate this port to the FEX ID we created earlier)

VLANS

vlan 10 (Creates vlan 10)
name storage (Creates a name of storage for vlan)
interface vlan 10 (Creates a vlan interface for vlan 10. Must have feature interface-vlan turned on)
ip address x.x.x.x/24 (Creates an ip address of x.x.x.x/24 for vlan interface 10)
description iSCSI-Storage (Creates a description of iSCSI-Storage for interface vlan 10)

Port-Channels

interface port-channel 1 (Creates an interface port-channel 1)
description Uplink-to-Cisco-3750 (Gives the port-channel interface a description)
switchport mode trunk (Turns the interface into vlan trunk mode)
switchport trunk allowed vlan 10 (Restrict which vlans are allowed over the trunk)
switchport trunk allowed vlan add 15 (Add vlan 15 to the restricted allowed vlans on the trunk)
switchport trunk native vlan 20 (Any untag packets will be placed in vlan 20)
switchport mode access (Turns the interface into access mode)
switchport access vlan 10 (Places the interface into vlan 10)
For spanning-tree options please refer to spanning-tree above
interface eth1/1-2 (Enter into interface range mode) eth1/1 and eth1/2
channel-group mode 1 active (Add the 2 interfaces into port-channel 1 using LACP) OR
channel-group mode 1 on (Add the 2 interfaces into port-channel 1. Forces the ports into a channel and does NOT use LACP)
show port-channel summary (Shows a summary of your port-channel interfaces and status)

VPC Virtual Port Channel

Make sure you have turn on feature vpc
Complete this on Switch 1:
vpc domain 100 (Create a unique VPC domain ID between the 2 Nexus 5Ks)
role priority 20 (Switch 1 will have a higher priority than switch 2)
peer-keepalive destination x.x.x.x source y.y.y.y (VPC keep-alive link x.x.x.x being the destination switch ip and y.y.y.y being the source switch ip)
interface port-channel 10 (Enter into the port-channel 10 interface)
vpc 10 (This port-channel belongs to vpc 10, this is not the same as VPC domain. These need to be unique among different port-channels)
interface port-channel 20 (Enter into the port-channel 20 interface)
vpc 20 (This port-channel belongs to vpc 20, this is not the same as VPC domain. These need to be unique among different port-channels)
Complete this on Switch 2:
vpc domain 100 (Create a unique VPC domain ID between the 2 Nexus 5Ks)
peer-keepalive destination y.y.y.y source x.x.x.x (VPC keep-alive link y.y.y.y being the destination switch ip and x.x.x.x being the source switch ip)
interface port-channel 10 (Enter into the port-channel 10 interface)
vpc 10 (This port-channel belongs to vpc 10, this is not the same as VPC domain. These need to be unique among different port-channels)
interface port-channel 20 (Enter into the port-channel 20 interface)
vpc 20 (This port-channel belongs to vpc 20, this is not the same as VPC domain. These need to be unique among different port-channels)
show run vpc (shows the vpc config from the running-config)
show vpc (show vpc information and status)

A nice explanation of what VPC is can be found by Jason Nash on his blog

SPAN or Port-Mirroring for Packet Capture

monitor session 1 (Creates a Monitor Session of 1)
source interface ethernet1/1 (Tells the monitor session which port to use as the source) OR
source interface port-channel 1 (Tells the monitor session which port-channel to use as the source) OR
source interface vlan 10 (Tells the monitor session which vlan to use as the source)
destination interface ethernet1/2 (Tells the monitor session which port to use as the destination, where you would plug your packet capture software such as wire shark)

Line Console

line console (Enter into line console)
speed 38400 (change baud rate to 38400. Might be used if changing the console logging level lower than warning)

Logging

logging console 7 (Turns logging on the console to debug. Must change line baud rate for this, see Line Console above)
logging monitor 7 (Turns logging on the monitor i.e. telnet or ssh, to debug)

Notes

Connecting a 1Gb SFP into a Nexus 5K requires the following command on the interface speed 1000
If you install the Layer 3 daughter card you must have this Base License LAN_BASE_SERVICES_PKG. It comes with the 5k by default, if it is not installed from the factory you may need to contact Cisco Licensing to get it issued to you as its free.

Additional Information
Cisco Nexus NXOS and Fixing Broken Switchto Syntax With Alias
SAN Port Channels from Nexus 5010 to MDS 9134

If you have any technical questions about this tutorial or any other tutorials on this site, please open a new thread in the forums and the community will be able to help you out.

Discussions

»Nexus 7706 and fabric extenders

How to Set Up Cisco Nexus Fabric Extender
Posted by Ethan Banks
January 28, 2013

In a data center that has deployed Cisco Nexus 5000 or 7000 switches, Cisco Nexus 2000 series fabric extenders (FEX) are commonly used for top of rack (ToR) connectivity. FEX units are priced attractively, as they serve a limited purpose and are not feature-rich switches. In fact, a FEX is not a switch, in that a FEX cannot switch traffic locally or be managed independently. A FEX functions only when connected to a Cisco Nexus 5K or 7K series. All traffic flowing into a FEX will be sent down to the parent 5K or 7K for forwarding, even if the destination is on the originating FEX. If you think of a FEX as a remote line card with no local switching capabilities, you've got the idea.

Having established that a FEX is not a switch, let's take a look at the process of installing a new FEX and bringing it online.

1. Rack the FEX

The official Cisco documentation demonstrates racking a FEX in a four-post rack, using the supplied rack ears and rail slides. I have successfully racked FEXs in open two-post racks using only the rack ears. If you choose to do this, be sure to mount the ears further back along the chassis for better weight balance. However you mount the FEX, provide enough clearance for the fiber uplink cables, as they stick out of the front of the chassis. Tight tolerances could prevent the rack door from shutting.

Do not power on the FEX until all cabling and uplink port provisioning on the uplinked 5K or 7K has been completed.

2. Install and cable the uplinks.

When uplinking to Nexus 5Ks, one topology option is to dual-home the physical Fabric Extender. Dual-homing a FEX provides path redundancy, but cuts in half the total number of Fabric Extenders you can deploy. A Nexus 5K supports 24 total connected FEX devices, meaning that two 5Ks could support 48 total single-homed physical FEX. When dual-homing, only 24 total FEX are supported between the two 5Ks.

If you choose to single-home the FEX, you lose uplink path redundancy for single-attached hosts. If single-attached hosts are not a concern, a common scenario is to deploy two FEX to ToR, each single-homed to a Nexus 5K or 7K. Multiattached servers then spread their uplinks across the two ToR FEX devices, and in that way enjoy uplink path redundancy. As of this writing, only Nexus 5Ks support dual-homing of FEX.

If you choose to dual-home the FEX, the uplink ports you select must match on both 5K-1 and 5K-2. For example, if you chose Eth1/1 on 5K-1, you must also use Eth1/1 on 5K-2. Another wise design choice is to spread your uplinks over multiple 5500 ASICs (the silicon inside the switch responsible for forwarding traffic). In the Nexus 5500s, ASICs are mapped to groups of 8 consecutive Ethernet ports on the front to the switch, 1-8, 9-16, etc. Therefore, spreading FEX uplinks over ports 1 and 9 is smarter than 1 and 2. If the ASIC servicing ports 1-8 fails, you won't lose both FEX uplinks.

Although options vary by model, FEX can be uplinked with a variety of media. When ordering, be aware of Cisco SKUs that bundle the FEX with uplink media, such as Fabric Extender Transceivers (FETs) or Twinax. Each option has a different price and distance constraint, so research this carefully to be sure you meet your installations requirements.

Another consideration is the number of uplinks to use. For example, a 2248TP FEX has 48 10/100/1000 access ports and four 10-Gbps uplink ports. If uplinking all four 10G ports, the oversubscription ratio of access to uplink ports is 1.2:1, which is very low. At the same time, youve used up four expensive 10G ports on the Nexus switch on the other end. Do your traffic patterns warrant using all four uplink ports, or can you get by with just two? Note that from a technical standpoint, the FEX will function correctly with only a single working uplink, but a sensible design uses at least two in a production environment.

The remainder of this example assumes a dual-homed FEX using all four uplinks, connected to a Nexus 5500 pair configured in a virtual port-channel domain.

3. Configure a virtual port-channel and add physical interfaces.

Two of the FEX uplinks will homed to one 5K, and two to the other 5K. Then, all four of the FEX uplinks will be combined into a single virtual port channel. Each FEX is assigned a number from 100-199.

The physical interface requires two specific commands to tell the hosting 5K that the interface is servicing a FEX. The command "switchport mode fex-fabric" lets the Nexus switch know that the device on the other end of the link is fabric extender. Note that if you use FETs as uplink media, the switch cant use these optical modules until this command is in place.

The command "fex associate " tells the Nexus switch which specific FEX is being uplinked to that port. The number selected must match for all uplink ports.

4. Apply the code below to both 5Ks.

Once this step is complete, you can optionally prepare your console session to watch the messages that will scroll as the FEX comes up for the first time by typing "term mon".



5. Power up the FEX.

Connect the power cables. Ensure that the redundant power supplies are uplinked to two different power sources.

6. Review console messages.

With the FEX powering up, you should see console messages scrolling by on the 5Ks that indicate a FEX has been discovered, ports are coming up (and probably going back down, then back up) and so on.

The first time the FEX boots, it will probably require a firmware upgrade. The Nexus switches to which the FEX is uplinked will upgrade the firmware automatically. The upgrade takes about 10 to 15 minutes. When this process is complete, the FEX will be online and ready for you to document and sanity check.

7. Set the FEX description.

This should be simply the name of the FEX. This name shows up in other CLI output, and is helpful in determining just which FEX you're working on.



8. Verify the FEX is uplinked and ready for use.

Here are some commands to help you verify the FEX is uplinked correctly. Following is an illustration.



9. Provision the FEX access ports.

Once the FEX is up and running to your satisfaction, you can provision the access ports according to your procedures. I ensure that all new ports are disabled and described as "AVAILABLE". This makes it easy for operators to see what ports can be used when required, and prevents server admins from plugging into a port at random.

Additional Info
Connecting a Nexus 2148 to a Nexus 5010

Concluding Thoughts

Note that on dual-homed FEXs, the FEX-related configuration commands on the Nexus switches must be identical. You can do this manually by configuring each switch in turn. Cisco also offers a configuration synchronization service to perform this task automatically, but setting up that service is beyond the scope of this article.

While the output of show fex detail shows that all FEX access ports are mapped to the virtual port channel, it is possible to map specific access ports to specific uplink ports using static-pinning.

Finally, Cisco offers a Nexus 5K/2K access layer design guide (in PDF file). I recommend you read it carefully to understand the options and caveats when deploying a Nexus infrastructure.

Ethan Banks is CCIE #20655, a host of the Packet Pushers podcast, and IT practitioner since 1995. He can be reached at @ecbanks.

Discussion

»Nexus 7706 and fabric extenders

US Robotics

Courier® External 56K* V.92 Global Dial-up Business Modem
56K* V.92 Serial Controller Dial-up External Faxmodem

Configure Cisco Router to receive dial-in through AUX and CONSOLE ports

Configuring a Modem on the AUX Port for EXEC Dialin Connectivity
Attaching a US Robotics Modem to the Console Port of a Cisco Router

WARP from FatPipe Networks

Here is a quote from the link.

---

Weighted Algorithm

This configures FatPipe WARP to balance traffic in proportion to the WAN weights defined by you. Each interface needs to be assigned a weight. (Default value for each interface is 1.) The ratio of these weights determines the ratio of downloaded traffic on the respective Internet lines, which the load balancing algorithm maintains. For each new outbound session, this algorithm finds the interface whose current throughput to total throughput ratio is farthest below the ratio determined by its weight, and send the session on that interface.

If weights for WAN1, WAN2, WAN3 are 1, 2, 3, respectively, and total download traffic amounts to 600kbps, the traffic will be balanced over respective lines as 100, 200, 300 kbps. Because FatPipe WARP balances sessions rather than packets, real world results will rarely achieve this ideal. In general, the greater the number of sessions, the closer the distribution of traffic will be to the specified weights.

---

Illustration

Let's assume the following
* There is a block of IP address 1.1.1.0/24 from ISP 1 and there is a block of IP address 2.2.2.0/24 from ISP 2
* You configure the ASA Outside interface and NAT statement using 1.1.1.0/24
* You may end up using IP address from 2.2.2.0/24 going out
* Such condition is possible due to the fact that ISP 2 is preferred when ISP 1 is kind of used up
* The 1.1.1.0/24 may be used after ISP 1 link is no longer heavily utilized

IOS-XE

One example is an 3650 switch. The 3650 has a dual-core CPU and IOS-XE is required to run on those CPUs. In terms of actually using the switch, you'll notice no difference between IOS and IOS-XE - they're functionally the same. You don't even know that Linux is running in the background. Following is a quote from Cisco website.

"It should go without saying that IOS-XE was designed for routers, switches, and appliances, and as such, it embraces all the field-tested capabilities and features of IOS, while adding new functionality and benefits traditionally found in a portable operating system interface (POSIX) environment. This was the most logical approach available to integrate network-aware applications into modern routing devices. As a result, IOS-XE seamlessly integrates a generic approach to network management into every function, borrowing heavily from the equally reliable POSIX operating system.

Furthermore, through the incorporation of a series of well-defined application programming interfaces (API), Cisco has improved IOS portability. Specifically, we are making reference to the operation of IOS across platforms as well as extending capabilities outside of IOS. This final component to IOS-XE creates a future where application integration will be simplified, integral, and commonplace.

IOS has been the center point for network expansion, configuration, and operation for decades, and this same functionality is now integrated into IOS-XE, thus preserving all the advantages of traditional IOS and its unparalleled history for delivering functionality for business-critical applications. All of this is done while retaining the same look and feel of IOS, but doing it while ensuring enhanced functionality. How is all this possible? IOS-XE runs a modern Linux operating system that employs a single daemon; the additional functionality we have been discussing will be run as isolated processes within the OS of the host. This means that we have all the capabilities we had in IOS with enhanced operations and functionality that will not require retraining.

At first glance, this might not seem to be that big of an improvement, but if we keep in mind that running IOS and these other applications as separate processes, it becomes apparent that we can now leverage symmetrical multiprocessing. This in itself means that we can garner the benefits of load balancing across multiple-core CPUs by binding processes to different cores. Thus, we create an operational environment where it is possible to support multithreading and multicore CPUs. This capability, coupled with how IOS-XE separates the control plane from the forwarding plane, ensures a level of management and control that could not possibly exist in the context of the traditional monolithic IOS."

Discussion
»[HELP] Catalyst-3650#request system shell
»[Info] NXOS and linux
»[HELP] Cisco Nexus Titanium Image

Typical redundant supervisor setup is Active/Standby. The idea is to have the Standby to take over in case the Active fails. However having two supervisors cost money and may not behave as expected.

If you need two supervisors as redundancy, then you may want to consider running VSS on two different switches that each has one supervisor. In addition, running VSS simplifies the network. Following is the FAQ.
»Cisco Forum FAQ »Catalyst VSS Technology

Discussion
»Re: Catalyst 6000 vs 6500

CSR 1000V Router

Cisco Documentation

Cisco CSR 1000V Series Cloud Services Router Software Configuration Guide

Discussion
»Cisco CSR 1000V - kicking tires

»10 gig switch recommendations

Cisco Documentation
Understanding Bridge Virtual Interface (BVI) and Bridge Domain Interface (BDI)

By default, router is a layer-3 device therefore two or more interfaces in a router cannot be on the same broadcast domain. In some cases however, you do need two router interfaces to be on the same broadcast domain while keeping the router's Layer-3 functionality intact. These cases include (but not limited to)

* You need to have a router between the ISP and your firewall while you only have one ISP-assigned block of Public IP address
* You need to simulate or extend Layer-2 connectivity over Layer-3 point-to-point circuits (routers) to support devices or some server's heartbeat that only do Layer-2

Traditional IOS-based Router Configuration

IOS-XE based Router Configuration

Netflow

When access to network sources is slow, you may want to know who causing it and for which purpose. In other words; you would like to know the source IP address (the machine causing the slowness), destination IP address and TCP/UDP ports (the purpose of access). In Cisco there is a technology called Netflow while in Juniper is called J-Flow.

Typically only routers support Netflow while some switches may also have the feature. Both Netflow and J-Flow requires a collector (i.e. Netflow collector or J-Flow collector) which is simply a server that stores all info collected from the router. This server runs some Netflow-compliant software with nice GUI to present the info in a way that humans understand; top talker based on IP address, applications (HTTP or some TCP/UDP protocols), or hosts.

In the following sample configuration, the Netflow collector IP address is 10.1.9.20, running on default UDP port 2055.

Sample Configuration

IOS XE

NX-OS Layer-3

Traditional IOS

»[Info] Setting proper expectations with our underlings