Cisco Nexus 2000 Series Fabric Extenders

This post is going to be focused on the Cisco Nexus 2000 Series Fabric Extenders.  All the information I'm presenting can be found in Cisco's documentation.

WHAT IT IS

The Nexus Fabric Extender is like an external line card that connects to and expands the capacity of the Nexus switches (5000, 6000, and 7000 series).  It looks like a typical 1U switch.  The ports on the Fabric Extender are divided into two types of ports.  Some of them are "fabric interfaces".  They look like regular ports.  Some interfaces are "host interfaces" that you are supposed to use to connect to servers.  The fabric interfaces are regular SFP+ or QSFP+, depending on the model.  They are typically located on the right side and will be grouped together.  They are used to connect to the parent Nexus switch.  The great thing about this architecture is that it doesn't use any weird stack cables or anything like that.  We just use regular network fiber cables and they connect to the Nexus switch using pretty much any of the integrated ports on the parent switch that you provisioned to be FEX ports. 

HOW IT WORKS

You provision ports on the parent switch to be "FEX" ports by issuing the command on the interface switchport mode fex-fabric.  You give it an ID using the command fex associate XXX, where XXX is the ID number you choose.  Then, you connect them to the Nexus Fabric Extender's fabric interfaces.

The Nexus Fabric Extender has its own power supply just like a regular switch.  Once it is powered up, the parent switch automatically uploads the firmware to the Fabric Extender.  Once the Fabric Extender is up and running, you configure it from the parent switch as if it was another line card.  The FEX ID you gave it is used to identify the chassis number.  For instance, if you gave it a FEX ID of 120, then you would configure port 5 on the Fabric Extender like so: interface ethernet 120/0/5.  This gives us the benefit of having to log into only one place to manage everything.  If you were adding actual switches to the network instead of Fabric Extenders, you would have to log into each of those switches separately to manage all of them.  What a pain.

The Nexus Fabric Extender doesn't perform the actual switching though. When traffic is received on one interface of the Fabric Extender and is destined for a host that is on another interface of the Fabric Extender, it is forwarded to the parent switch which forwards it back to the Fabric Extender and out the destination port. Therefore, all traffic must take a hop through the parent switch.

FABRIC INTERFACES

• The fabric interfaces on the Fabric Extender cannot be used for connecting to anything else other than the parent switch.  They have to be DIRECTLY connected to the FEX ports you configured on the parent switch.  You cannot put another switch or hub between them.
• You can either create an EtherChannel to aggregate all the FEX ports or you can use Static Pinning.  
• In Static Pinning, each host port on the Fabric Extender is automatically mapped by the parent switch to a fabric interface.  This gives us the benefit of assigning hosts to the host ports in a way where we can influence which fabric interface will be used for that host.  This is kind of a manual QoS reservation technique.  
• When using Static Pinning, if one of those fabric interfaces were to go down, the mapped host ports also go down until the problem is fixed.  This doesn't happen with EtherChannels.  
• You can configure Static Pinning while the switch is running, but any changes WILL disrupt the hosts connected to that Fabric Extender.
• CDP isnot support on fabric interfaces when connecting using vPC.

HOST PORTS

• BPDU-GUARD is enabled on host ports by default and cannot be disabled.  That means that you should not connect switches to them.  The host ports are only supposed to be used for servers.  The exception is if you’re connecting a switch that doesn’t use STP, but extra precaution will be needed to prevent loops.
• The host ports are also automatically configured with spanning-tree port type edge.   
• You can only enable inbound ACLs on the host ports if you are doing Static Pinning.
• Host ports can only be SPAN sources and not SPAN destinations.
• Up to 8 host interfaces can be combined in an EtherChannel.  You can use LACP or statically port channel them.
• You cannot port channel parent ports with Fabric Extender ports.  In earlier versions of NX-OS, you could only port channel host interfaces on the same Fabric Extender.  This limitation was removed in later versions.
• Beginning in 5.2, all host interfaces are L3 by default.  If you upgrade to 5.2, the already configured host ports retain their default port mode of L2.
• You can configure up to 32 sub-interfaces on a host interface.
• Host ports can be Access or Trunk ports.

LIMITATIONS

Because the total amount of bandwidth the host ports provide is greater than the total amount of bandwidth that the fabric interfaces can provide, you WILL have over-subscription.  How much over-subscription will be based on how many fabric interfaces you connect to the parent switch.

On the Nexus 7000, there are certain limitations regarding what kind of line cards you can configure FEX ports on.

SOFTWARE UPGRADES

You never have to manage the Fabric Extenders separately from the parent switch.  The configuration is saved on the parent switch.  And, when you upgrade the firmware on the parent switch, this will automatically upgrade the firmware on the Fabric Extenders as well.

CHANGE LOG

NX-OS 4.0(a)

• A fabric extender can only connect to and be managed by one parent switch.
• You CANNOT EtherChannel host ports on the Fabric Extender.

NX-OS 4.0(1a)N2(1)

• You cannot use the ports on the expansion modules of a Nexus 5500 series as FEX ports.  You have to use the integrated ports.

NX-OS 4.1(3)N1(1)

• vPC support
• You can dual-home Fabric Extenders to Nexus 5500 series switches using vPC on the FEX ports.  You cannot do this on the 7000 series.
• You can vPC host ports on separate FEXes, but only if the Fabric Extenders are single-homed.
• Locator Beacon feature added to identify the FEX.

NX-OS 4.2(1)N1(1)

• feature fex is disabled by default
• Support for new Nexus 2248TP and 2232PP models, which supports port channeling host ports on the same Fabric Extender.

NX-OS 5.1(3)N1(1)

• Support added for Enhanced vPC on Nexus 5500 series.  Host interface port channels can now be created across different Fabric Extenders which are themselves dual homed.

NX-OS 5.2(1)

• All host interfaces are by default L3 interfaces.

NX-OS 6.0(1)

• No new FEX features or changes that I can tell.

NX-OS 6.1(1)

• port-channel min-links command added to specify the minimum number of links threshold that causes the fabric port channel to be brought down to force a NIC switchover.  This is for dual-homed hosts connect to host ports in an active/standby configuration.
• Can now configure load balancing method for host interface port-channels.

NX-OS 6.2(2)

• FEX now supports queuing.  This allows us to connect a router to the host ports.  We can either create a routing adjacency with an L3 host interface.  Or, we can create a routing adjacency with an SVI and an L2 host interface.