Buffer-to-buffer credits (BB_credits) are a Fibre Channel link-level flow-control mechanism that ensures every frame (a Fibre Channel packet) that is sent has enough buffer space to be received. Each Fibre Channel link can be viewed as two unidirectional links each with their own set of BB_credits. During link initialization, each side informs the other side of the number of receive (Rx) BB_credits it has via the Exchange Link Parameters (ELP) and Accept (ELP) on an E port and FLOGI and Accept (FLOGI) on an F or NP port. When the Rx BB_credit number is received, it is stored as the transmit (Tx) BB_credit number. This way each side's Rx BB_credit number is the other side's Tx BB credit number for each direction on the link.

Each buffer location holds exactly one Fibre Channel frame regardless of size. As a frame is to be transmitted, the sender checks the remaining Tx BB_credit number. If it is greater than 0, a frame can be transmitted. The sender then decrements Tx BB_credit remaining number and transmits the frame. After the frame is received and processed such that the receiver's buffer location is cleared, the receiver transmits a R_RDY primitive (BB_credit). When the BB credit is received, the sender increments its Tx BB_credit remaining number. This mechanism guarantees that a sender never transmits a frame that the receiver does not have a buffer to hold it in.

Note

Cisco MDS switches support mechanisms to avoid loss of R_RDYs which may cause issues on links. For more information, see the Buffer-to-Buffer Credit Recovery section. Reconfiguring BB_credits on an active link is a disruptive operation. The number of Rx BB_credits does not have to match on each side of the link. Only the Rx BB_credits can be configured on an individual interface because those are the only credits that an interface has control over. If the transmitter decrements the remaining Tx BB_credit remaining number and hits zero, the Tx transition to zero counter will be incremented by one. This typically indicates some level of congestion at the receiving device. Although it could also indicate that there are insufficient buffers for the speed and distance of the link. If the receiver does not transmit R_RDYs to the sender of frames, then once the number of frames equals to the Rx BB_credits are received the sender must stop sending since it has hit 0 Tx BB_credit remaining. The receiver will also hit 0 Rx BB_credits remaining and will increment its Rx transition to zero counter by one. Long-distance links may must have the number of BB_credits increased on both sides to ensure maximum performance.

A port group is a set of contiguous ports that share common resources such as bandwidth and buffer credits from a global pool of buffers.

The global pool of buffers includes the global receive buffer pool. The global receive buffer pool includes the following buffer groups:

• Reserved internal buffers

• Allocated buffers for each Fibre Channel interface (user configured or assigned by default)

• Unallocated buffers, if any, to be used for additional buffers when required

Extended buffer-to-buffer credits are made possible by allocating extra buffers to specific interfaces. These extra buffers are taken from the unallocated buffer pool.

Note	The ENTERPRISE_PKG license is required to use extended buffer-to-buffer credits on 16 Gbps, 32 Gbps, and 64 Gbps switching modules.

All ports on the 16 Gbps and 32 Gbps switching modules support extended buffer-to-buffer credits. There is a limitation on the maximum number of extended buffer-to-buffer credits you can assign to a port. If necessary, you can configure interfaces to use minimum credits to make more extended buffer-to-buffer credits available to other ports.

For long-distance ISLs, the extended buffer-to-buffer credits feature allows you to configure the receive buffers up to the level you need and within hardware resource limits. When necessary, you can reduce the buffers on one port and assign them to another port in the same port group. However, you must have first released the buffers from the other ports before configuring larger extended buffer-to-buffer credits for a port.

Extended BB_credits are typically used on long-distance ISL ports (E ports). If you require additional BB_credits on a port or a group of ports, buffers may need to be made available.

To allow most buffers to be available, perform these steps:

1. Configure ISL ports over different port-groups and modules.

2. Configure ports that are connected to end devices (F ports) from mode auto to mode F by using the switchport mode f command.

In general, you can configure any port in a port group to dedicated rate mode. To do this, you must first release the buffers from the other ports before configuring larger extended buffer-to-buffer credits for a port. This will reduce the number of buffers allocated to those ports 500–32 (on most switch types) and add those saved buffers into the unallocated pool.

Without changing the default mode or speed, you can assign the remaining available BB_credits within the port group, using the switchport fcrxbbcredit extended credits command. Use the show port-resources module slot command to verify the updated BB_credits allocation among the interfaces.

For example, if there are 300 extended BB_credits available, we can assign these 300 BB_credits to a port that is having 500 BB_credits so that the port now has 800 BB_credits provided all other ports in the port group are configured with speed auto and mode auto.

switch# configure terminal
switch(config)# interface fc1/1
switch(config-if)# switchport fcrxbbcredit extended 800

When we configure ports in a port group to F Port mode, the reserved buffer credits for such ports are reduced from 500 BB_credits to 32 BB_credits and the remaining BB_credits are assigned to the unallocated interface buffer pool. If you need more BB_credits for that particular port group buffer pool, you can reduce the number of BB_credits being used by the F ports, using the switchport fcrxbbcredit credits command.

An alternate option is to configure the remaining ports in a port group to minimum credits to free up all BB_credits from these ports for extended BB_credits use.

Note

The ENTERPRISE_PKG license is required to use extended buffer-to-buffer credits on 16 Gbps and 32 Gbps switching modules. All ports on the 16 Gbps and 32 Gbps switching modules support extended buffer-to-buffer credits. There are no limitations for how many extended buffer-to-buffer credits you can assign to a port (except for the maximum and minimum limits). If necessary, you can configure interfaces to use minimum credits to make more extended buffer-to-buffer credits available to other ports.

This section describes how buffer credits are allocated on Cisco MDS 9000 Series Multilayer switches.

When long-distance ISLs are required, you must have sufficient BB_credits configured to ensure that the ISL can run at maximum capacity. The simplest formula or the rule of thumb for computing BB_credits for long-distance ISL assumes a full sized Fibre Channel frame of approximately 2 KB and factors in the interface operating speed and one way distance of the ISL.

Note	If the average frame size is less than 2 KB, the number of BB_credits must be increased.

As per the table, to operate a 16-Gbps Fibre Channel ISL over 50 km, you would multiply the one way distance (50) times the minimum number of BB_credits per km (8). That is, a 50 km 16-Gbps ISL requires 400 BB_credits when the average frame size is approximately 2 KB. This is the minimum number of BB_credits that are required for the link to function at its best when utilized to its maximum. To accommodate a smaller average frame size than the maximum (full sized) value, more BB_credits would be required proportionally. Since each buffer is for a Fibre Channel frame irrespective of its size, when Fibre Channel frames are not full sized, more BB_credits are required to achieve full link utilization. In this case, an approximate yet simple formula for calculating BB_credits is the following:

BB_credits = (Minimum number of BB_credits required per km for interface speed x One-way distance (km)) / ((Average receive frame size (bytes) / 2150 bytes))

The following example displays the BB_credits calculation for a 16 Gbps link that is 50 km long with an average input frame size of approximately 1 KB (1075 bytes):

(8 BB_credits per km at 16 Gbps x 50 km) / (1075 / 2150) = 800 BB_credits

switch# show interface fc 2/7 counters
fc2/7
    5 minutes input rate 1048060640 bits/sec, 131007580 bytes/sec, 94786 frames/sec
    5 minutes output rate 253368512 bits/sec, 31671064 bytes/sec, 47717 frames/sec
    14079632456 frames input, 18624775031572 bytes
      0 discards, 0 errors, 0 CRC/FCS
      0 unknown class, 0 too long, 0 too short
    8089598629 frames output, 6040401816628 bytes
      0 discards, 0 errors
    0 timeout discards, 0 credit loss
    0 input OLS, 0 LRR, 0 NOS, 0 loop inits
    0 output OLS, 0 LRR, 0 NOS, 0 loop inits
    0 link failures, 0 sync losses, 0 signal losses
    15031 Transmit B2B credit transitions to zero
    0 Receive B2B credit transitions to zero
    11192 2.5us TxWait due to lack of transmit credits
    Percentage TxWait not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
    500 receive B2B credit remaining
    481 transmit B2B credit remaining
    481 low priority transmit B2B credit remaining
    Last clearing of "show interface" counters: 2d09h

In the above example, the calculation for the average frame size is:

18624775031572 (bytes) / 14079632456 (frames) = 1323 bytes/frame which is approximately 1.3 KB/frame

To complete the calculation:

(8 BB_credits per km at 16 Gbps x 50 km) / (1323 average bytes per frame / 2150 bytes) = 650 BB_credits

Thus, for an 8 Gbps link traversing 50 km carrying an average frame size of 1323 bytes would take a minimum of 650 BB_credits.

Note	The required BB_credits for the other end of the same link may be different due to a different average frame size in the opposite direction. The average frame size must be calculated in a similar way from the adjacent interface.

For more information on how to change BB_credits, see the Extended Buffer-to-Buffer Credits section and see the switchport fcrxbbcredit std_bufs and switchport fcrxbbcredit extended ext_bufs command outputs.

Although Fibre Channel standards require low bit and frame error rates, there is a likelihood of errors occurring. When these errors affect certain Fibre Channel primitives, credit loss might occur. When credits are lost, performance degradation might occur. When all credits are lost, transmission of frames in that direction stops. The Fibre Channel standards introduces a feature for two attached ports to detect and correct such scenarios nondisruptively. This feature is called buffer-to-buffer credit recovery.

A credit can be lost in either of these scenarios:

• An error corrupts the start-of-frame (SoF) delimiter of a frame. The receiving port fails to recognize the frame and subsequently does not send a corresponding receiver ready (R_RDY) primitive to the sender. The sending port does not replenish the credit to the receiving port.

• An error corrupts an R_RDY primitive. The receiving port fails to recognize the R_RDY and does not replenish the corresponding credit to the sending port.

The Buffer-to-Buffer Credit Recovery feature can help recover from the two specified scenarios. It is a per-hop feature and is negotiated between two directly attached peer ports when the link comes up, by exchanging parameters. Buffer-to-buffer credit recovery is enabled when a receiver acknowledges a nonzero buffer-to-buffer state change number (BB_SC_N).

Buffer-to-buffer credit recovery functions as follows:

1. The local port and peer port agree to send checkpoint primitives to each other for frames and R_RDYs, starting from the time the link comes up.

2. If a port detects frame loss, it sends the corresponding number of R_RDYs to replenish the lost credits at the peer port.

3. If a port detects R_RDY loss, the port internally replenishes the lost credits to the interface buffer pool.

Buffer-to-buffer credit recovery implementation is as follows:

1. Buffer-to-buffer state change SOF (BB_SCs) primitives are transmitted every 2^BB_SC_N number of frames sent. This enables an attached port to determine if any frames are lost. If frames loss is detected, the receiver of the BB_SCs transmits the appropriate number of R_RDYs to compensate for the lost frames.

2. Buffer-to-buffer state change R_RDY (BB_SCr) primitives are transmitted every 2^BB_SC_N number of R_RDY primitives sent. This enables an attached port to determine if any R_RDY primitives are lost. If R_RDY primitive loss is detected, the receiver of the BB_SCr increments the number of transmit credits by the appropriate number to compensate for the lost R_RDYs.

The Buffer-to-Buffer Credit Recovery feature can be used on any nonarbitrated loop link. This feature is most useful on unreliable links, such as Metropolitan Area Networks (MANs) or WANs, but can also help on shorter, high-loss links, such as a link with a faulty fiber connection.

Note

The Buffer-to-Buffer Credit Recovery feature is not compatible with the distance extension (DE) feature, also known as buffer-to-buffer credit spoofing. If you use intermediate optical equipment, such as dense wavelength-division multiplexing (DWDM) or Fibre Channel bridges that use DE on Inter-Switch Links (ISLs) between switches, then buffer-to-buffer credit recovery on both sides of an ISL must be disabled using the no switchport fcbbscn command.

The following are the guidelines and restrictions for the Buffer-to-Buffer Credit Recovery feature:

• E ports

  • This feature is enabled by default on ISLs (E ports).

  • This feature works on an ISL between a Cisco switch and a peer switch from any vendor, provided this feature is supported on the peer switch.

  • This feature is supported only on links that are in R_RDY flow control mode. It is not supported on links that are in ER_RDY flow control mode.

• F ports

  • This feature is enabled by default on F ports starting from Cisco MDS NX-OS Release 8.2(1).

  • This feature works on an F port between a Cisco switch and a peer device from any vendor, provided this feature is supported on the peer device.

  Note	Some host bus adapters (HBAs) do not support the Buffer-to-Buffer Credit Recovery feature. Others support this feature at only certain speeds. Check with your HBA vendor about the exact configurations supported.

• NP ports

  • The adjacent N-PortID Virtualization (NPIV) F port must also support this feature. Prior to Cisco MDS NX-OS Release 8.4(1), N-PortID Virtualization (NPIV) ports do not support buffer-to-buffer credit recovery for Cisco N-Port Virtualizer (Cisco NPV) switch logins.

  • This feature is enabled by default on NP ports starting from Cisco MDS NX-OS Release 8.4(1).

The count of times the buffer-to-buffer credits have been recovered for both types of recovery can be displayed using the show interface counters detailed command:

Prior to Cisco MDS NX-OS Release 8.4(1a) and earlier release:

switch# show interface fc1/1 counters detailed
fc1/1
...
0 BB_SCs credit resend actions, 0 BB_SCr Tx credit increment actions

From Cisco MDS NX-OS Release 8.4(2) and later release:

switch# show interface fc1/1 counters detailed
fc1/1
...

Congestion Stats:
Tx Timeout discards: 0
Tx Credit loss: 0
BB_SCs credit resend actions: 0
BB_SCr Tx credit increment actions: 0

By default, the maximum data field size is configured for Fibre Channel interfaces and cannot be reconfigured.