Configuring Deduplication

This chapter describes how to configure deduplication on Cisco NX-OS devices.

This chapter contains the following sections:

About Deduplication

From Cisco NX-OS Release 10.4(1)F, deduplication feature can be used to remove duplicate copies of the data flow when they are going through switches in Nexus Data Broker (NDB).

Due to the continuous rise in data flow, navigating in the networks and the data processed in different applications leads to data duplication both in source and target. For efficient data management, security, and storage you can use deduplication to remove duplicate data.

The deduplication feature eliminates the duplicate traffic which is received from traffic analysers or data storage tools. It identifies duplicate flows which are transferred in the NDB switch. Deduplication supports two model types:

  • Inline deduplication Model (Interface Mode)

    You can configure the inline model on existing NDB switches. This eliminates duplicate packets using interfaces. Initial interface packets are forwarded and the duplicate interface packets from other interfaces are dropped.

  • Deduplication on a Stick Model (VLAN Mode)

    In this model, the packet flow is bound to a specific VLAN. The packet flow is permited on the first VLAN and duplicate flows on other VLANs are restricted. The packets are tagged with specific VLAN when the flow enters NDB switches. Each packet is tagged with a unique VLAN, the packets with VLAN tag are forwarded to the deduplication switch, and duplicate flow is removed on that switch.

Guidelines and Limitations

The below mentioned are the guidelines and limitations for the deduplication:

  • You can configure deduplication for inner packet flows only.

  • TCP and UDP flows are supported for flow deduplication.

  • Deduplication is supported for local SPAN and Optical TAP flows sessions.

  • Deduplication can be performed for VXLAN and GRE tunnel packets with a single encap.

  • Deduplication is supported for Cisco Nexus 9300-FX2/FX3 and 9300-GX/GX2 platorm switches.

  • Deduplication is not supported for:

    • Cisco Nexus 9500 platorm switches

    • Cisco Nexus 9300-C, 9300-EX, 9300-FX and EOR switches

    • Cisco N9K-C9408 switch

  • You cannot configure deduplication for duplicate packets on the same interface such as ERSPAN. It may terminate more than a single ERSPAN session on an interface or SVI. This impacts the copy of same flow ends up on the same interface or VLAN. A switch cannot differentiate different ERSPAN sessions for deduplication.

  • You cannot delete duplication flow for short lived flows.

  • Flow packets from different VRFs with similar 5 tuples cannot be deleted. As deduplication is for the flows performed using 5-tuple of a packet. It cannot identify VRF packets.

  • Deduplication stick model (VLAN model) is not supported for tunnel termination, and Q-in-Q ports.

  • Deduplication is not supported for IPv6 and multicast flows.

  • Dynamic aging is supported only for 128k flows.

  • During ISSU, deduplication is disabled by default. Post ISSU all flows are cleared and refreshed.

  • Deduplication to function appropriately on tunnel traffic, ensure that you enable flow terminate.

Configuring Deduplication

From Cisco NX-OS Release 10.4(1)F, deduplication feature can be used to remove duplicate copies of the data flow when they are going through switches in Nexus Data Broker (NDB).

Ensure that you reload switch after configuring deduplication, for the configuration changes to be effective.

SUMMARY STEPS

  1. configure terminal
  2. tap-aggregation flow-deduplication
  3. (Optional) absolute-timertime in minutes
  4. (Optional) dynamic-timertime in milli seconds
  5. mode {vlan|interface}
  6. clear hardware deduplication statistics{slot|module in number}

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

tap-aggregation flow-deduplication

Example:

switch(config-dedup)#
Enable flow-deduplication feature on a switch.

Step 3

(Optional) absolute-timertime in minutes

Example:

switch(config)# absolute-timer 10
switch(config-dedup)#
 (Optional)

Allows to configure absolute timer to deduplication of packet flows. The interval range is from 0 to 1440 minutes.

Step 4

(Optional) dynamic-timertime in milli seconds

Example:

switch(config)# dynamic-timer 2000
switch(config-dedup) mode interface#
 (Optional)

Allows to configure dynamic timer to deduplication of packet flows. The interval range is from 0 to 300000 milli seconds.

Step 5

mode {vlan|interface}

Example:

switch(config)# interface
switch(config-dedup)#

Allows to configure deduplication on required mode.

Note

 

Ensure to save configuration and reload the switch to configure deleting duplication on the switch.

Step 6

clear hardware deduplication statistics{slot|module in number}

To clear deduplication on the required slot or module. The slot or module number range is from 1 to 30.

Example

Below shows the sample output for reference:
switch# show hardware deduplication summary
slot 1
=======
Deduplication		    : Enabled
Dedup Mode 		      : Interface
Dynamic timer                 : 200000 milliseconds
Absolute timer                : 5 minutes
Max Supported Flows           : 240K
Total number of learned flows : 240000
Total number of dropped bytes : 65698869600
switch# show hardware deduplication detail
slot1
=========================================================================== 
			Dedup Flows
============================================================================
SourceIP 	Destination IP 	Ports(Src:Dst) Protocol    Interface     Learn-time
======================================================================================================================================================
33.1.1.2 	12.1.1.2 		3000 :3001 	6 	Eth1/1 	07/28/2023 11:47:09.532376
55.1.1.2 	12.1.1.2 		15000:15001 	17     Eth1/1 	07/28/2023 11:47:09.532229
11.1.1.2 	12.1.1.2 		1841 :1842 	6 	Eth1/1 	07/28/2023 11:47:09.532340
1.22.1.2 	1.12.1.2 		2000 :2001 	6 	Eth1/1 	07/28/2023 11:47:09.532428
1.44.1.2 	1.12.1.2 		4000 :4001 	6 	Eth1/23 	07/28/2023 11:47:09.532133
switch#show hardware deduplication age-history
slot 1
===========================================================================
				Dedup Flows
===========================================================================
Source 	Destination Ports 	Protocol Interface Timer 	  Learn-Time 			Aged-Time 
IP		IP	  (Src:Dst)		
===================================================================================================================================
1.44.1.2	1.12.1.2   4000:4001	6 	Eth1/17 Dynamic   08/05/2023 2:24:49.26020      08/05/2023 12:33:29.21904
33.1.1.2 	12.1.1.2  3000:3001       6 	Eth1/27 Dynamic  08/05/2023 12:24:49.126246    08/05/2023 12:33:29.21945
55.1.1.2 	12.1.1.2  15000:15001    17 	Eth1/5 Dynamic    08/05/2023 12:24:49.26070     08/05/2023 12:33:29.21957
1.22.1.2 	1.12.1.2   2000 :2001     6 	Eth1/5 Dynamic    08/05/2023 12:24:49.26115     08/05/2023 12:33:29.21969
11.1.1.2 	12.1.1.2   1841 :1842     6 	Eth1/17 Dynamic   08/05/2023 12:24:49.25949     08/05/2023 12:33:29.21979