The DI network is the private L2 network that interconnects the VMs. The DI network transports user traffic from the received
VM to the serving Session Manager on a different VM, and also transports CF to SF communications such as CLI commands, health
checks, status changes. If the link is compromised unexpected things can happen (such as slow response to CLI commands), potentially
resulting in service interruption.
The available monitoring capabilities to verify the health of the DI network are detailed here:
Inter-SF DI Network Tests
Each SF periodically sends non-blocking UDP test packets to each of other active and standby SFs, and keeps track of the responses
to calculate latency and packet loss. Test packets are sent once per second. Both jumbo and non-jumbo test packets are sent
alternately. A non-jumbo UDP test packet has a payload size of 200 bytes, and a jumbo test packet has a payload size of 4000
bytes. These statistics are recorded:
-
Dropped packet counts—On receiving a test packet from another SF, the receiving SF sends back a reply. If an SF does not receive
a test packet reply within one second, it marks the packet as dropped.
-
Dropped jumbo packets—Same calculation as dropped packet counts, but only counts jumbo test packets.
-
Number of packets with long latency—If the SF receives a test packet reply after 200 milliseconds, it marks the packet as
having long latency.
Note |
Counters are cleared after an SF reboot.
|
The reporting interval starts at 15 seconds and can range to 3600 seconds. If there is no error detected during an interval,
no warning log is generated and the reporting interval doubles until the interval is 3600 seconds. When an error is detected
during an interval, a warning log is generated and the reporting interval is reduced in half until there are no more packets
dropped.
If there are any packets lost or long latency counts, a WARNING event is generated. An example warning is shown here:
2016-Jan-10+22:00:01.477 [hat 3081 warning] [5/0/5146 <hatcpu:50> hatcpu.c:1307] [software
internal system syslog] Over the past 15 seconds, tests from card 5 to 4 had 1 total drops,
0 jumbo drops, 0 long latency.
Use the command show heartbeat stats card
cardnumber
cpu
cpunumber to view the statistics collected regarding inter-SF communications.
DI network monitoring is enabled by default. Use the command debug heartbeat test to stop and start SF packet tests on specific SFs, or to clear test packet counters on a specific SF.
You can also use the command show cloud monitor di-network to display the DI network monitoring statistics. Sample output from the show cloud monitor di-network summary command is shown here for Card number 3:
Card 3 Test Results:
ToCard Health 5MinLoss 60MinLoss
1 Good 0.0% 0.0%
2 Good 0.0% 0.0%
4 Bad 6.32% 5.36%
5 Good 0.0% 0.0%
6 Good 0.0% 0.0%
The display shows the test packet loss rate for the past five minutes and past 60 minutes. If the rate is larger than 1%,
the health status is marked as "Bad".
SF to Standby CF DI Network Tests
During an SF boot
up, each SF sends both non-jumbo and jumbo ping packets to the standby CF to
ensure that the standby CF is reachable.
During SF normal operation, the SF periodically sends non-blocking UDP test packets to the standby CF, and keeps track of
the responses to calculate latency and packet loss. This functionality is the same as described for the Inter-SF DI Network Tests.
SF Secondary IP Address DI Network Tests
During an SF boot up, each SF sends both non-jumbo and jumbo ping packets to the active CF using the SF primary IP address.
In addition, each SF also sends non-jumbo ping packets to active CF using each of its secondary IP addresses. If any of these
pings fails, the SF notifies the active CF and the SF reboots.
Standby CF to Active CF DI Network Tests
During the standby
CF boot up, the standby CF sends both non-jumbo and jumbo ping packets to the
active CF.
DI-Network Bulk Statistics
The mon-di-net schema provides the following bulk statistics for monitoring the health of the DI-network on a VPC-DI platform. This information
is similar to that provided in the output of the show cloud monitor di-network summary Exec mode command.
-
src-card – Source card slot number on which monitoring has been performed.
-
dest-card – Destination card slot number to which traffic was routed.
-
total-pkts-5mins – Total number of packets sent over the past 5 minutes.
-
total-drops-5mins – Total number of packets that were dropped over the past 5 minutes.
-
total-pkts-60mins – Total number of packets sent over the past 60 minutes.
-
total-drops-60mins – Total number of packets that were dropped over the past 60 minutes.
-
total-pkts – Total number of all packets sent.
-
total-pkts-jumbo – Total number of jumbo packets sent.
-
total-drops – Total number of jumbo and non-jumbo test packets that were dropped.
-
total-drops-jumbo – Number of jumbo test packets that were dropped.
-
latency-warnings – Total number of times the latency has exceeded the threshold.
-
long-rtt – Longest Round Trip Time (RTT) in milliseconds.
-
average-rtt – Average Round Trip Time (RTT) in milliseconds.
The mon-di-net BulkStats Mode command configures the collection of statistics for the Mon-DI-Net schema.
See the Bulk Statistics chapter for information about configuring bulk statistic collection.
DI-Network Heartbeat Thresholds
This feature adds the capability to define thresholds for the internal DI-network for percentage heartbeat loss in order to
monitor the card-to-card network health in a VPC-DI deployment.
When heartbeat loss (on any of the cards) crosses a set limit of threshold, this feature raises alarms/SNMP trap to indicate
the loss.
The internal High Availability Task (HAT) tracks the percentage heartbeat loss over the past 5 minutes and past 60 minutes
between cards and can generate SNMP alarms if a threshold has been crossed or a previous alarm has been cleared.
There can be multiple cards in the system and any card can raise this same trap ID but with different card information.
The scope of this funtionality is across the system. It is not specific to any service and is configured at the Global Configuration
mode.
See Configure DI-Network Heartbeat Thresholds for instructions to enable this feature.
Feedback