Information About NHRP
How NHRP and NBMA Networks Interact
Most WAN networks are a collection of point-to-point links. Virtual tunnel networks (for example Generic Routing Encapsulation (GRE) tunnels) are also a collection of point-to-point links. To effectively scale the connectivity of these point-to-point links, they are usually grouped into a single or multilayer hub-and-spoke network. Multipoint interfaces (for example, GRE tunnel interfaces) can be used to reduce the configuration on a hub router in such a network. This resulting network is a Non-Broadcast Multi-Access (NBMA) network.
Because there are multiple tunnel endpoints reachable through the single multipoint interface, there needs to be a mapping from the logical tunnel endpoint IP address to the physical tunnel endpoint IP address in order to forward packets out the multipoint GRE (mGRE) tunnel interfaces over this NBMA network. This mapping could be statically configured, but it is preferable if the mapping can be discovered or learned dynamically.
NHRP is an ARP-like protocol that alleviates these NBMA network problems. With NHRP, systems attached to an NBMA network dynamically learn the NBMA address of the other systems that are part of that network, allowing these systems to directly communicate without requiring traffic to use an intermediate hop.
Routers, access servers, and hosts can use NHRP to discover the addresses of other routers and hosts connected to an NBMA network. Partially meshed NBMA networks typically have multiple logical networks behind the NBMA network. In such configurations, packets traversing the NBMA network might have to make several hops over the NBMA network before arriving at the exit router (the router nearest the destination network). When NHRP is combined with IPsec, the NBMA network is basically a collection of point-to-point logical tunnel links over a physical IP network.
NHRP allows two functions to help support these NBMA networks:
-
NHRP Registration. NHRP allows Next Hop Clients (NHCs) to dynamically register with Next Hop Servers (NHSs). This registration function allows the NHCs to join the NBMA network without configuration changes on the NHSs, especially in cases where the NHC has a dynamic physical IP address or is behind a Network Address Translation (NAT) router that dynamically changes the physical IP address. In these cases, it would be impossible to preconfigure the logical virtual private network (VPN IP) to physical (NBMA IP) mapping for the NHC on the NHS. See the NHRP_Registration section for more information.
-
NHRP Resolution. NHRP allows one NHC (spoke) to dynamically discover the logical VPN IP to physical NBMA IP mapping for another NHC (spoke) within the same NBMA network. Without this discovery, IP packets traversing from hosts behind one spoke to hosts behind another spoke would have to traverse by way of the NHS (hub) router. This process would increase the utilization of the hub’s physical bandwidth and CPU to process these packets that enter and exit the hub on the multipoint interface. With NHRP, systems attached to an NBMA network dynamically learn the NBMA address of the other systems that are part of that network, allowing these systems to directly communicate without requiring traffic to use an intermediate hop. This function alleviates the load on the intermediate hop (NHS) and can increase the overall bandwidth of the NBMA network to be greater than the bandwidth of the hub router.
Dynamically Built Hub-and-Spoke Networks
With NHRP, the NBMA network is initially laid out as a hub-and-spoke network that can be multiple hierarchical layers of NHCs as spokes and NHSs as hubs. The NHCs are configured with static mapping information to reach their NHSs and will connect to their NHS and send an NHRP registration to the NHS. This configuration allows the NHS to dynamically learn the mapping information for the spoke, reducing the configuration needed on the hub and allowing the spoke to obtain a dynamic NBMA (physical) IP address.
Once the base hub-and-spoke network is dynamically built, NHRP resolution requests and responses can be used to dynamically discover spoke-to-spoke mapping information, which allows spokes to bypass the hub and contact each other directly. This process allows a dynamic mesh of connections between spokes to be built based on data traffic patterns without requiring a preconfigured static fully meshed network. Using a dynamic-mesh network allows smaller spoke routers to participate up to their capability in a large NBMA network when these smaller spoke routers do not have the resources to participate in a full mesh on the same size network. The smaller spoke routers do not need to build out all possible spoke-to-spoke links; these routers need to build only the ones they are currently using.
Next Hop Server Selection
NHRP resolution requests traverse one or more hops (hubs) within the base hub-and-spoke NBMA subnetwork before reaching the station that is expected to generate a response. Each station (including the source station) chooses a neighboring NHS to which it forwards the request. The NHS selection procedure typically involves performing a routing decision based upon the network layer destination address of the NHRP request. The NHRP resolution request eventually arrives at a station that generates an NHRP resolution reply. This responding station either serves the destination, or is the destination itself. The responding station generates a reply using the source address from within the NHRP packet to determine where the reply should be sent.
The Cisco implementation of NHRP also supports and extends the IETF RFC 2332, NBMA Next Hop Resolution Protocol (NHRP) .
The figure below illustrates four routers connected to an NBMA network. Within the network are IP routers necessary for the routers to communicate with each other by tunneling the IP data packets in GRE IP tunnel packets. The infrastructure layer routers support logical IP tunnel circuit connections represented by hops 1, 2, and 3. When router A attempts to forward an IP packet from the source host to the destination host, NHRP is triggered. On behalf of the source host, router A sends an NHRP resolution request packet encapsulated in a GRE IP packet, which takes three hops across the network to reach router D, connected to the destination host. After router A receives a positive NHRP resolution reply, router A determines that router D is the NBMA IP next hop, and router A sends subsequent data IP packets for the destination to router D in one GRE IP tunnel hop.
With NHRP, once the NBMA next hop is determined, the source either starts sending data packets to the destination (in a connectionless NBMA network such as GRE IP or SMDS) or establishes a virtual circuit (VC) connection to the destination. This connection is configured with the desired bandwidth and quality of service (QoS) characteristics for a connection-oriented NBMA network (such as Frame Relay or ATM) or with Dynamic Multipoint VPN (DMVPN) where an IPsec encryption peering must be established.
Other address resolution methods can be used while NHRP is deployed. IP hosts that rely upon the Logical IP Subnet (LIS) model might require ARP servers and services over the NBMA network, and deployed hosts might not implement NHRP, but might continue to support ARP variations. NHRP is designed to eliminate the suboptimal routing that results from the LIS model, and can be deployed with existing ARP services without interfering with them.
NHRP Registration
NHRP registrations are sent from NHCs to their configured NHSs every one-third of the NHRP holdtime (configured by the ip nhrp holdtime value command ), unless the ip nhrp registration timeout value command is configured, in which case registrations are sent out according to the configured timeout value. If an NHRP registration reply is not received for an NHRP registration request, the NHRP registration request is retransmitted at timeouts of 1, 2, 4, 8, 16, and 32 seconds, then the sequence starts over again at 1.
The NHS is declared down if an NHRP registration reply is not received after three retransmission (7 seconds), and an NHRP resolution packets will no longer be sent to or by way of that NHS. NHRP registrations will continue to be sent at 1-, 2-, 4-, 8-, 16-, and 32-second intervals, probing the NHS until an NHRP registration reply is received. As soon as an NHRP registration reply is received the NHS is immediately declared up, the NHRP registration requests revert to being sent every one-third of NHRP holdtime or the value configured in the ip nhrp registration timeout command, and the NHS can again be sent NHRP resolution requests. The show ip nhrp nhs detail command can be used to check the state of the NHRP NHSs.
NHRP Used with a DMVPN
NHRP can be used to help build a VPN. In this context, a VPN consists of a virtual Layer 3 network that is built on top of an actual Layer 3 network. The topology you use over the VPN is largely independent of the underlying network, and the protocols you run over it are completely independent of it. The Dynamic Multipoint VPN (DMVPN) is based on GRE IP logical tunnels that can be protected by adding in IPsec to encrypt the GRE IP tunnels.
Dynamic Spoke-to-Spoke Tunnels
Spoke-to-spoke tunnels are designed to be dynamic, in that they are created only when there is data traffic to use the tunnel and they are removed when there is no longer any data traffic using the tunnel.
In addition to NHRP registration of NHCs with NHSs, NHRP provides the capability for NHCs (spokes) to find a shortcut path over the infrastructure of the network (IP network, SMDS) or build a shortcut switched virtual circuit (SVC) over a switched infrastructure network (Frame Relay and ATM) directly to another NHC (spoke), bypassing hops through the NHSs (hubs). This capability allows the building of very large NHRP NBMA networks. In this way, the bandwidth and CPU limitations of the hub do not limit the overall bandwidth of the NHRP NBMA network. This capability effectively creates a full-mesh-capable network without having to discover all possible connections beforehand. This type of network is called a dynamic-mesh network, where there is a base hub-and-spoke network of NHCs and NHSs for transporting NHRP and dynamic routing protocol information (and data traffic) and dynamic direct spoke-to-spoke links that are built when there is data traffic to use the link and torn down when the data traffic stops.
The dynamic-mesh network allows individual spoke routers to directly connect to anywhere in the NBMA network, even though they are capable of connecting only to a limited number at the same time. This functionality allows each spoke in the network to participate in the whole network up to its capabilities without limiting another spoke from participating up to its capability. If a full-mesh network were to be built, then all spokes would have to be sized to handle all possible tunnels at the same time.
For example, in a network of 1000 nodes, a full-mesh spoke would need to be large and powerful because it must always support 999 tunnels (one to every other node). In a dynamic-mesh network, a spoke needs to support only a limited number of tunnels to its NHSs (hubs) plus any currently active tunnels to other spokes. Also, if a spoke cannot build more spoke-to-spoke tunnels, then it will send its data traffic by way of the spoke-hub-spoke path. This design ensures that connectivity is always preserved, even when the preferred single hop path is not available.
Developmental Phases of DMVPN and NHRP
The developmental phases described in this section are actually DMVPN phases combining mGRE plus NHRP and IPsec. Phase 2 is important because it provides the functionality needed to support dynamic spoke-to-spoke tunnels.
-
Phase 1 is the hub-and-spoke capability only. This phase will not be discussed here because phase 1 does not support spoke-to-spoke tunnels.
-
Phase 2 adds spoke-to-spoke capability.
NHRP gathers the information that it needs to build spoke-to-spoke tunnels by using NHRP resolution request and reply packets that are sent via the spoke-hub-spoke path through the NBMA network. NHRP also has to be triggered (or know when) to collect this information for building the spoke-to-spoke tunnels, because it brings up the spoke-to-spoke tunnel only when there is data traffic to use it. The two ways that NHRP does this are described the following sections.
NHRP gathers the information that it needs to build spoke-to-spoke tunnels by using NHRP resolution request and reply packets that are sent via the spoke-hub-spoke path through the NBMA network. NHRP also has to be triggered (or know when) to collect this information for building the spoke-to-spoke tunnels, because it brings up the spoke-to-spoke tunnel only when there is data traffic to use it.
The IP routing table and the routes learned by way of the hub are important when building spoke-to-spoke tunnels. Therefore, the availability of the NHSs (hubs) is critical for the functioning of an NHRP-based network. When there is only one hub and that hub goes down, the spoke removes the routes that it learned from the hub from its routing table, because it lost the hub as its routing neighbor. However, the spoke does not delete any of the spoke-to-spoke tunnels (NHRP mappings) that are now up. Even though the spoke-to-spoke tunnel is still there the spoke will not be able to use the tunnel because its routing table no longer has a route to the destination network. The spoke has a path (spoke-to-spoke tunnel), but does not know to use it (because there is no routing table entry).
In addition, when the routing entries are removed there is no trigger into NHRP for NHRP to remove NHRP mapping entries. Eventually NHRP will time out the current dynamic NHRP mapping entries that it had when the hub went down because they are not being used. Only at that time does NHRP remove the mapping entry.
In phase 2, if there still happened to be a route in the routing table (could be a static route) with the correct IP next hop, then the spoke could still use the spoke-to-spoke tunnel even when the hub is down. NHRP will not be able to refresh the mapping entry because the NHRP resolution request or response would need to go through the hub.
If you have two (or more) NHS hubs within a single NBMA network (single mGRE, Frame Relay, or ATM interface), then when the first (primary) hub goes down, the spoke router will still remove the routes from the routing table that it learned from this hub, but it will also be learning the same routes (higher metric) from the second (backup) hub, so it will immediately install these routes. Therefore the spoke-to-spoke traffic would continue going over the spoke-to-spoke tunnel and be unaffected by the primary hub outage.
In phase 2, NHRP brings up the NHC-to-NHS tunnel and a dynamic routing protocol is used to distribute routing information about all of the networks that are available behind the hub and all of the other spokes. Included in this information is the IP next hop of the destination spoke that is supporting a particular destination network.
When a data packet is forwarded, it obtains the outbound interface and the IP next hop from the matching routing table network entry. If the NHRP interface is the outbound interface, it looks for an NHRP mapping entry for that IP next hop. If there is no matching of an NHRP mapping entry, then NHRP is triggered to send an NHRP resolution request to get the mapping information (IP next-hop address to physical layer address). The NHRP registration reply packet contains this mapping information. When this information is received, the spoke has enough information to correctly encapsulate the data packet to go directly to the remote spoke, taking one hop across the infrastructure network. One of the disadvantages to this technique is that each spoke must have all of the individual routes in its routing table for all possible destination networks behind the hub and other spokes. Keeping this routing information distributed and up to date can put a significant load on the routing protocol running over the VPN.
Spoke Refresh Mechanism for Spoke-to-Spoke Tunnels
Spoke-to-spoke tunnels are designed to be dynamic, in that they are created only when there is data traffic to use the tunnel and they are removed when there is no longer any data traffic using the tunnel. This section describes the mechanism to refresh the spoke-to-spoke tunnel when it is still being used (no packet loss) and to detect and remove the spoke-to-spoke tunnel when it is no longer being used.
Process Switching
Each time a data packet is switched using an NHRP mapping entry, the “used” flag is set on the mapping entry. Then when the NHRP background process runs (every 60 seconds) the following actions occur:
-
If the expire time is >135 seconds and the “used” flag is set, then the “used” flag is cleared.
-
If the expire time is <= 135 seconds and the “used” flag is set, then the entry is refreshed.
-
If the expire time is <= 135 seconds and the “used” flag is not set, then nothing is done.
CEF Switching
NHRP has no knowledge about when a packet is Cisco Express Forwarding (CEF) switched through the spoke-to-spoke tunnel.
When the NHRP background process runs, the following actions occur:
-
If the expire time is > 135 seconds, then nothing is done.
-
If the expire time is <= 135 seconds, then the corresponding CEF adjacency is marked “stale”. If the CEF adjacency is then used to switch a packet, CEF will mark the adjacency “fresh” and trigger NHRP to refresh the mapping entry.
In both the process and CEF switching cases, refreshed means that another NHRP resolution request is sent and response is needed to keep the entry from expiring. If the expiration time goes to 0 then the NHRP mapping entry is deleted. Also, if this entry is the last mapping entry with this NBMA address and if the router is CEF switching, then the CEF adjacency will be cleared and marked incomplete.
If the IPsec tunnel protection ipsec profile name command is used on an NHRP mGRE interface, then the following actions also occur:
-
The corresponding crypto socket entry is deleted.
-
The corresponding crypto map entry is deleted.
-
The corresponding IPsec security associations (SAs) and Internet Security Association and Key Management Protocol (ISAKMP) SAs are deleted.
-
Just prior to removing the ISAKMP SA, phase 2 and phase 1 delete notify messages are sent to the ISAKMP peer.
-
The ISAKMP peer deletes the corresponding IPsec SAs and ISAKMP SAs.
-
Via the crypto socket, the ISAKMP peer’s NHRP mapping entry sets its expire time set to 5 seconds, unless it is a static NHRP mapping entry.
-
When the NHRP mapping entry expires and if it is the last mapping entry with this NBMA address, then the ISAKMP peer also performs items 1 through 5.