OMP Path Selection Problem when Egress Path is Enforced on vEdge
Available Languages
Introduction
This document describes the problem that occurs with a design of the redundancy when Overlay Management Protocol (OMP) path selection is enforced on a vEdge device and not on the vSmart controller that causes unwanted results and loss of reachability to the remote site in case of link failure even if the backup path is available. This problem is also alternatively known as "vSmart doesn't take into account TLOC state on remote vEdge".
Topology
In order to understand the problem better, here is a simple topology diagram that depicts the setup:
Configuration
Here you can find the brief description of the configuration.
- Site DC1 has TLOC colors "private1" and "biz-internet"
- Site DC2 has TLOC colors "private1" and "biz-internet"
- Site HQ has TLOC color "private1" only
- In DC1 and DC2 both colors are used for control connection to vSmart
Both DC sites (DC1 and DC2) advertise the same network, 198.51.100.0/24.
In each site, vEdge learns the router from the DC via some kind of dynamic routing protocol, e.g. Border Gateway Protocol (BGP).
Each DC site tags the prefix with a different metric:
At site DC1 vEdge set origin-metric 32
At site DC2 vEdge setorigin-metric 52
| hostname | site-id | system-ip |
| DC1 | 21 | 10.100.0.21 |
| DC2 | 41 | 10.100.0.41 |
| HQ | 100 | 10.100.0.100 |
| vSmart | 100 | 10.100.0.20 |
Problem
At the time of normal operation:
1. vSmart receives 198.51.100.0/24 from both DC1 and DC2.
vsmart1# show omp routes 198.51.100.0/24
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
U -> TLOC unresolved
PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------------------------------------
3 198.51.100.0/24 10.100.0.21 36 1003 C,R installed 10.100.0.21 biz-internet ipsec - <===== METRIC 32 (PREFERRED)
10.100.0.21 49 1003 C,R installed 10.100.0.21 private1 ipsec - <===== METRIC 32 (PREFERRED)
10.100.0.41 36 1003 R installed 10.100.0.41 biz-internet ipsec - <===== METRIC 52
10.100.0.41 49 1003 R installed 10.100.0.41 private1 ipsec - <===== METRIC 52
2. vSmart advertises to HQ the route with destination DC1 (via private1 and biz-internet) because it has the lowest origin-metric as per OMP route selection criteria.
vsmart1# show omp routes 198.51.100.0/24 vpn 3 detail
---------------------------------------------------
omp route entries for vpn 3 route 198.51.100.0/24
---------------------------------------------------
RECEIVED FROM: <================= RECEIVED FROM vEdge in DC1 in "biz-internet" color
peer 10.100.0.21
path-id 36
label 1003
status C,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 10.100.0.21
type installed
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 21
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
RECEIVED FROM: <================= RECEIVED FROM vEdge in DC1 in "private1" color
peer 10.100.0.21
path-id 49
label 1003
status C,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 10.100.0.21
type installed
tloc 10.100.0.21, private1, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 21
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
RECEIVED FROM: <================= RECEIVED FROM vEdge in DC2 in "biz-internet" color
peer 10.100.0.41
path-id 36
label 1003
status R
loss-reason origin-metric
lost-to-peer 10.100.0.21
lost-to-path-id 49
Attributes:
originator 10.100.0.41
type installed
tloc 10.100.0.41, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 41
preference not set
tag 1000030041
origin-proto eBGP
origin-metric 52
as-path "65001 65001 65001 65001 65001"
unknown-attr-len not set
RECEIVED FROM: <================= RECEIVED FROM vEdge in DC2 in "private1" color
peer 10.100.0.41
path-id 49
label 1003
status R
loss-reason tloc-id
lost-to-peer 10.100.0.41
lost-to-path-id 36
Attributes:
originator 10.100.0.41
type installed
tloc 10.100.0.41, private1, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 41
preference not set
tag 1000030041
origin-proto eBGP
origin-metric 52
as-path "65001 65001 65001 65001 65001"
unknown-attr-len not set
ADVERTISED TO: <================= WE ADVERTISE TO HQ vEdge ONLY BEST ROUTES WITH METRIC 32
peer 10.100.0.100
Attributes:
originator 10.100.0.21
label 1003
path-id 4410
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
site-id 21
overlay-id 1
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
Attributes:
originator 10.100.0.21
label 1003
path-id 4439
tloc 10.100.0.21, private1, ipsec
ultimate-tloc not set
domain-id not set
site-id 21
overlay-id 1
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
3. HQ vEdge flags the route with TLOC "biz-internet" as "Inv,U" because this vEdge does not have TLOC biz-internet.
4. HQ vEdge flags the route with TLOC "private1" as "C,I,R" and installs the route.
DC1 failure scenario:
1. In failure scenario, DC1 vEdge uplink in color "private1" fails (interface goes in down state) while "biz-internet" stays up.
2. vSmart receives 198.51.100.0/24 from DC1 (reachable only via biz-internet) and DC2 (biz-internet and private1).
3. vSmart advertises to HQ vEdge routes to DC1 (via biz-internet) because DC1 has the lowest metric.
vsmart1# show omp routes 198.51.100.0/24 detail
---------------------------------------------------
omp route entries for vpn 3 route 198.51.100.0/24
---------------------------------------------------
RECEIVED FROM:
peer 10.100.0.21
path-id 36
label 1003
status C,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 10.100.0.21
type installed
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 21
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
RECEIVED FROM:
peer 10.100.0.41
path-id 36
label 1003
status R
loss-reason origin-metric
lost-to-peer 10.100.0.21
lost-to-path-id 36
Attributes:
originator 10.100.0.41
type installed
tloc 10.100.0.41, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 41
preference not set
tag 1000030041
origin-proto eBGP
origin-metric 52
as-path "65001 65001 65001 65001 65001"
unknown-attr-len not set
RECEIVED FROM:
peer 10.100.0.41
path-id 49
label 1003
status R
loss-reason tloc-id
lost-to-peer 10.100.0.41
lost-to-path-id 36
Attributes:
originator 10.100.0.41
type installed
tloc 10.100.0.41, private1, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 41
preference not set
tag 1000030041
origin-proto eBGP
origin-metric 52
as-path "65001 65001 65001 65001 65001"
unknown-attr-len not set
ADVERTISED TO:
peer 10.100.0.31
Attributes:
originator 10.100.0.21
label 1003
path-id 5906
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
site-id 21
overlay-id 1
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
ADVERTISED TO:
peer 10.100.0.41
Attributes:
originator 10.100.0.21
label 1003
path-id 7689
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
site-id 21
overlay-id 1
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
ADVERTISED TO: <===== THIS IS WHAT WE ADVERTISE TO HQ SITE
peer 10.100.0.100
Attributes:
originator 10.100.0.21
label 1003
path-id 4410
tloc 10.100.0.21, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
site-id 21
overlay-id 1
preference not set
tag 1000030021
origin-proto eBGP
origin-metric 32
as-path "65001 65001 65001"
unknown-attr-len not set
4. HQ vEdge flags the route with TLOC "biz-internet" as "Inv,U" because this vEdge does not have TLOC biz-internet.
The result is that HQ vEdge cannot reach 198.51.100.0/24.
Solution
vSmart could have sent the routes towards DC2 (with less prefered higher metric) and in that case HQ vEdge would still reach the destination with the use of the "private1" TLOC via DC2, which is still up:
VEDGE-HQ-1# show bfd sessions site-id 41
SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.100.0.41 41 up private1 private1 192.168.11.1 192.168.41.1 12406 ipsec 7 1000 12:04:02:25 0
But there is no route via "private1" TLOC via DC2 on HQ vEdge installed because vSmart has already selected biz-internet route with lower metric as the best path. vSmart does not advertise OMP routes with different metrics by default, hence does not let receiving vEdge device decide which path to take (and take into account available TLOCs and its states). vSmart does not take into account the TLOC colors available on the remote device (HQ vEdge in our case) to which you advertise the route and does not take into account its state because there is no such mechanism to control this.
This is the OMP corner case that can be seen in similar topology with iBGP route reflector and peering on physical interfaces addresses.
The first solution option is to use add path like functionality (RFC7911) available in OMP and called "send-backup-paths" on vSmart:
omp send-backup-paths
It advertises all available paths, so remote HQ vEdge chooses the path based on TLOC availability.
The second solution option here is to remove route-policy action "set metric" for the corresponding prefix on DC1 and DC2 vEdges and then perform centralized route selection enforcement via vSmart control-policy as shown here for example:
policy
lists
site-list site_11
site-id 11
!
prefix-list PREFIX
ip-prefix 198.51.100.0/24
!
control-policy SET_PREF
sequence 10
match route
prefix-list PREFIX
site-id 21
!
action accept
set
preference 200
!
!
!
sequence 20
match route
prefix-list PREFIX
site-id 41
!
action accept
set
preference 100
!
!
!
default-action accept
!
apply-policy
site-list site_11
control-policy SET_PREF out
!
Here, site-id 11 is the HQ vEdge and prefix-list PREFIX contains prefixes which you want to be preferred over one TLOC color or another. Since both OMP routes are on HQ vEdge, once vEdge can not reach biz-internet anymore, it will install a route via private1 in the Routing Information Base (RIB) from it's OMP routes table.
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)