Overlapping Subnets on L3outs in Cisco ACI
Available Languages
Contents
Introduction
Cisco's Application Centric Infrastructure (ACI) facilitates for communication between internal Tenants and external routed networks, via L3outs(Layer 3 out). Such L3outs can also be configured to have one or more End Point Groups (EPGs). For ACI to know how to classify taffic coming in, as an L3out's EPG, explicit subnets need to be defined with certain flags enabled. This article aims to shed some light on the hardware implementation of L3out EPG's in the context of contract based policy application. We will specifically explore the flag 'external subnets for external EPGs' and the unexpected consequences of declaring overlapping prefixes as 'external' on separate EPGs.
Concept
The rule of thumb is: when deploying L3outs, separate EPGs in the same Virtual Routing and Forwarding (VRF) instance, should not have overlapping subnets marked as 'external subnet for external EPGs'. This also means that traffic sourced from a specific subnet should not come in through different EPGs. This may cause unexpected classification of traffic based on longest prefix match against subnets declared against unrelated EPGs. Let's look at a few scenarios to understand this in detail
Prerequisites
Basic understanding of ACI: L3outs, contracts and policy enforcement. Some useful terms are briefly explained below, more detailed information on these is beyong the scope of this document:
pcTag: ACI classifies traffic into pcTags and these are internal representations of EPGs. These values, by default have a scope of VRF - i.e, they are unique within a VRF, but may be reused across VRFs. However, if one EPG has a contract with another EPG in a differenet VRF / Tenant , then the pcTag value has a global scope - i.e, you will not find any other EPG in ACI with the same pcTag.
ELAM: Embedded Logic Analyser Module. This tool is used to capture one packet on ASIC based on filters and to check the headers/flags set on the packet. This tool also helps understand lookups / logic made by hardware based
sclass/dclass: when traffic comes in to a leaf, based on direction of policy enforcement and locally available prefix knowledge the leaf will mark source and destination traffic into EPGs - in ELAM captures this will be seen as sclass and dclass respectively
zoning-rule: These are internal representations of contracts and are similar to lines of an ACL. The SrcEpg and DstEpg values should match with sclass/dclass for traffic to hit a given rule and be allowed. By default in an enforced vrf there is an implicit deny as the last line , so any traffic not matching against a certain rule will hit the implicit deny and be dropped.
Setup and Topology
Two leafs - 101 and 102 , model: N9K-C93180YC-EX
- Version 3.2(4e)
- One VRF used -
- Policy Enforcement Preference : Enforced
- Policy Enforcement Direction: Ingress.
- VRF VNID(VxLAN Network Identifier): 2752513 ; pcTag: 32770
- L3out in Leaf1 (101) -
- Protocol: Open Shortest Path First (OSPF)
- L3 interface user for neighborship- eth1/22 (10.27.48.1/24)
- external EPG pcTag: 16387
- Application EPG on Leaf101
- Trunk - eth1/24
- pcTag: 49153
- IP End Point: 172.16.1.17
- Gateway: 172.16.1.254/24 - deployed on Bridge Domain (BD)
- BD has pcTag 32771
- L3out on Leaf2 (202) -
- Protocol: Enhanced Interior Gateway Routing Protocol (EIGRP)
- SVI used for neighborship with Path 1/16 - vlan 2747 (10.27.47.1/24)
- external EPG pcTag: 163869
Scenarios
Traffic sourced from overlapping subnets
In this scenario we look at potential mis-classification when traffic is sourced from overlapping subnets (from ACI's perspective)
OSPF advertises:
10.9.9.6/32
EIGRP advertises:
10.9.9.1/32
We start with the topology in Diagram 1, but without any contracts. For EPG on OSPF we define subnet 0.0.0.0/0 as 'external subnet for external EPGs' and 10.9.9.0/24 with the same flag for EIGRP's EPG. Here's what the tables on Leaf1 and 2 look like:
Leaf1:
leaf101# show end int eth1/24
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged L - local
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
48 vlan-2743 dcce.c15b.1e47 L eth1/24
shparanj:eigrp-test vlan-2743 172.16.1.17 L eth1/24
leaf101# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.9.9.1/32, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/128576], 05:31:49, bgp-65003, internal, tag 65003
10.9.9.6/32, ubest/mbest: 1/0
*via 10.27.48.2, eth1/22, [110/5], 05:09:51, ospf-default, intra
10.27.47.0/24, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/0], 05:31:49, bgp-65003, internal, tag 65003
10.27.48.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.48.1, eth1/22, [1/0], 05:31:46, direct
10.27.48.1/32, ubest/mbest: 1/0, attached
*via 10.27.48.1, eth1/22, [1/0], 05:31:46, local, local
172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.240.34%overlay-1, [1/0], 05:27:43, static
172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive
*via 172.16.1.254, vlan47, [1/0], 05:31:52, local, local
leaf101# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4173 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4174 0 0 implarp enabled 2752513 permit any_any_filter(17)
4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
4207 0 32771 implicit enabled 2752513 permit any_dest_any(16)
<<vsh>> (to go into vsh propmt , type: #vsh )
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Leaf2:
leaf102# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.9.9.1/32, ubest/mbest: 1/0
*via 10.27.47.10, vlan78, [90/128576], 06:13:41, eigrp-default, internal
10.9.9.6/32, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/5], 05:20:27, bgp-65003, internal, tag 65003
10.27.47.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.47.2, vlan78, [1/0], 3d21h, direct
10.27.47.2/32, ubest/mbest: 1/0, attached
*via 10.27.47.2, vlan78, [1/0], 3d21h, local, local
10.27.48.0/24, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/0], 05:35:06, bgp-65003, internal, tag 65003
leaf102# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4472 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4471 0 0 implarp enabled 2752513 permit any_any_filter(17)
4470 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
<<vsh>>
leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 37 0x80000025 Up shparanj:eigrp-test ::/0 15 False True False
2752513 37 0x25 Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
2752513 37 0x25 Up shparanj:eigrp-test 10.9.9.0/24 16386 False True False
Let's add contract B (contract in tenant , scope vrf - filer: common:default)
As soon as we add contract B - we see the eigrp EPG prefix added on leaf1:
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test 2752513 26 0x1a Up shparanj:eigrp-test 10.9.9.0/24 16386 False True False 2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False 2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Lets look at other polices:
Leaf 1 contracts:
leaf101# show zoning-rule scope 2752513 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4173 0 0 implicit enabled 2752513 deny,log any_any_any(21) 4174 0 0 implarp enabled 2752513 permit any_any_filter(17) 4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22) 4207 0 32771 implicit enabled 2752513 permit any_dest_any(16) 4604 49153 16386 default enabled 2752513 permit src_dst_any(9) 4605 16386 49153 default enabled 2752513 permit src_dst_any(9)
Leaf 2 contracts (remain unchanged):
leaf102# show zoning-rule scope 2752513 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4472 0 0 implicit enabled 2752513 deny,log any_any_any(21) 4471 0 0 implarp enabled 2752513 permit any_any_filter(17) 4470 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
In this scenario traffic coming in from ospf l3out , which we expect to be tagged with 16387 gets tagged with 16386 instead. This is because traffic hits the new prefix entry on Leaf1.
Ping from 10.9.9.6 to end-point 172.16.1.17:
# ping 172.16.1.17 vrf shp-ospf source 10.9.9.6 count 1000 interval 1 PING 172.16.1.17 (172.16.1.17) from 10.9.9.6: 56 data bytes 64 bytes from 172.16.1.17: icmp_seq=0 ttl=253 time=2.207 ms 64 bytes from 172.16.1.17: icmp_seq=1 ttl=253 time=1.443 ms 64 bytes from 172.16.1.17: icmp_seq=2 ttl=253 time=1.312 ms
Ping works even without a contract between ospf epg and app-epg. This is because it hits against policy for eigrp-epg and gets allowed.
ELAM:
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 10.9.9.6
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
module-1(DBG-elam-insel6)# report | grep sclass
sug_lurw_vec.info.nsh_special.sclass: 0x4002
sug_lurw_vec.info.ifabric_spine.sclass: 0x4002
sug_lurw_vec.info.ifabric_leaf.sclass: 0x4002
#dec 0x4002
16386
In this scenario the traffic ends up working due to classification into a pcTag that has a contract with the intended destination. However, if, for example, the compute leaf was a separate 3rd leaf, then our traffic would fail - as the entry for contract would only exist on the third leaf (ingress policy) or on leaf102 (egress policy).
Fabric with overlapping subnets declared as external on separate external EPGs
In this scenario we look at policy conflict and potential mis-classification due to overlapping or same subnets declared as external on different external EPGs.
OSPF advertises network:
10.9.1.0/24
EIGRP advertises network:
10.9.2.0/24
We start with the topology in Diagram 1, but without any contracts. We define subnet 10.9.0.0/16 as 'external subnet for external EPGs' for EPG on both L3outs.
Here's what the tables on Leaf1 and 2 look like:
Leaf 1:
leaf101# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.9.1.0/24, ubest/mbest: 1/0
*via 10.27.48.2, eth1/22, [110/5], 00:01:50, ospf-default, intra
10.9.2.0/24, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/128576], 00:00:32, bgp-65003, internal, tag 65003
10.27.47.0/24, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/0], 01:54:45, bgp-65003, internal, tag 65003
10.27.48.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.48.1, eth1/22, [1/0], 1d09h, direct
10.27.48.1/32, ubest/mbest: 1/0, attached
*via 10.27.48.1, eth1/22, [1/0], 1d09h, local, local
172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.240.34%overlay-1, [1/0], 1d09h, static
172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive
*via 172.16.1.254, vlan47, [1/0], 1d09h, local, local
leaf101# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4173 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4174 0 0 implarp enabled 2752513 permit any_any_filter(17)
4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
4207 0 32771 implicit enabled 2752513 permit any_dest_any(16)
<<vsh>>
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 26 0x1a Up shparanj:eigrp-test 10.9.0.0/16 16387 False True False
2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Leaf2:
leaf102# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.9.1.0/24, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/5], 00:05:29, bgp-65003, internal, tag 65003
10.9.2.0/24, ubest/mbest: 1/0
*via 10.27.47.10, vlan80, [90/128576], 00:04:10, eigrp-default, internal
10.27.47.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.47.2, vlan80, [1/0], 01:58:24, direct
10.27.47.2/32, ubest/mbest: 1/0, attached
*via 10.27.47.2, vlan80, [1/0], 01:58:24, local, local
10.27.48.0/24, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/0], 1d09h, bgp-65003, internal, tag 65003
leaf102# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4472 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4471 0 0 implarp enabled 2752513 permit any_any_filter(17)
4470 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
<<vsh>>
leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 37 0x80000025 Up shparanj:eigrp-test ::/0 15 False True False
2752513 37 0x25 Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
2752513 37 0x25 Up shparanj:eigrp-test 10.9.0.0/16 16386 False True False
In this state, without any contracts, we see no faults on either EPGs. No overlap in prefixes is detected yet!
If we add Contract B, we see a fault in the app-EPG (which consumes Contract B).
Topology:
Let's look at the change in tables:
leaf101# show zoning-rule scope 2752513 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4173 0 0 implicit enabled 2752513 deny,log any_any_any(21) 4174 0 0 implarp enabled 2752513 permit any_any_filter(17) 4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22) 4207 0 32771 implicit enabled 2752513 permit any_dest_any(16) 4605 49153 16386 default enabled 2752513 permit src_dst_any(9) 4604 16386 49153 default enabled 2752513 permit src_dst_any(9) <<vsh>> leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test 2752513 26 0x1a Up shparanj:eigrp-test 10.9.0.0/16 16387 False True False 2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False 2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Leaf2 remains unchanged.
This shows us that the zoning-rule corresponding to Contract B is installed. However the prefix cannot be added, as it already exists - marked against the OSPF EPG!
And that is exactly what the fault warns us, "prefix entry already used in another EPG" - the fault is only raised when there is a conflict on a particular leaf between policy (zoning-rules) and its application. The fault is raised on the consumer EPG.
If we start traffic from 10.9.2.1 , it gets dropped on Leaf101 due to policy deny:
# show logging ip access-list internal packet-log deny
[ Tue Feb 19 19:31:33 2019 234270 usecs]: CName: shparanj:eigrp-test(VXLAN: 2752513), VlanType: FD_VLAN, Vlan-Id: 48, SMac: 0xdccec15b1e47, DMac:0x0022bdf819ff, SIP: 172.16.1.17, DIP: 10.9.2.1, SPort: 0, DPort: 0, Src Intf: Ethernet1/24, Proto: 1, PktLen: 98 [ Tue Feb 19 19:31:31 2019 234310 usecs]: CName: shparanj:eigrp-test(VXLAN: 2752513), VlanType: FD_VLAN, Vlan-Id: 48, SMac: 0xdccec15b1e47, DMac:0x0022bdf819ff, SIP: 172.16.1.17, DIP: 10.9.2.1, SPort: 0, DPort: 0, Src Intf: Ethernet1/24, Proto: 1, PktLen: 98
We see that replies from EP 172.16.1.17 to 10.9.2.1 are dropped. This is because:
- Requests from 10.9.2.1 coming in from fabric are already classified with sclass 16386 - these hit the Rule ID 4604 and are allowed through
- Replies from 172.16.1.17 get marked with dclass 16387 - this is picked up based on policy-mgr prefix rules. There is no rule corresponding to 16387 and these are denied.
In this situation misclassification causes traffic to be dropped even though we seem to have the right config in place (if fault is ignored).
Fabric with 0.0.0.0/0 prefix declared as external on multiple external EPGs
In this scenario we look at potential mis-classification and unexpected security violations due to the application of 0.0.0.0/0 subnet as external on different external EPGs.
OSPF advertises network:
10.7.7.0/24
EIGRP advertises network:
10.8.8.0/24
We start with the topology in Diagram 1, but without any contracts. We define subnet 0.0.0.0/0 as 'external subnet for external EPGs' for EPG on both L3outs.
Here's what the tables on Leaf1 and 2 look like:
Leaf1:
leaf101# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4173 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4174 0 0 implarp enabled 2752513 permit any_any_filter(17)
4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
4207 0 32771 implicit enabled 2752513 permit any_dest_any(16)
leaf101# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.7.7.0/24, ubest/mbest: 1/0
*via 10.27.48.2, eth1/22, [110/5], 00:23:29, ospf-default, intra
10.8.8.0/24, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/128576], 00:02:30, bgp-65003, internal, tag 65003
10.27.47.0/24, ubest/mbest: 1/0
*via 10.0.248.0%overlay-1, [200/0], 00:02:33, bgp-65003, internal, tag 65003
10.27.48.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.48.1, eth1/22, [1/0], 1d07h, direct
10.27.48.1/32, ubest/mbest: 1/0, attached
*via 10.27.48.1, eth1/22, [1/0], 1d07h, local, local
172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.240.34%overlay-1, [1/0], 1d07h, static
172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive
*via 172.16.1.254, vlan47, [1/0], 1d07h, local, local
<<vsh>>
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Leaf2:
leaf102# show ip route vrf shparanj:eigrp-test
IP Route Table for VRF "shparanj:eigrp-test"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.7.7.0/24, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/5], 00:26:07, bgp-65003, internal, tag 65003
10.8.8.0/24, ubest/mbest: 1/0
*via 10.27.47.10, vlan80, [90/128576], 00:05:08, eigrp-default, internal
10.27.47.0/24, ubest/mbest: 1/0, attached, direct
*via 10.27.47.2, vlan80, [1/0], 00:05:11, direct
10.27.47.2/32, ubest/mbest: 1/0, attached
*via 10.27.47.2, vlan80, [1/0], 00:05:11, local, local
10.27.48.0/24, ubest/mbest: 1/0
*via 10.0.0.64%overlay-1, [200/0], 1d07h, bgp-65003, internal, tag 65003
leaf102# show zoning-rule scope 2752513
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4472 0 0 implicit enabled 2752513 deny,log any_any_any(21)
4471 0 0 implarp enabled 2752513 permit any_any_filter(17)
4470 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22)
<<vsh>>
leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test
2752513 37 0x80000025 Up shparanj:eigrp-test ::/0 15 False True False
2752513 37 0x25 Up shparanj:eigrp-test 0.0.0.0/0 15 False True False
If we add both contracts A & B, we still don't see any faults.
Let's look at the tables on Leafs:
Leaf1:
leaf101# show zoning-rule scope 2752513 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4173 0 0 implicit enabled 2752513 deny,log any_any_any(21) 4174 0 0 implarp enabled 2752513 permit any_any_filter(17) 4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22) 4207 0 32771 implicit enabled 2752513 permit any_dest_any(16) 4616 49153 15 default enabled 2752513 permit src_dst_any(9) 4617 32770 49153 default enabled 2752513 permit src_dst_any(9)
<<vsh>>
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test 2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False 2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False
Tables on Leaf2 remain unchanged.
We don't see any faults as there is actually no policy conflict from each leaf's perspective. The Rule IDs added when using 0.0.0.0/0 as external EPG are special.
- Traffic coming in to either border leaf from its respective EPG is marked with sclass 32770 - this is the VRF's pcTag.
- dclass on this traffic is 49153 - the app-EPG's pcTag.
- Return traffic from app-EPG has dclass of 15
ELAM on Leaf1:
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 10.7.7.1
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
module-1(DBG-elam-insel6)# report | grep sclass
sug_lurw_vec.info.nsh_special.sclass: 0x8002
sug_lurw_vec.info.ifabric_spine.sclass: 0x8002
sug_lurw_vec.info.ifabric_leaf.sclass: 0x8002
module-1(DBG-elam-insel6)# dec 0x8002
32770
module-1(DBG-elam-insel6)# reset
module-1(DBG-elam-insel6)# set outer ipv4 dst_ip 10.7.7.1
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Armed
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
module-1(DBG-elam-insel6)# report | grep dclass
sug_lurw_vec.info.nsh_special.dclass: 0xF
sug_lurw_vec.info.ifabric_leaf.dclass: 0xF
Even if we remove Contract A, 10.7.7.1 can continue communication with 172.16.1.17.
This is because removal of Contract A does not result in any changes on the zoning-rules on Leaf1.
leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test 2752513 26 0x1a Up shparanj:eigrp-test 0.0.0.0/0 15 False True False 2752513 26 0x8000001a Up shparanj:eigrp-test ::/0 15 False True False leaf101# exit leaf101# show zoning-rule scope 2752513 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4173 0 0 implicit enabled 2752513 deny,log any_any_any(21) 4174 0 0 implarp enabled 2752513 permit any_any_filter(17) 4175 0 15 implicit enabled 2752513 deny,log any_vrf_any_deny(22) 4207 0 32771 implicit enabled 2752513 permit any_dest_any(16) 4616 49153 15 default enabled 2752513 permit src_dst_any(9) 4617 32770 49153 default enabled 2752513 permit src_dst_any(9)
Further, traffic coming in on OSPF external EPG continues to be tagged with VRF pcTag, as the EPG still has 0.0.0.0/0 marked as external subnet.
This leads to a breach in security policy, i.e, two EPGs able to communicate without a contract in an enforced VRF.
Further Reading
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_010010.html
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
28-Oct-2019 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)