Routing with NSX using multiple sites (with iBGP between the UDLR and the ESG’s)
Introduction
In one of my previous articles I demonstrated the routing between two Data Centers with using eBGP.
This article will be about doing the same, but now with iBGP between the DLR control VM and the ESG’s of both sites.
Diagram
The following diagram will be used for our setup.
LAB100_-_NSX_Routing_based_on_iBGP.pdf

Components
I am still using the same components as in this previous article.
Use–Cases
The use-case that I will be testing is the same use-case as in the previous article (only now with iBGP)
Tenant one
Tenant one will have workloads in DC1 and in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site. When the primary site is down the traffic should exit from the secondary site.
Routing protocol options
All possible routing options are described in the previous article.
The routing option in this article will be:
Option 2
- iBGP peering between the UDLR and the ESG’s
- eBGP peering between the ESG’s and the external routers
- eBGP peering between the external routers and the CORE
Option 2
Because of the many options available in the setup I want to start with the following: Tenant 1 workloads with the use of UDLR-01 and option 1 routing. The other options will be outlined in other articles.
I have only documented the changes below compared to the previous article. The “show” outputs are given fully again of all the routing tables and the route path verification as well.
UDLR configuration
DC1
The UDLR is in AS# 65530 and it will stay this way. We will change the ESG’s to be in this same AS.
- Change the peering AS towards the ESG

DC2
The UDLR control VM does not exist in DC2.
ESG configuration
DC1
- Change the Local AS
- Configure the BGP peers towards the external routers and the UDLR.
- Disable BGP
- Enable BGP

DC2
- Change the Local AS
- Configure the BGP peers towards the external routers and the UDLR.
- Disable BGP
- Enable BGP

Cisco1000V configuration
DC1
!hostname rt-a-01!no router bgp 65511router bgp 65511 bgp router-id 10.11.11.31 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.100.19.2 remote-as 65530 neighbor 10.100.19.2 description ESG-A ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.100.19.2 activate exit-address-family!ip route 10.200.19.0 255.255.255.0 10.11.11.253ip route 10.200.21.0 255.255.255.0 10.11.11.253!!hostname rt-a-02!no router bgp 65511router bgp 65511 bgp router-id 10.11.11.32 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.100.21.2 remote-as 65530 neighbor 10.100.21.2 description ESG-A ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.100.21.2 activate exit-address-family!ip route 10.200.19.0 255.255.255.0 10.11.11.253ip route 10.200.21.0 255.255.255.0 10.11.11.253!DC2
!hostname rt-b-01!no router bgp 65512router bgp 65512 bgp router-id 10.11.11.33 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.200.19.2 remote-as 65530 neighbor 10.200.19.2 description ESG-B ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.200.19.2 activate exit-address-family!ip route 10.100.19.0 255.255.255.0 10.11.11.253ip route 10.100.21.0 255.255.255.0 10.11.11.253!!hostname rt-b-02!no router bgp 65512router bgp 65512 bgp router-id 10.11.11.34 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.200.21.2 remote-as 65530 neighbor 10.200.21.2 description ESG-B ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.200.21.2 activate exit-address-family!ip route 10.100.19.0 255.255.255.0 10.11.11.253ip route 10.100.21.0 255.255.255.0 10.11.11.253!Core configuration
Between the Core and the Cisco 1000V CSR’s the AS did not change and we are still doing eBGP there.
UDLR route peering verification
DC1
REGX-UDLR01-0> show ip bgp neighbors
BGP neighbor is 172.39.39.1, remote AS 65530,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 10 messages, Sent 11 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 1 Identifier 0x57572d5c Route refresh request:received 0 sent 0 Prefixes received 6 sent 7 advertised 7Connections established 1, dropped 2Local host: 172.39.39.13, Local port: 179Remote host: 172.39.39.1, Remote port: 21815
BGP neighbor is 172.39.39.2, remote AS 65530,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 10 messages, Sent 10 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 2 Identifier 0x57572d5c Route refresh request:received 0 sent 0 Prefixes received 6 sent 7 advertised 7Connections established 1, dropped 2Local host: 172.39.39.13, Local port: 179Remote host: 172.39.39.2, Remote port: 41244DC2
The UDLR control VM does not exist in DC2.
ESG route peering verification
DC1
REGA-ESG01-0> show ip bgp neighbors
BGP neighbor is 10.100.19.1, remote AS 65511,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 16 messages, Sent 14 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 1 Identifier 0xc2a60eac Route refresh request:received 0 sent 0 Prefixes received 2 sent 10 advertised 10Connections established 1, dropped 1Local host: 10.100.19.2, Local port: 61962Remote host: 10.100.19.1, Remote port: 179
BGP neighbor is 10.100.21.1, remote AS 65511,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 17 messages, Sent 14 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 2 Identifier 0xc2a60eac Route refresh request:received 0 sent 0 Prefixes received 2 sent 10 advertised 10Connections established 1, dropped 1Local host: 10.100.21.2, Local port: 27456Remote host: 10.100.21.1, Remote port: 179
BGP neighbor is 172.39.39.13, remote AS 65530,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 15 messages, Sent 18 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 3 Identifier 0xc2a60eac Route refresh request:received 0 sent 0 Prefixes received 7 sent 6 advertised 6Connections established 2, dropped 2Local host: 172.39.39.1, Local port: 21815Remote host: 172.39.39.13, Remote port: 179DC2
REGB-ESG01-0> show ip bgp neighbors
BGP neighbor is 10.200.19.1, remote AS 65512,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 18 messages, Sent 17 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 1 Identifier 0x359d76ec Route refresh request:received 0 sent 0 Prefixes received 2 sent 10 advertised 10Connections established 1, dropped 1Local host: 10.200.19.2, Local port: 43262Remote host: 10.200.19.1, Remote port: 179
BGP neighbor is 10.200.21.1, remote AS 65512,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 18 messages, Sent 15 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 2 Identifier 0x359d76ec Route refresh request:received 0 sent 0 Prefixes received 2 sent 10 advertised 10Connections established 1, dropped 1Local host: 10.200.21.2, Local port: 19183Remote host: 10.200.21.1, Remote port: 179
BGP neighbor is 172.39.39.13, remote AS 65530,BGP state = Established, upHold time is 180, Keep alive interval is 60 secondsNeighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0Received 18 messages, Sent 20 messagesDefault minimum time between advertisement runs is 30 secondsFor Address family IPv4 Unicast:advertised and received Index 3 Identifier 0x359d76ec Route refresh request:received 0 sent 0 Prefixes received 7 sent 6 advertised 6Connections established 2, dropped 2Local host: 172.39.39.2, Local port: 41244Remote host: 172.39.39.13, Remote port: 179Cisco1000V route peering verification
DC1
rt-a-01#show ip bgp summaryBGP router identifier 10.11.11.31, local AS number 65511BGP table version is 54, main routing table version 5415 network entries using 3720 bytes of memory15 path entries using 1800 bytes of memory3/3 BGP path/bestpath attribute entries using 720 bytes of memory3 BGP AS-PATH entries using 88 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 6328 total bytes of memoryBGP activity 15/0 prefixes, 34/19 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.11.11.253 4 65510 22 24 54 0 0 00:13:57 510.100.19.2 4 65530 17 18 54 0 0 00:10:26 10rt-a-01#rt-a-02#show ip bgp summaryBGP router identifier 10.11.11.32, local AS number 65511BGP table version is 54, main routing table version 5415 network entries using 3720 bytes of memory15 path entries using 1800 bytes of memory3/3 BGP path/bestpath attribute entries using 720 bytes of memory3 BGP AS-PATH entries using 88 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 6328 total bytes of memoryBGP activity 15/0 prefixes, 34/19 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.11.11.253 4 65510 22 23 54 0 0 00:13:34 510.100.21.2 4 65530 17 20 54 0 0 00:11:03 10rt-a-02#DC2
rt-b-01#show ip bgp summaryrt-b-01#show ip bgp summaryBGP router identifier 10.11.11.33, local AS number 65512BGP table version is 48, main routing table version 4815 network entries using 3720 bytes of memory22 path entries using 2640 bytes of memory3/3 BGP path/bestpath attribute entries using 720 bytes of memory3 BGP AS-PATH entries using 88 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 7168 total bytes of memoryBGP activity 15/0 prefixes, 38/16 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.11.11.253 4 65510 22 24 48 0 0 00:13:39 1210.200.19.2 4 65530 19 20 48 0 0 00:12:11 10rt-b-01#rt-b-02#show ip bgp summaryBGP router identifier 10.11.11.34, local AS number 65512BGP table version is 28, main routing table version 2815 network entries using 3720 bytes of memory22 path entries using 2640 bytes of memory3/3 BGP path/bestpath attribute entries using 720 bytes of memory3 BGP AS-PATH entries using 88 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 7168 total bytes of memoryBGP activity 15/0 prefixes, 38/16 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.11.11.253 4 65510 21 21 28 0 0 00:13:29 1210.200.21.2 4 65530 19 21 28 0 0 00:12:42 10rt-b-02#Core route peering verification
cs-01#show ip bgp summary BGP router identifier 10.11.11.253, local AS number 65510BGP table version is 402, main routing table version 40215 network entries using 2040 bytes of memory42 path entries using 2184 bytes of memory3/3 BGP path/bestpath attribute entries using 372 bytes of memory2 BGP AS-PATH entries using 48 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 4644 total bytes of memoryBGP activity 63/48 prefixes, 644/602 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.11.11.31 4 65511 27 25 402 0 0 00:16:20 1010.11.11.32 4 65511 25 24 402 0 0 00:15:20 1010.11.11.33 4 65512 25 23 402 0 0 00:14:43 1010.11.11.34 4 65512 21 22 402 0 0 00:14:02 10cs-01#UDLR routing tables
DC1
REGX-UDLR01-0> show ip route
Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
Total number of routes: 15
B 10.11.11.0/24 [200/0] via 172.39.39.1B 10.22.22.0/24 [200/0] via 172.39.39.1B 10.100.19.0/24 [200/0] via 172.39.39.1B 10.100.21.0/24 [200/0] via 172.39.39.1B 10.200.19.0/24 [200/0] via 172.39.39.2B 10.200.21.0/24 [200/0] via 172.39.39.2C 172.20.1.0/24 [0/0] via 172.20.1.254C 172.20.2.0/24 [0/0] via 172.20.2.254C 172.20.3.0/24 [0/0] via 172.20.3.254C 172.20.8.0/24 [0/0] via 172.20.8.254C 172.20.9.0/24 [0/0] via 172.20.9.254C 172.20.10.0/24 [0/0] via 172.20.10.254C 172.39.39.0/28 [0/0] via 172.39.39.13B 172.39.39.16/28 [200/0] via 172.39.39.1B 172.39.39.32/28 [200/0] via 172.39.39.2DC2
The UDLR control VM does not exist in DC2.
ESG routing tables
DC1
REGA-ESG01-0> show ip route
Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
Total number of routes: 12
B 10.11.11.0/24 [20/0] via 10.100.19.1B 10.11.11.0/24 [20/0] via 10.100.21.1B 10.22.22.0/24 [20/0] via 10.100.19.1B 10.22.22.0/24 [20/0] via 10.100.21.1C 10.100.19.0/24 [0/0] via 10.100.19.2C 10.100.21.0/24 [0/0] via 10.100.21.2B 172.20.1.0/24 [200/0] via 172.39.39.14B 172.20.2.0/24 [200/0] via 172.39.39.14B 172.20.3.0/24 [200/0] via 172.39.39.14B 172.20.8.0/24 [200/0] via 172.39.39.14B 172.20.9.0/24 [200/0] via 172.39.39.14B 172.20.10.0/24 [200/0] via 172.39.39.14C 172.39.39.0/28 [0/0] via 172.39.39.1C 172.39.39.16/28 [0/0] via 172.39.39.17DC2
REGB-ESG01-0> show ip route
Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
Total number of routes: 12
B 10.11.11.0/24 [20/0] via 10.200.19.1B 10.11.11.0/24 [20/0] via 10.200.21.1B 10.22.22.0/24 [20/0] via 10.200.19.1B 10.22.22.0/24 [20/0] via 10.200.21.1C 10.200.19.0/24 [0/0] via 10.200.19.2C 10.200.21.0/24 [0/0] via 10.200.21.2B 172.20.1.0/24 [200/0] via 172.39.39.14B 172.20.2.0/24 [200/0] via 172.39.39.14B 172.20.3.0/24 [200/0] via 172.39.39.14B 172.20.8.0/24 [200/0] via 172.39.39.14B 172.20.9.0/24 [200/0] via 172.39.39.14B 172.20.10.0/24 [200/0] via 172.39.39.14C 172.39.39.0/28 [0/0] via 172.39.39.2C 172.39.39.32/28 [0/0] via 172.39.39.33REGB-ESG01-0>Cisco1000V routing tables
DC1
rt-a-01#show ip routeCodes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masksC 10.11.11.0/24 is directly connected, GigabitEthernet2L 10.11.11.31/32 is directly connected, GigabitEthernet2B 10.22.22.0/24 [20/0] via 10.11.11.253, 00:17:32C 10.100.1.0/24 is directly connected, GigabitEthernet1L 10.100.1.101/32 is directly connected, GigabitEthernet1C 10.100.19.0/24 is directly connected, GigabitEthernet3L 10.100.19.1/32 is directly connected, GigabitEthernet3B 10.100.21.0/24 [20/0] via 10.100.19.2, 00:14:45S 10.200.19.0/24 [1/0] via 10.11.11.253S 10.200.21.0/24 [1/0] via 10.11.11.253 172.20.0.0/24 is subnetted, 6 subnetsB 172.20.1.0 [20/0] via 10.100.19.2, 00:14:02B 172.20.2.0 [20/0] via 10.100.19.2, 00:14:02B 172.20.3.0 [20/0] via 10.100.19.2, 00:14:02B 172.20.8.0 [20/0] via 10.100.19.2, 00:14:02B 172.20.9.0 [20/0] via 10.100.19.2, 00:14:02B 172.20.10.0 [20/0] via 10.100.19.2, 00:14:02 172.39.0.0/28 is subnetted, 3 subnetsB 172.39.39.0 [20/0] via 10.100.19.2, 00:14:45B 172.39.39.16 [20/0] via 10.100.19.2, 00:14:45B 172.39.39.32 [20/0] via 10.11.11.253, 00:14:47rt-a-01#rt-a-02#show ip routeCodes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masksC 10.11.11.0/24 is directly connected, GigabitEthernet2L 10.11.11.32/32 is directly connected, GigabitEthernet2B 10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:46C 10.100.1.0/24 is directly connected, GigabitEthernet1L 10.100.1.102/32 is directly connected, GigabitEthernet1B 10.100.19.0/24 [20/0] via 10.100.21.2, 00:15:18C 10.100.21.0/24 is directly connected, GigabitEthernet3L 10.100.21.1/32 is directly connected, GigabitEthernet3S 10.200.19.0/24 [1/0] via 10.11.11.253S 10.200.21.0/24 [1/0] via 10.11.11.253 172.20.0.0/24 is subnetted, 6 subnetsB 172.20.1.0 [20/0] via 10.100.21.2, 00:14:35B 172.20.2.0 [20/0] via 10.100.21.2, 00:14:35B 172.20.3.0 [20/0] via 10.100.21.2, 00:14:35B 172.20.8.0 [20/0] via 10.100.21.2, 00:14:35B 172.20.9.0 [20/0] via 10.100.21.2, 00:14:35B 172.20.10.0 [20/0] via 10.100.21.2, 00:14:35 172.39.0.0/28 is subnetted, 3 subnetsB 172.39.39.0 [20/0] via 10.100.21.2, 00:15:18B 172.39.39.16 [20/0] via 10.100.21.2, 00:15:18B 172.39.39.32 [20/0] via 10.11.11.253, 00:15:21rt-a-02#DC2
rt-b-01#show ip routeCodes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masksC 10.11.11.0/24 is directly connected, GigabitEthernet2L 10.11.11.33/32 is directly connected, GigabitEthernet2B 10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:51S 10.100.19.0/24 [1/0] via 10.11.11.253S 10.100.21.0/24 [1/0] via 10.11.11.253C 10.200.1.0/24 is directly connected, GigabitEthernet1L 10.200.1.101/32 is directly connected, GigabitEthernet1C 10.200.19.0/24 is directly connected, GigabitEthernet3L 10.200.19.1/32 is directly connected, GigabitEthernet3B 10.200.21.0/24 [20/0] via 10.200.19.2, 00:16:14 172.20.0.0/24 is subnetted, 6 subnetsB 172.20.1.0 [20/0] via 10.200.19.2, 00:15:05B 172.20.2.0 [20/0] via 10.200.19.2, 00:15:05B 172.20.3.0 [20/0] via 10.200.19.2, 00:15:05B 172.20.8.0 [20/0] via 10.200.19.2, 00:15:05B 172.20.9.0 [20/0] via 10.200.19.2, 00:15:05B 172.20.10.0 [20/0] via 10.200.19.2, 00:15:05 172.39.0.0/28 is subnetted, 3 subnetsB 172.39.39.0 [20/0] via 10.200.19.2, 00:16:14B 172.39.39.16 [20/0] via 10.11.11.253, 00:16:51B 172.39.39.32 [20/0] via 10.200.19.2, 00:16:14rt-b-01#rt-b-02#show ip routeCodes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masksC 10.11.11.0/24 is directly connected, GigabitEthernet2L 10.11.11.34/32 is directly connected, GigabitEthernet2B 10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:25S 10.100.19.0/24 [1/0] via 10.11.11.253S 10.100.21.0/24 [1/0] via 10.11.11.253C 10.200.1.0/24 is directly connected, GigabitEthernet1L 10.200.1.102/32 is directly connected, GigabitEthernet1B 10.200.19.0/24 [20/0] via 10.200.21.2, 00:16:25C 10.200.21.0/24 is directly connected, GigabitEthernet3L 10.200.21.1/32 is directly connected, GigabitEthernet3 172.20.0.0/24 is subnetted, 6 subnetsB 172.20.1.0 [20/0] via 10.200.21.2, 00:15:40B 172.20.2.0 [20/0] via 10.200.21.2, 00:15:40B 172.20.3.0 [20/0] via 10.200.21.2, 00:15:40B 172.20.8.0 [20/0] via 10.200.21.2, 00:15:40B 172.20.9.0 [20/0] via 10.200.21.2, 00:15:40B 172.20.10.0 [20/0] via 10.200.21.2, 00:15:40 172.39.0.0/28 is subnetted, 3 subnetsB 172.39.39.0 [20/0] via 10.200.21.2, 00:16:25B 172.39.39.16 [20/0] via 10.11.11.253, 00:16:25B 172.39.39.32 [20/0] via 10.200.21.2, 00:16:25rt-b-02#Core routing table
cs-01#show ip route bgp Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP + - replicated route, % - next hop override
Gateway of last resort is 10.11.11.254 to network 0.0.0.0
172.20.0.0/24 is subnetted, 6 subnetsB 172.20.1.0 [20/0] via 10.11.11.31, 00:16:19B 172.20.2.0 [20/0] via 10.11.11.31, 00:16:19B 172.20.3.0 [20/0] via 10.11.11.31, 00:16:19B 172.20.8.0 [20/0] via 10.11.11.31, 00:16:19B 172.20.9.0 [20/0] via 10.11.11.31, 00:16:19B 172.20.10.0 [20/0] via 10.11.11.31, 00:16:19 172.39.0.0/28 is subnetted, 3 subnetsB 172.39.39.0 [20/0] via 10.11.11.32, 00:16:50B 172.39.39.16 [20/0] via 10.11.11.32, 00:16:50B 172.39.39.32 [20/0] via 10.11.11.34, 00:16:51cs-01#UDLR BGP tables
DC1
REGX-UDLR01-0> show ip bgp
Status codes: s - suppressed, d - damped, > - best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight AS Path > 10.11.11.0/24 10.100.19.1 0 100 200 65511 65510 i 10.11.11.0/24 10.200.19.1 0 100 30 65512 65510 i > 10.22.22.0/24 10.100.19.1 0 100 200 65511 65510 i 10.22.22.0/24 10.200.19.1 0 100 30 65512 65510 i > 10.100.19.0/24 172.39.39.1 0 100 200 ? > 10.100.21.0/24 172.39.39.1 0 100 200 ? > 10.200.19.0/24 172.39.39.2 0 100 30 ? > 10.200.21.0/24 172.39.39.2 0 100 30 ? > 172.20.1.0/24 0.0.0.0 0 100 32768 ? > 172.20.2.0/24 0.0.0.0 0 100 32768 ? > 172.20.3.0/24 0.0.0.0 0 100 32768 ? > 172.20.8.0/24 0.0.0.0 0 100 32768 ? > 172.20.9.0/24 0.0.0.0 0 100 32768 ? > 172.20.10.0/24 0.0.0.0 0 100 32768 ? 172.39.39.0/28 172.39.39.1 0 100 200 ? 172.39.39.0/28 172.39.39.2 0 100 30 ? > 172.39.39.0/28 0.0.0.0 0 100 32768 ? > 172.39.39.16/28 172.39.39.1 0 100 200 ? > 172.39.39.32/28 172.39.39.2 0 100 30 ?REGX-UDLR01-0>DC2
The UDLR control VM does not exist in DC2.
ESG BGP tables
DC1
REGA-ESG01-0> show ip bgp
Status codes: s - suppressed, d - damped, > - best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight AS Path > 10.11.11.0/24 10.100.19.1 0 100 60 65511 65510 i 10.11.11.0/24 10.100.21.1 0 100 60 65511 65510 i > 10.22.22.0/24 10.100.19.1 0 100 60 65511 65510 i 10.22.22.0/24 10.100.21.1 0 100 60 65511 65510 i > 10.100.19.0/24 0.0.0.0 0 100 32768 ? > 10.100.21.0/24 0.0.0.0 0 100 32768 ? > 172.20.1.0/24 172.39.39.14 0 100 60 ? > 172.20.2.0/24 172.39.39.14 0 100 60 ? > 172.20.3.0/24 172.39.39.14 0 100 60 ? > 172.20.8.0/24 172.39.39.14 0 100 60 ? > 172.20.9.0/24 172.39.39.14 0 100 60 ? > 172.20.10.0/24 172.39.39.14 0 100 60 ? 172.39.39.0/28 172.39.39.14 0 100 60 ? > 172.39.39.0/28 0.0.0.0 0 100 32768 ? > 172.39.39.16/28 0.0.0.0 0 100 32768 ?REGA-ESG01-0>DC2
REGB-ESG01-0> show ip bgp
Status codes: s - suppressed, d - damped, > - best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight AS Path > 10.11.11.0/24 10.200.19.1 0 100 60 65512 65510 i 10.11.11.0/24 10.200.21.1 0 100 60 65512 65510 i > 10.22.22.0/24 10.200.19.1 0 100 60 65512 65510 i 10.22.22.0/24 10.200.21.1 0 100 60 65512 65510 i > 10.200.19.0/24 0.0.0.0 0 100 32768 ? > 10.200.21.0/24 0.0.0.0 0 100 32768 ? > 172.20.1.0/24 172.39.39.14 0 100 60 ? > 172.20.2.0/24 172.39.39.14 0 100 60 ? > 172.20.3.0/24 172.39.39.14 0 100 60 ? > 172.20.8.0/24 172.39.39.14 0 100 60 ? > 172.20.9.0/24 172.39.39.14 0 100 60 ? > 172.20.10.0/24 172.39.39.14 0 100 60 ? 172.39.39.0/28 172.39.39.14 0 100 60 ? > 172.39.39.0/28 0.0.0.0 0 100 32768 ? > 172.39.39.32/28 0.0.0.0 0 100 32768 ?REGB-ESG01-0>Cisco1000V BGP tables
DC1
rt-a-01#show ip bgpBGP table version is 54, local router ID is 10.11.11.31Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i r> 10.100.19.0/24 10.100.19.2 0 65530 ? *> 10.100.21.0/24 10.100.19.2 0 65530 ? r> 10.200.19.0/24 10.11.11.253 0 65510 65512 65530 ? r> 10.200.21.0/24 10.11.11.253 0 65510 65512 65530 ? *> 172.20.1.0/24 10.100.19.2 0 65530 ? *> 172.20.2.0/24 10.100.19.2 0 65530 ? *> 172.20.3.0/24 10.100.19.2 0 65530 ? *> 172.20.8.0/24 10.100.19.2 0 65530 ? *> 172.20.9.0/24 10.100.19.2 0 65530 ? *> 172.20.10.0/24 10.100.19.2 0 65530 ? *> 172.39.39.0/28 10.100.19.2 0 65530 ? Network Next Hop Metric LocPrf Weight Path *> 172.39.39.16/28 10.100.19.2 0 65530 ? *> 172.39.39.32/28 10.11.11.253 0 65510 65512 65530 ?rt-a-01#rt-a-02#show ip bgpBGP table version is 54, local router ID is 10.11.11.32Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i *> 10.100.19.0/24 10.100.21.2 0 65530 ? r> 10.100.21.0/24 10.100.21.2 0 65530 ? r> 10.200.19.0/24 10.11.11.253 0 65510 65512 65530 ? r> 10.200.21.0/24 10.11.11.253 0 65510 65512 65530 ? *> 172.20.1.0/24 10.100.21.2 0 65530 ? *> 172.20.2.0/24 10.100.21.2 0 65530 ? *> 172.20.3.0/24 10.100.21.2 0 65530 ? *> 172.20.8.0/24 10.100.21.2 0 65530 ? *> 172.20.9.0/24 10.100.21.2 0 65530 ? *> 172.20.10.0/24 10.100.21.2 0 65530 ? *> 172.39.39.0/28 10.100.21.2 0 65530 ? Network Next Hop Metric LocPrf Weight Path *> 172.39.39.16/28 10.100.21.2 0 65530 ? *> 172.39.39.32/28 10.11.11.253 0 65510 65512 65530 ?rt-a-02#DC2
rt-b-01#show ip bgpBGP table version is 48, local router ID is 10.11.11.33Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i r> 10.100.19.0/24 10.11.11.253 0 65510 65511 65530 ? r> 10.100.21.0/24 10.11.11.253 0 65510 65511 65530 ? r> 10.200.19.0/24 10.200.19.2 0 65530 ? *> 10.200.21.0/24 10.200.19.2 0 65530 ? *> 172.20.1.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.2.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? Network Next Hop Metric LocPrf Weight Path *> 172.20.3.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.8.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.9.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.10.0/24 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.0/28 10.200.19.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.16/28 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.32/28 10.200.19.2 0 65530 ?rt-b-01#rt-b-02#show ip bgprt-b-02#show ip bgpBGP table version is 28, local router ID is 10.11.11.34Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i r> 10.100.19.0/24 10.11.11.253 0 65510 65511 65530 ? r> 10.100.21.0/24 10.11.11.253 0 65510 65511 65530 ? *> 10.200.19.0/24 10.200.21.2 0 65530 ? r> 10.200.21.0/24 10.200.21.2 0 65530 ? *> 172.20.1.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.2.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? Network Next Hop Metric LocPrf Weight Path *> 172.20.3.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.8.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.9.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.20.10.0/24 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.0/28 10.200.21.2 0 65530 ? * 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.16/28 10.11.11.253 0 65510 65511 65530 ? *> 172.39.39.32/28 10.200.21.2 0 65530 ?rt-b-02#Core BGP tables
cs-01#show ip bgpBGP table version is 402, local router ID is 10.11.11.253Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, x best-externalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path*> 10.11.11.0/24 0.0.0.0 0 32768 i*> 10.22.22.0/24 0.0.0.0 0 32768 ir 10.100.19.0/24 10.11.11.31 0 65511 65530 ?r> 10.11.11.32 0 65511 65530 ?r 10.100.21.0/24 10.11.11.31 0 65511 65530 ?r> 10.11.11.32 0 65511 65530 ?r 10.200.19.0/24 10.11.11.33 0 65512 65530 ?r> 10.11.11.34 0 65512 65530 ?r 10.200.21.0/24 10.11.11.33 0 65512 65530 ?r> 10.11.11.34 0 65512 65530 ?* 172.20.1.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.20.2.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.20.3.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.20.8.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.20.9.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.20.10.0/24 10.11.11.32 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.31 0 65511 65530 ?* 172.39.39.0/28 10.11.11.31 0 65511 65530 ?* 10.11.11.33 0 65512 65530 ?* 10.11.11.34 0 65512 65530 ?*> 10.11.11.32 0 65511 65530 ?* 172.39.39.16/28 10.11.11.31 0 65511 65530 ?*> 10.11.11.32 0 65511 65530 ?* 172.39.39.32/28 10.11.11.33 0 65512 65530 ?*> 10.11.11.34 0 65512 65530 ?cs-01#Routing path verifications
Routing path verification from the external client VM to T1–WEB–1 and T1–WEB–2
The expectation is that the traffic will route through DC1.
C:\Users\Administrator>tracert 172.20.1.1
Tracing route to 172.20.1.1 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 2 ms <1 ms <1 ms 172.20.1.1
Trace complete.C:\Users\Administrator>tracert 172.20.1.2
Tracing route to 172.20.1.2 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 6 ms <1 ms <1 ms 172.20.1.2
Trace complete.Routing path verification from the T1–WEB–1 and T1–WEB–2 to the external client VM
root@Web01:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.184 ms 0.060 ms 0.149 ms 2 172.39.39.1 (172.39.39.1) 0.454 ms 0.408 ms 0.438 ms 3 10.100.19.1 (10.100.19.1) 1.412 ms 1.450 ms 1.434 ms 4 10.11.11.50 (10.11.11.50) 1.454 ms * *root@Web01:~#root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.132 ms 0.105 ms 0.082 ms 2 172.39.39.1 (172.39.39.1) 0.922 ms 0.880 ms 0.771 ms 3 10.100.19.1 (10.100.19.1) 1.110 ms 1.716 ms 1.704 ms 4 10.11.11.50 (10.11.11.50) 2.738 ms * *We are seeing that the ingress AND egress paths that are taken is the path trough DC1. And this was expected…
Routing path verification from the external client VM to T1–WEB–3 and T1–WEB–4
The expectation is that the traffic will route through DC1.
C:\Users\Administrator>tracert 172.20.1.3
Tracing route to 172.20.1.3 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 3 ms 1 ms <1 ms 172.20.1.3
Trace complete.C:\Users\Administrator>tracert 172.20.1.4
Tracing route to 172.20.1.4 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 6 ms 1 ms <1 ms 172.20.1.4
Trace complete.Routing path verification from the T1–WEB–3 and T1–WEB–4 to the external client VM
The expectation is that the traffic will route through DC1.
root@Web03:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.133 ms 0.257 ms 0.137 ms 2 172.39.39.1 (172.39.39.1) 0.510 ms 0.460 ms 0.601 ms 3 10.100.21.1 (10.100.21.1) 1.298 ms 1.319 ms 1.266 ms 4 10.11.11.50 (10.11.11.50) 1.380 ms * *root@Web04:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.145 ms 0.101 ms 0.094 ms 2 172.39.39.1 (172.39.39.1) 0.557 ms 0.631 ms 0.818 ms 3 10.100.19.1 (10.100.19.1) 1.287 ms 1.290 ms 1.315 ms 4 10.11.11.50 (10.11.11.50) 1.393 ms * *The routing components in DC1 may fail or in case of a disaster the full site may go down. In that case, the routing should flow through DC2.
Let’s test this…
Turn RT-A-01 off. Verify paths again. The expectation is that the traffic will route through RT-A-02.
Routing path verification from the external client VM to T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4
C:\Users\Administrator>tracert 172.20.1.1
Tracing route to 172.20.1.1 over a maximum of 30 hops
1 3 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 3 ms <1 ms <1 ms 172.20.1.1
Trace complete.C:\Users\Administrator>tracert 172.20.1.2
Tracing route to 172.20.1.2 over a maximum of 30 hops
1 1 ms 1 ms 4 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms 1 ms <1 ms 172.39.39.14 5 3 ms <1 ms <1 ms 172.20.1.2
Trace complete.C:\Users\Administrator>tracert 172.20.1.3
Tracing route to 172.20.1.3 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 3 <1 ms <1 ms <1 ms 10.100.21.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 2 ms <1 ms <1 ms 172.20.1.3C:\Users\Administrator>tracert 172.20.1.4
Tracing route to 172.20.1.4 over a maximum of 30 hops
1 1 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 3 <1 ms <1 ms <1 ms 10.100.21.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 1 ms 1 ms <1 ms 172.20.1.4
Trace complete.Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root@Web01:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.273 ms 0.157 ms 0.323 ms 2 172.39.39.1 (172.39.39.1) 0.314 ms 0.248 ms 0.575 ms 3 10.100.21.1 (10.100.21.1) 0.608 ms 0.594 ms 0.550 ms 4 10.11.11.50 (10.11.11.50) 0.596 ms * *root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.189 ms 0.090 ms 0.097 ms 2 172.39.39.1 (172.39.39.1) 0.408 ms 0.361 ms 0.441 ms 3 10.100.21.1 (10.100.21.1) 0.668 ms 0.636 ms 0.774 ms 4 10.11.11.50 (10.11.11.50) 0.733 ms * *root@Web03:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.160 ms 0.144 ms 0.124 ms 2 172.39.39.1 (172.39.39.1) 0.512 ms 0.500 ms 0.478 ms 3 10.100.21.1 (10.100.21.1) 1.095 ms 1.115 ms 1.153 ms 4 10.11.11.50 (10.11.11.50) 1.907 ms * *root@Web04:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.155 ms 0.144 ms 0.109 ms 2 172.39.39.1 (172.39.39.1) 4.136 ms 4.021 ms 3.921 ms 3 10.100.21.1 (10.100.21.1) 6.588 ms 6.605 ms 6.631 ms 4 10.11.11.50 (10.11.11.50) 6.878 ms * *Turn RT-A-02 off. Verify paths again. The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
C:\Users\Administrator>tracert 172.20.1.1
Tracing route to 172.20.1.1 over a maximum of 30 hops
1 <1 ms 1 ms 2 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 75 ms 1 ms 1 ms 172.20.1.1
Trace complete.C:\Users\Administrator>tracert 172.20.1.2
Tracing route to 172.20.1.2 over a maximum of 30 hops
1 3 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 1 ms <1 ms <1 ms 10.200.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 4 ms 1 ms <1 ms 172.20.1.2
Trace complete.C:\Users\Administrator>tracert 172.20.1.3
Tracing route to 172.20.1.3 over a maximum of 30 hops
1 4 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.3
Trace complete.C:\Users\Administrator>tracert 172.20.1.4
Tracing route to 172.20.1.4 over a maximum of 30 hops
1 2 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 3 ms 1 ms <1 ms 172.20.1.4
Trace complete.Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root@Web01:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.280 ms 0.217 ms 0.209 ms 2 172.39.39.2 (172.39.39.2) 0.646 ms 0.787 ms 0.767 ms 3 10.200.19.1 (10.200.19.1) 1.025 ms 1.678 ms 1.604 ms 4 10.11.11.50 (10.11.11.50) 1.577 ms * *root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.135 ms 0.137 ms 0.056 ms 2 172.39.39.2 (172.39.39.2) 0.669 ms 0.651 ms 0.727 ms 3 10.200.19.1 (10.200.19.1) 2.106 ms 2.067 ms 2.061 ms 4 10.11.11.50 (10.11.11.50) 2.195 ms * *root@Web03:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.185 ms 0.150 ms 0.179 ms 2 172.39.39.2 (172.39.39.2) 0.267 ms 0.285 ms 0.209 ms 3 10.200.19.1 (10.200.19.1) 0.871 ms 0.707 ms 0.746 ms 4 10.11.11.50 (10.11.11.50) 0.850 ms * *root@Web04:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.119 ms 0.077 ms 0.079 ms 2 172.39.39.2 (172.39.39.2) 0.322 ms 0.382 ms 0.447 ms 3 10.200.19.1 (10.200.19.1) 1.584 ms 1.540 ms 1.495 ms 4 10.11.11.50 (10.11.11.50) 1.479 ms * *Turn RT-A-01 and RT-A-02 back on and turn ESG-A off. Verify paths again. The expectation is that the traffic will route through DC2 the only ESG in DC1 is down.
Before I turned off ESG-A I have verified if the traffic was flowing through DC1 again as an extra verification step. This is not shown below. For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
With the ESG-A turned off…
C:\Users\Administrator>tracert 172.20.1.1
Tracing route to 172.20.1.1 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 254 ms 1 ms 1 ms 172.20.1.1
Trace complete.C:\Users\Administrator>tracert 172.20.1.2
Tracing route to 172.20.1.2 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms 1 ms <1 ms 172.20.1.2
Trace complete.C:\Users\Administrator>tracert 172.20.1.3
Tracing route to 172.20.1.3 over a maximum of 30 hops
1 3 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.3
Trace complete.C:\Users\Administrator>tracert 172.20.1.4
Tracing route to 172.20.1.4 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 5 ms <1 ms <1 ms 172.20.1.4
Trace complete.Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root@Web01:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.154 ms 0.115 ms 0.102 ms 2 172.39.39.2 (172.39.39.2) 0.665 ms 0.660 ms 0.637 ms 3 10.200.19.1 (10.200.19.1) 0.969 ms 0.998 ms 0.977 ms 4 10.11.11.50 (10.11.11.50) 1.196 ms * *root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 1.620 ms 1.536 ms 1.488 ms 2 172.39.39.2 (172.39.39.2) 0.452 ms 0.502 ms 0.510 ms 3 10.200.19.1 (10.200.19.1) 0.763 ms 0.718 ms 0.686 ms 4 10.11.11.50 (10.11.11.50) 0.993 ms * *root@Web03:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.171 ms 0.131 ms 0.143 ms 2 172.39.39.2 (172.39.39.2) 0.326 ms 0.362 ms 0.483 ms 3 10.200.19.1 (10.200.19.1) 1.043 ms 1.063 ms 0.974 ms 4 10.11.11.50 (10.11.11.50) 1.055 ms * *root@Web04:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.131 ms 3.804 ms 3.799 ms 2 172.39.39.2 (172.39.39.2) 0.443 ms 0.319 ms 0.333 ms 3 10.200.19.1 (10.200.19.1) 0.575 ms 0.508 ms 0.772 ms 4 10.11.11.50 (10.11.11.50) 0.782 ms * *Turn ESG-A back on. Verify paths again. The expectation is that everything is back to normal now.
For some reason after turning on the ESG on DC1 the routes kept flowing through DC2. The same “issue” we saw before. So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
C:\Users\Administrator>tracert 172.20.1.1
Tracing route to 172.20.1.1 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.1
Trace complete.C:\Users\Administrator>tracert 172.20.1.2
Tracing route to 172.20.1.2 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 4 ms <1 ms <1 ms 172.20.1.2
Trace complete.C:\Users\Administrator>tracert 172.20.1.3
Tracing route to 172.20.1.3 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 2 ms 1 ms <1 ms 172.20.1.3
Trace complete.C:\Users\Administrator>tracert 172.20.1.4
Tracing route to 172.20.1.4 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 3 <1 ms <1 ms <1 ms 10.100.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 2 ms <1 ms <1 ms 172.20.1.4
Trace complete.Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root@Web01:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.163 ms 0.203 ms 0.123 ms 2 172.39.39.1 (172.39.39.1) 0.214 ms 0.255 ms 0.266 ms 3 10.100.21.1 (10.100.21.1) 0.647 ms 0.558 ms 0.401 ms 4 10.11.11.50 (10.11.11.50) 0.798 ms * *root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.183 ms 0.117 ms 0.119 ms 2 172.39.39.1 (172.39.39.1) 0.290 ms 0.241 ms 0.319 ms 3 10.100.21.1 (10.100.21.1) 0.609 ms 0.555 ms 0.602 ms 4 10.11.11.50 (10.11.11.50) 0.737 ms * *root@Web03:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.246 ms 0.102 ms 0.065 ms 2 172.39.39.1 (172.39.39.1) 0.465 ms 0.521 ms 0.568 ms 3 10.100.21.1 (10.100.21.1) 0.686 ms 0.840 ms 0.812 ms 4 10.11.11.50 (10.11.11.50) 0.983 ms * *root@Web02:~# traceroute 10.11.11.50traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.165 ms 0.177 ms 0.211 ms 2 172.39.39.1 (172.39.39.1) 0.433 ms 0.500 ms 0.549 ms 3 10.100.19.1 (10.100.19.1) 0.860 ms 0.956 ms 0.949 ms 4 10.11.11.50 (10.11.11.50) 1.073 ms * *Use–Cases
The end conclusion is that routing with this topology (with iBGP between the DLR and the ESG) works fine. When DC1 becomes unavailable DC2 will take over. For some reason when DC1 is available again the switch-over does not happen automatically, and we need to “kick” it by disabling BGP on the ESG-B.
Originally published on the old wiki at /index.php/Routing_with_NSX_using_multiple_sites_(with_iBGP_between_the_UDLR_and_the_ESG's).