“Mastering Azure Route Server: Navigating the Path to Efficient Networking with NVAs in Azure”

Azure Route Server addresses the challenge of efficiently managing and directing traffic between Network Virtual Appliances (NVAs) and Azure virtual networks.

Before Azure route server rollout, NVA firewall configurations faced certain limitations and challenges. Azure administrators had to rely on layer 4 Network Security Groups (NSGs) for securing critical resources, and the routing tables to firewalls were often static, lacking flexibility in adapting to changing network conditions.

Azure Route Server is the glue that holds together the routing tables being learned from the ExpressRoute to on-prem, the native Azure spoke virtual network routes, the default route, and any site-to-site VPN routes advertised from the NVA firewalls. ARS uses BGP to communicate these routes so it is very predictable and stable.

If two or more routes are programmed in the Azure virtual machines, they will use Equal Cost MultiPathing (ECMP) to choose one of the NVA instances for every traffic flow. As a consequence, SNAT is a must in this design if traffic symmetry is a requirement.

Without SNAT, outbound traffic from the VMs may take different paths through NVA1 and NVA2 based on ECMP.

If a VM sends traffic to the internet, it might use NVA1 for one connection and NVA2 for another connection, leading to potential asymmetry.

How does ARS BGP advertising works?

Let’s say you’ve got 2 on-premise environments and these environments have got ER GW and VPN GW respectively as figure shows.

• The router at the VPN destination uses BGP to share its connected routes.
• VPN Gateway receives and integrates these routes into its internal routing table.
• Relevant routes are then sent from VPN Gateway to Azure Route Server (ARS) using BGP.
• ARS incorporates these routes into its own routing table and reflects them in the associated Virtual Network (VNet).
• The received routes are also shared with the ExpressRoute Gateway via BGP, becoming part of its routing table.
• ExpressRoute Gateway communicates these routes to the Microsoft Enterprise Edge (MSEE) through BGP.
• After necessary adjustments, the routes are disseminated to the Customer Edge (CE) corresponding to the remote location.
• This interconnected system ensures proper communication, allowing traffic to flow accurately from Customer Edge through MSEE to the VNet, with the VPN serving as the return path.

Role of ARS in Spokes

Typically, User-Defined Routes (UDR) are configured in spoke VNets to direct traffic through security devices in the hub. Managing these routes across multiple spokes can be complex for administrators.

• Traditionally, without the Route Server, user-defined routes would be configured in each spoke to direct traffic through the NVA, often involving a default route for 0.0.0.0/0 for security inspection.
• With the Route Server in the hub VNet, the need for user-defined routes is eliminated.
• The NVA advertises network prefixes to the Route Server.
• The Route Server injects these routes into the effective routes of any virtual machine deployed in the hub VNet or spoke VNets that are peered with the hub VNet using the “Use the remote virtual network’s gateway or Route Server” setting.

Limitations in ARS

• ARS is capable of supporting a maximum of 8 BGP neighbors so if we are using going to scale out our NVAs for additional capacity we will have a maximum of eight NVAs.
• The other downside to ARS is that while spoke VMs will use equal cost multi-path to route to the NVAs we do not have a good way of truly load balancing our East-West traffic between our active-active NVAs.Without effective load balancing, East-West traffic from Spoke VM1 to Spoke VM2 may consistently go through NVA1 in the hub.

Solution —

You create default route UDRs in the spoke networks pointing to an internal load balancer (ILB) in front of NVA1 and NVA2 in the hub.

These UDRs have “Disable Gateway Propagation” enabled, ensuring that they don’t interfere with the ECMP-based routing for outbound traffic through Azure Route Server.