Network Working Group K. Majumdar Internet Draft Oracle Intended status: Standard Track L. Dunbar Expires: August 18, 2025 Futurewei V.Kasiviswanathan Arista A. Ramchandra Microsoft A. Choudhary Aviatrix February 18, 2025 Multi-segment SD-WAN via Cloud DCs draft-ietf-rtgwg-multisegment-sdwan-02 Abstract This document describes a method for SD-WAN Customer Premises Equipment (CPEs) to use GENEVE encapsulation (RFC8926) to transport IPsec-encrypted packets across a Cloud Backbone without requiring decryption at Cloud Gateways (GWs). In this approach, SD-WAN CPEs encapsulate IPsec-encrypted payloads within GENEVE headers and forward them to their nearest Cloud GWs. These Cloud GWs then steer the encrypted traffic through the Cloud Backbone while preserving its confidentiality, ensuring seamless transport to the destination Cloud GWs. The egress Cloud GWs subsequently deliver the original IPsec- encrypted payloads to the receiving CPEs. This mechanism enables the Cloud Backbone to interconnect multiple SD-WAN segments efficiently, eliminating the need for Cloud GWs to decrypt and re-encrypt payloads, thus enhancing the performance. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. xxx, et al. Expires August 18, 2025 [Page 1] Internet-Draft Multi-segment SD-WAN Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on Dec 18, 2025. Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction..............................................3 2. Conventions used in this document.........................5 3. Use Cases.................................................6 3.1. Multi-segment SD-WAN via a Single Cloud GW...........6 3.2. Multi-segment SD-WAN via Cloud Backbone..............7 3.3. Traffic Steering Challenges in Multi-Segment SD-WAN..8 4. Data Plane encoding for SD-WAN Transit....................9 4.1. Multi-Segment SD-WAN Option Class....................9 4.2. SD-WAN Tunnel Endpoint Sub-TLV......................11 4.3. SD-WAN Tunnel Originator Sub-TLV....................12 Dunbar, et al. Expires Dec 18, 2025 [Page 2] Internet-Draft Multi-segment SD-WAN 4.4. Egress GW Sub-TLV...................................13 4.5. Include Transit Sub-TLV.............................14 4.6. Exclude Transit Sub-TLV.............................15 5. Packet Header Processing.................................16 6. Error Handling...........................................18 7. Control Plane considerations.............................19 7.1. Control Plane for CPEs..............................19 7.2. Control Plane between CPEs and Cloud GWs............19 8. Observability Consideration..............................19 9. Security Considerations..................................20 9.1. Threat Analysis.....................................20 9.2. HMAC-based Integrity and Authentication.............22 10. Manageability Considerations............................24 11. IANA Considerations.....................................25 12. References..............................................26 12.1. Normative References...............................26 12.2. Informative References.............................27 13. Acknowledgments.........................................27 Appendix A: Illustration of Packets through Cloud GWs.......28 A.1 Single Hop Cloud GW.....................................28 A.2 Multi-hop Transit GWs...................................29 Appendix B: Illustration from Private VPN to IPsec Tunnel...31 1. Introduction SD-WAN is widely deployed to connect enterprises' on-premises CPEs with services in Cloud DCs. As described in Section 4.1 of [Net2Cloud], enterprises have multiple options for connecting to Cloud DCs: - Direct Interconnect model, - Direct Interconnect model with Enterprise's virtual appliances in the Cloud, - Indirect Interconnect model via SD-WAN paths and - Managed Hybrid WAN model using Enterprise's existing VPN connections. In many deployments, enterprises operate multiple SD-WAN segments to optimize network performance, enhance security, and meet compliance requirements. Multi-segment SD-WAN refers to SD-WAN deployments where different segments-such as branch offices, cloud regions, and data centers-are independently managed and connected through a backbone network. This Dunbar, et al. Expires Dec 18, 2025 [Page 3] Internet-Draft Multi-segment SD-WAN architecture is often necessary when enterprises span multiple geographic regions, use different SD-WAN vendors, or have regulatory constraints requiring segmented traffic management. Interconnecting these SD-WAN segments via a Cloud Backbone provides several key benefits: a) Seamless connectivity - Enterprises can integrate geographically dispersed SD-WAN segments into a unified network without complex manual configurations. b) Scalability - The Cloud Backbone's elasticity accommodates increased traffic demands without requiring extensive on- premises infrastructure. c) Simplified operations - Centralized orchestration streamlines policy enforcement and network management across all segments. To ensure security, enterprise traffic between CPEs remains encrypted and inaccessible to third parties, including Cloud DCs. However, for encrypted packets to be steered through the Cloud Backbone, routing information must be included in the packet headers. Since the IPsec Security Association (SA) exists only between CPEs and is not accessible to Cloud GWs, an additional tunnel between the source CPE and ingress Cloud GW is required. This tunnel could be another layer of IPsec, but decrypting the outer IPsec tunnel solely for routing introduces significant drawbacks: - Processing Overhead - Cloud GWs must decrypt the outer IPsec layer just to extract routing information, increasing CPU load and reducing efficiency. - Latency - The additional decryption and re-encryption steps introduce delays, degrading performance. - Scalability Constraints -decrypting every IPsec packet at Cloud GWs becomes a bottleneck, making it inefficient for high-volume traffic. - Cloud IPsec Gateway Limits - Many cloud providers cap IPsec processing per instance, requiring additional gateways at extra cost when limits are exceeded. AWS Dunbar, et al. Expires Dec 18, 2025 [Page 4] Internet-Draft Multi-segment SD-WAN restricts SAs and throughput per CGW instance, Azure imposes per-gateway limits, and Google Cloud enforces per-tunnel throughput caps. If encrypted traffic is forwarded through the Cloud Backbone without decryption or re-encryption at Cloud GWs, it reduces processing overhead, latency, and scalability constraints while maintaining end-to-end encryption. Since the payload remains encrypted between CPEs, security and compliance requirements are upheld without exposing sensitive data at transit points. This document introduces a method for SD-WAN CPEs to encapsulate IPsec-encrypted packets using GENEVE Encapsulation (RFC8926). CPEs send the GENEVE encapsulated packets to their nearest Cloud GWs, where Sub-TLVs in the GENEVE header indicate whether the packets should traverse the Cloud Backbone without decryption. If backbone traversal is required, the encrypted payload is forwarded to the optimal egress Cloud GWs, which then deliver the original IPsec-encrypted payload to the destination CPEs. This method enables seamless multi-segment SD-WAN connectivity over the Cloud Backbone while eliminating the need for Cloud GWs to decrypt and re-encrypt payloads. GENEVE is chosen for encapsulation due to its widespread use in Cloud DC environments. 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. The following acronyms and terms are used in this document: Dunbar, et al. Expires Dec 18, 2025 [Page 5] Internet-Draft Multi-segment SD-WAN Cloud DC: Off-Premises Data Center, managed by the third party, that hosts applications, services, and workload for different organizations or tenants. CPE: Customer (Edge) Premises Equipment. OnPrem: On Premises data centers and branch offices. RR Route Reflector. SA IPsec Security Association SD-WAN An overlay connectivity service that optimizes transport of IP Packets over one or more Underlay Connectivity Services and determining forwarding behavior by applying Policies to them. [MEF-70.1] VPN Virtual Private Network. 3. Use Cases 3.1. Multi-segment SD-WAN via a Single Cloud GW Enterprise branches with established SD-WAN paths to a Cloud GW for accessing cloud services can also use the Cloud GW to interconnect with one another, as shown in Figure 1. This approach offers several advantages: - Reliable Connectivity - The public internet between branches may suffer from limited bandwidth, unpredictable performance, and security risks. In contrast, SD-WAN paths to a Cloud GW provide more stable, monitored connections with enhanced network functions. - Integrated Security - Cloud-based security services, such as firewalls and DDoS protection, can be centrally applied to enforce consistent policies across workloads and branches. - Advanced Threat Intelligence - Some cloud environments offer proprietary security tools and SaaS-based analytics that help monitor and mitigate threats in network traffic. Dunbar, et al. Expires Dec 18, 2025 [Page 6] Internet-Draft Multi-segment SD-WAN (^^^^^^^^^^^^) ( Cloud ) ( +----+ +----+ ) + -----(-|Edge| + GW | ) Direct | ( +----+ +/--\+ ) Connect | (^^^^^^^/^^^^\^) {-+---} / \ SD-WAN Path CPE<->GW { VPN } / \ {-+---} / IPsec Tunnel +-------+----/------+ \ | / | \ ++--/+ | +-\--+ |CPE1| +----+CPE2| +----+ +----+ Client Route: 11.1.1.x 10.1.1.x 21.1.1.x 20.1.1.x 30.1.1.x Figure 1 Multi-Segment SD-WAN stitching via a Cloud GW 3.2. Multi-segment SD-WAN via Cloud Backbone For geographically distant enterprise branches that have established SD-WAN paths to their respective Cloud GWs for accessing cloud services, the Cloud Backbone provides an efficient way to interconnect these branches, as shown in Figure 2. As outlined in the Introduction section, this approach enhances network integration, supports dynamic scaling, and simplifies overall management, making it well- suited for multi-segment SD-WAN deployments across different regions. Dunbar, et al. Expires Dec 18, 2025 [Page 7] Internet-Draft Multi-segment SD-WAN (^^^^^^^^^^^^^^^) ( Cloud ) ( +----+ +----+ ) +-----+ + ---(-|Edge|==| GW1|=================== GW2 | Direct | ( +----+ +/--\+ ) +--|--+ Connect | (^^^^^^^/^^^^\^) | {-+---} / \ | { VPN } / \ +-----+ {-+---} / IPsec Tunnel |CPE10| +-------+--/--------+ \ +-----+ | / | \ 10.2.1.x ++/--+ | +\---+ 20.2.1.x |CPE1| +----+CPE2| 30.2.1.x +----+ +----+ Client Route: 11.1.1.x 10.1.1.x 21.1.1.x 20.1.1.x 30.1.1.x Figure 2 Multi-Segment SD-WAN Stitching via Cloud Backbone 3.3. Traffic Steering Challenges in Multi-Segment SD-WAN Many well-established traffic engineering methods, such as SRv6 and MPLS-TE, effectively steer traffic through specific network nodes when the entire network operates under a single administrative domain. However, in SD-WAN deployments where on-premises CPEs connect to Cloud GWs over the public internet, traffic forwarding is typically destination-based, limiting the ability to enforce precise traffic steering policies. Even when branch CPEs have dedicated SD-WAN paths to Cloud GWs, the paths between branches may still be routed over the public internet via any available route, bypassing the Cloud Backbone entirely. This lack of predictable routing makes traffic steering between branch offices highly challenging. Unlike private MPLS networks or provider-controlled backbones, SD-WAN cannot inherently dictate the intermediate paths for branch-to- Dunbar, et al. Expires Dec 18, 2025 [Page 8] Internet-Draft Multi-segment SD-WAN branch traffic. As a result, policies intended to optimize performance, enforce security, or ensure compliance can be difficult to implement. To address this issue, this document proposes a method where Cloud GWs explicitly interconnect SD-WAN segments, ensuring that branch-to-branch traffic is steered through the Cloud Backbone rather than taking unpredictable internet routes. This approach provides greater control over traffic flows, improving reliability, security, and policy enforcement. 4. Data Plane encoding for SD-WAN Transit To enable Cloud GWs to distinguish between packets requiring decryption for internal cloud services and transit packets that should be forwarded to destination CPEs, proper packet marking is essential. Since GENEVE Encapsulation [RFC8926] is widely supported by major Cloud Service Providers, it is chosen as the encapsulation method. This allows Cloud GWs to efficiently steer IPsec-encrypted packets between CPEs without decryption, reducing processing overhead and improving performance while maintaining end-to-end encryption. 4.1. Multi-Segment SD-WAN Option Class Geneve header format is specified in Section 3 of [RFC8926]. This document introduces a new GENEVE Option Class (Type value = 0x0163) to indicate that Multi-Segment SD-WAN- specific Sub-TLVs are encoded within the GENEVE header. This enables Cloud GWs to interpret and process SD-WAN transit packets efficiently without requiring decryption. Dunbar, et al. Expires Dec 18, 2025 [Page 9] Internet-Draft Multi-segment SD-WAN 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Multi-seg-SD-WAN Option Class |C| Type |R|R|R| Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ SD-WAN Tunnel Endpoint Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Optional SD-WAN Tunnel Originator Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Optional Egress GW Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // // // Optional Type Length Value objects (variable) // // // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3 Multi Segment SD-WAN Option Class Multi-seg-SD-WAN Option Class is a newly defined GENEVE Option Class (16 bits, Type value = 0x0163, assigned by IANA) that identifies packets carrying Multi-Segment SD-WAN- specific Sub-TLVs within the GENEVE header . - C-bit: Must be set to ensure that a receiving node drops the packet if it does not recognize the option, as per [RFC8926]. - Type (8 bits): Specifies the multi-segment SD-WAN forwarding model: Type = 1: Single-hop transit SD-WAN Type = 2: Multi-Hop transit SD-WAN with an explicitly specified egress Cloud GW (via Sub-TLV). Type = 3: Multi-hop transit SD-WAN without an explicitly specified egress Cloud GW. - Length (5 bits): Indicates the total length of the option fields in 4-byte units. If no options are present, this field is zero [RFC8926]. Dunbar, et al. Expires Dec 18, 2025 [Page 10] Internet-Draft Multi-segment SD-WAN Note: the payload following the multi-seg-SD-WAN Option Class can be IPv4 or IPv6. The Protocol Type of the GENEVE header is set to 50, indicating the GENEVE payload carries IPsec ESP [RFC8926][IPsecOverGENEVE]. 4.2. SD-WAN Tunnel Endpoint Sub-TLV The SD-WAN Endpoint sub-TLV indicates the destination CPE, which is the endpoint of the IPsec Tunnel between branch CPEs. This Sub-TLV is used by the Cloud Backbone to determine the optimal egress Cloud GW for forwarding the encrypted traffic. For example, in an SD-WAN deployment where CPE1 establishes an IPsec SA with CPE2 (as shown in Figure 1), this Sub-TLV within the GENEVE header contains CPE2's IP address, ensuring that encrypted traffic is correctly routed to the terminating CPE of the IPsec tunnel while enabling the Cloud Backbone to steer the packet to the most suitable egress Cloud GW. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SD-WAN Endpoint| length | Reserved | TTL | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SD-WAN Dst Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ | SD-WAN end point Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 SD-WAN Endpoint Sub-TLV - SD-WAN Endpoint (8 bits): Identifies the SD-WAN Tunnel Endpoint Sub-TLV with a Type value of 1. - Length (8 bits): Specifies the total length of the value field in 4-byte units. Dunbar, et al. Expires Dec 18, 2025 [Page 11] Internet-Draft Multi-segment SD-WAN - TTL (Time to Live): set by the SD-WAN Tunnel Originator (e.g., CPE1) to track the number transit nodes or cloud regions/zones that the packet traverses. Each transit node or region (visible to the CPEs) decrement the TTL to allow the egress Cloud GW to determine the number of logical transits the packet has traversed. Enterprises can also define the TTL value to enforce the maximum transit nodes/regions the packets traverse. 4.3. SD-WAN Tunnel Originator Sub-TLV The SD-WAN Tunnel Originator Sub-TLV is an optional Sub-TLV within the multi-seg-SD-WAN Option Class to indicate the originating CPE of the IPsec Tunnel. For example, in an SD-WAN deployment where CPE1 establishes an IPsec SA with CPE2 (as shown in Figure 1), this Sub-TLV within the GENEVE header carries CPE1's address, allowing transit nodes and Cloud GWs to recognize the source of the encrypted traffic. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SDWAN Origin | length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SD-WAN Org Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ | SD-WAN Tunnel Originator Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5 SD-WAN Tunnel Originator Sub-TLV - SDWAN Origin (8 bits): Identifies the SDWAN Tunnel Originator Sub-TLV with a Type value of 2. - Length (8 bits): Specifies the total length of the value field in 4-byte units, excluding the first 4 bytes, which include the SD-WAN Origin (1 byte), Length (1 byte), and Reserved (2 bytes) fields. - Reserved (16 bits): Reserved for future. Must set to 0. Ignored by recipients. Dunbar, et al. Expires Dec 18, 2025 [Page 12] Internet-Draft Multi-segment SD-WAN This Sub-TLV allows Cloud GWs and transit nodes to identify the packet's source, allowing them to apply source specific policies for forwarding. These policies may include routing optimization based on the origin, security enforcement tailored to the source, or traffic engineering rules specific to the originating CPE. 4.4. Egress GW Sub-TLV In a multi-segment SD-WAN deployment over the Cloud Backbone, the originating CPE can use the Egress GW Sub-TLV to explicitly specify the egress Cloud GW responsible for forwarding traffic to the destination CPE. This ensures predictable routing and optimized packet delivery across the Cloud Backbone. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SDWAN EgressGW | length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Egress GW Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ | Egress GW Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6 SD-WAN Egress GW Sub-TLV - SDWAN EgressGW (8 bits): Identifies the Egress GW Sub- TLV with a Type value of 3. - Length (8 bits): Specifies the total length of the value field in 4-byte units, excluding the first 4 bytes, which include the SD-WAN Origin Sub-TLV Type (1 byte), Length (1 byte), and Reserved (2 bytes) fields . - Reserved (16 bits): Reserved for future. Must set to 0. Ignored by recipients. The Egress GW Sub-TLV allows the originating CPE to specify the egress Cloud GW responsible for forwarding traffic to the destination CPE. This egress GW address can be either preconfigured or dynamically discovered through a control plane protocol exchange with the destination CPE. By explicitly defining the egress GW, this Sub-TLV ensures predictable traffic steering, reducing reliance on destination-based routing and optimizing packet delivery Dunbar, et al. Expires Dec 18, 2025 [Page 13] Internet-Draft Multi-segment SD-WAN across the Cloud Backbone. The details of the control plane protocol used for egress GW discovery are beyond the scope of this document. 4.5. Include Transit Sub-TLV Include-Transit Sub-TLV is an optional field that allows explicitly specifying a list of Cloud Availability Regions or Zones that a packet must traverse when forwarded through the Cloud Backbone. This can be useful for: - Enhancing observability and security by directing traffic through regions with specific OAM and security functions. - Ensuring compliance with regulatory or business requirements that mandate traffic to pass through designated locations. Multiple Include-Transit Sub-TLVs can be included within a single GENEVE header to indicate multiple required transit nodes or regions. However, these Sub-TLVs form an unordered set, meaning there is no enforced sequence in which the specified regions must be traversed. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Include-Transit| length |Transit_Type |I|Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Transit node ID | ~ ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7 Include-Transit Sub-TLV - Include-Transit (8 bits): identifies the Include-Transit Sub-TLV with a Type value of 4. - Length (8 bits): Specifies the total length of the value field in 4-byte units, excluding the first 4 bytes, which include the Include-Transit Sub-TLV Type (1 byte), Length (1 byte), Transit_Type (1 byte), I-bit (1 bit), and Reserved bits (7 bits). - Transit_type (8 bits): Defines how the transit node is identified: Dunbar, et al. Expires Dec 18, 2025 [Page 14] Internet-Draft Multi-segment SD-WAN o Transit_type = 1: The Transit Node ID is represented as a numeric identifier, such as a Cloud Availability Region or Zone number assigned by the cloud provider. o Transit_type = 2: The Transit Node ID is represented as an IP address. - I-bit (1 bit) - Inclusion Requirement: o 0: The transit node is a preferred hop, meaning the Cloud Backbone will make a best-effort attempt to route traffic through it. o 1: The transit node is mandatory, meaning the packet must traverse this node. If the node cannot be reached, an alert or alarm must be generated to notify the enterprise via an out-of-band mechanism. The specifics of these alerts are beyond the scope of this document. 4.6. Exclude Transit Sub-TLV Exclude-Transit Sub-TLV is an optional field that explicitly specifies a list of Cloud Availability Regions or Zones that must be avoided when forwarding packets through the Cloud Backbone. This can be used for: - Regulatory compliance, ensuring traffic does not traverse restricted or non-compliant regions. - Risk mitigation, preventing traffic from passing through regions with known security, performance, or geopolitical concerns. Multiple Exclude-Transit Sub-TLVs can be included within a single GENEVE header to specify multiple excluded transit nodes or regions. However, these Sub-TLVs form an unordered set, meaning there is no defined sequence in which exclusions are applied. Dunbar, et al. Expires Dec 18, 2025 [Page 15] Internet-Draft Multi-segment SD-WAN 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Exclude-Transit| length |Transit_Type |E| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Transit node ID | ~ ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8 Exclude-Transit Sub-TLV - Exclude -Transit (8 bits): identifies the Exclude - Transit Sub-TLV with a Type value of 5. - Length (8 bits): Specifies the total length of the value field in 4-byte units, excluding the first 4 bytes, which include the Exclude-Transit Sub-TLV Type (1 byte), Length (1 byte), Transit_Type (1 byte), E-bit (1 bit), and Reserved bits (7 bits). - Transit_type (8 bits): Defines how the transit node is identified: Transit_type = 1: The Transit Node ID is represented as a numeric identifier, such as a Cloud Availability Region or Zone number assigned by the cloud provider. Transit_type = 2: The Transit Node ID is represented as an IP address. - E-bit (1 bit) - Exclusion Requirement: o 0: The specified transit node is undesirable, and the Cloud Backbone will make a best-effort attempt to route traffic around it. o 1: The specified transit node must be avoided. If the node cannot be bypassed, an alert or alarm must be generated to notify the enterprise via an out- of-band mechanism. The specifics of these alerts are beyond the scope of this document. 5. Packet Header Processing As illustrated in Figure 1, when a Cloud GW receives a GENEVE-encapsulated packet with Protocol Type = 50 (ESP), it processes the packet as follows: Processing at the Ingress Cloud GW Dunbar, et al. Expires Dec 18, 2025 [Page 16] Internet-Draft Multi-segment SD-WAN - Authenticate the packet using a preconfigured authentication method. - Extract the destination CPE address from the SD-WAN Endpoint Sub-TLV within the GENEVE header. - Check if the Egress GW Sub-TLV is present: o If the Egress GW Sub-TLV exists, the Cloud Backbone uses it to identify the egress Cloud GW. o If the Egress GW Sub-TLV is not present, the Cloud Backbone determines the optimal egress Cloud GW based on the destination CPE address. - Change the destination address in the outer IP header of the GENEVE packet to the address of the identified egress Cloud GW (either extracted from the GENEVE header or determined by the Cloud Backbone).. - Preserve the GENEVE header without modification. - Forward the packet to the egress Cloud GW. To prevent unauthorized access, the Cloud GW SHOULD drop any packets containing unrecognized source addresses or invalid values in the GENEVE Sub-TLVs, ensuring that only registered entities can utilize Cloud services. Processing at the Egress Cloud GW: - Decapsulate the GENEVE header to extract the IPsec- encrypted payload. - Perform additional security checks: - Validate that the SD-WAN Tunnel Endpoint Sub-TLV corresponds to a registered destination CPE. - Ensure the source Cloud GW is an authorized forwarding node to prevent unauthorized traffic injection. - Forward the IPsec-encrypted payload to the destination CPE, preserving the end-to-end encryption. - Drop any packet that lacks a valid destination CPE, or originates from an untrusted source. By enforcing these processing steps at both the ingress and egress Cloud GWs, the system ensures secure, efficient, and policy-compliant forwarding of SD-WAN traffic across the Cloud Backbone. Dunbar, et al. Expires Dec 18, 2025 [Page 17] Internet-Draft Multi-segment SD-WAN 6. Error Handling To ensure secure and efficient traffic forwarding through the Cloud Backbone, the Cloud GW SHOULD enforce the following error handling measures: - Drop packets with unregistered or invalid source/destination addresses to prevent unauthorized access. - Reject packets originating from unpaid or unregistered CPEs to enforce service subscription policies. - Validate the SD-WAN Endpoint Sub-TLV and drop packets if the destination CPE is unauthorized, unreachable, or mismatched. - Discard malformed packets with incorrect GENEVE headers, invalid Sub-TLV formats, or authentication failures. - Drop packets with expired TTL values to prevent routing loops and log repeated occurrences. - Reject misrouted packets if the Cloud Backbone cannot determine an optimal egress Cloud GW or if the specified egress GW is unreachable. - Enforce rate limits on excessive traffic from a single source to prevent congestion and abuse. - Verify compliance with transit node policies (e.g., ensuring mandatory transit nodes are included and excluded nodes are avoided). - Mitigate replay attacks by tracking sequence numbers and rejecting duplicate packets. By implementing these error handling mechanisms, Cloud GWs ensure network stability, security, and efficient resource utilization while preventing misconfigurations, abuse, and performance degradation. Dunbar, et al. Expires Dec 18, 2025 [Page 18] Internet-Draft Multi-segment SD-WAN 7. Control Plane considerations 7.1. Control Plane for CPEs The control plane enables SD-WAN CPEs to discover their network attributes, establish connectivity, and exchange routing information. In an SD-WAN deployment, on-premises CPEs and virtual CPEs (vCPEs) in Cloud DCs may be managed under a common iBGP administrative domain, facilitating route propagation and policy enforcement. Mechanisms such as BGP-based SD-WAN Edge Discovery [SD-WAN- Edge-Discovery] allow CPEs to dynamically discover each other's properties, improving automation and reducing manual configurations. Additionally, IPsec SAs parameters between CPEs and Cloud GWs can be exchanged through the iBGP control plane using a RR to simplify security policy management. 7.2. Control Plane between CPEs and Cloud GWs In multi-segment SD-WAN deployments, the control plane between CPEs and Cloud GWs must support dynamic routing updates and secure tunnel establishment. - eBGP sessions are commonly used between enterprise CPEs and Cloud GWs to facilitate dynamic route exchange. - If an IPsec tunnel is required between a Cloud GW and a vCPE, the IKEv2 key exchange must be performed to establish the tunnel securely. - Control plane mechanisms must ensure that Cloud GWs can identify and authenticate SD-WAN CPEs, validate SD-WAN metadata, and apply appropriate routing policies based on dynamic network conditions. By leveraging these control plane considerations, enterprises can ensure efficient route discovery, security enforcement, and seamless SD-WAN interconnectivity across the Cloud Backbone. 8. Observability Consideration Observability considerations encompass monitoring, analysis, and reporting mechanisms to gain insights into the behavior and performance of the multi-segment SD-WAN infrastructure. Key observability aspects include: Dunbar, et al. Expires Dec 18, 2025 [Page 19] Internet-Draft Multi-segment SD-WAN - Performance Metrics: Monitor and collect performance metrics related to link utilization, latency, and packet loss across the SD-WAN segments and Cloud DC backbone. This data provides insights into the overall health and efficiency of the network. IP Flow Information Export (IPFIX) [RFC7011] is one of the standardized methods to expose traffic flow over the network. - Global Network Topology Visualization: Utilize visualization tools to depict the global network topology, showcasing the interconnections and traffic flows between different SD-WAN segments and Cloud DCs. - Control Plane Monitoring: Monitor the control plane for both CPEs and the communication between CPEs and Cloud GWs. This includes tracking route discovery, path selection, and any changes in network state to ensure proper functioning of the SD-WAN control plane. - Security Event Logging: The security event logging is to capture and analyze security-related events, including threat detection, authentication failures, and any unauthorized access attempts. Syslog [RFC5424] is a valuable tool for security monitoring and auditing. These considerations contribute to the overall success of the multi-segment SD-WAN deployment connecting edge devices via a Cloud DC backbone. 9. Security Considerations 9.1. Threat Analysis As shown in Figure 3, the information carried by the GENEVE Header is not encrypted, which is susceptible to Man-in-the- Middle (MitM) attacks. An attacker can intercept and potentially alter the information in the GENEVE header between the branch CPEs and the Cloud GWs without the enterprise and the Cloud provider's knowledge or consent. Here is the threat analysis of the MitM attacks between CPEs and Cloud GWs: Dunbar, et al. Expires Dec 18, 2025 [Page 20] Internet-Draft Multi-segment SD-WAN a) Eavesdropping: Attackers can get knowledge of the enterprise's branch locations and their respective contracted Cloud GWs. As the payload between the CPEs is encrypted, attackers can't get any data exchanged between CPEs. This threat is no different from direct IPsec SAs between two CPEs. b) Data Manipulation: Attackers alter the content (Sub-TLVs) in the GENEVE header. As packets with unrecognized source addresses or invalid values in the Sub-TLVs of the GENEVE header are dropped by Cloud GWs, there might be a higher packet drop rate between the CPEs. Packet drop is not a new problem. The transport layer, such as TCP or QUIC, can handle packet drop well. c) Potential steeling of Cloud Backbone bandwidth: A threat actor might want to leverage Cloud Backbones to transport its own traffic between two locations without paying for the services. For example, a legitimate Cloud subscriber pays for the Cloud Backbone transport services for traffic between CPE-A and CPE-B. The attacker, who has two locations far apart (say Node-A and Node-B), can use CPE-A's address as the source address and CPE-B as the value in the SD-WAN Endpoint Sub-TLV for a packet from Node-A to Node-B before reaching the ingress Cloud GW. When the packet is sent from the egress Cloud GW via the Internet towards CPE-B, the actor can change the source address back to Node-A and the destination address to Node- B. By doing so, Node-A and Node-B can maintain the IPsec tunnel via the Cloud Backbone without paying for the service. Therefore, it is necessary to have some level data integrity and authentication for traffic between CPEs and Cloud GWs even though it is not necessary for Cloud GWs to decrypt and re-encrypt the payload between CPEs. [RFC2403] and [RFC2404] define the authentication algorithms used in AH and ESP. SHA2 224/256/384/512 are some of the cryptographic hashing algorithms. They are part of a Hashed Message Authentication Code. Dunbar, et al. Expires Dec 18, 2025 [Page 21] Internet-Draft Multi-segment SD-WAN 9.2. HMAC-based Integrity and Authentication HMAC (Hash-based Message Authentication Code), a widely used cryptographic technique for ensuring both data integrity and authentication, can be used to ensure the integrity and authenticity of data between CPEs and Cloud GWs to verify that GENEVE header has not been tampered with. The basic idea behind HMAC is to combine a secret key and a hash function to produce a fixed-size authentication code for the GENEVE header between CPEs and the Cloud GW. This authentication code is then sent along with the data itself. When the Cloud GW and the destination CPEs receive the data and the authentication code, they can independently compute the HMAC using the same key and hash function. If the computed HMAC matches the received authentication code, it indicates that the data has not been altered, as long as the secret key remains confidential. The HMAC authentication code can be carried by an HMAC Sub- TLV in the GENEVE Header, as specified below: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HMAC-Auth-Val | length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ ~ | HMAC Authentication Value | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9 Multi Segment SD-WAN HMAC Sub-TLV HMAC-Auth-Val (8 bits): HMAC Authentication Value Sub-TLV Type =6 (Assigned by this document). Length (8 bits): total length of the value field, which is the length of the HMAC Authentication Value in bytes plus 2 Reserved bytes. It is 6 bytes by default. However, in some deployments where security requirements are high, a longer authentication value can be considered. The HMAC Authentication Value (4 bytes or 8 bytes): is computed including the entire encapsulation header, excluding the HMAC-auth-Val Sub-TLV, based on a pre-configured algorithm. The advantages of using HMAC are: Dunbar, et al. Expires Dec 18, 2025 [Page 22] Internet-Draft Multi-segment SD-WAN - Data Integrity: HMAC provides a strong mechanism for verifying the integrity of data. By hashing the message and using a secret key, it generates a fixed-size hash value that ensures the data has not been tampered with during transmission. - Authentication: HMAC also verifies the authenticity of the sender. Since HMAC requires a shared secret key between the sender and receiver, it confirms that the sender is who they claim to be. - Efficiency: HMAC is computationally efficient, making it suitable for real-time and resource-constrained devices. It uses simple bitwise operations and hash functions. - Resistance to Tampering: HMAC is designed to resist various forms of tampering, including replay attacks, message insertion, and message deletion. Any change in the message will result in a different HMAC value. - Flexibility: HMAC can be used with various hash functions, such as SHA-256 or SHA-512, depending on the desired level of security requirements. - Widely Supported: HMAC is a well-established and widely supported authentication mechanism, making it easy to integrate into different systems. Here are some common problems associated with using the HMAC and why their risks are acceptable in the scenario described in this draft. - Key Management: The security of HMAC depends heavily on the confidentiality and management of the shared secret key. If the key is compromised, the data packets from CPEs to Cloud GW can be dropped but not compromised because the user payloads are protected by IPsec SA encryption. - Lack of Non-Repudiation: HMAC provides data integrity and sender authentication but does not provide non-repudiation. Non-repudiation is the ability to prove that a message was sent by a specific sender, which HMAC alone cannot guarantee. This risk is same as two IPsec protected traffic between CPEs. - Limited to Symmetric Cryptography: HMAC relies on symmetric key cryptography, which means that both parties must share the same secret key. As the Cloud backbone interconnecting Dunbar, et al. Expires Dec 18, 2025 [Page 23] Internet-Draft Multi-segment SD-WAN CPEs are paid services, there are established channels to distribute the symmetric key. - No Protection Against Eavesdropping: While HMAC ensures data integrity and sender authentication, it does not provide encryption. Eavesdropping does pose additional risks to payloads encrypted by IPsec SA. In summary, HMAC-based integrity and authentication offer strong security benefits in terms of data integrity and sender authentication. Even though it does not provide non- repudiation or protection against eavesdropping, the IPsec encrypted payload between CPEs won't be impacted. 9.3. AH based Integrity and Authentication For enterprises or Cloud providers worrying about secret HMAC keys being compromised, they can add another layer of AH encryption [RFC4301] or ESP-NULL [RFC2410] [RFC6071] on top of the IPsec encryption between the two CPEs. Both AH and ESP-NULL IPsec encryption require pairwise IPsec key management between Cloud GWs and the CPEs, therefore requiring more processing on Cloud GWs and CPEs. In addition, the AH encrypted packets can't traverse NAT because of outer IP address changes. 10. Manageability Considerations The following manageability considerations are crucial for the successful deployment and ongoing operation of the proposed strategies outlined in this document: - Centralized Orchestration: A centralized orchestration system is needed to manage and authenticate multiple SD-WAN segments through the Cloud GWs. - Policy-based Configuration: Utilize policy-driven configurations to streamline the deployment of SD-WAN segments and their connectivity options. This approach allows for efficient management of network policies, ensuring consistent and coherent behavior across diverse deployment scenarios. [RFC8192] can be used to automate the security policy configurations. Dunbar, et al. Expires Dec 18, 2025 [Page 24] Internet-Draft Multi-segment SD-WAN - Real-time Monitoring and Analytics: Integrate robust monitoring and analytics tools to provide real-time visibility into the performance and health of SD-WAN segments. This includes monitoring bandwidth utilization, latency, packet loss, and other key performance indicators to promptly identify and address any issues. - Automated Alerting and Reporting: Implement automated alerting mechanisms to promptly notify network administrators of potential issues or anomalies within the SD-WAN infrastructure. Additionally, generate regular reports to facilitate performance analysis, capacity planning, and compliance monitoring. 11. IANA Considerations IANA is requested to assign a new GENEVE Option Class from the IETF Review range as shown below: Option Class Description Assignee/Contact Reference ------ ------------------- ------------- ----------- 0x0163 Multi Segment SD-WAN IETF [this document] IANA is requested to create the registry below and place it in a new Multi Segment SD-WAN Geneve Option Class registry group: Registry: Multi Segment SD-WAN Sub-TLVs Assignment Policy: IETF Review Reference: [this document] Sub-TLV Type Description Reference ------------ ---------------------- --------------- 0 Reserved 1 SD-WAN Endpoint [Section 4.2] 2 SD-WAN Originator [Section 4.3] 3 SD-WAN Egress GW [Section 4.4] Dunbar, et al. Expires Dec 18, 2025 [Page 25] Internet-Draft Multi-segment SD-WAN 4 Include Transit [Section 4.5] 5 Exclude Transit [Section 4.6] 6 Multi SD-WAN-HMAC [Section 9.2] 5-254 Unassigned 255 Reserved 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2403] C. Madson, R. Glenn, "The Use of HMAC-MD5-96 within ESP and AH", RFC2403, Nov. 1998. [RFC2404] C. Madson, R. Glenn, "The Use of HMAC-SHA-1-96 within ESP and AH", RFC2404, Nov. 1998. [RFC4301] S. Kent and K. Seo, "Security Architecture for the Internet Protocol", RFC4301, Dec. 2005. [RFC4303] S. Kent, "IP Encapsulating Security Payload (ESP)". RFC4303, Dec. 2005. [RFC5424] R. Gerhards, "The Syslog Protocol", RFC5424, March 2009. [RFC7011] B. Claise, B. Trammell, and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", RFC7011, Sept 2013. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8926] J. Gross, et al, "Geneve: Generic Network Virtualization Encapsulation", RFC8926, Nov 2020. Dunbar, et al. Expires Dec 18, 2025 [Page 26] Internet-Draft Multi-segment SD-WAN 12.2. Informative References [IPsecOverGENEVE] S. Boutros, et al, "IPsec over GENEVE Encapsulation", draft-boutros-nvo3-ipsec-over- geneve-01, work-in-progress, Jan, 2018. [RFC2410] R. Glenn and S. Kent, "The NULL encryption Algorithm and Its Use with IPsec", RFC2310, Nov. 1998. [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and Internet Key Exchange (IKE) Document Roadmap", Feb. 2011. [RFC8192] S. Hares, et al, "Interface to Network Security Functions (I2NSF) Problem Statement and Use Cases", July 2017 [MEF-70.1] MEF 70.1 SD-WAN Service Attributes and Service Framework. Nov. 2021. [Net2Cloud] L. Dunbar and A. Malis, "Dynamic Networks to Hybrid Cloud DCs Problem Statement", draft-ietf- rtgwg-net2cloud-problem-statement-39, April, 2024. [SD-WAN-Edge-Discovery] L. Dunbar, et al, "BGP UPDATE for SD- WAN Edge Discovery", draft-ietf-idr-sdwan-edge- discovery-12, Oct. 2023. 13. Acknowledgments Acknowledgements to Adrian Farrel, Donald Eastlake, Stephen Farrell for their extensive review and suggestions. This document was prepared using 2-Word-v2.0.template.dot. Dunbar, et al. Expires Dec 18, 2025 [Page 27] Internet-Draft Multi-segment SD-WAN Appendix A: Illustration of Packets through Cloud GWs This section illustrates Cloud GWs connecting traffic flow carried by the IPsec tunnels. A.1 Single Hop Cloud GW Assuming that all CPEs are under one administrative control (e.g., iBGP). Using Figure 1 as an example: - There is a bidirectional IPsec tunnel between CPE1 and Cloud GW; with IPsec SA1 for the traffic from the CPE1 to the Cloud-GW; and IPsec SA2 for the traffic from the Cloud-GW to the CPE1. - There is a bidirectional IPsec tunnel between CPE2 and Cloud GW; with IPsec SA3 for the traffic from the CPE2 to the Cloud-GW; and IPsec SA4 for the traffic from the Cloud-GW to the CPE2. - All the CPEs are under one iBGP administrative domain, with a Route Reflector (RR) as their controller. The CPEs notify their peers of their corresponding Cloud GW addresses (which is out of the scope of this document). When 11.1.1.x and 10.1.1.x need to communicate with each other, CPE1 and CPE2 establish a bidirectional IPsec Tunnel, with SA5 for the traffic from CPE1 to CPE2 and SA6 for the traffic from CPE2 to CPE1. Assume the IPsec ESP Tunnel Mode is used. A packet from 11.1.1.1 to 10.1.1.2 has the following outer header: Dunbar, et al. Expires Dec 18, 2025 [Page 28] Internet-Draft Multi-segment SD-WAN Outer IP header: +---------------------------+ | protocol = 17(UDP) | | src = CPE1 | | dst = Cloud GW | +---------------------------+ | Source Port =xxxx | | Dst Port = 6081 (GENEVE) | +===========================+ | GENEVE Header | | Protocol=50(ESP payload) | +- - -- -- - - -- - --+ |MultiSeg-SDWAN Option Class| +- - -- -- - - -- - --+ |SD-WAN EndPt SubTLV (CPE2) | +---------------------------+ |GENEVE Hdr Authentication |<-validated by GW +---------------------------+ < ----------+ |SPI(Security Parameter Idx)| | +---------------------------+ | | sequence number | | +---------------------------+ <-+ | | payload IP header: | | | | src = 11.1.1.1 | | | | dst = 10.1.1.2 | | | +---------------------------+ Encrypted | | TCP header + | | | ~ payload (variable) ~ | | | | | | +===========================+ <-+ -------+ |Integrity Check Value (ICV)|<-Generated by CPE1, +---------------------------+ validated by CPE2 Figure 10 Packet header illustration to Cloud GWs A.2 Multi-hop Transit GWs Traffic to/from geographic apart CPEs can cross multiple Cloud DCs via Cloud backbone. The on-premises CPEs are under one administrative control (e.g., iBGP). Using Figure 2 as an example: Dunbar, et al. Expires Dec 18, 2025 [Page 29] Internet-Draft Multi-segment SD-WAN - There is a bidirectional IPsec tunnel between CPE1 and the Cloud GW1; with IPsec SA1 for the traffic from the CPE1 to the Cloud-GW1; and IPsec SA2 for the traffic from the Cloud-GW1 to the CPE1. - There is a bidirectional IPsec tunnel between CPE10 and the Cloud GW2; with IPsec SA3 for the traffic from the CPE10 to the Cloud-GW2; and IPsec SA4 for the traffic from the Cloud-GW2 to the CPE10. - All the CPEs are under one iBGP administrative domain, with a Route Reflector (RR) as their controller. CPEs notify their peers of their corresponding Cloud GW addresses. When 11.1.1.x and 10.2.1.x need to communicate with each other, CPE1 and CPE10 establish a bidirectional IPsec Tunnel, with SA5 for the traffic from CPE1 to CPE10 and SA6 for the traffic from CPE10 to CPE1. Assume the IPsec ESP Tunnel Mode is used, a packet from 11.1.1.1 to 10.2.1.2 has the following outer header: Dunbar, et al. Expires Dec 18, 2025 [Page 30] Internet-Draft Multi-segment SD-WAN Outer IP header: +---------------------------+ | proto = 17 (UDP) | | src = CPE1 | | dst = Cloud GW1 | +===========================+ | GENEVE Header | | Proto=50(ESP payload) | +- - -- -- - - -- - --+ |MultiSeg-SDWAN Option Class| +- - -- -- - - -- - --+ |SD-WAN EndPt SubTLV (CPE10)| +---------------------------+ | EgressGW-SubTLV | +---------------------------+ |GENEVE Hdr Authentication |<-validated by GW +---------------------------+ < ----------+ |SPI(Security Parameter Idx)| | +---------------------------+ | | sequence number | | +---------------------------+ <-+ | | payload IP header: | | | | src = 11.1.1.1 | | | | dst = 10.2.1.2 | | | +---------------------------+ Encrypted | | TCP header + | | | ~ payload (variable) ~ | | | | | | +===========================+ <-+ -------+ |Integrity Check Value (ICV)|<- validated by CPE10 +---------------------------+ Figure 11 GENEVE header encapsulated IPsec packet Appendix B: Illustration from Private VPN to IPsec Tunnel This section illustrates a Cloud GW connecting client traffic from a branch CPE via a Private VPN to another CPE via an IPsec tunnel. Using Figure 1 as an example: - CPE1 send traffic via a Private VPN (Direct Connect to the Cloud Edge) to the Cloud GW. The traffic is not encrypted. Dunbar, et al. Expires Dec 18, 2025 [Page 31] Internet-Draft Multi-segment SD-WAN - There is a bidirectional IPsec tunnel between CPE2 and the Cloud GW; with IPsec SA1 for the traffic from the CPE2 to the Cloud-GW; and IPsec SA2 for the traffic from the Cloud-GW to the CPE2. - All the CPEs are under one iBGP administrative domain, with a Route Reflector (RR) as their controller. CPEs notify their peers of their corresponding Cloud GW addresses. Assume the IPsec ESP Tunnel Mode is used for the IPsec SA between Cloud GW and CPE2. For a packet from 11.1.1.1 to 10.2.1.2, the following header is added by CPE1 sending over the Private VPN: Outer IP header: +---------------------------+ | proto = 17 (UDP) | | src = CPE1 | | dst = Cloud GW | +===========================+ | GENEVE Header | | Protocol=50(ESP payload) | +- - -- -- - - -- - --+ |MultiSeg-SDWAN Option Class| +- - -- -- - - -- - --+ |SD-WAN EndPt SubTLV (CPE2) | +---------------------------+ |GENEVE Hdr Authentication |<-validated by GW +---------------------------+ < -+ | payload IP header: | | | src = 11.1.1.1 | | | dst = 10.2.1.2 | | +---------------------------+ Not Encrypted | TCP header + | | ~ payload (variable) ~ | | | | +===========================+ <-+ Figure 12 Illustration of packet through VPN Upon receiving the GENEVE encapsulated packet with the "Multi-Segment-SD-WAN" option, the Cloud GW extracts the destination CPE from the GENEVE header and encrypts the packet with the IPsec SA2 to forward to the destination (i.e., CPE2). The GENEVE Header is carried to the CPE2. Dunbar, et al. Expires Dec 18, 2025 [Page 32] Internet-Draft Multi-segment SD-WAN Outer IP header: +---------------------------+ | proto = 17 (UDP) | | src = Cloud GW | | dst = CPE2 | +===========================+ | GENEVE Header | | Proto=50(ESP payload) | +- - -- -- - - -- - --+ |MultiSeg-SDWAN Option Class| +- - -- -- - - -- - --+ |SD-WAN EndPt SubTLV (CPE2) | +---------------------------+ |GENEVE Hdr Authentication |<-validated by GW +---------------------------+ < ----------+ |SPI(Security Parameter Idx)| | +---------------------------+ | | sequence number | | +---------------------------+ <-+ | | payload IP header: | | | | src = 11.1.1.1 | | | | dst = 10.2.1.2 | | | +---------------------------+ Encrypted | | TCP header + | | | ~ payload (variable) ~ | | | | | | +===========================+ <-+ -------+ |Integrity Check Value (ICV)|<-validated by CPE2 +---------------------------+ Figure 13 Illustration of packet from the Egress Cloud GW Dunbar, et al. Expires Dec 18, 2025 [Page 33] Internet-Draft Multi-segment SD-WAN Authors' Addresses Linda Dunbar Futurewei Email: ldunbar@futurewei.com Kausik Majumdar Argus Networks Email: kausikm.ietf@gmail.com Venkit Kasiviswanathan Arista Email: venkit@arista.com Ashok Ramchandra Microsoft Email: aramchandra@microsoft.com Aseem Choudhary Aviatrix Email: achoudhary@aviatrix.com Contributors' Addresses Dunbar, et al. Expires Dec 18, 2025 [Page 34]