Windowless Cumulative ACK Extension for RDMA Retransmission

Abstract

Traditional RDMA Reliable Connection (RC) uses selective retransmission, while TCP SACK depends on sliding window and RTO timer tuning. Both mechanisms waste network bandwidth when window or RTO parameters mismatch real network conditions. This document defines a windowless cumulative selective acknowledgement extension (T/C AETH) embedded within the RDMA ETH header, paired with dual-trigger ACK reporting logic driven by cumulative receive time threshold and cumulative received packet count threshold.¶

The receiver generates extended SACK reports without sliding window constraints. The sender accurately identifies lost PSN segments through multi-segment confirmation fields carried in T/C AETH, then only retransmits missing packets. This design removes window dependency, eliminates redundant retransmissions, and improves bandwidth utilization and end-to-end throughput for high-speed RDMA workloads in data centers and wide-area networks.¶

2. Mechanism Overview

The complete end-to-end workflow contains two symmetric operation planes:¶

• Receiver Plane (Section 5): Accumulate received PSNs, monitor dual time/count thresholds, construct T/C AETH extended ACK and transmit to sender when any threshold is hit.¶

• Sender Plane (Section 6): Transmit packets at fixed preset rate without sliding window limitation; parse multi-segment blocks in T/C AETH to extract missing PSN ranges, perform selective retransmission only for identified lost segments.¶

Core technical advantages:¶

• Windowless sender transmission: No per-flow credit window tracking required on sender side.¶

• Batch ACK reporting: Dual-threshold batching drastically reduces reverse-path ACK traffic volume.¶

• Multi-segment compact SACK: Single ACK frame carries multiple discontinuous received PSN ranges to enable precise loss recovery.¶

High-Level End-to-End Flow:¶

• Sender continuously transmits RDMA data packets at negotiated fixed preset rate.¶

• Receiver accumulates received data packets, tracks cumulative receive time and cumulative packet count.¶

• If cumulative time >= T_thresh OR cumulative count >= C_thresh: a. Receiver constructs RDMA ACK frame with embedded T/C AETH target fragment. b. Transmit extended ACK (First Message) back to sender QP.¶

• Sender parses T/C AETH segments, calculates PSN gaps between all confirmed receive blocks.¶

• Sender selectively retransmits only PSN ranges identified as lost.¶

• Fallback timeout retransmission activates if sender receives no extended ACK after fixed timeout window expires.¶

3. Receiver Operation

For threshold configuration, during RC QP setup handshake:¶

• Sender advertises its preset packet send rate to receiver via CM REQ/REP.¶

• Receiver computes and configures T_thresh (cumulative time threshold) and C_thresh (cumulative count threshold) proportional to sender transmission rate.¶

Receiver maintains per-QP batch state variables:¶

• BatchStartTime: Timestamp of the first packet received in current batch cycle. - BatchPacketCount: Total valid unique PSNs received in current batch.¶

• SegmentList: Ordered list of continuous received PSN blocks, sorted by ascending PSN value.¶

For each incoming valid data packet:¶

• Record PSN if packet is valid and not duplicate.¶

• Merge new PSN into existing continuous segment in SegmentList, or create a new standalone segment block.¶

• Increment BatchPacketCount counter by one.¶

• Calculate current cumulative receive time = CurrentTime - BatchStartTime.¶

After batch receive state tracking, it transmits T/C AETH extended ACK when either condition evaluates to true:¶

• Condition A: Cumulative receive time >= T_thresh¶

• Condition B: Cumulative receive count >= C_thresh¶

After sending extended ACK frame:¶

• Reset BatchStartTime to current system timestamp.¶

• Clear BatchPacketCount to zero.¶

• Flush SegmentList to empty for new batch collection cycle.¶

To complete the extended ACK transmission, the sender needs to construct an extended ACK frame.¶

• Base frame: Standard RDMA ACK ETH header sequence defined in RFC5040.¶

• Insert new T/C AETH Target Fragment between standard AETH and Atomic AETH fields inside ETH header.¶

• Populate all T/C AETH fields with data extracted from SegmentList, detailed format specified in Section 7.¶

• Encapsulate complete ACK frame and transmit to corresponding sender QP.¶

4. Sender Operation

In this draft, we design a windowless continuous transmission. After QP establishment handshake completes:¶

• Sender transmits data packets at pre-negotiated fixed preset rate continuously.¶

• No sliding window credit check is executed before sending new outbound packets.¶

• Cache all transmitted PSNs within send buffer for potential selective retransmission processing.¶

Upon receiving T/C AETH extended ACK frame:¶

• Parse SegmentNum field to read total number of valid receive confirmation segments stored inside Target Fragment.¶

• Extract Last Finished PSN, representing end PSN of first continuous fully received block.¶

• Extract each subsequent SegX Left / SegX Right pair representing discontinuous received PSN blocks.¶

• Compute all PSN gaps between adjacent confirmed segments; these gaps are defined as lost packet ranges.¶

• Trigger selective retransmission for all PSNs inside identified gap ranges only.¶

The designed multi-segment loss calculation logic is as follows:¶

• Case 1: SegmentNum = 1 (single continuous received block) All PSNs smaller than Last Finished PSN are fully received without loss. Lost packets = all PSNs greater than Last Finished PSN up to latest transmitted PSN.¶

• Case 2: SegmentNum > 1 (multiple discontinuous receive segments) Lost range 1: (Last Finished PSN, Seg1 Left PSN) Lost range N: (SegN Right PSN, SegN+1 Left PSN) All PSNs inside above intervals shall be retransmitted by sender.¶

5. T/C AETH Target Fragment Header Format

The T/C AETH fragment occupies fixed 30 Bytes inside RDMA ETH header, inserted between standard AETH and Atomic AETH subheaders. All fields use big-endian byte order, aligned to 24-bit unless otherwise specified.¶

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SegmentNum(24b)| Last Finished PSN (24bit)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seg1 Left(24b)| Seg1 Right(24b)                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seg2 Left(24b)| Seg2 Right(24b)                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seg3 Left(24b)| Seg3 Right(24b)        Reserved (24 bits)     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6. Retransmission Decision

Sender strictly retransmits only PSNs within gap ranges calculated from T/C AETH segment fields. No extra packets outside lost intervals are retransmitted, eliminating redundant retransmission overhead existing in Go-Back-N and oversized sliding window implementations.¶

For example, sender transmits PSN range 100 ~ 599 at fixed preset rate. Receiver receives continuous block 100~199, loses 200~299, receives 300~399, loses 400~499, receives 500~599.¶

Corresponding T/C AETH field values:¶

• SegmentNum = 3¶

• Seg1 Left=300, Seg1 Right=399¶

• Seg2 Left=500, Seg2 Right=599¶

Sender identifies lost ranges: 200~299, 400~499; retransmits only these two PSN intervals.¶

To handle potential loss of extended ACK frames on reverse data path:¶

• Sender maintains per-QP ACK timeout timer configured during QP setup.¶

• If sender transmits PSN range [X, Y] and receives no T/C AETH frame after timeout expires: a. Retransmit the latest transmitted sequential PSN block (highest PSNs sent). b. Restart ACK timeout timer immediately after retransmission.¶

• Fallback timeout mechanism serves only as safety backup; primary loss recovery relies on extended T/C AETH SACK reports.¶

9. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.