Ultra Ethernet is ready

Contents

The Ultra Ethernet Consortium (UEC) has published specification 1.0, setting a milestone for high-performance and AI-optimized network infrastructures. The new specification makes the classic Ethernet protocol fit for high-speed networks and is intended to provide a scalable and interoperable communication stack across all layers of the network that is specially designed for bandwidth-hungry, latency- and packet-loss-sensitive applications such as AI clusters and high-performance computing (HPC).

Background

While Ethernet has been used for AI clusters in recent years, UEC members wanted to better design its characteristics for this use case. More than 100 companies and 1000 participants contributed to Ultra Ethernet 1.0. The 562-page document is available under Creative Commons license (CC BY-ND 4.0) on the UEC website.

In a video, Hugh Holbrook, Chair of the Technical Advisory Committee of the Ultra Ethernet Consortium and Vice President of Software Engineering at Arista Networks, explains the background: Why should Ethernet be adapted at all? He mentions the multi-vendor support, the broad support in switches, NICs, hosts, test equipment and a broad understanding of the protocol as the basis. Ultra Ethernet is fundamentally based on Ethernet. There are no changes to the core protocol, as it is only about better suitability for HPC/KI in the future.

At the heart of Ultra Ethernet 1.0 is not a revolution on the classic Ethernet layer, but the introduction of a new transport protocol. It is called Ultra Ethernet Transport (UET). UET makes it possible to transfer data directly from the network to the application memory and vice versa – without any complex software intervention. This capability already exists under the buzzword RDMA (Remote Direct Memory Access), but UET is intended to take it to a new level.

A characteristic feature of UET is Multipath RDMA with Relaxed Delivery Ordering: both ordered, and unordered delivery are possible, which makes load distribution more flexible. Thanks to modern congestion control and rapid loss recovery, UET should be able to better detect bottlenecks across Ethernet and IP and react quickly if packet loss is detected. Within a single round-trip time, the data transmission should reach the full data rate on the transmission medium (Wirerate) and adapt quickly if necessary, i.e., also down-regulate again. This should offer decisive advantages over TCP with its slow-start behavior.

Packet spraying ensures that all available paths to the destination are used. Hot-spot avoidance ensures optimum distribution of very large flows: Just enough entropy is generated to avoid bottlenecks on individual links without unnecessarily impairing the response speed due to too much entropy.

Loss detection in combination with packet trimming is a particularly interesting detail. If a packet arrives at an overloaded switch, it is not simply discarded, but shortened to small fragments with a 64th of the actual size and moved to a high-priority queue. The recipient thus recognizes packet losses almost in real time and can request only the missing data without having to retransmit entire frames.

The overload control itself can be operated either sender- or receiver-based. In default mode, it takes place on the transmitter side. This results in a rapid increase and a rapid deceleration of the flows. Delay, markers, and trimming are used as indicators for overload.

Optionally, the receiver can exert additional influence via a credit system. This results in optimistic transmission even before all credits have been confirmed, and it adjusts the speed as soon as new capacity information arrives.

Finally, UET with Zero-RTT-Connection-Setup enables a connection to be established without the classic three-way handshake of TCP, i.e., similar to QUIC. This allows connections to be established more quickly and user data to be transmitted directly in the first packet.

Security functions in UE

Every UET connection is encrypted end-to-end with AES-GCM (Galois/Counter Mode) as standard. Keys are derived via KDF (Key Derivation Function) and anti-replay functions are also available. UET also works with group keys, whereby a group key is valid for a security domain. Each member of a group trusts all others in the same group.

Ultra Ethernet integrates seamlessly into the software ecosystem of modern high-performance networks by acting as a provider directly below the OpenFabric Alliance's libfabric 2.0 API. libfabric provides a standardized interface for delivering high-performance network services to applications.

There have also been changes to the lower layers. Specifically, this involves link layer reliability (LLR) and credit-based flow control. Data flows in HPC/AI environments are sensitive to latency. A single link with suboptimal performance, for example due to temporarily higher bit error rates (BER), can slow down the entire parallel application. LLR provides a fast hardware-based response that mitigates link performance issues. Credit-based flow control is a mechanism used in networks to control data transmission by regulating the rate at which a sender can transmit data based on the availability of packet buffers at the receiver.

Outlook

Several promising developments are already emerging that will make the specifications even more powerful in future versions. There are already ideas for improved telemetry, overload control, bindings for storage protocols and in-network compute. All information on Ultra Ethernet 1.0 can be found here.

()