Post-Quantum in TLS 1.3: The X25519MLKEM768 Group and the MTU Realities

Putting post-quantum key exchange into TLS 1.3 is, cryptographically, almost anticlimactic: the hybrid exchange is exposed as just another named group, and the protocol machinery accommodates it without redesign. The disruption is physical. The post-quantum public key and ciphertext are each about a kilobyte, an order of magnitude larger than the thirty-two-byte elliptic-curve values they accompany, and that suddenly-large handshake collides with a network quietly built on the assumption that a ClientHello fits in one small packet. The hard part of this migration is not the algorithm but the megabytes-per-million-connections of extra handshake and the brittle middleboxes it trips. This article covers how the group fits and, more importantly, the size realities that break naive rollouts.

The hybrid group in the handshake

TLS 1.3 establishes keys by having the client offer key shares for one or more named groups in its opening message, the server pick one and return its own share, and both derive the session keys from the result. A hybrid post-quantum group slots directly into this mechanism: it is a named group like any other, whose key share is the concatenation of a classical elliptic-curve share and a post-quantum key-encapsulation public key. The client offers it, and if the server supports it, the server returns a share that concatenates the classical value and the post-quantum ciphertext.

Both sides then combine the classical and post-quantum secrets through the handshake's key schedule, using the robust combiner of the previous article, and proceed with an otherwise standard TLS 1.3 handshake. No new handshake message types are needed and the state machine is unchanged; from the protocol's structural viewpoint the hybrid group is a drop-in. This is by design — reusing the existing negotiation is what makes deployment tractable at all.

The elegance of the fit is genuine, but it conceals the cost, which lives entirely in the size of the shares. The classical share is tiny; the post-quantum public key and ciphertext are each roughly a kilobyte. Everything difficult about the rollout flows from that size difference, which the rest of the article examines.

The size problem: a ClientHello over the MTU

The concrete numbers explain the difficulty. A classical elliptic-curve key share is about thirty-two bytes, and a traditional ClientHello comfortably fits within a few hundred bytes — well inside a single network packet. The post-quantum public key in the recommended parameter set is over eleven hundred bytes, so the hybrid client share alone is roughly one and a quarter kilobytes, and the whole ClientHello swells past a kilobyte. That crosses a threshold: it no longer fits in the space a single typical packet provides once headers are accounted for.

This matters because a great deal of network machinery assumes the ClientHello arrives in the first packet of the connection. When it must be split across multiple packets, or when it approaches the maximum transmission unit — the largest packet a network path will carry without fragmentation — a range of latent problems surface. The server's response is affected too: it returns a roughly one-kilobyte ciphertext, so the size pressure is bidirectional, though the ClientHello growth is usually the more troublesome because it happens at the very start of the connection before anything is established.

None of this is a cryptographic weakness; the exchange is exactly as secure as the previous article described. It is an integration problem: a protocol field that was small for a generation has become large, and the surrounding network was implicitly optimized for it being small. The threshold-crossing from one packet to several is where naive rollouts begin to fail.

Middlebox and MTU realities

The enlarged handshake exposes three classes of network fragility. The first is middlebox ossification: firewalls, load balancers, and inspection devices that were written assuming a small, single-packet ClientHello may mishandle one that spans multiple packets or is fragmented, silently dropping or corrupting the connection. Because TLS has been deployed for decades, a long tail of such devices exists, and they cannot be fixed by the endpoints — they are on the path. This ossification is the reason large ClientHellos fail in ways that are hard to diagnose from either end.

The second is amplification limits. To prevent a server from being used to flood a spoofed victim, transports such as the modern encrypted transport restrict how much a server may send before it has validated the client's address — typically a small multiple of what the client first sent. A larger ClientHello raises the client's initial contribution, which interacts with these limits in subtle ways, and a server response that includes a kilobyte ciphertext must fit within the permitted amplification budget or incur extra round trips. The size pressure and the anti-amplification rules are in tension.

The third is buffer and fragmentation assumptions in software and hardware along the path that were sized for small handshakes. When the ClientHello must be reassembled from several segments, code paths that were rarely exercised become hot, and bugs that never mattered begin to bite. The common thread is that none of these is a flaw in the cryptography or even in TLS 1.3 itself; they are consequences of a network ecosystem tuned for a smaller handshake meeting a larger one, and they are exactly what a careful rollout must test for against real paths, not just in a lab.

Downgrade and retry costs

Two protocol-level consequences deserve attention. The first is downgrade resistance. TLS 1.3 hashes the entire handshake transcript, including the ClientHello and the negotiated group, into the keys, so an on-path attacker cannot silently strip the hybrid group and substitute a classical-only one without the tampering being detected when the transcripts fail to match. This protects the integrity of the negotiation, though it does not prevent an attacker from causing a connection to fail, which is a denial rather than a downgrade — a distinction the threat-modeling series returns to.

The second is the cost of a group mismatch. If a client offers only the hybrid group and the server prefers or requires a different one, the server issues a retry request, and the client must send a fresh ClientHello with the acceptable group, costing an extra network round trip before the handshake even properly begins. Because the hybrid ClientHello is already large, an unnecessary retry doubles the expensive first exchange, so clients are configured to offer the hybrid group in a way that avoids gratuitous retries against common servers.

Both considerations reinforce that the migration is as much about careful negotiation configuration as about the cryptography. Offering the right groups, in the right order, sized to avoid retries and to survive the network path, is the practical craft of deploying post-quantum TLS, and it is learned by measuring against real clients and servers rather than assumed.

Deployment status and guidance

The hybrid group pairing the widely used elliptic-curve exchange with the recommended post-quantum parameter set has a standardized codepoint and is already deployed at scale by major browsers and content networks, so it is a proven, interoperable choice rather than an experiment. That real-world deployment is what surfaced and drove fixes for the middlebox and size issues above, and it is why adopting the standardized group is far safer than assembling a custom hybrid.

The deployment guidance follows from the realities. Enable the standardized hybrid group and test it against real network paths, especially those traversing older middleboxes, rather than only in controlled environments. Monitor for connection failures that correlate with handshake size, keep classical groups available for graceful negotiation without allowing a silent security downgrade, and be prepared for the extra bytes to matter at high connection volume. The cryptography is settled; the operational validation is the work.

For autonomous AI systems this lands directly on the infrastructure where TLS terminates: model-serving endpoints and agent service meshes that establish enormous numbers of connections. At that volume the extra kilobyte per handshake is a real bandwidth and latency consideration, and the middlebox fragility is more likely to be encountered simply because more paths are traversed. The guidance is to adopt the standardized hybrid group at these termination points, validate it against the actual network paths agents use, and treat handshake size as a monitored operational metric — so that post-quantum protection is gained without the connection failures that a size-unaware rollout invites.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

tls_hybrid_rollout:
  group: x25519_ml_kem_768_standardized
  clienthello_size: monitored_metric
  path_testing: real_middleboxes_required
  amplification: reply_within_budget
  negotiation: avoid_gratuitous_retry
downgrade:
  transcript_hash_protects: true
  classical_fallback: no_silent_downgrade

Glossary

Named group

A key-exchange option negotiated in TLS 1.3; the hybrid group's key share concatenates a classical and a post-quantum value.

ClientHello key share

The client's offered key-exchange material; with a hybrid group it grows from about 32 bytes to over a kilobyte.

MTU (maximum transmission unit)

The largest packet a network path carries without fragmentation; the enlarged ClientHello approaches or crosses it.

Middlebox ossification

On-path devices written for small single-packet ClientHellos that mishandle larger or fragmented ones, breaking connections the endpoints cannot fix.

Amplification limit

A transport rule capping how much a server may send before validating the client's address, in tension with the larger post-quantum reply.

HelloRetryRequest

A server message asking the client to resend with an acceptable group, costing an extra round trip on an already-large exchange.

References

1. IETF draft-ietf-tls-hybrid-design, Hybrid Key Exchange in TLS 1.3
2. Stebila, Fluhrer & Gueron, Hybrid Key Exchange in TLS 1.3 (IACR ePrint 2018/903)
3. NIST SP 800-56C Rev. 2, Recommendation for Key-Derivation Methods (2020)
4. NIST FIPS 203, Module-Lattice-Based Key-Encapsulation Mechanism Standard (2024)
5. NIST IR 8547, Transition to Post-Quantum Cryptography Standards (2024)