Protocol stacks

Overview

All serving lives in the binary's uring/ module. HTTP/1.1 is a custom monoio + httparse codec; HTTP/2 and cleartext h2c use the native hj-h2 stack through its monoio serve path; HTTP/3 is uring/h3.rs, a quinn-proto QUIC loop over io_uring UDP with quinn-udp socket metadata (GSO/GRO). Protocol selection happens at the connection boundary: ALPN over TLS, a non-destructive preface sniff in cleartext, and QUIC on the UDP listener.

stateDiagram-v2
  direction LR
  [*] --> Accepted: multishot accept
  Accepted --> ProxyHeader: listener has proxyProtocol=1
  Accepted --> Handshake: secure listener
  Accepted --> Sniff: plain listener
  ProxyHeader --> Handshake: header valid (else close)
  ProxyHeader --> Sniff
  Handshake --> ALPN: rustls (TLS 1.2 / 1.3, SNI, optional client cert)
  ALPN --> H1: http/1.1
  ALPN --> H2: h2
  Sniff --> H2: 24-byte h2c preface
  Sniff --> H1: anything else
  H1 --> H1: keep-alive ≤ maxKeepAliveReq
  H2 --> H2: streams multiplexed on one task
  H1 --> Closed: Connection close · timeout · error
  H2 --> Closed: GOAWAY drain · timeout
  Closed --> [*]

Connection front doors

Two per-listener features sit before protocol detection. Both are httpjet-native — LiteSpeed has no listener-level equivalent — and both change who httpjet thinks the client is.

PROXY protocol v2 / v1

A listener with <proxyProtocol>1</proxyProtocol> requires every connection to open with a PROXY header. The header is consumed and the claimed client address installed before the TLS handshake on secure listeners and before the H1/h2c discriminator on plain ones. Missing header → the connection is closed (fail-closed both ways). Since 2026-09-16 a header that claims a loopback or unspecified source is refused at the parser: those addresses feed the mTLS exemption and the loopback purge gate, so accepting them would let a proxy hop forge local trust.

HTTP over Unix sockets

A listener whose address is unix:/path/sock (or uds:), or the --http-uds flag, serves plain HTTP over a Unix-domain socket: H1 and h2c prior knowledge, one accept core (SO_REUSEPORT does not exist for AF_UNIX), socket mode 0600 group-writable, socket-activated fds adopted by path. A stale socket file is unlinked only after a connect(2) proves it dead, and cleanup is bound to the owned directory and socket identity. Peers have no address: ACLs, the per-IP throttle and REMOTE_ADDR see a fabricated loopback value and the access log renders unix:. TLS over UDS is unsupported and linted.

HTTP/1.1 & h2c detection

The H1 codec parses with httparse, enforces strict framing (single Content-Length, conflicting-length rejection, bounded incomplete reads), supports keep-alive and pipelining, honors maxKeepAliveReq (0 = unlimited), decodes chunked bodies, and streams large responses from the bridge without buffering. Since the 2026-10-06 review it also enforces the RFC 9112 request-target and Host rules at intake: only origin-form targets (plus * for OPTIONS) and http/https absolute-form (reduced to origin-form, its authority replacing Host); CONNECT answers 405, authority-form and a raw # answer 400. HTTP/1.1 requires Host, and any Host must be UTF-8 and pass the same authority validation the h2/h3 :authority rule uses (HTTP/1.0 may omit it). A header line terminated by a bare LF is rejected 400 before the head is materialized.

Cleartext HTTP/2 by prior knowledge has no upgrade handshake — the client simply sends the connection preface. httpjet detects it non-destructively: the bytes already read are handed to whichever stack wins, so h1 and h2 both parse the connection as if no router sat in front.

// over TLS: ALPN decides the TCP stack
match alpn {
    b"h2"       => hj_h2::server::serve_local_with_prefix(io, prefix, service, cfg).await,
    _           => handle_h1_bridged(io, prefix, service).await,   // http/1.1 or none
}

// cleartext: sniff the 24-byte preface, pass the observed bytes along
match sniff_cleartext(&mut stream).await? {
    Proto::Http2(prefix) => hj_h2::server::serve_local_with_prefix(stream, prefix, service, cfg).await,
    Proto::Http1(prefix) => handle_h1_bridged(stream, prefix, service).await,
}

Beyond framing, the H1 path implements the newer semantics end to end:

• 103 Early Hints — the same preload/preconnect Link matcher the h2 stack uses emits an interim 103 on h1; 101 upgrades bypass it; canonical/alternate links are excluded.
• Ranges, single and multipart — RFC 9110 multi-range sets are parsed, clamped, coalesced and framed as multipart/byteranges, capped at 8 ranges / 8 MiB of part bytes with a full-200 fallback. An invalid component poisons the whole header (→ 200); an unsatisfiable set is 416.
• Request decompression — gzip bodies decode by default; brotli/zstd opt in with --request-decompression-extra. Bounded and buffered; see Compression.
• Bandwidth pacing — a per-connection token bucket (burst 2× rate) paces Body::Full as 64 KiB slices and streams per chunk when bandwidthLimit is set; a matched context overrides the server-wide rate.
• Explicit CONNECT rejection (405) and stalled-sendfile backoff with a 60 s zero-progress bound.

Native HTTP/2

hj-h2 is httpjet's own HTTP/2 server. Every h2 connection — ALPN h2 over TLS and cleartext h2c — lands here through the monoio local serve path. The design targets the CDN→origin shape: few long-lived connections, many concurrent streams, so it optimizes one connection's multiplexing rather than spreading work across many.

• One task per connection. All streams multiplex through FuturesUnordered; a 100-stream connection costs one task.
• Send-side flow control honors per-stream and connection windows; frame and body deadlines are enforced on the monoio loop (parity with the tokio loop landed 2026-09-16).
• Streaming bodies — LSAPI, proxy, SSE and async file bodies stream without buffering; the response-completion hook keeps spans open through the final flush.
• One vectored write per wake; large bodies are zero-copy from page-cache Bytes. The vendored monoio-rustls fork adds a real writev so the flush lands in rustls with one plaintext copy.
• Inbound RFC 9113 validation with graceful GOAWAY drain; CONTINUATION without an open header block and stream-id reuse are connection errors; the HTTP/2 Bomb pattern (CVE-2026-49975) and no-forward-progress cases are bounded.
• Egress pacing is per connection at the OutQueue flush with the server-wide rate (a per-response override is ill-defined under multiplexing). HTTP/3 is not paced.

HPACK

Header compression is RFC 7541 HPACK, implemented as the hpack submodule of hj-h2 (once a standalone crate, folded in so the codec and the connection state evolve together). It is zero-allocation — fields are decoded and encoded by slicing frame bytes in place — and pure safe Rust with #![forbid(unsafe_op_in_unsafe_fn)]. Both the decoder and the round-trip are fuzz targets (Conformance).

HTTP/3 & QUIC

HTTP/3 is uring/h3.rs: quinn-proto for QUIC state, quinn-udp for socket metadata, UDP driven from monoio/io_uring, QPACK encode/decode, and the same bridge into the pipeline. Small, cached and sub-threshold bodies are buffered and sent whole; large, SSE and proxy bodies stream — HEADERS goes out immediately and DATA follows as the backend produces it, with backpressured writeback. Retained response memory is bounded per connection (2026-09-06). The process-wide connection budget (shared with TCP accepts) is charged at Connected — after address validation — so spoofed Initials cannot hold budget for the idle timeout; until then a connection costs only one of 256 per-core handshake slots, and once 64 connections on a core are handshaking an unvalidated Initial receives a stateless Retry (token-bearing Initials are still accepted). A full budget closes with H3_EXCESSIVE_LOAD; handshakes are closed after 10 s.

Discovery is via Alt-Svc on TLS responses (the last response transform), so an h2 client learns it can upgrade to QUIC. Connection migration is disabled because listener steering does not yet preserve ownership across an address change. Each connection pins its trust generation and mTLS requirement at acceptance; a trust reload replaces quinn's server configuration only for future handshakes while the bound UDP workers and established connection IDs survive.

File egress: sendfile & kTLS

An uncached Body::File keeps its pinned inode and byte range across the tokio→monoio bridge. On plain TCP HTTP/1 it is sent with bounded nonblocking sendfile(2); unsupported descriptor/filesystem combinations resume at the exact offset with positional io_uring reads, and short files or client aborts close the connection rather than completing an incorrect Content-Length. A stalled sendfile backs off exponentially and gives up after 60 s of zero progress.

A --features ktls build extends this to TLS 1.3 H1: after the rustls handshake the socket is upgraded to kernel TLS, so the same sendfile writer feeds ciphertext without a userspace AEAD copy. A peer KeyUpdate queued during a transfer is applied before the next response writes, fencing TX rekey from sendfile. Policy is --ktls=auto|on|off: auto (default in feature builds) resolves the IPv4 default-route interface, reads its driver and features via ethtool ioctls, refuses known virtual NICs, and enables kTLS only when tls-hw-tx-offload is active — otherwise it logs one reason and stays on rustls. TLS 1.2, H2, H3 and UDS keep their userspace paths.

TLS in one paragraph

hj-tls builds one rustls::ServerConfig for TCP and QUIC with aws-lc-rs as the crypto provider. SNI maps directly to configured chains (no cert-name-match, like OLS), with dual ECDSA+RSA chains per name; client verification is optional mTLS against the listener CA with CRL enforcement; TLS 1.3 offers exactly TLS_AES_128_GCM_SHA256; resumption uses single-use tickets with 0-RTT off. Certificates, ACME and OCSP have their own page: TLS, ACME & OCSP.