Architecture

Workspace shape

Responsibilities are split so the contract a handler depends on stays tiny while transports, protocols, gateways and the cache evolve behind it. Four libraries are deliberately isolated — hj-rewrite, hj-log, hj-geo and hj-cache take no hj-* dependency, so their logic is portable and testable on its own. hj-pagecache is transport-agnostic and builds on hj-cache's sharded LRU; its request-aware glue lives in the binary.

Crate dependency graph

hj-config parses the LiteSpeed XML tree into one normalized model; hj-core re-exports it so every downstream crate imports the configuration model through core — one import surface, never inverted. The binary is the only crate that sees everything.

Crate map

Transport paths

The pure io_uring transport is the only serving transport. There is no runtime switch and no URING=0 rollback build; reverting is a rebuild from an older revision. One monoio runtime runs per inherited or self-bound SO_REUSEPORT listener fd, pinned to its core when the worker count equals the CPU count (--no-core-pinning is the kill switch).

flowchart TB
  classDef core fill:#13203a,stroke:#818cf8,color:#fff
  classDef k fill:#0a0d17,stroke:#2c3856,color:#c2cadd
  subgraph sockets ["listeners: systemd socket activation or self-bound, SO_REUSEPORT ×N workers"]
    direction LR
    tcp80["TCP :80"]:::k
    tcp443["TCP :443"]:::k
    udp443["UDP :443"]:::k
    uds["unix:/path"]:::k
  end
  subgraph cores ["one monoio io_uring runtime per listener fd, pinned to its core"]
    direction TB
    pp["PROXY protocol strip (per-listener opt-in)"]
    tls["monoio-rustls handshake · ALPN · optional kTLS"]
    sniff["cleartext sniff: 24-byte h2 preface?"]
    h1["H1 codec (httparse, sendfile)"]
    h2["hj-h2 · one task per connection"]
    quic["quinn-proto QUIC driver · HTTP/3 · QPACK"]
    memo["on-core fast path: finished-response memo · static · cache hit"]
  end
  tcp80 --> pp
  tcp443 --> pp
  uds --> sniff
  udp443 --> quic
  pp -- plain --> sniff
  pp -- secure --> tls
  sniff -- "http/1.x" --> h1
  sniff -- "h2c" --> h2
  tls -- "http/1.1" --> h1
  tls -- "h2" --> h2
  h1 --> memo
  h2 --> memo
  quic --> memo
  memo -- "miss or dynamic" --> bridge["uring::bridge — hand the request to the tokio runtime"]:::core
  bridge --> pipe["pipeline::handle() · ServerState · LSAPI pool · page cache · reload · metrics"]:::core
  pipe -- "response: buffered ≤ 256 KiB, streamed above" --> bridge

Ring setup is tunable but deliberately plain: --uring-ring legacy (default) uses flagless rings because the measured coop and defer variants were noise or a ~2% loss on the virtualized production guest; ring size stays at the default because ring memory is memlock-charged. Multishot accept is on by default (--no-multishot-accept reverts to one SQE per connection).

The frozen contract — hj-core

This is the seam that lets handlers compose without knowing the transport. Changing these signatures means updating every dependent, so they are treated as frozen:

enum Body { Empty, Full(Bytes), Stream(BoxBody), File(FileBody) }
type Request  = http::Request<IncomingBody>;
type Response = http::Response<Body>;

// every terminal handler (StaticFiles, Lsapi, FastCgi, ProxyHandler)
trait Handler { async fn handle(&self, &mut ReqCtx, Request) -> Result<Response, HandlerError>; }

// every post-handler munger (promote → expires → sub_filter → compress → deny-CDN-cache → Alt-Svc)
trait ResponseTransform { async fn transform(&self, &ReqCtx, &mut Response); }

FileBody may carry an open descriptor that pins the selected inode across a concurrent cache version swap, recompression or unlink — that is what lets sendfile and the file tier coexist safely. ReqCtx carries the per-request state: server/vhost (Arcs of the pinned generation), peer_ip/client_ip (post-forwarding, or peer_unix), is_tls, protocol, trusted_proxy, and an env vec that holds rewrite [E=] sets and CGI overrides. Router::resolve(listener, key) maps SNI/Host to a vhost.

The request pipeline

pipeline::handle() is the integration point: vhost resolution → mTLS trust boundary → client-IP/ACL/geo/throttle → optional WAF inspection → extension pre-handlers → dispatch() → response transforms → access log, metrics and spans. dispatch() mirrors OpenLiteSpeed's ordering with the page-cache seams inserted once rewrite and access state are known:

The terminals are intentionally not a uniform matcher registry. Their predicates differ (request headers for WebSocket, URI prefix for proxy contexts, script-path split for suffix routing), their fall-through rules differ, and their post-processing differs, so an explicit if/return chain keeps the security-relevant ordering legible. Two fall-throughs matter: a proxy context that cannot resolve its ext-processor falls through to static, while a script whose handler has no pool hard-fails 503 and must never fall through — falling through would serve the raw PHP source.

The cache seams are inert unless serve runs with --page-cache: lscache::cache_lookup runs after rewrite/ACL but before any backend (a hit short-circuits dispatch), and lscache::cache_store runs after finalize_response at each terminal return. Both key off a single CacheCtx built once per dispatch.

Module organization

Configuration generations

Live state is published as immutable generations behind an ArcSwap. A SIGHUP or an API transaction builds a complete candidate (config, rewrite caches, health schedulers) with background work inactive, validates it, then publishes it atomically under one coordinator; requests already in flight finish on the generation they pinned at acceptance.

stateDiagram-v2
  direction LR
  [*] --> Candidate: SIGHUP · PUT /v1/config
  Candidate --> Rejected: parse error · restart_required · stale revision · missing vhost file
  Candidate --> Published: exact revision recheck passes
  Rejected --> [*]: serving generation untouched
  Published --> Serving
  Serving --> Retiring: a newer generation publishes
  Retiring --> [*]: in-flight requests finish on their pinned generation
  note right of Published
    caches keyed by epoch + generation:
    old renders cannot overwrite new entries
  end note

The full ownership inventory and the transactional replacement sequence are on Admin API & reload.

Code conventions

• Formatting: rustfmt-clean (default settings, edition 2024); cargo fmt --all --check is a CI gate.
• Errors: thiserror types at system boundaries; internal invariant violations panic with a message rather than pretending to recover.
• unsafe: rare, and never in HPACK. The real sites are hj-h2's pin projection and writev/iovec plumbing, the validated fixed-size mmaps in hj-pagecache/diskstore.rs and hj-lsapi/pool.rs, and small libc shims — each with its safety invariant documented above the block.
• Async: tokio tasks for independent work, FuturesUnordered for multiplexing with coordinated shutdown, no per-stream spawn on hot paths.
• Hashing: rustc-hash only for internal maps keyed by trusted values (h2 stream ids); anything keyed by untrusted input keeps SipHash.
• Allocator: mimalloc globally, with a background mi_collect trim that only runs above an RSS threshold (--mimalloc-trim-*).