Operations & deployment

Installation

No signed packages are published yet; build from source and install the binary at a stable path outside Cargo's target/ so a later cargo clean cannot delete what a running unit depends on.

cargo build --locked --release -p httpjet
sudo useradd --system --home-dir /var/lib/httpjet --no-create-home --shell /usr/sbin/nologin httpjet
sudo install -d /usr/local/httpjet/bin /usr/local/httpjet/conf /usr/local/httpjet/logs
sudo install -m0755 target/release/httpjet /usr/local/httpjet/bin/httpjet

# an existing LiteSpeed tree works as-is — validate it first, as root AND as the service user
sudo /usr/local/httpjet/bin/httpjet --root /usr/local/lsws check --strict
sudo -u httpjet /usr/local/httpjet/bin/httpjet --root /usr/local/lsws check --strict

# minimal loopback example unit (no TLS, no PHP, no cache)
sudo install -m0644 packaging/systemd/httpjet-example.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now httpjet-example

The systemd model

flowchart TB
  classDef sock fill:#0e1424,stroke:#22d3ee,color:#e8ecf6
  classDef svc fill:#13203a,stroke:#818cf8,color:#fff
  classDef gate fill:#1a1f33,stroke:#fbbf73,color:#e8ecf6
  BR["httpjet-boot-ready.service<br/>oneshot gate: waits for the LSAPI pool (hard), DB/Redis (soft, 300 s)"]:::gate
  LS["httpjet-lsphp.socket<br/>owns the LSAPI unix listener · backlog 8192 · 0600"]:::sock
  LSV["httpjet-lsphp.service<br/>httpjet lsphp · Type=notify · ExecReload = lsphp-reload"]:::svc
  S["httpjet.socket<br/>N× :80 TCP · N× :443 TCP · N× :443 UDP · ReusePort"]:::sock
  W["httpjet.service<br/>httpjet serve --lsphp-external … · User=nobody · Restart=always"]:::svc
  LS --> LSV
  LSV --> BR
  BR --> S
  S --> W
  LSV -. "Wants / After" .-> W
  W -- "systemctl restart: fds stay open, PHP stays warm" --> W
  LSV -- "systemctl reload: candidate-first generation swap" --> LSV

The persistent PHP pool

Static and proxy deployments need no PHP. For LSAPI, run the pool as its own unit and point the web tier at it; the units above wire the socket, the root-only control socket and the 16-byte generation file. Routine opcache recycle is systemctl reload httpjet-lsphp.service — synchronous and candidate-first. Reserve a hard restart of the pool service for supervisor binary/unit changes; verify the listener inode is unchanged before and after. Reference: LSAPI & PHP.

before=$(stat -Lc '%d:%i' /run/httpjet/php.sock)
systemctl daemon-reload
systemctl restart httpjet-lsphp.service                 # the SERVICE only — never the .socket
test "$(stat -Lc '%d:%i' /run/httpjet/php.sock)" = "$before"
test "$(stat -Lc %s /run/httpjet/lsphp.generation)" = 16
ss -xlnH | awk '$5 == "/run/httpjet/php.sock" {print $4}'    # Send-Q should equal the XML backlog

Safe change workflow

1. Edit in a reviewable change

   Configuration tree or application content.

2. check --strict

   The deploy gate. SIGHUP with a broken config keeps the old generation, but the check should catch it first.

3. Test in isolation

   Alternate ports, a separate PHP socket, a separate cache store path, low worker/child counts. Kill test instances by exact PID; a leaked SO_REUSEPORT instance silently shares the port with independent cache and htaccess state.

4. Apply

   systemctl reload httpjet (SIGHUP) for vhost/rewrite/access/expires/tuning/certificate changes; systemctl restart httpjet for the binary or boot-owned resources; the write API for validated in-memory replacement.

5. Verify on loopback, then watch

   curl -sk -H 'Host: …' https://127.0.0.1/…, the journal, httpjet_requests_total and 5xx counters. Purge any outer cache if an incorrect cacheable response reached it.

Binary deploy and rollback

flowchart TB
  classDef gate fill:#1a1f33,stroke:#fbbf73,color:#e8ecf6
  classDef ok fill:#0f2a24,stroke:#34d399,color:#fff
  classDef bad fill:#2a1420,stroke:#e879f9,color:#fff
  A["cargo build --release -p httpjet<br/>(or scripts/build-pgo.sh)"] --> B["httpjet --root … check --strict"]:::gate
  B -- findings --> STOP[fix config first]:::bad
  B -- clean --> C["install to a stable path outside target/<br/>keep the previous binary beside it"]
  C --> D["systemctl restart httpjet<br/>(socket activation keeps :80/:443 open; PHP pool untouched)"]
  D --> E["health probe on loopback<br/>curl -sk -H 'Host: …' https://127.0.0.1/"]:::gate
  E -- fail --> RB["restore previous binary → systemctl restart"]:::bad
  E -- ok --> F["cache gates (only with --page-cache)<br/>cache_xurl · cache_scheme · cache_xhost · cache_private · cache_persist"]:::gate
  F -- fail --> RB
  F -- ok --> G["purge probe: POST /__hj_cache_purge → 204"]:::gate
  G --> H["done — watch httpjet_requests_total, 5xx, hit ratio"]:::ok
  RB --> I["if wrong content reached the CDN: purge the edge too"]

The production build uses profile-guided optimization (scripts/build-pgo.sh: instrument → drive a representative loopback workload → rebuild). PGO scratch data is kept off tmpfs. Never benchmark a non-PGO binary against a PGO one.

What SIGHUP reloads

Vhost maps, rewrites, document roots, contexts, access/expires/tuning settings, cache policy gates, and TLS certificates on an unchanged listener topology. A change that needs a restart (listener addresses, QUIC enablement, PHP topology, cache-store identity, privileges, log paths) is logged as "requires a restart — keeping current config" and nothing changes. A reload that would 404 a mapped vhost because its per-vhost file failed to load is refused. The dictionary registry is retained across SIGHUP (rotation needs a restart). Full semantics: Admin API & reload.

Observability checklist

curl -s http://127.0.0.1:9090/metrics | grep '^httpjet_'          # Prometheus text
curl -s http://127.0.0.1:9090/cache-entries                        # cache totals, budgets, health line
curl -s http://127.0.0.1:9091/v1/status | jq .                     # with --admin-addr: generation, fingerprint, upstream health
curl -sk https://127.0.0.1/__hj_cache_ready -o /dev/null -w '%{http_code}\n'
journalctl -u httpjet -n 100

Metric families, log formats, the request-id header, telemetry TSV, PHP slow log, OpenTelemetry and profiling are all on Observability. Pair the counters with a process monitor (the tree ships a monit check whose start timeout stays above the boot gate's worst case).

Hardened OCI example

packaging/oci/ is an adoption example, not a prescription. The image runs as numeric UID/GID 10001:10001, binds unprivileged ports, needs no capabilities; the compose file adds a read-only root filesystem, drops every capability, sets no-new-privileges, and mounts a small noexec,nosuid,nodev /tmp. Because Moby's default seccomp allowlist omits io_uring, seccomp-httpjet.json is the pinned Moby default plus only io_uring_setup, io_uring_enter and io_uring_register (ptrace/process_vm_* gated on CAP_SYS_PTRACE).

docker compose -f packaging/oci/compose.yaml up --build     # publishes 8080/tcp, 8443/tcp AND 8443/udp
docker run … --security-opt seccomp=/abs/path/seccomp-httpjet.json …   # a hand-written run must pass the profile explicitly
bash scripts/oci-smoke.sh                                    # read-only root, UID 10001, caps dropped, real HTTP/3 through the UDP port

A seccomp denial is a startup/test failure — do not work around it with --privileged, --cap-add or seccomp=unconfined; narrow the runtime policy or pick a supported host. Rootless engines may add memlock, UDP or io_uring restrictions of their own. The image health check calls /__hj_cache_ready, so readiness means the cache boot scan finished, not that a user URL was warmed.

Release and upgrade discipline

A public release must provide portable x86_64 and aarch64 artifacts, checksums, signatures or attestations, an SBOM, the GPL-3.0 license text and a documented Linux/kernel/libc floor; architecture-tuned PGO helpers must not be uploaded as generic binaries. Upgrades preserve the tree, dictionaries and state; stage and validate the new binary before restart; keep a known-good binary. Dictionary rotation: write the new file atomically, validate, restart, expect old-generation cache entries to become safe misses. Dependency upgrades run through scripts/update-deps.sh and the full suite (the 0.0.5 sweep refreshed aws-lc-rs, zstd and opentelemetry, replaced the unmaintained rustls-pemfile and fxhash, and lifted the vendored monoio forks off socket2 0.5/mio 0.8; a daily dependency-advisory job now fails the build on any RustSec finding).