LLMhop
One port, many models: A tiny, stateless HTTP router for OpenAI-compatible LLM inference backends.
LLMhop reads the model field of an incoming request and reverse-proxies it to the matching backend.
It is built for single-model servers like vLLM and SGLang, but works with any OpenAI-compatible backend, self-hosted or remote.
- Pure Go, no third-party packages, no CGO, stateless and safe behind any load balancer.
- Body and path routing, a models API served from the config, health and readiness endpoints, graceful draining.
- Ships as a static binary, a Docker image and a hardened NixOS module running llama.cpp, SGLang or vLLM behind llmhop, SMG, Bifrost or no gateway.
How it works
LLMhop looks up the request’s model in its config and forwards the request verbatim.
Unknown models return 404, invalid JSON or a missing model return 400.
GET /v1/models and GET /v1/models/{model} are answered from the config and never proxied.
Model names may contain /, as in Hugging Face names such as org/name.
A request under /route/{model}/ is routed by path instead, with the prefix stripped and the body left unparsed, so POST /route/production/detect reaches production as POST /detect.
The longest configured model name that prefixes the path wins, and a / in a model name may be written raw or as %2F.
A model marked "unlisted": true is routed but left out of the models API, which suits services such as a watermark detector.
GET /health returns { "status": "ok", "models": 3 } with the number of listed models.
GET /readiness probes the healthPath of every model that sets one and answers 200 only if all respond 2xx, else 503.
Both never require a token.
LLMhop sends READY=1 to systemd once its listeners are bound, and drains in-flight requests on SIGTERM or SIGINT.
Errors
Every error is JSON in the shape of OpenAI, as in {"error": {"message": "...", "type": "not_found_error", "code": "model_not_found"}}.
| Status | Meaning |
|---|---|
400 | invalid JSON, a missing model or an unreadable body |
401 | missing or invalid bearer token, with WWW-Authenticate: Bearer |
404 | unknown model |
408 | the body did not arrive within a minute |
413 | the body exceeds maxBodyBytes |
429 | the model reached maxConcurrentRequests, with Retry-After: 1 |
502 | the backend is unreachable |
503 | the maxBufferedBytes budget is exhausted, with Retry-After: 1 |
Configuration
{
"listen": ["127.0.0.1:8080", "unix:///run/llmhop/http.sock"],
"authTokens": { "client": "${cred:llmhop.client-token}" },
"models": {
"llama-3-8b": { "url": "http://localhost:30000", "healthPath": "/health" },
"meta-llama/Llama-3.1-8B": { "url": "http://localhost:30001" },
"qwen3-8b": {
"url": "unix:///run/llmhop/vllm-qwen3-8b/http.sock",
"maxConcurrentRequests": 16
},
"openai-gpt-4o": {
"url": "https://api.openai.com",
"headers": { "Authorization": "Bearer ${cred:openai-key}" }
}
}
}
listen holds host:port addresses, with [::1]:8080 for IPv6 and :8080 for every interface, or unix:///<socket path>.
It has no default, and serve uses the sockets of systemd socket activation when present, else listen.
Setting both or neither is an error, and check does not require listen.
A url is an absolute http(s) URL or unix:///<socket path>.
With authTokens, a map of names to tokens, set, every route but /health and /readiness requires a matching bearer token, which is stripped before forwarding.
authTokenCredentials, a systemd credential glob with a single trailing * as in ImportCredential=, adds every matching credential as a token named by the part the * matched, read at startup, and keeps tokens required while none matches.
A model’s headers are applied last, so a configured Authorization replaces the client’s.
Unknown keys are rejected, and llmhop check config.json validates a config without binding a port or expanding secrets.
Secret references
Token values in authTokens and strings in models.*.headers are expanded at startup, and an unresolved reference is a startup error:
${cred:name}: the systemd credentialnamefrom$CREDENTIALS_DIRECTORY.${env:NAME}: an environment variable.${file:/absolute/path}: a file, with trailing newlines trimmed.$$: a literal$.
Request limits
maxBodyBytes(100 MiB): larger bodies get413.maxBufferedBytes(1 GiB): bytes of body-routed requests buffered at once, further ones get503withRetry-After: 1. A body is buffered until the backend starts its response, and must arrive within a minute, or the request gets408.models.*.maxConcurrentRequests(unlimited): proxied requests per model, further ones get429withRetry-After: 1.
0 disables a limit.
Running
llmhop serve config.json
nix run github:mirkolenz/llmhop -- serve config.json
docker run --rm -p 8080:8080 -v ./config.json:/config.json ghcr.io/mirkolenz/llmhop
llmhop version prints the version.
The image runs serve /config.json, so the mounted config must set "listen": [":8080"].
NixOS and system-manager modules
Add the flake input github:mirkolenz/llmhop and import llmhop.nixosModules.default on NixOS or llmhop.systemManagerModules.default on other distributions with system-manager:
services.llmhop = {
gateway.type = "llmhop";
gateway.llmhop.listen.default.host = "";
gateway.llmhop.listen.default.openFirewall = true;
llama-cpp.enable = true;
llama-cpp.models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};
Both include quadlet-nix for containers, which every backend and gateway selects with runtime = "native" or "quadlet", defaulting to the global services.llmhop.runtime, "native".
Container setups can also be bundled for hosts without Nix.
With system-manager, the host provides the configuration of podman in /etc/containers.
See the options reference for every option and the module internals for the design rationale.
Gateways
services.llmhop.gateway.type selects at most one gateway, configured by the option set of the same name, which routes every model of every backend by its name.
The gateway runs only while some backend serves a model, which the read-only services.llmhop.gateway.active reports.
type | Gateway | Unix sockets | Detectors |
|---|---|---|---|
null | none, clients connect to each model | ||
"llmhop" | llmhop | native only | yes |
"smg" | the Shepherd Model Gateway | no | no |
"bifrost" | Bifrost | no | no |
Each gateway runs natively or as a container, as selected by gateway.<type>.runtime.
Routed models must stay on loopback and leave authentication to the gateway, and gateways without socket support need every model to set a port.
SMG and Bifrost listen on 127.0.0.1:8080 by default, configured through host, port and openFirewall.
- llmhop runs as the
llmhopuser with one socket-activatedllmhop-<name>.socketper entry oflisten, as inlisten.caddy.socketGroup = "caddy"for a unix socket, or as a container receiving the same sockets, which loads an image archive of the same build, unlesscontainer.imagenames a registry reference such asghcr.io/mirkolenz/llmhop:latest. Thedefaultlistener serves127.0.0.1:8080andenvironmentandenvironmentFileare forwarded to the service. Its generated config is validated at build time withllmhop check. Reference secrets insettingsas${cred:name}and grant them throughcredentials.settings.authTokenCredentials = "llmhop.token.*"imports the tokens of/etc/credstore/llmhop.token.<client>, or encrypted ones of/etc/credstore.encrypted, so clients come and go without a rebuild by adding or deleting their file. Natively,systemctl reload llmhopthen re-imports the credentials and swaps the tokens without a restart, while a container needssystemctl restart llmhop, whose sockets hold new connections meanwhile. - SMG reads its workers from a generated discovery file.
settingstakes its CLI flags, where${cred:name}resolves to a file path, as insettings.api-key-file = "\${cred:smg.api-key}", which requires an SMG build newer than v1.13.0. The nativepackageis amkUvEnvenvironment. - Bifrost gets every model as a key of its
vllmprovider, orsglfor SGLang.settingsis itsconfig.json, with secrets asenv.NAMEreferences loaded fromenvironmentFile. The nativepackagecomes from Bifrost’s own flake.
Without a gateway, each model serves clients on its own socket or port, and socketClients lists the users, such as a reverse proxy, that may connect to every model socket.
Inference backends
Each backend serves the entries of its models as isolated workers, and all backends can run side by side:
services.llmhop = {
llama-cpp = {
enable = true;
models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};
vllm = {
enable = true;
package = inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; };
models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 19001;
settings.reasoning-parser = "qwen3";
};
};
sglang = {
enable = true;
runtime = "quadlet";
container.image = "docker.io/lmsysorg/sglang:latest";
models."llama-3-8b" = {
model = "meta-llama/Meta-Llama-3-8B-Instruct";
port = 19002;
};
};
};
Each model runs as the unit <engine>-<name>.
Native workers are hardened services running as the backend’s user, which defaults to the engine name, while container workers run rootful, with user defaulting to root, through quadlet-nix with the backend’s container.image, managed with quadletctl.
Switching a backend’s runtime therefore changes the owner of its cache directories, but not of the files a previous fetch left in them, so a native fetcher cannot update a cache filled by a container until chown -R hands it to the new user and group, or the cache is deleted for a fresh download.
A worker stays activating until it answers its health endpoint, and startupOrdering, enabled by default for every backend, starts workers one at a time by name, so two never profile or size their GPU memory at once.
Evaluation rejects any port, unit or model name claimed twice.
Customize the generated units directly, as in systemd.services.vllm-qwen3-8b.serviceConfig.MemoryHigh = "64G" or virtualisation.quadlet.containers.vllm-qwen3-8b.containerConfig.
llama.cpp and vLLM workers default to port = null, binding a unix socket below socketDirectory that only the backend’s workers, llmhop and socketClients can reach.
Set port to bind <host>:<port> instead, which SGLang always needs.
Fetching and offline servers
Each model has a fetcher <engine>-<name>-fetch that downloads everything into the model’s own cache cache.directory/<name>, after which the server runs offline from it.
A model that is an absolute path has no fetcher.
extraRepositories lists further repositories that settings reference, such as a draft model or LoRA adapter, and revision pins a branch, tag or commit:
services.llmhop.vllm.models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
revision = "main";
extraRepositories = [ "Qwen/Qwen3-0.6B" ];
settings.speculative-config = {
model = "Qwen/Qwen3-0.6B";
num_speculative_tokens = 4;
};
};
Only fetchers get the backend’s fetch.credentials, fetch.environment and the Hugging Face token of gated repositories, which fetch.hfTokenCredential = "llmhop.hf-token" imports for every engine.
<backend>-update, such as vllm-update, reruns every fetcher on the fetch.schedule (OnCalendar=), weekly by default, prunes what the model no longer references, and restarts each running server after a successful fetch, even if its model did not change.
An interrupted download leaves a running server alone, and a successful retry refreshes it.
An explicitly stopped server stays stopped.
fetch.enable = false drops the fetchers, refresh and update units, so each server downloads its repositories itself when it starts, which needs offline = false, its default then.
A container backend can then also set cache.directory = null, keeping each cache in the named volume <engine>-<name> without tmpfiles rules, and container.imageUnit = false, so each container references container.image directly rather than through the image unit <engine>-image.
Together they leave one unit per model, as in the bundle packages.x86_64-linux.vllm-bundle of nix/bundles/vllm.nix.
With fetchers, servers never download from the hub: they get HF_HUB_OFFLINE, and with offline = true, the default, no network.
A native server on a socket gets no network at all and on a port only loopback, while a container gets Network=none.
Only a socket or a native loopback host can be isolated like this, so any other workload defaults to offline = false and rejects offline = true.
offline = false lifts the network restrictions for features like media URLs in requests.
Tagged container images are labeled AutoUpdate=registry for virtualisation.quadlet.autoUpdate, and are otherwise only pulled while missing, so a start needs no registry.
Settings rendering
settings, over the backend’s modelSettings, becomes the server’s CLI flags, while the listener and the served name always win.
Strings and paths are passed verbatim and everything else as JSON, so attribute sets work for options like --speculative-config.
truebecomes--<key>, whilenulland empty lists are dropped.- llama.cpp and vLLM render
falseas--no-<key>, while SGLang drops it, so write the negated key, as indisable-radix-cache = true. - vLLM and SGLang pass a list to one flag (
--<key> a b), llama.cpp repeats the flag (--<key> a --<key> b).
Containers and user namespaces
Every container backend and gateway runs as root by default, which a user namespace maps to an unprivileged host range.
container.config adds [Container] keys to every container of a backend, servers, fetchers and detectors alike, or of a gateway, and container.mountOptions adds idmapped mounts let the namespace own the cache and read the credentials:
services.llmhop = {
vllm = {
runtime = "quadlet";
container = {
config.UserNS = "auto:size=65536";
mountOptions = {
cache = [ "idmap" ];
credentials = [ "idmap" ];
};
};
};
gateway.llmhop = {
runtime = "quadlet";
container = {
config.UserNS = "auto";
mountOptions.credentials = [ "idmap" ];
};
};
};
UIDMap= and GIDMap= work the same way in place of UserNS=.
A non-root User= inside the namespace takes the mapping idmap=uids=0-<uid>-1;gids=0-<gid>-1 in both lists, so the cache stays owned by host root.
Only a cache on a filesystem without idmapped mounts, such as NFS, needs the backend’s user and group set to the host IDs that an explicit UIDMap= and GIDMap= give the container user.
Rootful UserNS=auto needs subordinate IDs for a containers user, which NixOS does not declare.
Hosts without Nix
The container backends and gateways can be rendered into plain files for any host with podman 5 and systemd, such as Ubuntu 25.04:
packages.x86_64-linux.llmhop-bundle = inputs.llmhop.legacyPackages.x86_64-linux.mkQuadletBundle {
modules = [
{
services.llmhop = {
gateway.type = "llmhop";
vllm = {
enable = true;
container.image = "docker.io/vllm/vllm-openai:latest";
models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 8001;
};
};
};
}
];
};
It is the bundle of quadlet-nix named llmhop, which every system importing llmhop’s modules exposes as virtualisation.quadlet.bundle.package, with every rootful Quadlet unit of that system.
It defaults services.llmhop.runtime to "quadlet".
The bundle holds the Quadlet files, update and refresh units, timers, the llmhop gateway’s socket units and tmpfiles rules below etc/, the scripts and generated configs below /usr/local/lib/llmhop, and an install.sh that copies them into place, starts the servers, restarts every running unit it changed and retires those a previous bundle had but this one lacks.
The uninstall.sh installed beside the assets removes the bundle again, keeping the model caches.
Native backends and gateways cannot be bundled, and firewall and user options have no effect there.
Native vLLM and SGLang from prebuilt wheels
Native vLLM and SGLang run from upstream’s wheels, pinned in a small uv workspace and built with mkUvEnv:
# vllm-env/pyproject.toml, then run `uv lock`
[project]
name = "vllm-env"
requires-python = "==3.12.*"
dependencies = ["vllm==0.16.2"]
mkUvEnv builds against the lowest admitted Python and bakes the GPU driver runpath into the wheels, while the driver itself comes from hardware.graphics and your vendor configuration.
models.<name>.package overrides the package per model, for example with a nightly.
- JIT compilers find
nvccthroughCUDA_HOME, whichmkCudaHome { packages = [ ... ]; }builds from thecudaPackages_*set named by the environment’spassthru.cudaPackagesAttr. - AMD and Intel GPUs work with vLLM’s ROCm or XPU wheels, added as an explicit
[[tool.uv.index]]per vLLM’s installation docs. - sdists without a build system need their build dependencies under
[tool.uv.extra-build-dependencies], as inantlr4-python3-runtime = ["setuptools"]. - flashinfer packages must be pinned to the
flashinfer-pythonversion vLLM pins, or flashinfer refuses to start. - Native libraries come from
buildInputsfor linked ones,runtimePathsfor ones loaded by a baredlopen, such aslibsndfileorrdma-core,driverLibsfor host driver sonames andoptionalLibsfor libraries that load on demand only. The build fails naming every unresolved library.
Inference server credentials
Credentials are granted per model, so no worker can read another’s keys.
credentials.<name> maps a name to a file outside the Nix store, or to null to import it from the system credential store, and ${cred:name} in settings or environment expands to its read-only path:
services.llmhop.llama-cpp.models."qwen3-8b" = {
model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
credentials.api-keys = "/run/secrets/qwen-api-keys";
settings.api-key-file = "\${cred:api-keys}";
};
Provision an imported credential as a root-only file or encrypted:
(umask 077; systemd-ask-password -n > /etc/credstore/llmhop.hf-token)
systemd-ask-password -n | systemd-creds encrypt --name=llmhop.hf-token - /etc/credstore.encrypted/llmhop.hf-token
vLLM and SGLang read a YAML file with api-key through settings.config, as in credentials."config.yaml" with settings.config = "\${cred:config.yaml}", while llama.cpp uses api-key-file.
Containers see their credentials at /run/llmhop/credentials, named by $CREDENTIALS_DIRECTORY as natively, and readable through a user namespace with the backend’s or gateway’s container.mountOptions.credentials, see user namespaces.
vLLM watermarking
Both vLLM backends can watermark a model’s output and run detectors serving POST /detect beside the workers.
A watermark holds vLLM’s WatermarkConfig without the key as settings, plus the credential holding the key, and a detector tests every one of its profiles:
services.llmhop.vllm = {
models.qwen = {
model = "Qwen/Qwen3-8B";
revision = "main";
watermark = {
credential = "vllm.watermark-key.2026-10";
settings.algorithm = "dual_key_gumbel";
};
};
detectors.production = {
profiles.qwen = {
tokenizer = "Qwen/Qwen3-8B";
inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
};
settings.p-value-threshold = 0.01;
};
};
A key is an unsigned 64-bit integer credential, never entering the Nix store or a command line:
(umask 077; od -An -N8 -tu8 /dev/urandom | tr -d ' \n' > /etc/credstore/vllm.watermark-key.2026-10)
A detector reports the most significant profile with its Bonferroni-corrected p_value, score, num_scored_tokens and is_watermarked.
To rotate a key, provision it under a new credential name and keep a profile with the old one while old text still needs checking.
llmhop routes detectors as unlisted models, as in POST /route/production/detect, while other gateways leave them to direct clients.
Watermarking requires vLLM 0.30.0 or newer.