LLMhop
One port, many models: A tiny, stateless HTTP router for OpenAI-compatible LLM inference backends.
LLMhop reads the model field of an incoming request and reverse-proxies it to the matching backend.
It is built for single-model servers like vLLM and SGLang, but works with any OpenAI-compatible backend, self-hosted or remote.
- Pure Go, no third-party packages, no CGO, stateless and safe behind any load balancer.
- Body and path routing, a models API served from the config, health and readiness endpoints, graceful draining.
- Ships as a static binary, a Docker image and a hardened NixOS module running llama.cpp, SGLang or vLLM behind llmhop, SMG, Bifrost or no gateway.
How it works
LLMhop looks up the request’s model in its config and forwards the request verbatim.
Unknown models return 404, invalid JSON or a missing model return 400.
GET /v1/models and GET /v1/models/{model} are answered from the config and never proxied.
Model names may contain /, as in Hugging Face names such as org/name.
A request under /route/{model}/ is routed by path instead, with the prefix stripped and the body left unparsed, so POST /route/production/detect reaches production as POST /detect.
The longest configured model name that prefixes the path wins, and a / in a model name may be written raw or as %2F.
A model marked "unlisted": true is routed but left out of the models API, which suits services such as a watermark detector.
GET /health returns { "status": "ok", "models": 3 } with the number of listed models.
GET /readiness probes the healthPath of every model that sets one and answers 200 only if all respond 2xx, else 503.
Both never require a token.
LLMhop sends READY=1 to systemd once its listeners are bound, and drains in-flight requests on SIGTERM or SIGINT.
Errors
Every error is JSON in the shape of OpenAI, as in {"error": {"message": "...", "type": "not_found_error", "code": "model_not_found"}}.
| Status | Meaning |
|---|---|
400 | invalid JSON, a missing model or an unreadable body |
401 | missing or invalid bearer token, with WWW-Authenticate: Bearer |
404 | unknown model |
408 | the body did not arrive within a minute |
413 | the body exceeds maxBodyBytes |
429 | the model reached maxConcurrentRequests, with Retry-After: 1 |
502 | the backend is unreachable |
503 | the maxBufferedBytes budget is exhausted, with Retry-After: 1 |
Configuration
{
"listen": ["127.0.0.1:8080", "unix:///run/llmhop/http.sock"],
"authTokens": { "client": "${cred:llmhop.client-token}" },
"models": {
"llama-3-8b": { "url": "http://localhost:30000", "healthPath": "/health" },
"meta-llama/Llama-3.1-8B": { "url": "http://localhost:30001" },
"qwen3-8b": {
"url": "unix:///run/llmhop/vllm-qwen3-8b/http.sock",
"maxConcurrentRequests": 16
},
"openai-gpt-4o": {
"url": "https://api.openai.com",
"headers": { "Authorization": "Bearer ${cred:openai-key}" }
}
}
}
listen holds host:port addresses, with [::1]:8080 for IPv6 and :8080 for every interface, or unix:///<socket path>.
It has no default, and serve uses the sockets of systemd socket activation when present, else listen.
Setting both or neither is an error, and check does not require listen.
A url is an absolute http(s) URL or unix:///<socket path>.
With authTokens, a map of names to tokens, set, every route but /health and /readiness requires a matching bearer token, which is stripped before forwarding.
authTokenCredentials, a systemd credential glob with a single trailing * as in ImportCredential=, adds every matching credential as a token named by the part the * matched, read at startup, and keeps tokens required while none matches.
A model’s headers are applied last, so a configured Authorization replaces the client’s.
Unknown keys are rejected, and llmhop check config.json validates a config without binding a port or expanding secrets.
Secret references
Token values in authTokens and strings in models.*.headers are expanded at startup, and an unresolved reference is a startup error:
${cred:name}: the systemd credentialnamefrom$CREDENTIALS_DIRECTORY.${env:NAME}: an environment variable.${file:/absolute/path}: a file, with trailing newlines trimmed.$$: a literal$.
Request limits
maxBodyBytes(100 MiB): larger bodies get413.maxBufferedBytes(1 GiB): bytes of body-routed requests buffered at once, further ones get503withRetry-After: 1. A body is buffered until the backend starts its response, and must arrive within a minute, or the request gets408.models.*.maxConcurrentRequests(unlimited): proxied requests per model, further ones get429withRetry-After: 1.
0 disables a limit.
Running
llmhop serve config.json
nix run github:mirkolenz/llmhop -- serve config.json
docker run --rm -p 8080:8080 -v ./config.json:/config.json ghcr.io/mirkolenz/llmhop
llmhop version prints the version.
The image runs serve /config.json, so the mounted config must set "listen": [":8080"].
NixOS and system-manager modules
Add the flake input github:mirkolenz/llmhop and import llmhop.nixosModules.default on NixOS or llmhop.systemManagerModules.default on other distributions with system-manager:
services.llmhop = {
gateway.type = "llmhop";
gateway.llmhop.listen.default.host = "";
gateway.llmhop.listen.default.openFirewall = true;
llama-cpp.enable = true;
llama-cpp.models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};
Both include quadlet-nix for containers, which every backend and gateway selects with runtime = "native" or "quadlet", defaulting to the global services.llmhop.runtime, "native".
Container setups can also be bundled for hosts without Nix.
With system-manager, the host provides the configuration of podman in /etc/containers.
See the options reference for every option and the module internals for the design rationale.
Gateways
services.llmhop.gateway.type selects at most one gateway, configured by the option set of the same name, which routes every model of every backend by its name.
The gateway runs only while some backend serves a model, which the read-only services.llmhop.gateway.active reports.
type | Gateway | Unix sockets | Detectors |
|---|---|---|---|
null | none, clients connect to each model | ||
"llmhop" | llmhop | native only | yes |
"smg" | the Shepherd Model Gateway | no | no |
"bifrost" | Bifrost | no | no |
Each gateway runs natively or as a container, as selected by gateway.<type>.runtime.
Routed models must stay on loopback and leave authentication to the gateway, and gateways without socket support need every model to set a port.
SMG and Bifrost listen on 127.0.0.1:8080 by default, configured through host, port and openFirewall.
- llmhop runs as the
llmhopuser with one socket-activatedllmhop-<name>.socketper entry oflisten, as inlisten.caddy.socketGroup = "caddy"for a unix socket, or as a container receiving the same sockets, which loads an image archive of the same build, unlesscontainer.imagenames a registry reference such asghcr.io/mirkolenz/llmhop:latest. Thedefaultlistener serves127.0.0.1:8080andenvironmentandenvironmentFileare forwarded to the service. Its generated config is validated at build time withllmhop check. Reference secrets insettingsas${cred:name}and grant them throughcredentials.settings.authTokenCredentials = "llmhop.token.*"imports the tokens of/etc/credstore/llmhop.token.<client>, or encrypted ones of/etc/credstore.encrypted, so clients come and go without a rebuild by adding or deleting their file. Natively,systemctl reload llmhopthen re-imports the credentials and swaps the tokens without a restart, while a container needssystemctl restart llmhop, whose sockets hold new connections meanwhile. - SMG reads its workers from a generated discovery file.
settingstakes its CLI flags, where${cred:name}resolves to a file path, as insettings.api-key-file = "\${cred:smg.api-key}", which requires an SMG build newer than v1.13.0. The nativepackageis amkUvEnvenvironment. - Bifrost gets every model as a key of its
vllmprovider, orsglfor SGLang.settingsis itsconfig.json, with secrets asenv.NAMEreferences loaded fromenvironmentFile. The nativepackagecomes from Bifrost’s own flake.
Without a gateway, each model serves clients on its own socket or port, and socketClients lists the users, such as a reverse proxy, that may connect to every model socket.
Inference backends
Each backend serves the entries of its models as isolated workers, and all backends can run side by side:
services.llmhop = {
llama-cpp = {
enable = true;
models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};
vllm = {
enable = true;
package = inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; };
models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 19001;
settings.reasoning-parser = "qwen3";
};
};
sglang = {
enable = true;
runtime = "quadlet";
container.image = "docker.io/lmsysorg/sglang:latest";
models."llama-3-8b" = {
model = "meta-llama/Meta-Llama-3-8B-Instruct";
port = 19002;
};
};
};
Each model runs as the unit <engine>-<name>.
Native workers are hardened services running as the backend’s user, which defaults to the engine name, while container workers run rootful, with user defaulting to root, through quadlet-nix with the backend’s container.image, managed with quadletctl.
Switching a backend’s runtime therefore changes the owner of its cache directories, but not of the files a previous fetch left in them, so a native fetcher cannot update a cache filled by a container until chown -R hands it to the new user and group, or the cache is deleted for a fresh download.
A worker stays activating until it answers its health endpoint, and startupOrdering, enabled by default for every backend, starts workers one at a time by name, so two never profile or size their GPU memory at once.
Evaluation rejects any port, unit or model name claimed twice.
Customize the generated units directly, as in systemd.services.vllm-qwen3-8b.serviceConfig.MemoryHigh = "64G" or virtualisation.quadlet.containers.vllm-qwen3-8b.containerConfig.
llama.cpp and vLLM workers default to port = null, binding a unix socket below socketDirectory that only the backend’s workers, llmhop and socketClients can reach.
Set port to bind <host>:<port> instead, which SGLang always needs.
Fetching and offline servers
Each model has a fetcher <engine>-<name>-fetch that downloads everything into the model’s own cache cache.directory/<name>, after which the server runs offline from it.
A model that is an absolute path has no fetcher.
extraRepositories lists further repositories that settings reference, such as a draft model or LoRA adapter, and revision pins a branch, tag or commit:
services.llmhop.vllm.models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
revision = "main";
extraRepositories = [ "Qwen/Qwen3-0.6B" ];
settings.speculative-config = {
model = "Qwen/Qwen3-0.6B";
num_speculative_tokens = 4;
};
};
Only fetchers get the backend’s fetch.credentials, fetch.environment and the Hugging Face token of gated repositories, which fetch.hfTokenCredential = "llmhop.hf-token" imports for every engine.
<backend>-update, such as vllm-update, reruns every fetcher on the fetch.schedule (OnCalendar=), weekly by default, prunes what the model no longer references, and restarts each running server after a successful fetch, even if its model did not change.
An interrupted download leaves a running server alone, and a successful retry refreshes it.
An explicitly stopped server stays stopped.
fetch.enable = false drops the fetchers, refresh and update units, so each server downloads its repositories itself when it starts, which needs offline = false, its default then.
A container backend can then also set cache.directory = null, keeping each cache in the named volume <engine>-<name> without tmpfiles rules, and container.imageUnit = false, so each container references container.image directly rather than through the image unit <engine>-image.
Together they leave one unit per model, as in the bundle packages.x86_64-linux.vllm-bundle of nix/bundles/vllm.nix.
With fetchers, servers never download from the hub: they get HF_HUB_OFFLINE, and with offline = true, the default, no network.
A native server on a socket gets no network at all and on a port only loopback, while a container gets Network=none.
Only a socket or a native loopback host can be isolated like this, so any other workload defaults to offline = false and rejects offline = true.
offline = false lifts the network restrictions for features like media URLs in requests.
Tagged container images are labeled AutoUpdate=registry for virtualisation.quadlet.autoUpdate, and are otherwise only pulled while missing, so a start needs no registry.
Settings rendering
settings, over the backend’s modelSettings, becomes the server’s CLI flags, while the listener and the served name always win.
Strings and paths are passed verbatim and everything else as JSON, so attribute sets work for options like --speculative-config.
truebecomes--<key>, whilenulland empty lists are dropped.- llama.cpp and vLLM render
falseas--no-<key>, while SGLang drops it, so write the negated key, as indisable-radix-cache = true. - vLLM and SGLang pass a list to one flag (
--<key> a b), llama.cpp repeats the flag (--<key> a --<key> b).
Containers and user namespaces
Every container backend and gateway runs as root by default, which a user namespace maps to an unprivileged host range.
container.config adds [Container] keys to every container of a backend, servers, fetchers and detectors alike, or of a gateway, and container.mountOptions adds idmapped mounts let the namespace own the cache and read the credentials:
services.llmhop = {
vllm = {
runtime = "quadlet";
container = {
config.UserNS = "auto:size=65536";
mountOptions = {
cache = [ "idmap" ];
credentials = [ "idmap" ];
};
};
};
gateway.llmhop = {
runtime = "quadlet";
container = {
config.UserNS = "auto";
mountOptions.credentials = [ "idmap" ];
};
};
};
UIDMap= and GIDMap= work the same way in place of UserNS=.
A non-root User= inside the namespace takes the mapping idmap=uids=0-<uid>-1;gids=0-<gid>-1 in both lists, so the cache stays owned by host root.
Only a cache on a filesystem without idmapped mounts, such as NFS, needs the backend’s user and group set to the host IDs that an explicit UIDMap= and GIDMap= give the container user.
Rootful UserNS=auto needs subordinate IDs for a containers user, which NixOS does not declare.
Hosts without Nix
The container backends and gateways can be rendered into plain files for any host with podman 5 and systemd, such as Ubuntu 25.04:
packages.x86_64-linux.llmhop-bundle = inputs.llmhop.legacyPackages.x86_64-linux.mkQuadletBundle {
modules = [
{
services.llmhop = {
gateway.type = "llmhop";
vllm = {
enable = true;
container.image = "docker.io/vllm/vllm-openai:latest";
models."qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 8001;
};
};
};
}
];
};
It is the bundle of quadlet-nix named llmhop, which every system importing llmhop’s modules exposes as virtualisation.quadlet.bundle.package, with every rootful Quadlet unit of that system.
It defaults services.llmhop.runtime to "quadlet".
The bundle holds the Quadlet files, update and refresh units, timers, the llmhop gateway’s socket units and tmpfiles rules below etc/, the scripts and generated configs below /usr/local/lib/llmhop, and an install.sh that copies them into place, starts the servers, restarts every running unit it changed and retires those a previous bundle had but this one lacks.
The uninstall.sh installed beside the assets removes the bundle again, keeping the model caches.
Native backends and gateways cannot be bundled, and firewall and user options have no effect there.
Native vLLM and SGLang from prebuilt wheels
Native vLLM and SGLang run from upstream’s wheels, pinned in a small uv workspace and built with mkUvEnv:
# vllm-env/pyproject.toml, then run `uv lock`
[project]
name = "vllm-env"
requires-python = "==3.12.*"
dependencies = ["vllm==0.16.2"]
mkUvEnv builds against the lowest admitted Python and bakes the GPU driver runpath into the wheels, while the driver itself comes from hardware.graphics and your vendor configuration.
models.<name>.package overrides the package per model, for example with a nightly.
- JIT compilers find
nvccthroughCUDA_HOME, whichmkCudaHome { packages = [ ... ]; }builds from thecudaPackages_*set named by the environment’spassthru.cudaPackagesAttr. - AMD and Intel GPUs work with vLLM’s ROCm or XPU wheels, added as an explicit
[[tool.uv.index]]per vLLM’s installation docs. - sdists without a build system need their build dependencies under
[tool.uv.extra-build-dependencies], as inantlr4-python3-runtime = ["setuptools"]. - flashinfer packages must be pinned to the
flashinfer-pythonversion vLLM pins, or flashinfer refuses to start. - Native libraries come from
buildInputsfor linked ones,runtimePathsfor ones loaded by a baredlopen, such aslibsndfileorrdma-core,driverLibsfor host driver sonames andoptionalLibsfor libraries that load on demand only. The build fails naming every unresolved library.
Inference server credentials
Credentials are granted per model, so no worker can read another’s keys.
credentials.<name> maps a name to a file outside the Nix store, or to null to import it from the system credential store, and ${cred:name} in settings or environment expands to its read-only path:
services.llmhop.llama-cpp.models."qwen3-8b" = {
model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
credentials.api-keys = "/run/secrets/qwen-api-keys";
settings.api-key-file = "\${cred:api-keys}";
};
Provision an imported credential as a root-only file or encrypted:
(umask 077; systemd-ask-password -n > /etc/credstore/llmhop.hf-token)
systemd-ask-password -n | systemd-creds encrypt --name=llmhop.hf-token - /etc/credstore.encrypted/llmhop.hf-token
vLLM and SGLang read a YAML file with api-key through settings.config, as in credentials."config.yaml" with settings.config = "\${cred:config.yaml}", while llama.cpp uses api-key-file.
Containers see their credentials at /run/llmhop/credentials, named by $CREDENTIALS_DIRECTORY as natively, and readable through a user namespace with the backend’s or gateway’s container.mountOptions.credentials, see user namespaces.
vLLM watermarking
Both vLLM backends can watermark a model’s output and run detectors serving POST /detect beside the workers.
A watermark holds vLLM’s WatermarkConfig without the key as settings, plus the credential holding the key, and a detector tests every one of its profiles:
services.llmhop.vllm = {
models.qwen = {
model = "Qwen/Qwen3-8B";
revision = "main";
watermark = {
credential = "vllm.watermark-key.2026-10";
settings.algorithm = "dual_key_gumbel";
};
};
detectors.production = {
profiles.qwen = {
tokenizer = "Qwen/Qwen3-8B";
inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
};
settings.p-value-threshold = 0.01;
};
};
A key is an unsigned 64-bit integer credential, never entering the Nix store or a command line:
(umask 077; od -An -N8 -tu8 /dev/urandom | tr -d ' \n' > /etc/credstore/vllm.watermark-key.2026-10)
A detector reports the most significant profile with its Bonferroni-corrected p_value, score, num_scored_tokens and is_watermarked.
To rotate a key, provision it under a new credential name and keep a profile with the old one while old text still needs checking.
llmhop routes detectors as unlisted models, as in POST /route/production/detect, while other gateways leave them to direct clients.
Watermarking requires vLLM 0.30.0 or newer.
Module internals
Decisions behind the NixOS and system-manager modules in nix/modules/ that are hard to re-derive from the code.
Both import the matching module of quadlet-nix, so they share all of llmhop’s modules.
Every backend and gateway selects its runtime as native or quadlet, defaulting to services.llmhop.runtime, and mkQuadletBundle defaults that to quadlet.
CLI dialects
Each engine declares how its parser reads booleans (negateBools) and lists (listStyle).
nixpkgs cannot render both: lib.cli.toCommandLine always repeats list flags and never sees the value when naming a flag, and lib.cli.toGNUCommandLine is deprecated.
Non-string values go through JSON, which keeps 0.6 from becoming 0.600000.
Worker hardening
Native units share a systemd-exec(5) baseline, while containers rely on Podman’s isolation.
The baseline denies binding IP sockets with SocketBindDeny = "any", which a worker with a port lifts for that port.
GPU relaxations
PrivateDevices = falseand therenderandvideogroups give access to the CUDA, ROCm and Level Zero device nodes. A model’snative.devicesnarrows it withDevicePolicy = "closed"andDeviceAllow. CUDA needs/dev/nvidiactlandchar-nvidia-uvmbeside each/dev/nvidia<N>, ROCm/dev/kfdbeside each/dev/dri/renderD<N>, and Level Zero only the latter.PrivateUsers = "identity", sincetruewould map those groups tonobody.MemoryDenyWriteExecute = falsefor the JIT compilers of torch, triton, CUDA and SYCL.LimitMEMLOCK = "infinity", since every stack pins host buffers and the default 8 MiB reports OOM.ProcSubset = "all"for/proc/meminfoand/proc/cpuinfo.
NCCL relaxations
getifaddrs() needs AF_NETLINK, and NCCL binds ephemeral TCP ports, which the bind hook sees as port 0, so all TCP is allowed.
NCCL and Gloo are kept on loopback and NCCL off InfiniBand, which RCCL honors too.
Gloo would otherwise bind the address the hostname resolves to, such as 127.0.0.2 on NixOS, which the IP address filter of an offline server drops.
Caches
Every accelerator stack caches compiled kernels below $HOME, which is read-only, so each cache variable points into the workload’s cache directory, set unconditionally since an absent stack never reads it.
Unix sockets
Each socket lives in a directory of its own below socketDirectory, so no worker can delete or squat another’s socket.
The root is root:root with mode 0711 and each directory 0710, since clients know the name, and servers run with umask 0007, since they never chmod their socket.
A native worker’s socket belongs to the backend’s group, which every user of socketClients joins, since systemd drops ACLs when it chowns an exec directory.
A container user maps to an unpredictable host ID, so container sockets rely on a default ACL user:<client>:-wx on the root instead.
That is why llmhop runs as a named user rather than a DynamicUser.
Each directory is a RuntimeDirectory= of its unit, removed on every stop, so no stale socket causes EADDRINUSE, which is why socketDirectory must stay below /run.
Python backends built from wheels
The wheel-based backends get a compiler, ninja and bash on PATH, since triton, torch and flashinfer compile at runtime and look them up the FHS way, and HOME points at the cache directory.
LD_LIBRARY_PATH covers driver libraries loaded by name, such as libnvidia-ml.so.1, and TRITON_LIBCUDA_PATH replaces triton’s ldconfig -p lookup.
They cannot use DynamicUser, whose ID-mapped state and cache mounts are noexec, while these runtimes load the kernels they compile there.
Lifecycle
Servers use Restart = "always", since vLLM and SGLang exit 0 after an engine crash, with a start limit of 3 per hour against crash loops.
A native server is Type=exec with an ExecStartPost= of llmhop internal wait-healthy that polls /health and also watches $MAINPID, so a crashed server does not stay activating until the timeout.
Containers use Podman’s Notify=healthy.
Startup is chained by name, since two workers profiling GPU memory on one device both see it free and run out of memory.
The llmhop gateway is PartOf= its listeners, so restarting a socket also restarts the gateway that holds it.
Listener configuration changes also trigger a native gateway restart on NixOS.
Fetching
A fetcher is a oneshot with RemainAfterExit=, which its server Wants= and is ordered after, but does not Requires=, so a failed fetch leaves a running server alone.
It shares its server’s identity, so both see the cache under one owner, and every workload has its own cache, so no fetch changes what another server loaded.
Fetchers keep systemd’s default start limit, since they do not restart automatically and may need several manual retries.
Every start or restart of a fetcher also starts <unit>-refresh, which it Wants=.
The refresh unit Requires= and follows the fetcher, so it only acts on a complete download.
It prunes the cache to the workload’s repositories and revisions, then restarts the server.
Telling a changed model from an unchanged one would need state, since a moved ref can leave its previous commit pinned by another revision, so every successful fetch restarts a running server, which the weekly default of fetch.schedule keeps rare.
It resets the server’s start limit first, so planned restarts do not use it up, and try-restart leaves a stopped or failed server down, so a server that hit its start limit waits for a manual start.
The server starts after it, so a first download finds the server still waiting rather than restarting it.
The fetcher controls everything in the cache, so no command touching it runs with more privileges than the cache owner, or a planted symlink could redirect a privileged write or deletion.
The refresh unit prunes as the workload’s user, writing only the hub cache and opening no socket, since a root owner could otherwise reach systemd or Podman.
Only its systemctl step runs as root, and it reads no file.
huggingface_hub moves a ref before downloading its snapshot, so an interrupted download leaves the ref on an incomplete snapshot, which an offline server fails to load until the next complete download.
<backend>-update restarts every fetcher one at a time in startup order, so the server restarts keep to the startup chain.
Claims
Ports, unit names and model names admit one owner each, but NixOS merges clashing units silently or fails opaquely, and port clashes only fail at boot.
So every owner adds its option path to the internal services.llmhop.claims.<kind>.<key>, and an assertion names all owners of a clash.
Endpoints are keyed by owner too, so a name clash reaches that assertion instead of failing the merge.
Directory layout
Each workload’s cache.directory/<name> is the HF_HOME of its fetcher and server, holding the hub cache in hub/, which the server can only read, beside the server’s runtime caches.
It is no CacheDirectory=, which would have to stay below /var/cache, and containers mount it at the same path.
Without cache.directory, a container mounts the named volume <unit> there instead, which Podman creates on first use, so no tmpfiles rule is needed.
A fetcher rules this out, since its refresh unit prunes the hub cache on the host.
Offline servers
Servers get HF_HUB_OFFLINE and llama.cpp --offline only with fetch.enable, since otherwise they download their repositories themselves, and vLLM always gets VLLM_HOST_IP, since it would otherwise derive its address from a route to a public host.
A socket server gets its own network namespace, keeping lo for NCCL.
A native server on a loopback port is limited to 127.0.0.1 and ::1 by the cgroup IP filter, not all of 127.0.0.0/8, which holds the systemd-resolved stub, and cannot see /run/nscd and /run/systemd/resolve.
The filter applies to both directions, so it cannot isolate a server that accepts clients beyond loopback, and a container’s published port needs a network.
The internal workload option isolable captures this, deriving the default of offline and the one assertion rejecting offline = true where it would not hold, so no runtime silently runs an offline server with network access.
Gateways
At most one gateway runs, since chained gateways duplicate routing and auth, and a model reachable beside the gateway bypasses its auth.
Backends publish their workloads to the internal services.llmhop.endpoints, which the selected gateway renders, so a new gateway needs no backend changes.
The gateway.type enum in core.nix lists every gateway module.
Only llmhop routes detectors, which neither SMG’s discovery nor Bifrost’s providers can describe.
SMG gets a discovery file, since it gives up on --worker-urls after a startup timeout shorter than model loading.
External gateways run as dynamic users or on the host network in a container, reaching models on loopback.
Both llmhop runtimes share their listen options, socket units, claims, firewall rules and tmpfiles in gateways/llmhop.nix, and never bind themselves, since their config must not set listen.
Podman passes the sockets of the units listed in the container’s Sockets= into the container, which stays on the host network.
Watermarking
Models and detectors share the runtime renderers, with an explicit workload kind selecting their execution settings. Catalog visibility is derived separately when publishing endpoints.
mkVllmWatermark’s serve merges the key into vLLM’s watermark config in-process, and its detect replaces vLLM’s example detector, which lacks unix sockets, health, file-based keys, several profiles and most algorithms.
Native builds check the script against the vLLM environment, while containers run it unchecked with the image’s Python.
Bundles
Bundles come from quadlet-nix, which renders every rootful Quadlet unit of the system, the NixOS services named by bundle.services with their timers, the sockets named by bundle.sockets, and its bundle.tmpfiles, and copies the store paths they reference below bundle.assetDirectory.
Every container backend names its update and refresh units there, which the bundle renders without the store PATH= of NixOS, so they run systemctl and llmhop by their absolute paths.
llmhop names the bundle, adds its 10-llmhop tmpfiles rules and rejects native backends and gateways through bundle.assertions, which only the bundle checks.
The container llmhop gateway names its llmhop-<name> listeners in bundle.sockets, so the bundle carries their socket units.
The build fails if a copied file still references the store, so only self-contained files such as scripts, configs, the static llmhop binary and its image archive can be bundled.
Containers
The llmhop container loads an archive of the image built with the binary that validated its config, so the running version always matches, it needs no registry, and it updates with the system or bundle rather than through podman auto-update.
The container rootfs stays writable, since ML runtimes scatter caches across HOME, and /tmp is a tmpfs.
Every container of a backend runs the backend’s image unit, which quadlet-nix starts only when a dependent starts.
Backend containers, fetchers and gateways are PartOf= their image unit, so an image restart also restarts its running dependents.
A registry image unit pulls with Policy=missing, since podman image pull otherwise contacts the registry on every start, so a boot during an outage would fail.
container.imageUnit = false references the image directly with Podman’s default Pull=missing, which pulls within the server’s start timeout and drops that restart handle, while a changed image still changes and thereby restarts every container.
Core
services.llmhop.runtime
Default runtime of every backend and gateway, natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
"native"
services.llmhop.socketClients
Users that may connect to the unix socket of every workload without a
port, such as a reverse proxy in front of them. Each joins the group
of every native backend and gets a default ACL on socketDirectory
for the containers. A gateway reaching sockets adds its own user.
Type: list of string
Default:
[ ]
Example:
[
"caddy"
]
services.llmhop.socketDirectory
Directory of every unix socket a workload or llmhop serves, with a
RuntimeDirectory= per workload without a port, hence below /run.
Type: string matching the pattern /run(/[[:alnum:]][[:alnum:].-]*)+
Default:
"/run/llmhop"
gateway
services.llmhop.gateway.active
Whether the gateway of type runs, which needs at least one
endpoint, so a host whose backends are all disabled, such as a test
VM without GPUs, keeps its gateway config without running it. Units
relying on the gateway, such as clients joining its socket group,
can follow it.
Type: boolean (read only)
Default:
config.services.llmhop.gateway.type != null && config.services.llmhop.endpoints != { }
services.llmhop.gateway.bifrost.package
Package providing bifrost-http as its main program, taken from
Bifrost’s own flake, since nixpkgs lacks it.
Only used if runtime is "native".
Type: package
services.llmhop.gateway.bifrost.container.config
[Container] keys of every container of Bifrost, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.bifrost.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/maximhq/bifrost:v2.3.2"
services.llmhop.gateway.bifrost.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
Bifrost, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.bifrost.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.bifrost.environment
Environment variables of Bifrost. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.bifrost.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to Bifrost.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.bifrost.host
IP address Bifrost listens on.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.gateway.bifrost.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.bifrost.port
TCP port Bifrost listens on.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
8080
services.llmhop.gateway.bifrost.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.bifrost.settings
Bifrost’s config.json, with every listed model as a key of its
vllm or sgl provider. Changes from its web UI or API persist
until the matching section of this file changes, while
source_of_truth = "config.json" prunes them and
config_store.enabled = false disables them. Use upstream env.
references with environmentFile for secrets.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
encryption_key = "env.BIFROST_ENCRYPTION_KEY";
governance = {
virtual_keys = [
{
id = "default";
is_active = true;
name = "default";
value = "env.BIFROST_VIRTUAL_KEY";
}
];
};
}
services.llmhop.gateway.llmhop.package
The llmhop package to use.
Type: package
Default:
config.services.llmhop.package
services.llmhop.gateway.llmhop.container.config
[Container] keys of every container of the llmhop gateway, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.llmhop.container.image
Container image reference, including a tag or digest, or null
for an archive of package, which validated the config at build
time and never auto-updates.
Only used if runtime is "quadlet".
Type: null or string
Default:
null
Example:
"ghcr.io/mirkolenz/llmhop:latest"
services.llmhop.gateway.llmhop.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
the llmhop gateway, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.llmhop.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
llmhop reads its own config, so in settings the reference expands
to the credential’s contents rather than to its path.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.llmhop.environment
Environment variables of the llmhop gateway. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.llmhop.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to the llmhop gateway.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.llmhop.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.gateway.llmhop.uid
services.llmhop.gateway.llmhop.group
Primary group of user.
The module declares it while it keeps the name llmhop, and otherwise the
deployer declares it. Only used if runtime is "native".
Type: string
Default:
config.services.llmhop.gateway.llmhop.user
services.llmhop.gateway.llmhop.listen
Listeners of the gateway, each a llmhop-<name>.socket unit handing
its socket to it through socket activation. The default one always
exists, on port 8080 unless configured otherwise.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
caddy = {
socketGroup = "caddy";
};
default = {
host = "";
};
}
services.llmhop.gateway.llmhop.listen.<name>.host
IP address to bind port to. An empty string binds every interface,
leaving access control to the firewall. IPv6 literals are written
plain (e.g. ::1).
Type: string
Default:
"127.0.0.1"
Example:
""
services.llmhop.gateway.llmhop.listen.<name>.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.llmhop.listen.<name>.port
TCP port to listen on. null listens on the unix socket socket
instead. The default listener defaults to 8080.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
Example:
8080
services.llmhop.gateway.llmhop.listen.<name>.socket
Unix socket to listen on while port is null.
Type: string
Default:
"${config.services.llmhop.socketDirectory}/‹name›.sock"
services.llmhop.gateway.llmhop.listen.<name>.socketGroup
Group of socket, typically the one of a reverse proxy in front of llmhop.
Type: string
Default:
"root"
Example:
"caddy"
services.llmhop.gateway.llmhop.listen.<name>.socketMode
Mode of socket. Connecting needs write permission.
Type: string
Default:
"0660"
services.llmhop.gateway.llmhop.listen.<name>.socketUser
Owner of socket.
Type: string
Default:
"root"
services.llmhop.gateway.llmhop.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.llmhop.settings
llmhop’s JSON config, validated at build time by the binary itself,
with every endpoint of the backends as a model. Its listeners come
from listen, so settings must not set it.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
authTokens = {
client = "\${cred:llmhop.client-token}";
};
models = {
"openai/gpt-4" = {
url = "https://api.openai.com";
};
};
}
services.llmhop.gateway.llmhop.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.gateway.llmhop.user
System user the gateway runs as.
The module declares it while it keeps the name llmhop, and otherwise the
deployer declares it. Only used if runtime is "native".
Type: string
Default:
"llmhop"
services.llmhop.gateway.smg.package
Python environment providing the smg package, built with the
flake’s mkUvEnv, since nixpkgs lacks SMG.
Only used if runtime is "native".
Type: package
services.llmhop.gateway.smg.container.config
[Container] keys of every container of SMG, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.smg.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"ghcr.io/smg-project/smg:nightly"
services.llmhop.gateway.smg.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
SMG, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.smg.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.smg.environment
Environment variables of SMG. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.smg.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to SMG.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.smg.host
IP address SMG listens on.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.gateway.smg.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.smg.port
TCP port SMG listens on.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
8080
services.llmhop.gateway.smg.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.smg.settings
CLI flags of SMG, rendered as --<key> <value> with false
dropped. The listener and the discovery file of every listed model
always win, while metrics on 127.0.0.1:29000 and disabled
tokenizer downloads are defaults. Credential references
resolve to file paths for upstream options such as api-key-file
and tenant-api-keys-file, which require a supporting SMG build.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
api-key-file = "\${cred:smg.api-key}";
policy = "cache_aware";
}
services.llmhop.gateway.type
The one gateway in front of every backend, configured by the option
set of the same name, or null for none, so clients connect to each
workload directly. It routes every listed model, which must then
stay on loopback and leave authentication to the gateway.
Type: null or one of “llmhop”, “smg”, “bifrost”
Default:
null
Example:
"llmhop"
llama-cpp
services.llmhop.llama-cpp.enable
Whether to enable llama.cpp model serving.
Type: boolean
Default:
false
Example:
true
services.llmhop.llama-cpp.package
The llama-cpp package. Only used if runtime is "native".
Type: package
Default:
pkgs.llama-cpp
services.llmhop.llama-cpp.cache.directory
Root of the cache directories of this backend’s workloads, created
with mode 0711, each <name> the HF_HOME of one fetcher and its
server, created with mode 0700 for user and group. The
fetcher writes its hub/, which the server only reads. Point it at
a data disk for large models.
null keeps each cache in the Podman named volume <unit> instead,
which needs no tmpfiles rules but only works for a container backend
without fetch.enable.
Type: null or absolute path not in the Nix store
Default:
"/var/cache/llama-cpp"
services.llmhop.llama-cpp.container.config
[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.llama-cpp.container.devices
Devices exposed to every model container as Quadlet AddDevice=
lines, CDI references such as nvidia.com/gpu=all or host device
paths. Per-model container.devices replaces it.
Only used if runtime is "quadlet".
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.llama-cpp.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"ghcr.io/ggml-org/llama.cpp:server-cuda"
services.llmhop.llama-cpp.container.imageUnit
Whether every container runs image through the image unit
llama-cpp-image, which pulls it once for all of them and restarts
them when restarted. Otherwise each container references image
directly and pulls it while it is missing.
Only used if runtime is "quadlet".
Type: boolean
Default:
true
services.llmhop.llama-cpp.container.mountOptions.cache
Podman volume options of the cache mounts below
cache.directory, such as idmap so a user namespace owns
them, or SELinux relabeling.
Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.llama-cpp.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
this backend, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.llama-cpp.environment
Environment variables set on every model service, under the
environment of each model, where ${cred:<name>} expands to the
path of a credential.
Type: attribute set of string
Default:
{ }
services.llmhop.llama-cpp.environmentFile
File in KEY=VALUE format forwarded to every server, loaded before
the environmentFile of each model. Prefer credentials for any
secret a server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
Example:
"/etc/llama-cpp/.env"
services.llmhop.llama-cpp.fetch.enable
Whether every workload has a fetcher <unit>-fetch downloading
its repositories before its server starts, with a refresh unit and
llama-cpp-update, after which the server runs offline. Otherwise
each server downloads its repositories itself when it starts, which
requires offline = false, and none of the other fetch options
apply.
Type: boolean
Default:
true
Example:
true
services.llmhop.llama-cpp.fetch.credentials
Credentials granted to every fetcher, in the format of a
workload’s credentials. Reference them from fetch.environment
as ${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.llama-cpp.fetch.environment
Environment variables set on every fetcher, such as a proxy. Values
may reference fetch.credentials as ${cred:<name>}.
The servers never see them.
Type: attribute set of string
Default:
{ }
Example:
{
HTTPS_PROXY = "http://proxy.example:3128";
}
services.llmhop.llama-cpp.fetch.environmentFile
File in KEY=VALUE format forwarded to every fetcher, for
variables that only exist as values.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.llama-cpp.fetch.hfTokenCredential
Credential holding a Hugging Face token for gated repositories,
imported from the system credential store unless
fetch.credentials names a file for it. Only the fetchers receive
it, as HF_TOKEN, and skip it while it is missing.
Type: null or string
Default:
null
Example:
"llmhop.hf-token"
services.llmhop.llama-cpp.fetch.schedule
OnCalendar= schedule of llama-cpp-update, which runs every
fetcher and restarts each running server after a successful fetch,
even if its model did not change.
null disables scheduled fetching.
Type: null or string
Default:
"weekly"
Example:
"daily"
services.llmhop.llama-cpp.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.llama-cpp.uid
services.llmhop.llama-cpp.group
Primary group of user.
If runtime is "native", the module declares it while it keeps the name llama-cpp, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.llama-cpp.runtime == "native" then config.services.llmhop.llama-cpp.user else "root"
services.llmhop.llama-cpp.modelSettings
CLI flags forwarded to the server of every model, under the
settings of each model.
Rendered as --<key> <value>, with false as --no-<key> and a list repeating the flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.llama-cpp.models
Models to serve, each as the unit llama-cpp-<name> beside its
fetcher llama-cpp-<name>-fetch, natively a hardened system service
and otherwise a container, whose units
virtualisation.quadlet.containers.<unit> override.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
}
services.llmhop.llama-cpp.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.llama-cpp.models.<name>.package
Package providing this model’s server, overriding the
backend-wide package, e.g. a nightly build for a
just-released architecture.
Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.llama-cpp.package
services.llmhop.llama-cpp.models.<name>.container.devices
Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".
Type: list of string
Default:
config.services.llmhop.llama-cpp.container.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.llama-cpp.models.<name>.container.shmSize
Size of the container’s private /dev/shm tmpfs, which
PyTorch uses for tensor-parallel inference.
Only used if the backend’s runtime is "quadlet".
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.llama-cpp.models.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.llama-cpp.models.<name>.environment
Environment variables of this model’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.llama-cpp.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.llama-cpp.models.<name>.extraRepositories
Further repositories the fetcher downloads beside model, such as a
speculative draft model or a LoRA adapter that settings
reference, since the server cannot download them itself.
Type: list of string
Default:
[ ]
Example:
[
"Qwen/Qwen3-0.6B"
]
services.llmhop.llama-cpp.models.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.llama-cpp.models.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.llama-cpp.models.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.llama-cpp.models.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.llama-cpp.models.<name>.model
Hugging Face repository with an optional quantization tag, passed as
--hf-repo, or an absolute path to a GGUF file, passed as --model.
A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.
Type: string
Example:
"unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL"
services.llmhop.llama-cpp.models.<name>.name
Name of this model in its unit llama-cpp-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.llama-cpp.models.<name>.native.devices
Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".
Type: null or (list of string)
Default:
config.services.llmhop.llama-cpp.native.devices
Example:
[
"/dev/nvidia0"
]
services.llmhop.llama-cpp.models.<name>.offline
Whether this model’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.llama-cpp.models.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this model.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.llama-cpp.models.<name>.port
Host port the server binds to on host.
null binds the unix socket socket instead, which claims no
port, only the gateway and services.llmhop.socketClients can
connect to, and keeps the server off any network.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.llama-cpp.models.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.llama-cpp.models.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.llama-cpp.models.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.llama-cpp.models.<name>.settings
CLI flags of this model’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false as --no-<key> and a list repeating the flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.llama-cpp.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/llama-cpp-<name>/http.sock while port is null, else null
services.llmhop.llama-cpp.native.devices
Devices every model service may access as systemd DeviceAllow=
entries under DevicePolicy=closed, while null allows all.
Per-model native.devices replaces it.
Only used if runtime is "native".
Type: null or (list of string)
Default:
null
Example:
[
"/dev/nvidiactl"
"char-nvidia-uvm"
"/dev/nvidia0"
]
services.llmhop.llama-cpp.openFilesLimit
File descriptor limit (LimitNOFILE) of every server of this
backend. Increase it if a server logs accept: Too many open files.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.llama-cpp.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.llama-cpp.startupOrdering
Whether to start model services one at a time by ascending name,
so two workers profiling memory on the same GPU do not race into an
OOM. Disable only when each model pins itself to a dedicated device,
natively via its own native.devices or environment, e.g.
CUDA_VISIBLE_DEVICES, or in a container via its own
container.devices.
Type: boolean
Default:
true
services.llmhop.llama-cpp.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.llama-cpp.user
System user the native units and any refresh units run as, which owns their directories.
If runtime is "native", the module declares it while it keeps the name llama-cpp, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.llama-cpp.runtime == "native" then "llama-cpp" else "root"
sglang
services.llmhop.sglang.enable
Whether to enable SGLang model serving.
Type: boolean
Default:
false
Example:
true
services.llmhop.sglang.package
Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server, built from
a uv workspace with the flake’s mkUvEnv, as in
inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./sglang-env; }.
Only used if runtime is "native".
Type: package
services.llmhop.sglang.cache.directory
Root of the cache directories of this backend’s workloads, created
with mode 0711, each <name> the HF_HOME of one fetcher and its
server, created with mode 0700 for user and group. The
fetcher writes its hub/, which the server only reads. Point it at
a data disk for large models.
null keeps each cache in the Podman named volume <unit> instead,
which needs no tmpfiles rules but only works for a container backend
without fetch.enable.
Type: null or absolute path not in the Nix store
Default:
"/var/cache/sglang"
services.llmhop.sglang.container.config
[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.sglang.container.devices
Devices exposed to every model container as Quadlet AddDevice=
lines, CDI references such as nvidia.com/gpu=all or host device
paths. Per-model container.devices replaces it.
Only used if runtime is "quadlet".
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.sglang.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/lmsysorg/sglang:latest"
services.llmhop.sglang.container.imageUnit
Whether every container runs image through the image unit
sglang-image, which pulls it once for all of them and restarts
them when restarted. Otherwise each container references image
directly and pulls it while it is missing.
Only used if runtime is "quadlet".
Type: boolean
Default:
true
services.llmhop.sglang.container.mountOptions.cache
Podman volume options of the cache mounts below
cache.directory, such as idmap so a user namespace owns
them, or SELinux relabeling.
Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.sglang.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
this backend, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.sglang.environment
Environment variables set on every model service, under the
environment of each model, where ${cred:<name>} expands to the
path of a credential.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.environmentFile
File in KEY=VALUE format forwarded to every server, loaded before
the environmentFile of each model. Prefer credentials for any
secret a server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
Example:
"/etc/sglang/.env"
services.llmhop.sglang.fetch.enable
Whether every workload has a fetcher <unit>-fetch downloading
its repositories before its server starts, with a refresh unit and
sglang-update, after which the server runs offline. Otherwise
each server downloads its repositories itself when it starts, which
requires offline = false, and none of the other fetch options
apply.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang.fetch.credentials
Credentials granted to every fetcher, in the format of a
workload’s credentials. Reference them from fetch.environment
as ${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.sglang.fetch.environment
Environment variables set on every fetcher, such as a proxy. Values
may reference fetch.credentials as ${cred:<name>}.
The servers never see them.
Type: attribute set of string
Default:
{ }
Example:
{
HTTPS_PROXY = "http://proxy.example:3128";
}
services.llmhop.sglang.fetch.environmentFile
File in KEY=VALUE format forwarded to every fetcher, for
variables that only exist as values.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang.fetch.hfTokenCredential
Credential holding a Hugging Face token for gated repositories,
imported from the system credential store unless
fetch.credentials names a file for it. Only the fetchers receive
it, as HF_TOKEN, and skip it while it is missing.
Type: null or string
Default:
null
Example:
"llmhop.hf-token"
services.llmhop.sglang.fetch.schedule
OnCalendar= schedule of sglang-update, which runs every
fetcher and restarts each running server after a successful fetch,
even if its model did not change.
null disables scheduled fetching.
Type: null or string
Default:
"weekly"
Example:
"daily"
services.llmhop.sglang.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.sglang.uid
services.llmhop.sglang.group
Primary group of user.
If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.sglang.runtime == "native" then config.services.llmhop.sglang.user else "root"
services.llmhop.sglang.modelSettings
CLI flags forwarded to the server of every model, under the
settings of each model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.sglang.models
Models to serve, each as the unit sglang-<name> beside its
fetcher sglang-<name>-fetch, natively a hardened system service
and otherwise a container, whose units
virtualisation.quadlet.containers.<unit> override.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b".model = "Qwen/Qwen3-8B";
}
services.llmhop.sglang.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang.models.<name>.package
Package providing this model’s server, overriding the
backend-wide package, e.g. a nightly build for a
just-released architecture.
Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.sglang.package
services.llmhop.sglang.models.<name>.container.devices
Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".
Type: list of string
Default:
config.services.llmhop.sglang.container.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.sglang.models.<name>.container.shmSize
Size of the container’s private /dev/shm tmpfs, which
PyTorch uses for tensor-parallel inference.
Only used if the backend’s runtime is "quadlet".
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.sglang.models.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.sglang.models.<name>.environment
Environment variables of this model’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang.models.<name>.extraRepositories
Further repositories the fetcher downloads beside model, such as a
speculative draft model or a LoRA adapter that settings
reference, since the server cannot download them itself.
Type: list of string
Default:
[ ]
Example:
[
"Qwen/Qwen3-0.6B"
]
services.llmhop.sglang.models.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.sglang.models.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.sglang.models.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.sglang.models.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.sglang.models.<name>.model
Hugging Face repository passed as --model-path, or an absolute path.
A repository is downloaded by the fetcher, while an absolute path,
in a container one inside it, is served as is.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.sglang.models.<name>.name
Name of this model in its unit sglang-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.sglang.models.<name>.native.devices
Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".
Type: null or (list of string)
Default:
config.services.llmhop.sglang.native.devices
Example:
[
"/dev/nvidia0"
]
services.llmhop.sglang.models.<name>.offline
Whether this model’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.sglang.models.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this model.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.sglang.models.<name>.port
Host port the server binds to on host.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.sglang.models.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.sglang.models.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.sglang.models.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.sglang.models.<name>.revision
Branch, tag or commit of model that the fetcher downloads and the
server loads. null follows the default branch, so fetch.schedule
picks up upstream changes, while a commit pins the model.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.sglang.models.<name>.settings
CLI flags of this model’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.sglang.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null
services.llmhop.sglang.native.devices
Devices every model service may access as systemd DeviceAllow=
entries under DevicePolicy=closed, while null allows all.
Per-model native.devices replaces it.
Only used if runtime is "native".
Type: null or (list of string)
Default:
null
Example:
[
"/dev/nvidiactl"
"char-nvidia-uvm"
"/dev/nvidia0"
]
services.llmhop.sglang.openFilesLimit
File descriptor limit (LimitNOFILE) of every server of this
backend. Increase it if a server logs accept: Too many open files.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.sglang.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.sglang.startupOrdering
Whether to start model services one at a time by ascending name,
so two workers profiling memory on the same GPU do not race into an
OOM. Disable only when each model pins itself to a dedicated device,
natively via its own native.devices or environment, e.g.
CUDA_VISIBLE_DEVICES, or in a container via its own
container.devices.
Type: boolean
Default:
true
services.llmhop.sglang.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.sglang.user
System user the native units and any refresh units run as, which owns their directories.
If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.sglang.runtime == "native" then "sglang" else "root"
vllm
services.llmhop.vllm.enable
Whether to enable vLLM model serving.
Type: boolean
Default:
false
Example:
true
services.llmhop.vllm.package
Package providing the vllm CLI at bin/vllm, built from
a uv workspace with the flake’s mkUvEnv, as in
inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; }.
Only used if runtime is "native".
Type: package
services.llmhop.vllm.cache.directory
Root of the cache directories of this backend’s workloads, created
with mode 0711, each <name> the HF_HOME of one fetcher and its
server, created with mode 0700 for user and group. The
fetcher writes its hub/, which the server only reads. Point it at
a data disk for large models.
null keeps each cache in the Podman named volume <unit> instead,
which needs no tmpfiles rules but only works for a container backend
without fetch.enable.
Type: null or absolute path not in the Nix store
Default:
"/var/cache/vllm"
services.llmhop.vllm.container.config
[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.vllm.container.devices
Devices exposed to every model container as Quadlet AddDevice=
lines, CDI references such as nvidia.com/gpu=all or host device
paths. Per-model container.devices replaces it.
Only used if runtime is "quadlet".
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.vllm.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/vllm/vllm-openai:latest"
services.llmhop.vllm.container.imageUnit
Whether every container runs image through the image unit
vllm-image, which pulls it once for all of them and restarts
them when restarted. Otherwise each container references image
directly and pulls it while it is missing.
Only used if runtime is "quadlet".
Type: boolean
Default:
true
services.llmhop.vllm.container.mountOptions.cache
Podman volume options of the cache mounts below
cache.directory, such as idmap so a user namespace owns
them, or SELinux relabeling.
Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.vllm.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
this backend, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.vllm.detectors
Standalone watermark detection services. The llmhop gateway routes them as unlisted models, while every other gateway leaves them to direct clients.
Type: attribute set of (submodule)
Default:
{ }
services.llmhop.vllm.detectors.<name>.enable
Whether to enable watermark detector ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.detectors.<name>.package
Environment this detector runs in. Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.detectors.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.detectors.<name>.environment
Environment variables of this watermark detector’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.detectors.<name>.environmentFile
File in KEY=VALUE format forwarded to this watermark detector’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.detectors.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.vllm.detectors.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.vllm.detectors.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.detectors.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.vllm.detectors.<name>.name
Name of this watermark detector in its unit vllm-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.detectors.<name>.offline
Whether this watermark detector’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.vllm.detectors.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this watermark detector.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.vllm.detectors.<name>.port
Host port the server binds to on host.
null binds the unix socket socket instead, which claims no
port, only the gateway and services.llmhop.socketClients can
connect to, and keeps the server off any network.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.detectors.<name>.profiles
Generation configurations to test, keyed by the name a response
reports as profile. Profiles sharing a tokenizer and revision
share its encoding of the text, and the reported p-value is that
of the most significant profile multiplied by their number.
Type: attribute set of (submodule)
Example:
{
qwen = {
tokenizer = "Qwen/Qwen3-8B";
inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
};
}
services.llmhop.vllm.detectors.<name>.profiles.<name>.revision
Branch, tag or commit of tokenizer, which must match the
revision of the generating model, or its
settings.tokenizer-revision. null follows the default branch.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.vllm.detectors.<name>.profiles.<name>.tokenizer
Tokenizer used to encode candidate text, a Hugging Face repository or an absolute path. It must exactly match the tokenizer used for watermarked generation. The fetcher downloads only the files of a repository that a tokenizer may need, not its weights.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark
The watermark of the generating model.
Type: submodule
Default:
{ }
Example:
{
credential = "vllm.watermark-key.2026-10";
settings = {
algorithm = "dual_key_gumbel";
};
}
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.credential
Credential holding the key, an unsigned 64-bit integer, imported from
the system credential store unless the service’s credentials names
a file for it. A new name per key, such as a date suffix, restarts
every service using it when the key is rotated.
Type: string
Default:
"vllm.watermark-key"
Example:
"vllm.watermark-key.2026-10"
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.settings
vLLM’s WatermarkConfig without key.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.detectors.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.vllm.detectors.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.vllm.detectors.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.vllm.detectors.<name>.script
Detector server run by the detector’s Python interpreter. It
receives detect and settings, the profiles among them as
JSON, and must serve GET /health and POST /detect.
p-value-threshold is the only detection-side setting of the
default.
Type: absolute path
Default:
the script of mkVllmWatermark, built for package if the backend’s runtime is "native"
services.llmhop.vllm.detectors.<name>.settings
CLI flags of this watermark detector’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.detectors.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null
services.llmhop.vllm.environment
Environment variables set on every model service, under the
environment of each model, where ${cred:<name>} expands to the
path of a credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.environmentFile
File in KEY=VALUE format forwarded to every server, loaded before
the environmentFile of each model. Prefer credentials for any
secret a server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
Example:
"/etc/vllm/.env"
services.llmhop.vllm.fetch.enable
Whether every workload has a fetcher <unit>-fetch downloading
its repositories before its server starts, with a refresh unit and
vllm-update, after which the server runs offline. Otherwise
each server downloads its repositories itself when it starts, which
requires offline = false, and none of the other fetch options
apply.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.fetch.credentials
Credentials granted to every fetcher, in the format of a
workload’s credentials. Reference them from fetch.environment
as ${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.fetch.environment
Environment variables set on every fetcher, such as a proxy. Values
may reference fetch.credentials as ${cred:<name>}.
The servers never see them.
Type: attribute set of string
Default:
{ }
Example:
{
HTTPS_PROXY = "http://proxy.example:3128";
}
services.llmhop.vllm.fetch.environmentFile
File in KEY=VALUE format forwarded to every fetcher, for
variables that only exist as values.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.fetch.hfTokenCredential
Credential holding a Hugging Face token for gated repositories,
imported from the system credential store unless
fetch.credentials names a file for it. Only the fetchers receive
it, as HF_TOKEN, and skip it while it is missing.
Type: null or string
Default:
null
Example:
"llmhop.hf-token"
services.llmhop.vllm.fetch.schedule
OnCalendar= schedule of vllm-update, which runs every
fetcher and restarts each running server after a successful fetch,
even if its model did not change.
null disables scheduled fetching.
Type: null or string
Default:
"weekly"
Example:
"daily"
services.llmhop.vllm.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.vllm.uid
services.llmhop.vllm.group
Primary group of user.
If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.vllm.runtime == "native" then config.services.llmhop.vllm.user else "root"
services.llmhop.vllm.modelSettings
CLI flags forwarded to the server of every model, under the
settings of each model.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.models
Models to serve, each as the unit vllm-<name> beside its
fetcher vllm-<name>-fetch, natively a hardened system service
and otherwise a container, whose units
virtualisation.quadlet.containers.<unit> override.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b".model = "Qwen/Qwen2.5-7B-Instruct";
}
services.llmhop.vllm.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.models.<name>.package
Package providing this model’s server, overriding the
backend-wide package, e.g. a nightly build for a
just-released architecture.
Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.models.<name>.container.devices
Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".
Type: list of string
Default:
config.services.llmhop.vllm.container.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.vllm.models.<name>.container.shmSize
Size of the container’s private /dev/shm tmpfs, which
PyTorch uses for tensor-parallel inference.
Only used if the backend’s runtime is "quadlet".
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.vllm.models.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.models.<name>.environment
Environment variables of this model’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.models.<name>.extraRepositories
Further repositories the fetcher downloads beside model, such as a
speculative draft model or a LoRA adapter that settings
reference, since the server cannot download them itself.
Type: list of string
Default:
[ ]
Example:
[
"Qwen/Qwen3-0.6B"
]
services.llmhop.vllm.models.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.vllm.models.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.vllm.models.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.models.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.vllm.models.<name>.model
Hugging Face repository passed to vllm serve, or an absolute path.
A repository is downloaded by the fetcher, while an absolute path,
in a container one inside it, is served as is.
Type: string
Example:
"Qwen/Qwen2.5-7B-Instruct"
services.llmhop.vllm.models.<name>.name
Name of this model in its unit vllm-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.models.<name>.native.devices
Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".
Type: null or (list of string)
Default:
config.services.llmhop.vllm.native.devices
Example:
[
"/dev/nvidia0"
]
services.llmhop.vllm.models.<name>.offline
Whether this model’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.vllm.models.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this model.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.vllm.models.<name>.port
Host port the server binds to on host.
null binds the unix socket socket instead, which claims no
port, only the gateway and services.llmhop.socketClients can
connect to, and keeps the server off any network.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.models.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.vllm.models.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.vllm.models.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.vllm.models.<name>.revision
Branch, tag or commit of model that the fetcher downloads and the
server loads. null follows the default branch, so fetch.schedule
picks up upstream changes, while a commit pins the model.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.vllm.models.<name>.settings
CLI flags of this model’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null
services.llmhop.vllm.models.<name>.watermark
The settings of vLLM’s WatermarkConfig without key, which a
detector profile must repeat, and the credential holding the key.
Type: null or (submodule)
Default:
null
Example:
{
credential = "vllm.watermark-key.2026-10";
settings = {
algorithm = "dual_key_gumbel";
};
}
services.llmhop.vllm.models.<name>.watermark.credential
Credential holding the key, an unsigned 64-bit integer, imported from
the system credential store unless the service’s credentials names
a file for it. A new name per key, such as a date suffix, restarts
every service using it when the key is rotated.
Type: string
Default:
"vllm.watermark-key"
Example:
"vllm.watermark-key.2026-10"
services.llmhop.vllm.models.<name>.watermark.settings
vLLM’s WatermarkConfig without key.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.native.devices
Devices every model service may access as systemd DeviceAllow=
entries under DevicePolicy=closed, while null allows all.
Per-model native.devices replaces it.
Only used if runtime is "native".
Type: null or (list of string)
Default:
null
Example:
[
"/dev/nvidiactl"
"char-nvidia-uvm"
"/dev/nvidia0"
]
services.llmhop.vllm.openFilesLimit
File descriptor limit (LimitNOFILE) of every server of this
backend. Increase it if a server logs accept: Too many open files.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.vllm.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.vllm.startupOrdering
Whether to start model services one at a time by ascending name,
so two workers profiling memory on the same GPU do not race into an
OOM. Disable only when each model pins itself to a dedicated device,
natively via its own native.devices or environment, e.g.
CUDA_VISIBLE_DEVICES, or in a container via its own
container.devices.
Type: boolean
Default:
true
services.llmhop.vllm.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.vllm.user
System user the native units and any refresh units run as, which owns their directories.
If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.vllm.runtime == "native" then "vllm" else "root"