Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

LLMhop

One port, many models: A tiny, stateless HTTP router for OpenAI-compatible LLM inference backends.

LLMhop reads the model field of an incoming request and reverse-proxies it to the matching backend. It is built for single-model servers like vLLM and SGLang, but works with any OpenAI-compatible backend, self-hosted or remote.

  • Pure Go, no third-party packages, no CGO, stateless and safe behind any load balancer.
  • Body and path routing, a models API served from the config, health and readiness endpoints, graceful draining.
  • Ships as a static binary, a Docker image and a hardened NixOS module running llama.cpp, SGLang or vLLM behind llmhop, SMG, Bifrost or no gateway.

How it works

LLMhop looks up the request’s model in its config and forwards the request verbatim. Unknown models return 404, invalid JSON or a missing model return 400. GET /v1/models and GET /v1/models/{model} are answered from the config and never proxied. Model names may contain /, as in Hugging Face names such as org/name.

A request under /route/{model}/ is routed by path instead, with the prefix stripped and the body left unparsed, so POST /route/production/detect reaches production as POST /detect. The longest configured model name that prefixes the path wins, and a / in a model name may be written raw or as %2F. A model marked "unlisted": true is routed but left out of the models API, which suits services such as a watermark detector.

GET /health returns { "status": "ok", "models": 3 } with the number of listed models. GET /readiness probes the healthPath of every model that sets one and answers 200 only if all respond 2xx, else 503. Both never require a token. LLMhop sends READY=1 to systemd once its listeners are bound, and drains in-flight requests on SIGTERM or SIGINT.

Errors

Every error is JSON in the shape of OpenAI, as in {"error": {"message": "...", "type": "not_found_error", "code": "model_not_found"}}.

StatusMeaning
400invalid JSON, a missing model or an unreadable body
401missing or invalid bearer token, with WWW-Authenticate: Bearer
404unknown model
408the body did not arrive within a minute
413the body exceeds maxBodyBytes
429the model reached maxConcurrentRequests, with Retry-After: 1
502the backend is unreachable
503the maxBufferedBytes budget is exhausted, with Retry-After: 1

Configuration

{
  "listen": ["127.0.0.1:8080", "unix:///run/llmhop/http.sock"],
  "authTokens": { "client": "${cred:llmhop.client-token}" },
  "models": {
    "llama-3-8b": { "url": "http://localhost:30000", "healthPath": "/health" },
    "meta-llama/Llama-3.1-8B": { "url": "http://localhost:30001" },
    "qwen3-8b": {
      "url": "unix:///run/llmhop/vllm-qwen3-8b/http.sock",
      "maxConcurrentRequests": 16
    },
    "openai-gpt-4o": {
      "url": "https://api.openai.com",
      "headers": { "Authorization": "Bearer ${cred:openai-key}" }
    }
  }
}

listen holds host:port addresses, with [::1]:8080 for IPv6 and :8080 for every interface, or unix:///<socket path>. It has no default, and serve uses the sockets of systemd socket activation when present, else listen. Setting both or neither is an error, and check does not require listen. A url is an absolute http(s) URL or unix:///<socket path>. With authTokens, a map of names to tokens, set, every route but /health and /readiness requires a matching bearer token, which is stripped before forwarding. authTokenCredentials, a systemd credential glob with a single trailing * as in ImportCredential=, adds every matching credential as a token named by the part the * matched, read at startup, and keeps tokens required while none matches. A model’s headers are applied last, so a configured Authorization replaces the client’s. Unknown keys are rejected, and llmhop check config.json validates a config without binding a port or expanding secrets.

Secret references

Token values in authTokens and strings in models.*.headers are expanded at startup, and an unresolved reference is a startup error:

  • ${cred:name}: the systemd credential name from $CREDENTIALS_DIRECTORY.
  • ${env:NAME}: an environment variable.
  • ${file:/absolute/path}: a file, with trailing newlines trimmed.
  • $$: a literal $.

Request limits

  • maxBodyBytes (100 MiB): larger bodies get 413.
  • maxBufferedBytes (1 GiB): bytes of body-routed requests buffered at once, further ones get 503 with Retry-After: 1. A body is buffered until the backend starts its response, and must arrive within a minute, or the request gets 408.
  • models.*.maxConcurrentRequests (unlimited): proxied requests per model, further ones get 429 with Retry-After: 1.

0 disables a limit.

Running

llmhop serve config.json
nix run github:mirkolenz/llmhop -- serve config.json
docker run --rm -p 8080:8080 -v ./config.json:/config.json ghcr.io/mirkolenz/llmhop

llmhop version prints the version. The image runs serve /config.json, so the mounted config must set "listen": [":8080"].

NixOS and system-manager modules

Add the flake input github:mirkolenz/llmhop and import llmhop.nixosModules.default on NixOS or llmhop.systemManagerModules.default on other distributions with system-manager:

services.llmhop = {
  gateway.type = "llmhop";
  gateway.llmhop.listen.default.host = "";
  gateway.llmhop.listen.default.openFirewall = true;
  llama-cpp.enable = true;
  llama-cpp.models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};

Both include quadlet-nix for containers, which every backend and gateway selects with runtime = "native" or "quadlet", defaulting to the global services.llmhop.runtime, "native". Container setups can also be bundled for hosts without Nix. With system-manager, the host provides the configuration of podman in /etc/containers. See the options reference for every option and the module internals for the design rationale.

Gateways

services.llmhop.gateway.type selects at most one gateway, configured by the option set of the same name, which routes every model of every backend by its name. The gateway runs only while some backend serves a model, which the read-only services.llmhop.gateway.active reports.

typeGatewayUnix socketsDetectors
nullnone, clients connect to each model
"llmhop"llmhopnative onlyyes
"smg"the Shepherd Model Gatewaynono
"bifrost"Bifrostnono

Each gateway runs natively or as a container, as selected by gateway.<type>.runtime.

Routed models must stay on loopback and leave authentication to the gateway, and gateways without socket support need every model to set a port. SMG and Bifrost listen on 127.0.0.1:8080 by default, configured through host, port and openFirewall.

  • llmhop runs as the llmhop user with one socket-activated llmhop-<name>.socket per entry of listen, as in listen.caddy.socketGroup = "caddy" for a unix socket, or as a container receiving the same sockets, which loads an image archive of the same build, unless container.image names a registry reference such as ghcr.io/mirkolenz/llmhop:latest. The default listener serves 127.0.0.1:8080 and environment and environmentFile are forwarded to the service. Its generated config is validated at build time with llmhop check. Reference secrets in settings as ${cred:name} and grant them through credentials. settings.authTokenCredentials = "llmhop.token.*" imports the tokens of /etc/credstore/llmhop.token.<client>, or encrypted ones of /etc/credstore.encrypted, so clients come and go without a rebuild by adding or deleting their file. Natively, systemctl reload llmhop then re-imports the credentials and swaps the tokens without a restart, while a container needs systemctl restart llmhop, whose sockets hold new connections meanwhile.
  • SMG reads its workers from a generated discovery file. settings takes its CLI flags, where ${cred:name} resolves to a file path, as in settings.api-key-file = "\${cred:smg.api-key}", which requires an SMG build newer than v1.13.0. The native package is a mkUvEnv environment.
  • Bifrost gets every model as a key of its vllm provider, or sgl for SGLang. settings is its config.json, with secrets as env.NAME references loaded from environmentFile. The native package comes from Bifrost’s own flake.

Without a gateway, each model serves clients on its own socket or port, and socketClients lists the users, such as a reverse proxy, that may connect to every model socket.

Inference backends

Each backend serves the entries of its models as isolated workers, and all backends can run side by side:

services.llmhop = {
  llama-cpp = {
    enable = true;
    models."qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
  };
  vllm = {
    enable = true;
    package = inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; };
    models."qwen3-8b" = {
      model = "Qwen/Qwen3-8B";
      port = 19001;
      settings.reasoning-parser = "qwen3";
    };
  };
  sglang = {
    enable = true;
    runtime = "quadlet";
    container.image = "docker.io/lmsysorg/sglang:latest";
    models."llama-3-8b" = {
      model = "meta-llama/Meta-Llama-3-8B-Instruct";
      port = 19002;
    };
  };
};

Each model runs as the unit <engine>-<name>. Native workers are hardened services running as the backend’s user, which defaults to the engine name, while container workers run rootful, with user defaulting to root, through quadlet-nix with the backend’s container.image, managed with quadletctl. Switching a backend’s runtime therefore changes the owner of its cache directories, but not of the files a previous fetch left in them, so a native fetcher cannot update a cache filled by a container until chown -R hands it to the new user and group, or the cache is deleted for a fresh download. A worker stays activating until it answers its health endpoint, and startupOrdering, enabled by default for every backend, starts workers one at a time by name, so two never profile or size their GPU memory at once. Evaluation rejects any port, unit or model name claimed twice. Customize the generated units directly, as in systemd.services.vllm-qwen3-8b.serviceConfig.MemoryHigh = "64G" or virtualisation.quadlet.containers.vllm-qwen3-8b.containerConfig.

llama.cpp and vLLM workers default to port = null, binding a unix socket below socketDirectory that only the backend’s workers, llmhop and socketClients can reach. Set port to bind <host>:<port> instead, which SGLang always needs.

Fetching and offline servers

Each model has a fetcher <engine>-<name>-fetch that downloads everything into the model’s own cache cache.directory/<name>, after which the server runs offline from it. A model that is an absolute path has no fetcher. extraRepositories lists further repositories that settings reference, such as a draft model or LoRA adapter, and revision pins a branch, tag or commit:

services.llmhop.vllm.models."qwen3-8b" = {
  model = "Qwen/Qwen3-8B";
  revision = "main";
  extraRepositories = [ "Qwen/Qwen3-0.6B" ];
  settings.speculative-config = {
    model = "Qwen/Qwen3-0.6B";
    num_speculative_tokens = 4;
  };
};

Only fetchers get the backend’s fetch.credentials, fetch.environment and the Hugging Face token of gated repositories, which fetch.hfTokenCredential = "llmhop.hf-token" imports for every engine. <backend>-update, such as vllm-update, reruns every fetcher on the fetch.schedule (OnCalendar=), weekly by default, prunes what the model no longer references, and restarts each running server after a successful fetch, even if its model did not change. An interrupted download leaves a running server alone, and a successful retry refreshes it. An explicitly stopped server stays stopped.

fetch.enable = false drops the fetchers, refresh and update units, so each server downloads its repositories itself when it starts, which needs offline = false, its default then. A container backend can then also set cache.directory = null, keeping each cache in the named volume <engine>-<name> without tmpfiles rules, and container.imageUnit = false, so each container references container.image directly rather than through the image unit <engine>-image. Together they leave one unit per model, as in the bundle packages.x86_64-linux.vllm-bundle of nix/bundles/vllm.nix.

With fetchers, servers never download from the hub: they get HF_HUB_OFFLINE, and with offline = true, the default, no network. A native server on a socket gets no network at all and on a port only loopback, while a container gets Network=none. Only a socket or a native loopback host can be isolated like this, so any other workload defaults to offline = false and rejects offline = true. offline = false lifts the network restrictions for features like media URLs in requests. Tagged container images are labeled AutoUpdate=registry for virtualisation.quadlet.autoUpdate, and are otherwise only pulled while missing, so a start needs no registry.

Settings rendering

settings, over the backend’s modelSettings, becomes the server’s CLI flags, while the listener and the served name always win. Strings and paths are passed verbatim and everything else as JSON, so attribute sets work for options like --speculative-config.

  • true becomes --<key>, while null and empty lists are dropped.
  • llama.cpp and vLLM render false as --no-<key>, while SGLang drops it, so write the negated key, as in disable-radix-cache = true.
  • vLLM and SGLang pass a list to one flag (--<key> a b), llama.cpp repeats the flag (--<key> a --<key> b).

Containers and user namespaces

Every container backend and gateway runs as root by default, which a user namespace maps to an unprivileged host range. container.config adds [Container] keys to every container of a backend, servers, fetchers and detectors alike, or of a gateway, and container.mountOptions adds idmapped mounts let the namespace own the cache and read the credentials:

services.llmhop = {
  vllm = {
    runtime = "quadlet";
    container = {
      config.UserNS = "auto:size=65536";
      mountOptions = {
        cache = [ "idmap" ];
        credentials = [ "idmap" ];
      };
    };
  };
  gateway.llmhop = {
    runtime = "quadlet";
    container = {
      config.UserNS = "auto";
      mountOptions.credentials = [ "idmap" ];
    };
  };
};

UIDMap= and GIDMap= work the same way in place of UserNS=. A non-root User= inside the namespace takes the mapping idmap=uids=0-<uid>-1;gids=0-<gid>-1 in both lists, so the cache stays owned by host root. Only a cache on a filesystem without idmapped mounts, such as NFS, needs the backend’s user and group set to the host IDs that an explicit UIDMap= and GIDMap= give the container user. Rootful UserNS=auto needs subordinate IDs for a containers user, which NixOS does not declare.

Hosts without Nix

The container backends and gateways can be rendered into plain files for any host with podman 5 and systemd, such as Ubuntu 25.04:

packages.x86_64-linux.llmhop-bundle = inputs.llmhop.legacyPackages.x86_64-linux.mkQuadletBundle {
  modules = [
    {
      services.llmhop = {
        gateway.type = "llmhop";
        vllm = {
          enable = true;
          container.image = "docker.io/vllm/vllm-openai:latest";
          models."qwen3-8b" = {
            model = "Qwen/Qwen3-8B";
            port = 8001;
          };
        };
      };
    }
  ];
};

It is the bundle of quadlet-nix named llmhop, which every system importing llmhop’s modules exposes as virtualisation.quadlet.bundle.package, with every rootful Quadlet unit of that system. It defaults services.llmhop.runtime to "quadlet". The bundle holds the Quadlet files, update and refresh units, timers, the llmhop gateway’s socket units and tmpfiles rules below etc/, the scripts and generated configs below /usr/local/lib/llmhop, and an install.sh that copies them into place, starts the servers, restarts every running unit it changed and retires those a previous bundle had but this one lacks. The uninstall.sh installed beside the assets removes the bundle again, keeping the model caches. Native backends and gateways cannot be bundled, and firewall and user options have no effect there.

Native vLLM and SGLang from prebuilt wheels

Native vLLM and SGLang run from upstream’s wheels, pinned in a small uv workspace and built with mkUvEnv:

# vllm-env/pyproject.toml, then run `uv lock`
[project]
name = "vllm-env"
requires-python = "==3.12.*"
dependencies = ["vllm==0.16.2"]

mkUvEnv builds against the lowest admitted Python and bakes the GPU driver runpath into the wheels, while the driver itself comes from hardware.graphics and your vendor configuration. models.<name>.package overrides the package per model, for example with a nightly.

  • JIT compilers find nvcc through CUDA_HOME, which mkCudaHome { packages = [ ... ]; } builds from the cudaPackages_* set named by the environment’s passthru.cudaPackagesAttr.
  • AMD and Intel GPUs work with vLLM’s ROCm or XPU wheels, added as an explicit [[tool.uv.index]] per vLLM’s installation docs.
  • sdists without a build system need their build dependencies under [tool.uv.extra-build-dependencies], as in antlr4-python3-runtime = ["setuptools"].
  • flashinfer packages must be pinned to the flashinfer-python version vLLM pins, or flashinfer refuses to start.
  • Native libraries come from buildInputs for linked ones, runtimePaths for ones loaded by a bare dlopen, such as libsndfile or rdma-core, driverLibs for host driver sonames and optionalLibs for libraries that load on demand only. The build fails naming every unresolved library.

Inference server credentials

Credentials are granted per model, so no worker can read another’s keys. credentials.<name> maps a name to a file outside the Nix store, or to null to import it from the system credential store, and ${cred:name} in settings or environment expands to its read-only path:

services.llmhop.llama-cpp.models."qwen3-8b" = {
  model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
  credentials.api-keys = "/run/secrets/qwen-api-keys";
  settings.api-key-file = "\${cred:api-keys}";
};

Provision an imported credential as a root-only file or encrypted:

(umask 077; systemd-ask-password -n > /etc/credstore/llmhop.hf-token)
systemd-ask-password -n | systemd-creds encrypt --name=llmhop.hf-token - /etc/credstore.encrypted/llmhop.hf-token

vLLM and SGLang read a YAML file with api-key through settings.config, as in credentials."config.yaml" with settings.config = "\${cred:config.yaml}", while llama.cpp uses api-key-file. Containers see their credentials at /run/llmhop/credentials, named by $CREDENTIALS_DIRECTORY as natively, and readable through a user namespace with the backend’s or gateway’s container.mountOptions.credentials, see user namespaces.

vLLM watermarking

Both vLLM backends can watermark a model’s output and run detectors serving POST /detect beside the workers. A watermark holds vLLM’s WatermarkConfig without the key as settings, plus the credential holding the key, and a detector tests every one of its profiles:

services.llmhop.vllm = {
  models.qwen = {
    model = "Qwen/Qwen3-8B";
    revision = "main";
    watermark = {
      credential = "vllm.watermark-key.2026-10";
      settings.algorithm = "dual_key_gumbel";
    };
  };
  detectors.production = {
    profiles.qwen = {
      tokenizer = "Qwen/Qwen3-8B";
      inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
    };
    settings.p-value-threshold = 0.01;
  };
};

A key is an unsigned 64-bit integer credential, never entering the Nix store or a command line:

(umask 077; od -An -N8 -tu8 /dev/urandom | tr -d ' \n' > /etc/credstore/vllm.watermark-key.2026-10)

A detector reports the most significant profile with its Bonferroni-corrected p_value, score, num_scored_tokens and is_watermarked. To rotate a key, provision it under a new credential name and keep a profile with the old one while old text still needs checking. llmhop routes detectors as unlisted models, as in POST /route/production/detect, while other gateways leave them to direct clients. Watermarking requires vLLM 0.30.0 or newer.

Module internals

Decisions behind the NixOS and system-manager modules in nix/modules/ that are hard to re-derive from the code. Both import the matching module of quadlet-nix, so they share all of llmhop’s modules. Every backend and gateway selects its runtime as native or quadlet, defaulting to services.llmhop.runtime, and mkQuadletBundle defaults that to quadlet.

CLI dialects

Each engine declares how its parser reads booleans (negateBools) and lists (listStyle). nixpkgs cannot render both: lib.cli.toCommandLine always repeats list flags and never sees the value when naming a flag, and lib.cli.toGNUCommandLine is deprecated. Non-string values go through JSON, which keeps 0.6 from becoming 0.600000.

Worker hardening

Native units share a systemd-exec(5) baseline, while containers rely on Podman’s isolation. The baseline denies binding IP sockets with SocketBindDeny = "any", which a worker with a port lifts for that port.

GPU relaxations

  • PrivateDevices = false and the render and video groups give access to the CUDA, ROCm and Level Zero device nodes. A model’s native.devices narrows it with DevicePolicy = "closed" and DeviceAllow. CUDA needs /dev/nvidiactl and char-nvidia-uvm beside each /dev/nvidia<N>, ROCm /dev/kfd beside each /dev/dri/renderD<N>, and Level Zero only the latter.
  • PrivateUsers = "identity", since true would map those groups to nobody.
  • MemoryDenyWriteExecute = false for the JIT compilers of torch, triton, CUDA and SYCL.
  • LimitMEMLOCK = "infinity", since every stack pins host buffers and the default 8 MiB reports OOM.
  • ProcSubset = "all" for /proc/meminfo and /proc/cpuinfo.

NCCL relaxations

getifaddrs() needs AF_NETLINK, and NCCL binds ephemeral TCP ports, which the bind hook sees as port 0, so all TCP is allowed. NCCL and Gloo are kept on loopback and NCCL off InfiniBand, which RCCL honors too. Gloo would otherwise bind the address the hostname resolves to, such as 127.0.0.2 on NixOS, which the IP address filter of an offline server drops.

Caches

Every accelerator stack caches compiled kernels below $HOME, which is read-only, so each cache variable points into the workload’s cache directory, set unconditionally since an absent stack never reads it.

Unix sockets

Each socket lives in a directory of its own below socketDirectory, so no worker can delete or squat another’s socket. The root is root:root with mode 0711 and each directory 0710, since clients know the name, and servers run with umask 0007, since they never chmod their socket.

A native worker’s socket belongs to the backend’s group, which every user of socketClients joins, since systemd drops ACLs when it chowns an exec directory. A container user maps to an unpredictable host ID, so container sockets rely on a default ACL user:<client>:-wx on the root instead. That is why llmhop runs as a named user rather than a DynamicUser.

Each directory is a RuntimeDirectory= of its unit, removed on every stop, so no stale socket causes EADDRINUSE, which is why socketDirectory must stay below /run.

Python backends built from wheels

The wheel-based backends get a compiler, ninja and bash on PATH, since triton, torch and flashinfer compile at runtime and look them up the FHS way, and HOME points at the cache directory. LD_LIBRARY_PATH covers driver libraries loaded by name, such as libnvidia-ml.so.1, and TRITON_LIBCUDA_PATH replaces triton’s ldconfig -p lookup. They cannot use DynamicUser, whose ID-mapped state and cache mounts are noexec, while these runtimes load the kernels they compile there.

Lifecycle

Servers use Restart = "always", since vLLM and SGLang exit 0 after an engine crash, with a start limit of 3 per hour against crash loops. A native server is Type=exec with an ExecStartPost= of llmhop internal wait-healthy that polls /health and also watches $MAINPID, so a crashed server does not stay activating until the timeout. Containers use Podman’s Notify=healthy. Startup is chained by name, since two workers profiling GPU memory on one device both see it free and run out of memory. The llmhop gateway is PartOf= its listeners, so restarting a socket also restarts the gateway that holds it. Listener configuration changes also trigger a native gateway restart on NixOS.

Fetching

A fetcher is a oneshot with RemainAfterExit=, which its server Wants= and is ordered after, but does not Requires=, so a failed fetch leaves a running server alone. It shares its server’s identity, so both see the cache under one owner, and every workload has its own cache, so no fetch changes what another server loaded. Fetchers keep systemd’s default start limit, since they do not restart automatically and may need several manual retries.

Every start or restart of a fetcher also starts <unit>-refresh, which it Wants=. The refresh unit Requires= and follows the fetcher, so it only acts on a complete download. It prunes the cache to the workload’s repositories and revisions, then restarts the server. Telling a changed model from an unchanged one would need state, since a moved ref can leave its previous commit pinned by another revision, so every successful fetch restarts a running server, which the weekly default of fetch.schedule keeps rare. It resets the server’s start limit first, so planned restarts do not use it up, and try-restart leaves a stopped or failed server down, so a server that hit its start limit waits for a manual start. The server starts after it, so a first download finds the server still waiting rather than restarting it.

The fetcher controls everything in the cache, so no command touching it runs with more privileges than the cache owner, or a planted symlink could redirect a privileged write or deletion. The refresh unit prunes as the workload’s user, writing only the hub cache and opening no socket, since a root owner could otherwise reach systemd or Podman. Only its systemctl step runs as root, and it reads no file.

huggingface_hub moves a ref before downloading its snapshot, so an interrupted download leaves the ref on an incomplete snapshot, which an offline server fails to load until the next complete download.

<backend>-update restarts every fetcher one at a time in startup order, so the server restarts keep to the startup chain.

Claims

Ports, unit names and model names admit one owner each, but NixOS merges clashing units silently or fails opaquely, and port clashes only fail at boot. So every owner adds its option path to the internal services.llmhop.claims.<kind>.<key>, and an assertion names all owners of a clash. Endpoints are keyed by owner too, so a name clash reaches that assertion instead of failing the merge.

Directory layout

Each workload’s cache.directory/<name> is the HF_HOME of its fetcher and server, holding the hub cache in hub/, which the server can only read, beside the server’s runtime caches. It is no CacheDirectory=, which would have to stay below /var/cache, and containers mount it at the same path. Without cache.directory, a container mounts the named volume <unit> there instead, which Podman creates on first use, so no tmpfiles rule is needed. A fetcher rules this out, since its refresh unit prunes the hub cache on the host.

Offline servers

Servers get HF_HUB_OFFLINE and llama.cpp --offline only with fetch.enable, since otherwise they download their repositories themselves, and vLLM always gets VLLM_HOST_IP, since it would otherwise derive its address from a route to a public host. A socket server gets its own network namespace, keeping lo for NCCL. A native server on a loopback port is limited to 127.0.0.1 and ::1 by the cgroup IP filter, not all of 127.0.0.0/8, which holds the systemd-resolved stub, and cannot see /run/nscd and /run/systemd/resolve. The filter applies to both directions, so it cannot isolate a server that accepts clients beyond loopback, and a container’s published port needs a network. The internal workload option isolable captures this, deriving the default of offline and the one assertion rejecting offline = true where it would not hold, so no runtime silently runs an offline server with network access.

Gateways

At most one gateway runs, since chained gateways duplicate routing and auth, and a model reachable beside the gateway bypasses its auth. Backends publish their workloads to the internal services.llmhop.endpoints, which the selected gateway renders, so a new gateway needs no backend changes. The gateway.type enum in core.nix lists every gateway module. Only llmhop routes detectors, which neither SMG’s discovery nor Bifrost’s providers can describe. SMG gets a discovery file, since it gives up on --worker-urls after a startup timeout shorter than model loading. External gateways run as dynamic users or on the host network in a container, reaching models on loopback. Both llmhop runtimes share their listen options, socket units, claims, firewall rules and tmpfiles in gateways/llmhop.nix, and never bind themselves, since their config must not set listen. Podman passes the sockets of the units listed in the container’s Sockets= into the container, which stays on the host network.

Watermarking

Models and detectors share the runtime renderers, with an explicit workload kind selecting their execution settings. Catalog visibility is derived separately when publishing endpoints.

mkVllmWatermark’s serve merges the key into vLLM’s watermark config in-process, and its detect replaces vLLM’s example detector, which lacks unix sockets, health, file-based keys, several profiles and most algorithms. Native builds check the script against the vLLM environment, while containers run it unchecked with the image’s Python.

Bundles

Bundles come from quadlet-nix, which renders every rootful Quadlet unit of the system, the NixOS services named by bundle.services with their timers, the sockets named by bundle.sockets, and its bundle.tmpfiles, and copies the store paths they reference below bundle.assetDirectory. Every container backend names its update and refresh units there, which the bundle renders without the store PATH= of NixOS, so they run systemctl and llmhop by their absolute paths. llmhop names the bundle, adds its 10-llmhop tmpfiles rules and rejects native backends and gateways through bundle.assertions, which only the bundle checks. The container llmhop gateway names its llmhop-<name> listeners in bundle.sockets, so the bundle carries their socket units. The build fails if a copied file still references the store, so only self-contained files such as scripts, configs, the static llmhop binary and its image archive can be bundled.

Containers

The llmhop container loads an archive of the image built with the binary that validated its config, so the running version always matches, it needs no registry, and it updates with the system or bundle rather than through podman auto-update.

The container rootfs stays writable, since ML runtimes scatter caches across HOME, and /tmp is a tmpfs. Every container of a backend runs the backend’s image unit, which quadlet-nix starts only when a dependent starts. Backend containers, fetchers and gateways are PartOf= their image unit, so an image restart also restarts its running dependents. A registry image unit pulls with Policy=missing, since podman image pull otherwise contacts the registry on every start, so a boot during an outage would fail. container.imageUnit = false references the image directly with Podman’s default Pull=missing, which pulls within the server’s start timeout and drops that restart handle, while a changed image still changes and thereby restarts every container.

Core

services.llmhop.runtime

Default runtime of every backend and gateway, natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

"native"

services.llmhop.socketClients

Users that may connect to the unix socket of every workload without a port, such as a reverse proxy in front of them. Each joins the group of every native backend and gets a default ACL on socketDirectory for the containers. A gateway reaching sockets adds its own user.

Type: list of string

Default:

[ ]

Example:

[
  "caddy"
]

services.llmhop.socketDirectory

Directory of every unix socket a workload or llmhop serves, with a RuntimeDirectory= per workload without a port, hence below /run.

Type: string matching the pattern /run(/[[:alnum:]][[:alnum:].-]*)+

Default:

"/run/llmhop"

gateway

services.llmhop.gateway.active

Whether the gateway of type runs, which needs at least one endpoint, so a host whose backends are all disabled, such as a test VM without GPUs, keeps its gateway config without running it. Units relying on the gateway, such as clients joining its socket group, can follow it.

Type: boolean (read only)

Default:

config.services.llmhop.gateway.type != null && config.services.llmhop.endpoints != { }

services.llmhop.gateway.bifrost.package

Package providing bifrost-http as its main program, taken from Bifrost’s own flake, since nixpkgs lacks it. Only used if runtime is "native".

Type: package

services.llmhop.gateway.bifrost.container.config

[Container] keys of every container of Bifrost, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.bifrost.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/maximhq/bifrost:v2.3.2"

services.llmhop.gateway.bifrost.container.mountOptions.credentials

Podman volume options of the credential mount of every container of Bifrost, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.bifrost.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.bifrost.environment

Environment variables of Bifrost. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.bifrost.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to Bifrost.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.bifrost.host

IP address Bifrost listens on.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.gateway.bifrost.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.bifrost.port

TCP port Bifrost listens on.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

8080

services.llmhop.gateway.bifrost.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.bifrost.settings

Bifrost’s config.json, with every listed model as a key of its vllm or sgl provider. Changes from its web UI or API persist until the matching section of this file changes, while source_of_truth = "config.json" prunes them and config_store.enabled = false disables them. Use upstream env. references with environmentFile for secrets.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  encryption_key = "env.BIFROST_ENCRYPTION_KEY";
  governance = {
    virtual_keys = [
      {
        id = "default";
        is_active = true;
        name = "default";
        value = "env.BIFROST_VIRTUAL_KEY";
      }
    ];
  };
}

services.llmhop.gateway.llmhop.package

The llmhop package to use.

Type: package

Default:

config.services.llmhop.package

services.llmhop.gateway.llmhop.container.config

[Container] keys of every container of the llmhop gateway, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.llmhop.container.image

Container image reference, including a tag or digest, or null for an archive of package, which validated the config at build time and never auto-updates. Only used if runtime is "quadlet".

Type: null or string

Default:

null

Example:

"ghcr.io/mirkolenz/llmhop:latest"

services.llmhop.gateway.llmhop.container.mountOptions.credentials

Podman volume options of the credential mount of every container of the llmhop gateway, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.llmhop.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

llmhop reads its own config, so in settings the reference expands to the credential’s contents rather than to its path.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.llmhop.environment

Environment variables of the llmhop gateway. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.llmhop.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to the llmhop gateway.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.llmhop.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.gateway.llmhop.uid

services.llmhop.gateway.llmhop.group

Primary group of user. The module declares it while it keeps the name llmhop, and otherwise the deployer declares it. Only used if runtime is "native".

Type: string

Default:

config.services.llmhop.gateway.llmhop.user

services.llmhop.gateway.llmhop.listen

Listeners of the gateway, each a llmhop-<name>.socket unit handing its socket to it through socket activation. The default one always exists, on port 8080 unless configured otherwise.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  caddy = {
    socketGroup = "caddy";
  };
  default = {
    host = "";
  };
}

services.llmhop.gateway.llmhop.listen.<name>.host

IP address to bind port to. An empty string binds every interface, leaving access control to the firewall. IPv6 literals are written plain (e.g. ::1).

Type: string

Default:

"127.0.0.1"

Example:

""

services.llmhop.gateway.llmhop.listen.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.llmhop.listen.<name>.port

TCP port to listen on. null listens on the unix socket socket instead. The default listener defaults to 8080.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

Example:

8080

services.llmhop.gateway.llmhop.listen.<name>.socket

Unix socket to listen on while port is null.

Type: string

Default:

"${config.services.llmhop.socketDirectory}/‹name›.sock"

services.llmhop.gateway.llmhop.listen.<name>.socketGroup

Group of socket, typically the one of a reverse proxy in front of llmhop.

Type: string

Default:

"root"

Example:

"caddy"

services.llmhop.gateway.llmhop.listen.<name>.socketMode

Mode of socket. Connecting needs write permission.

Type: string

Default:

"0660"

services.llmhop.gateway.llmhop.listen.<name>.socketUser

Owner of socket.

Type: string

Default:

"root"

services.llmhop.gateway.llmhop.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.llmhop.settings

llmhop’s JSON config, validated at build time by the binary itself, with every endpoint of the backends as a model. Its listeners come from listen, so settings must not set it.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  authTokens = {
    client = "\${cred:llmhop.client-token}";
  };
  models = {
    "openai/gpt-4" = {
      url = "https://api.openai.com";
    };
  };
}

services.llmhop.gateway.llmhop.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.gateway.llmhop.user

System user the gateway runs as. The module declares it while it keeps the name llmhop, and otherwise the deployer declares it. Only used if runtime is "native".

Type: string

Default:

"llmhop"

services.llmhop.gateway.smg.package

Python environment providing the smg package, built with the flake’s mkUvEnv, since nixpkgs lacks SMG. Only used if runtime is "native".

Type: package

services.llmhop.gateway.smg.container.config

[Container] keys of every container of SMG, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.smg.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"ghcr.io/smg-project/smg:nightly"

services.llmhop.gateway.smg.container.mountOptions.credentials

Podman volume options of the credential mount of every container of SMG, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.smg.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.smg.environment

Environment variables of SMG. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.smg.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to SMG.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.smg.host

IP address SMG listens on.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.gateway.smg.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.smg.port

TCP port SMG listens on.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

8080

services.llmhop.gateway.smg.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.smg.settings

CLI flags of SMG, rendered as --<key> <value> with false dropped. The listener and the discovery file of every listed model always win, while metrics on 127.0.0.1:29000 and disabled tokenizer downloads are defaults. Credential references resolve to file paths for upstream options such as api-key-file and tenant-api-keys-file, which require a supporting SMG build.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  api-key-file = "\${cred:smg.api-key}";
  policy = "cache_aware";
}

services.llmhop.gateway.type

The one gateway in front of every backend, configured by the option set of the same name, or null for none, so clients connect to each workload directly. It routes every listed model, which must then stay on loopback and leave authentication to the gateway.

Type: null or one of “llmhop”, “smg”, “bifrost”

Default:

null

Example:

"llmhop"

llama-cpp

services.llmhop.llama-cpp.enable

Whether to enable llama.cpp model serving.

Type: boolean

Default:

false

Example:

true

services.llmhop.llama-cpp.package

The llama-cpp package. Only used if runtime is "native".

Type: package

Default:

pkgs.llama-cpp

services.llmhop.llama-cpp.cache.directory

Root of the cache directories of this backend’s workloads, created with mode 0711, each <name> the HF_HOME of one fetcher and its server, created with mode 0700 for user and group. The fetcher writes its hub/, which the server only reads. Point it at a data disk for large models. null keeps each cache in the Podman named volume <unit> instead, which needs no tmpfiles rules but only works for a container backend without fetch.enable.

Type: null or absolute path not in the Nix store

Default:

"/var/cache/llama-cpp"

services.llmhop.llama-cpp.container.config

[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.llama-cpp.container.devices

Devices exposed to every model container as Quadlet AddDevice= lines, CDI references such as nvidia.com/gpu=all or host device paths. Per-model container.devices replaces it. Only used if runtime is "quadlet".

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.llama-cpp.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"ghcr.io/ggml-org/llama.cpp:server-cuda"

services.llmhop.llama-cpp.container.imageUnit

Whether every container runs image through the image unit llama-cpp-image, which pulls it once for all of them and restarts them when restarted. Otherwise each container references image directly and pulls it while it is missing. Only used if runtime is "quadlet".

Type: boolean

Default:

true

services.llmhop.llama-cpp.container.mountOptions.cache

Podman volume options of the cache mounts below cache.directory, such as idmap so a user namespace owns them, or SELinux relabeling. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.llama-cpp.container.mountOptions.credentials

Podman volume options of the credential mount of every container of this backend, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.llama-cpp.environment

Environment variables set on every model service, under the environment of each model, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.llama-cpp.environmentFile

File in KEY=VALUE format forwarded to every server, loaded before the environmentFile of each model. Prefer credentials for any secret a server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

Example:

"/etc/llama-cpp/.env"

services.llmhop.llama-cpp.fetch.enable

Whether every workload has a fetcher <unit>-fetch downloading its repositories before its server starts, with a refresh unit and llama-cpp-update, after which the server runs offline. Otherwise each server downloads its repositories itself when it starts, which requires offline = false, and none of the other fetch options apply.

Type: boolean

Default:

true

Example:

true

services.llmhop.llama-cpp.fetch.credentials

Credentials granted to every fetcher, in the format of a workload’s credentials. Reference them from fetch.environment as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.llama-cpp.fetch.environment

Environment variables set on every fetcher, such as a proxy. Values may reference fetch.credentials as ${cred:<name>}. The servers never see them.

Type: attribute set of string

Default:

{ }

Example:

{
  HTTPS_PROXY = "http://proxy.example:3128";
}

services.llmhop.llama-cpp.fetch.environmentFile

File in KEY=VALUE format forwarded to every fetcher, for variables that only exist as values.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.llama-cpp.fetch.hfTokenCredential

Credential holding a Hugging Face token for gated repositories, imported from the system credential store unless fetch.credentials names a file for it. Only the fetchers receive it, as HF_TOKEN, and skip it while it is missing.

Type: null or string

Default:

null

Example:

"llmhop.hf-token"

services.llmhop.llama-cpp.fetch.schedule

OnCalendar= schedule of llama-cpp-update, which runs every fetcher and restarts each running server after a successful fetch, even if its model did not change. null disables scheduled fetching.

Type: null or string

Default:

"weekly"

Example:

"daily"

services.llmhop.llama-cpp.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.llama-cpp.uid

services.llmhop.llama-cpp.group

Primary group of user. If runtime is "native", the module declares it while it keeps the name llama-cpp, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.llama-cpp.runtime == "native" then config.services.llmhop.llama-cpp.user else "root"

services.llmhop.llama-cpp.modelSettings

CLI flags forwarded to the server of every model, under the settings of each model. Rendered as --<key> <value>, with false as --no-<key> and a list repeating the flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.llama-cpp.models

Models to serve, each as the unit llama-cpp-<name> beside its fetcher llama-cpp-<name>-fetch, natively a hardened system service and otherwise a container, whose units virtualisation.quadlet.containers.<unit> override.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b".model = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
}

services.llmhop.llama-cpp.models.<name>.enable

Whether to enable model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.llama-cpp.models.<name>.package

Package providing this model’s server, overriding the backend-wide package, e.g. a nightly build for a just-released architecture. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.llama-cpp.package

services.llmhop.llama-cpp.models.<name>.container.devices

Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".

Type: list of string

Default:

config.services.llmhop.llama-cpp.container.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.llama-cpp.models.<name>.container.shmSize

Size of the container’s private /dev/shm tmpfs, which PyTorch uses for tensor-parallel inference. Only used if the backend’s runtime is "quadlet".

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.llama-cpp.models.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.llama-cpp.models.<name>.environment

Environment variables of this model’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.llama-cpp.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.llama-cpp.models.<name>.extraRepositories

Further repositories the fetcher downloads beside model, such as a speculative draft model or a LoRA adapter that settings reference, since the server cannot download them itself.

Type: list of string

Default:

[ ]

Example:

[
  "Qwen/Qwen3-0.6B"
]

services.llmhop.llama-cpp.models.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.llama-cpp.models.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.llama-cpp.models.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.llama-cpp.models.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.llama-cpp.models.<name>.model

Hugging Face repository with an optional quantization tag, passed as --hf-repo, or an absolute path to a GGUF file, passed as --model.

A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.

Type: string

Example:

"unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL"

services.llmhop.llama-cpp.models.<name>.name

Name of this model in its unit llama-cpp-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.llama-cpp.models.<name>.native.devices

Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".

Type: null or (list of string)

Default:

config.services.llmhop.llama-cpp.native.devices

Example:

[
  "/dev/nvidia0"
]

services.llmhop.llama-cpp.models.<name>.offline

Whether this model’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.llama-cpp.models.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this model. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.llama-cpp.models.<name>.port

Host port the server binds to on host. null binds the unix socket socket instead, which claims no port, only the gateway and services.llmhop.socketClients can connect to, and keeps the server off any network.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.llama-cpp.models.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.llama-cpp.models.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.llama-cpp.models.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.llama-cpp.models.<name>.settings

CLI flags of this model’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false as --no-<key> and a list repeating the flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.llama-cpp.models.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/llama-cpp-<name>/http.sock while port is null, else null

services.llmhop.llama-cpp.native.devices

Devices every model service may access as systemd DeviceAllow= entries under DevicePolicy=closed, while null allows all. Per-model native.devices replaces it. Only used if runtime is "native".

Type: null or (list of string)

Default:

null

Example:

[
  "/dev/nvidiactl"
  "char-nvidia-uvm"
  "/dev/nvidia0"
]

services.llmhop.llama-cpp.openFilesLimit

File descriptor limit (LimitNOFILE) of every server of this backend. Increase it if a server logs accept: Too many open files.

Type: positive integer, meaning >0

Default:

1048576

services.llmhop.llama-cpp.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.llama-cpp.startupOrdering

Whether to start model services one at a time by ascending name, so two workers profiling memory on the same GPU do not race into an OOM. Disable only when each model pins itself to a dedicated device, natively via its own native.devices or environment, e.g. CUDA_VISIBLE_DEVICES, or in a container via its own container.devices.

Type: boolean

Default:

true

services.llmhop.llama-cpp.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.llama-cpp.user

System user the native units and any refresh units run as, which owns their directories. If runtime is "native", the module declares it while it keeps the name llama-cpp, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.llama-cpp.runtime == "native" then "llama-cpp" else "root"

sglang

services.llmhop.sglang.enable

Whether to enable SGLang model serving.

Type: boolean

Default:

false

Example:

true

services.llmhop.sglang.package

Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server, built from a uv workspace with the flake’s mkUvEnv, as in inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./sglang-env; }. Only used if runtime is "native".

Type: package

services.llmhop.sglang.cache.directory

Root of the cache directories of this backend’s workloads, created with mode 0711, each <name> the HF_HOME of one fetcher and its server, created with mode 0700 for user and group. The fetcher writes its hub/, which the server only reads. Point it at a data disk for large models. null keeps each cache in the Podman named volume <unit> instead, which needs no tmpfiles rules but only works for a container backend without fetch.enable.

Type: null or absolute path not in the Nix store

Default:

"/var/cache/sglang"

services.llmhop.sglang.container.config

[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.sglang.container.devices

Devices exposed to every model container as Quadlet AddDevice= lines, CDI references such as nvidia.com/gpu=all or host device paths. Per-model container.devices replaces it. Only used if runtime is "quadlet".

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.sglang.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/lmsysorg/sglang:latest"

services.llmhop.sglang.container.imageUnit

Whether every container runs image through the image unit sglang-image, which pulls it once for all of them and restarts them when restarted. Otherwise each container references image directly and pulls it while it is missing. Only used if runtime is "quadlet".

Type: boolean

Default:

true

services.llmhop.sglang.container.mountOptions.cache

Podman volume options of the cache mounts below cache.directory, such as idmap so a user namespace owns them, or SELinux relabeling. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.sglang.container.mountOptions.credentials

Podman volume options of the credential mount of every container of this backend, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.sglang.environment

Environment variables set on every model service, under the environment of each model, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.environmentFile

File in KEY=VALUE format forwarded to every server, loaded before the environmentFile of each model. Prefer credentials for any secret a server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

Example:

"/etc/sglang/.env"

services.llmhop.sglang.fetch.enable

Whether every workload has a fetcher <unit>-fetch downloading its repositories before its server starts, with a refresh unit and sglang-update, after which the server runs offline. Otherwise each server downloads its repositories itself when it starts, which requires offline = false, and none of the other fetch options apply.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang.fetch.credentials

Credentials granted to every fetcher, in the format of a workload’s credentials. Reference them from fetch.environment as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.sglang.fetch.environment

Environment variables set on every fetcher, such as a proxy. Values may reference fetch.credentials as ${cred:<name>}. The servers never see them.

Type: attribute set of string

Default:

{ }

Example:

{
  HTTPS_PROXY = "http://proxy.example:3128";
}

services.llmhop.sglang.fetch.environmentFile

File in KEY=VALUE format forwarded to every fetcher, for variables that only exist as values.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.sglang.fetch.hfTokenCredential

Credential holding a Hugging Face token for gated repositories, imported from the system credential store unless fetch.credentials names a file for it. Only the fetchers receive it, as HF_TOKEN, and skip it while it is missing.

Type: null or string

Default:

null

Example:

"llmhop.hf-token"

services.llmhop.sglang.fetch.schedule

OnCalendar= schedule of sglang-update, which runs every fetcher and restarts each running server after a successful fetch, even if its model did not change. null disables scheduled fetching.

Type: null or string

Default:

"weekly"

Example:

"daily"

services.llmhop.sglang.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.sglang.uid

services.llmhop.sglang.group

Primary group of user. If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.sglang.runtime == "native" then config.services.llmhop.sglang.user else "root"

services.llmhop.sglang.modelSettings

CLI flags forwarded to the server of every model, under the settings of each model. Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.sglang.models

Models to serve, each as the unit sglang-<name> beside its fetcher sglang-<name>-fetch, natively a hardened system service and otherwise a container, whose units virtualisation.quadlet.containers.<unit> override.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b".model = "Qwen/Qwen3-8B";
}

services.llmhop.sglang.models.<name>.enable

Whether to enable model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang.models.<name>.package

Package providing this model’s server, overriding the backend-wide package, e.g. a nightly build for a just-released architecture. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.sglang.package

services.llmhop.sglang.models.<name>.container.devices

Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".

Type: list of string

Default:

config.services.llmhop.sglang.container.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.sglang.models.<name>.container.shmSize

Size of the container’s private /dev/shm tmpfs, which PyTorch uses for tensor-parallel inference. Only used if the backend’s runtime is "quadlet".

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.sglang.models.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.sglang.models.<name>.environment

Environment variables of this model’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.sglang.models.<name>.extraRepositories

Further repositories the fetcher downloads beside model, such as a speculative draft model or a LoRA adapter that settings reference, since the server cannot download them itself.

Type: list of string

Default:

[ ]

Example:

[
  "Qwen/Qwen3-0.6B"
]

services.llmhop.sglang.models.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.sglang.models.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.sglang.models.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.sglang.models.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.sglang.models.<name>.model

Hugging Face repository passed as --model-path, or an absolute path. A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.

Type: string

Example:

"Qwen/Qwen3-8B"

services.llmhop.sglang.models.<name>.name

Name of this model in its unit sglang-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.sglang.models.<name>.native.devices

Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".

Type: null or (list of string)

Default:

config.services.llmhop.sglang.native.devices

Example:

[
  "/dev/nvidia0"
]

services.llmhop.sglang.models.<name>.offline

Whether this model’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.sglang.models.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this model. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.sglang.models.<name>.port

Host port the server binds to on host.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.sglang.models.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.sglang.models.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.sglang.models.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.sglang.models.<name>.revision

Branch, tag or commit of model that the fetcher downloads and the server loads. null follows the default branch, so fetch.schedule picks up upstream changes, while a commit pins the model.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.sglang.models.<name>.settings

CLI flags of this model’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.sglang.models.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null

services.llmhop.sglang.native.devices

Devices every model service may access as systemd DeviceAllow= entries under DevicePolicy=closed, while null allows all. Per-model native.devices replaces it. Only used if runtime is "native".

Type: null or (list of string)

Default:

null

Example:

[
  "/dev/nvidiactl"
  "char-nvidia-uvm"
  "/dev/nvidia0"
]

services.llmhop.sglang.openFilesLimit

File descriptor limit (LimitNOFILE) of every server of this backend. Increase it if a server logs accept: Too many open files.

Type: positive integer, meaning >0

Default:

1048576

services.llmhop.sglang.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.sglang.startupOrdering

Whether to start model services one at a time by ascending name, so two workers profiling memory on the same GPU do not race into an OOM. Disable only when each model pins itself to a dedicated device, natively via its own native.devices or environment, e.g. CUDA_VISIBLE_DEVICES, or in a container via its own container.devices.

Type: boolean

Default:

true

services.llmhop.sglang.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.sglang.user

System user the native units and any refresh units run as, which owns their directories. If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.sglang.runtime == "native" then "sglang" else "root"

vllm

services.llmhop.vllm.enable

Whether to enable vLLM model serving.

Type: boolean

Default:

false

Example:

true

services.llmhop.vllm.package

Package providing the vllm CLI at bin/vllm, built from a uv workspace with the flake’s mkUvEnv, as in inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; }. Only used if runtime is "native".

Type: package

services.llmhop.vllm.cache.directory

Root of the cache directories of this backend’s workloads, created with mode 0711, each <name> the HF_HOME of one fetcher and its server, created with mode 0700 for user and group. The fetcher writes its hub/, which the server only reads. Point it at a data disk for large models. null keeps each cache in the Podman named volume <unit> instead, which needs no tmpfiles rules but only works for a container backend without fetch.enable.

Type: null or absolute path not in the Nix store

Default:

"/var/cache/vllm"

services.llmhop.vllm.container.config

[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.vllm.container.devices

Devices exposed to every model container as Quadlet AddDevice= lines, CDI references such as nvidia.com/gpu=all or host device paths. Per-model container.devices replaces it. Only used if runtime is "quadlet".

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.vllm.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/vllm/vllm-openai:latest"

services.llmhop.vllm.container.imageUnit

Whether every container runs image through the image unit vllm-image, which pulls it once for all of them and restarts them when restarted. Otherwise each container references image directly and pulls it while it is missing. Only used if runtime is "quadlet".

Type: boolean

Default:

true

services.llmhop.vllm.container.mountOptions.cache

Podman volume options of the cache mounts below cache.directory, such as idmap so a user namespace owns them, or SELinux relabeling. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.vllm.container.mountOptions.credentials

Podman volume options of the credential mount of every container of this backend, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.vllm.detectors

Standalone watermark detection services. The llmhop gateway routes them as unlisted models, while every other gateway leaves them to direct clients.

Type: attribute set of (submodule)

Default:

{ }

services.llmhop.vllm.detectors.<name>.enable

Whether to enable watermark detector ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.detectors.<name>.package

Environment this detector runs in. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.vllm.package

services.llmhop.vllm.detectors.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.detectors.<name>.environment

Environment variables of this watermark detector’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.detectors.<name>.environmentFile

File in KEY=VALUE format forwarded to this watermark detector’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.detectors.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.vllm.detectors.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.vllm.detectors.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.detectors.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.vllm.detectors.<name>.name

Name of this watermark detector in its unit vllm-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm.detectors.<name>.offline

Whether this watermark detector’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.vllm.detectors.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this watermark detector. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.vllm.detectors.<name>.port

Host port the server binds to on host. null binds the unix socket socket instead, which claims no port, only the gateway and services.llmhop.socketClients can connect to, and keeps the server off any network.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.detectors.<name>.profiles

Generation configurations to test, keyed by the name a response reports as profile. Profiles sharing a tokenizer and revision share its encoding of the text, and the reported p-value is that of the most significant profile multiplied by their number.

Type: attribute set of (submodule)

Example:

{
  qwen = {
    tokenizer = "Qwen/Qwen3-8B";
    inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
  };
}

services.llmhop.vllm.detectors.<name>.profiles.<name>.revision

Branch, tag or commit of tokenizer, which must match the revision of the generating model, or its settings.tokenizer-revision. null follows the default branch.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.vllm.detectors.<name>.profiles.<name>.tokenizer

Tokenizer used to encode candidate text, a Hugging Face repository or an absolute path. It must exactly match the tokenizer used for watermarked generation. The fetcher downloads only the files of a repository that a tokenizer may need, not its weights.

Type: string

Example:

"Qwen/Qwen3-8B"

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark

The watermark of the generating model.

Type: submodule

Default:

{ }

Example:

{
  credential = "vllm.watermark-key.2026-10";
  settings = {
    algorithm = "dual_key_gumbel";
  };
}

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.credential

Credential holding the key, an unsigned 64-bit integer, imported from the system credential store unless the service’s credentials names a file for it. A new name per key, such as a date suffix, restarts every service using it when the key is rotated.

Type: string

Default:

"vllm.watermark-key"

Example:

"vllm.watermark-key.2026-10"

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.settings

vLLM’s WatermarkConfig without key.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.detectors.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.vllm.detectors.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.vllm.detectors.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.vllm.detectors.<name>.script

Detector server run by the detector’s Python interpreter. It receives detect and settings, the profiles among them as JSON, and must serve GET /health and POST /detect. p-value-threshold is the only detection-side setting of the default.

Type: absolute path

Default: the script of mkVllmWatermark, built for package if the backend’s runtime is "native"

services.llmhop.vllm.detectors.<name>.settings

CLI flags of this watermark detector’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.detectors.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null

services.llmhop.vllm.environment

Environment variables set on every model service, under the environment of each model, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.environmentFile

File in KEY=VALUE format forwarded to every server, loaded before the environmentFile of each model. Prefer credentials for any secret a server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

Example:

"/etc/vllm/.env"

services.llmhop.vllm.fetch.enable

Whether every workload has a fetcher <unit>-fetch downloading its repositories before its server starts, with a refresh unit and vllm-update, after which the server runs offline. Otherwise each server downloads its repositories itself when it starts, which requires offline = false, and none of the other fetch options apply.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.fetch.credentials

Credentials granted to every fetcher, in the format of a workload’s credentials. Reference them from fetch.environment as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.fetch.environment

Environment variables set on every fetcher, such as a proxy. Values may reference fetch.credentials as ${cred:<name>}. The servers never see them.

Type: attribute set of string

Default:

{ }

Example:

{
  HTTPS_PROXY = "http://proxy.example:3128";
}

services.llmhop.vllm.fetch.environmentFile

File in KEY=VALUE format forwarded to every fetcher, for variables that only exist as values.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.fetch.hfTokenCredential

Credential holding a Hugging Face token for gated repositories, imported from the system credential store unless fetch.credentials names a file for it. Only the fetchers receive it, as HF_TOKEN, and skip it while it is missing.

Type: null or string

Default:

null

Example:

"llmhop.hf-token"

services.llmhop.vllm.fetch.schedule

OnCalendar= schedule of vllm-update, which runs every fetcher and restarts each running server after a successful fetch, even if its model did not change. null disables scheduled fetching.

Type: null or string

Default:

"weekly"

Example:

"daily"

services.llmhop.vllm.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.vllm.uid

services.llmhop.vllm.group

Primary group of user. If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.vllm.runtime == "native" then config.services.llmhop.vllm.user else "root"

services.llmhop.vllm.modelSettings

CLI flags forwarded to the server of every model, under the settings of each model. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.models

Models to serve, each as the unit vllm-<name> beside its fetcher vllm-<name>-fetch, natively a hardened system service and otherwise a container, whose units virtualisation.quadlet.containers.<unit> override.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b".model = "Qwen/Qwen2.5-7B-Instruct";
}

services.llmhop.vllm.models.<name>.enable

Whether to enable model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.models.<name>.package

Package providing this model’s server, overriding the backend-wide package, e.g. a nightly build for a just-released architecture. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.vllm.package

services.llmhop.vllm.models.<name>.container.devices

Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".

Type: list of string

Default:

config.services.llmhop.vllm.container.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.vllm.models.<name>.container.shmSize

Size of the container’s private /dev/shm tmpfs, which PyTorch uses for tensor-parallel inference. Only used if the backend’s runtime is "quadlet".

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.vllm.models.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.models.<name>.environment

Environment variables of this model’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.models.<name>.extraRepositories

Further repositories the fetcher downloads beside model, such as a speculative draft model or a LoRA adapter that settings reference, since the server cannot download them itself.

Type: list of string

Default:

[ ]

Example:

[
  "Qwen/Qwen3-0.6B"
]

services.llmhop.vllm.models.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.vllm.models.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.vllm.models.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.models.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.vllm.models.<name>.model

Hugging Face repository passed to vllm serve, or an absolute path. A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.

Type: string

Example:

"Qwen/Qwen2.5-7B-Instruct"

services.llmhop.vllm.models.<name>.name

Name of this model in its unit vllm-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm.models.<name>.native.devices

Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".

Type: null or (list of string)

Default:

config.services.llmhop.vllm.native.devices

Example:

[
  "/dev/nvidia0"
]

services.llmhop.vllm.models.<name>.offline

Whether this model’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.vllm.models.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this model. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.vllm.models.<name>.port

Host port the server binds to on host. null binds the unix socket socket instead, which claims no port, only the gateway and services.llmhop.socketClients can connect to, and keeps the server off any network.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.models.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.vllm.models.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.vllm.models.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.vllm.models.<name>.revision

Branch, tag or commit of model that the fetcher downloads and the server loads. null follows the default branch, so fetch.schedule picks up upstream changes, while a commit pins the model.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.vllm.models.<name>.settings

CLI flags of this model’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.models.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null

services.llmhop.vllm.models.<name>.watermark

The settings of vLLM’s WatermarkConfig without key, which a detector profile must repeat, and the credential holding the key.

Type: null or (submodule)

Default:

null

Example:

{
  credential = "vllm.watermark-key.2026-10";
  settings = {
    algorithm = "dual_key_gumbel";
  };
}

services.llmhop.vllm.models.<name>.watermark.credential

Credential holding the key, an unsigned 64-bit integer, imported from the system credential store unless the service’s credentials names a file for it. A new name per key, such as a date suffix, restarts every service using it when the key is rotated.

Type: string

Default:

"vllm.watermark-key"

Example:

"vllm.watermark-key.2026-10"

services.llmhop.vllm.models.<name>.watermark.settings

vLLM’s WatermarkConfig without key.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.native.devices

Devices every model service may access as systemd DeviceAllow= entries under DevicePolicy=closed, while null allows all. Per-model native.devices replaces it. Only used if runtime is "native".

Type: null or (list of string)

Default:

null

Example:

[
  "/dev/nvidiactl"
  "char-nvidia-uvm"
  "/dev/nvidia0"
]

services.llmhop.vllm.openFilesLimit

File descriptor limit (LimitNOFILE) of every server of this backend. Increase it if a server logs accept: Too many open files.

Type: positive integer, meaning >0

Default:

1048576

services.llmhop.vllm.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.vllm.startupOrdering

Whether to start model services one at a time by ascending name, so two workers profiling memory on the same GPU do not race into an OOM. Disable only when each model pins itself to a dedicated device, natively via its own native.devices or environment, e.g. CUDA_VISIBLE_DEVICES, or in a container via its own container.devices.

Type: boolean

Default:

true

services.llmhop.vllm.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.vllm.user

System user the native units and any refresh units run as, which owns their directories. If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.vllm.runtime == "native" then "vllm" else "root"