services.llmhop.vllm.enable
Whether to enable vLLM model serving.
Type: boolean
Default:
false
Example:
true
services.llmhop.vllm.package
Package providing the vllm CLI at bin/vllm, built from
a uv workspace with the flake’s mkUvEnv, as in
inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; }.
Only used if runtime is "native".
Type: package
services.llmhop.vllm.cache.directory
Root of the cache directories of this backend’s workloads, created
with mode 0711, each <name> the HF_HOME of one fetcher and its
server, created with mode 0700 for user and group. The
fetcher writes its hub/, which the server only reads. Point it at
a data disk for large models.
null keeps each cache in the Podman named volume <unit> instead,
which needs no tmpfiles rules but only works for a container backend
without fetch.enable.
Type: null or absolute path not in the Nix store
Default:
"/var/cache/vllm"
services.llmhop.vllm.container.config
[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.vllm.container.devices
Devices exposed to every model container as Quadlet AddDevice=
lines, CDI references such as nvidia.com/gpu=all or host device
paths. Per-model container.devices replaces it.
Only used if runtime is "quadlet".
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.vllm.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/vllm/vllm-openai:latest"
services.llmhop.vllm.container.imageUnit
Whether every container runs image through the image unit
vllm-image, which pulls it once for all of them and restarts
them when restarted. Otherwise each container references image
directly and pulls it while it is missing.
Only used if runtime is "quadlet".
Type: boolean
Default:
true
services.llmhop.vllm.container.mountOptions.cache
Podman volume options of the cache mounts below
cache.directory, such as idmap so a user namespace owns
them, or SELinux relabeling.
Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.vllm.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
this backend, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.vllm.detectors
Standalone watermark detection services. The llmhop gateway routes them as unlisted models, while every other gateway leaves them to direct clients.
Type: attribute set of (submodule)
Default:
{ }
services.llmhop.vllm.detectors.<name>.enable
Whether to enable watermark detector ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.detectors.<name>.package
Environment this detector runs in. Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.detectors.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.detectors.<name>.environment
Environment variables of this watermark detector’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.detectors.<name>.environmentFile
File in KEY=VALUE format forwarded to this watermark detector’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.detectors.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.vllm.detectors.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.vllm.detectors.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.detectors.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.vllm.detectors.<name>.name
Name of this watermark detector in its unit vllm-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.detectors.<name>.offline
Whether this watermark detector’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.vllm.detectors.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this watermark detector.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.vllm.detectors.<name>.port
Host port the server binds to on host.
null binds the unix socket socket instead, which claims no
port, only the gateway and services.llmhop.socketClients can
connect to, and keeps the server off any network.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.detectors.<name>.profiles
Generation configurations to test, keyed by the name a response
reports as profile. Profiles sharing a tokenizer and revision
share its encoding of the text, and the reported p-value is that
of the most significant profile multiplied by their number.
Type: attribute set of (submodule)
Example:
{
qwen = {
tokenizer = "Qwen/Qwen3-8B";
inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
};
}
services.llmhop.vllm.detectors.<name>.profiles.<name>.revision
Branch, tag or commit of tokenizer, which must match the
revision of the generating model, or its
settings.tokenizer-revision. null follows the default branch.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.vllm.detectors.<name>.profiles.<name>.tokenizer
Tokenizer used to encode candidate text, a Hugging Face repository or an absolute path. It must exactly match the tokenizer used for watermarked generation. The fetcher downloads only the files of a repository that a tokenizer may need, not its weights.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark
The watermark of the generating model.
Type: submodule
Default:
{ }
Example:
{
credential = "vllm.watermark-key.2026-10";
settings = {
algorithm = "dual_key_gumbel";
};
}
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.credential
Credential holding the key, an unsigned 64-bit integer, imported from
the system credential store unless the service’s credentials names
a file for it. A new name per key, such as a date suffix, restarts
every service using it when the key is rotated.
Type: string
Default:
"vllm.watermark-key"
Example:
"vllm.watermark-key.2026-10"
services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.settings
vLLM’s WatermarkConfig without key.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.detectors.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.vllm.detectors.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.vllm.detectors.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.vllm.detectors.<name>.script
Detector server run by the detector’s Python interpreter. It
receives detect and settings, the profiles among them as
JSON, and must serve GET /health and POST /detect.
p-value-threshold is the only detection-side setting of the
default.
Type: absolute path
Default:
the script of mkVllmWatermark, built for package if the backend’s runtime is "native"
services.llmhop.vllm.detectors.<name>.settings
CLI flags of this watermark detector’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.detectors.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null
services.llmhop.vllm.environment
Environment variables set on every model service, under the
environment of each model, where ${cred:<name>} expands to the
path of a credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.environmentFile
File in KEY=VALUE format forwarded to every server, loaded before
the environmentFile of each model. Prefer credentials for any
secret a server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
Example:
"/etc/vllm/.env"
services.llmhop.vllm.fetch.enable
Whether every workload has a fetcher <unit>-fetch downloading
its repositories before its server starts, with a refresh unit and
vllm-update, after which the server runs offline. Otherwise
each server downloads its repositories itself when it starts, which
requires offline = false, and none of the other fetch options
apply.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.fetch.credentials
Credentials granted to every fetcher, in the format of a
workload’s credentials. Reference them from fetch.environment
as ${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.fetch.environment
Environment variables set on every fetcher, such as a proxy. Values
may reference fetch.credentials as ${cred:<name>}.
The servers never see them.
Type: attribute set of string
Default:
{ }
Example:
{
HTTPS_PROXY = "http://proxy.example:3128";
}
services.llmhop.vllm.fetch.environmentFile
File in KEY=VALUE format forwarded to every fetcher, for
variables that only exist as values.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.fetch.hfTokenCredential
Credential holding a Hugging Face token for gated repositories,
imported from the system credential store unless
fetch.credentials names a file for it. Only the fetchers receive
it, as HF_TOKEN, and skip it while it is missing.
Type: null or string
Default:
null
Example:
"llmhop.hf-token"
services.llmhop.vllm.fetch.schedule
OnCalendar= schedule of vllm-update, which runs every
fetcher and restarts each running server after a successful fetch,
even if its model did not change.
null disables scheduled fetching.
Type: null or string
Default:
"weekly"
Example:
"daily"
services.llmhop.vllm.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.vllm.uid
services.llmhop.vllm.group
Primary group of user.
If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.vllm.runtime == "native" then config.services.llmhop.vllm.user else "root"
services.llmhop.vllm.modelSettings
CLI flags forwarded to the server of every model, under the
settings of each model.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.models
Models to serve, each as the unit vllm-<name> beside its
fetcher vllm-<name>-fetch, natively a hardened system service
and otherwise a container, whose units
virtualisation.quadlet.containers.<unit> override.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b".model = "Qwen/Qwen2.5-7B-Instruct";
}
services.llmhop.vllm.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.models.<name>.package
Package providing this model’s server, overriding the
backend-wide package, e.g. a nightly build for a
just-released architecture.
Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.models.<name>.container.devices
Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".
Type: list of string
Default:
config.services.llmhop.vllm.container.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.vllm.models.<name>.container.shmSize
Size of the container’s private /dev/shm tmpfs, which
PyTorch uses for tensor-parallel inference.
Only used if the backend’s runtime is "quadlet".
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.vllm.models.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.vllm.models.<name>.environment
Environment variables of this model’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.models.<name>.extraRepositories
Further repositories the fetcher downloads beside model, such as a
speculative draft model or a LoRA adapter that settings
reference, since the server cannot download them itself.
Type: list of string
Default:
[ ]
Example:
[
"Qwen/Qwen3-0.6B"
]
services.llmhop.vllm.models.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.vllm.models.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.vllm.models.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.models.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.vllm.models.<name>.model
Hugging Face repository passed to vllm serve, or an absolute path.
A repository is downloaded by the fetcher, while an absolute path,
in a container one inside it, is served as is.
Type: string
Example:
"Qwen/Qwen2.5-7B-Instruct"
services.llmhop.vllm.models.<name>.name
Name of this model in its unit vllm-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.models.<name>.native.devices
Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".
Type: null or (list of string)
Default:
config.services.llmhop.vllm.native.devices
Example:
[
"/dev/nvidia0"
]
services.llmhop.vllm.models.<name>.offline
Whether this model’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.vllm.models.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this model.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.vllm.models.<name>.port
Host port the server binds to on host.
null binds the unix socket socket instead, which claims no
port, only the gateway and services.llmhop.socketClients can
connect to, and keeps the server off any network.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.models.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.vllm.models.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.vllm.models.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.vllm.models.<name>.revision
Branch, tag or commit of model that the fetcher downloads and the
server loads. null follows the default branch, so fetch.schedule
picks up upstream changes, while a commit pins the model.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.vllm.models.<name>.settings
CLI flags of this model’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null
services.llmhop.vllm.models.<name>.watermark
The settings of vLLM’s WatermarkConfig without key, which a
detector profile must repeat, and the credential holding the key.
Type: null or (submodule)
Default:
null
Example:
{
credential = "vllm.watermark-key.2026-10";
settings = {
algorithm = "dual_key_gumbel";
};
}
services.llmhop.vllm.models.<name>.watermark.credential
Credential holding the key, an unsigned 64-bit integer, imported from
the system credential store unless the service’s credentials names
a file for it. A new name per key, such as a date suffix, restarts
every service using it when the key is rotated.
Type: string
Default:
"vllm.watermark-key"
Example:
"vllm.watermark-key.2026-10"
services.llmhop.vllm.models.<name>.watermark.settings
vLLM’s WatermarkConfig without key.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.vllm.native.devices
Devices every model service may access as systemd DeviceAllow=
entries under DevicePolicy=closed, while null allows all.
Per-model native.devices replaces it.
Only used if runtime is "native".
Type: null or (list of string)
Default:
null
Example:
[
"/dev/nvidiactl"
"char-nvidia-uvm"
"/dev/nvidia0"
]
services.llmhop.vllm.openFilesLimit
File descriptor limit (LimitNOFILE) of every server of this
backend. Increase it if a server logs accept: Too many open files.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.vllm.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.vllm.startupOrdering
Whether to start model services one at a time by ascending name,
so two workers profiling memory on the same GPU do not race into an
OOM. Disable only when each model pins itself to a dedicated device,
natively via its own native.devices or environment, e.g.
CUDA_VISIBLE_DEVICES, or in a container via its own
container.devices.
Type: boolean
Default:
true
services.llmhop.vllm.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.vllm.user
System user the native units and any refresh units run as, which owns their directories.
If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.vllm.runtime == "native" then "vllm" else "root"