services.llmhop.vllm.enable
Whether to enable vLLM model serving via systemd (native host process), fronted by llmhop.
Type: boolean
Default:
false
Example:
true
services.llmhop.vllm.package
Package providing the vllm CLI at bin/vllm.
No default on purpose: vLLM has no one-derivation-fits-all (new model architectures routinely need dev snapshots, and the wheels come in per-accelerator variants), so you build the package from a uv workspace and pin / follow upstream there. The flake exposes a helper:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./vllm-env; # your pyproject.toml + uv.lock
}
Individual models may override this with models.<name>.package.
Type: package
Example:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./vllm-env;
}
services.llmhop.vllm.detectors
Standalone watermark detection services.
Type: attribute set of (submodule)
Default:
{ }
services.llmhop.vllm.detectors.<name>.enable
Whether to enable watermark detector ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.detectors.<name>.package
Environment this detector runs in.
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.detectors.<name>.credentials
Credentials granted exclusively to this service through systemd.
A path outside the Nix store uses LoadCredential=. The attribute form
can select LoadCredentialEncrypted= for a systemd-creds encrypted
source, which must be encrypted under the same name, or omit source
to import the credential of that name from the system credential store.
That store is shared by every service, so prefix an imported name with
its service, as in llmhop.hf-token.
Reference the resulting read-only file from settings as
${cred:<name>}. The module resolves the reference to the native or
container credential path without copying its contents to the Nix store
or command line.
Type: attribute set of ((submodule) or absolute path convertible to it)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
tls-key = {
source = "/run/secrets/tls-key.cred";
encrypted = true;
};
"llmhop.hf-token" = { };
}
services.llmhop.vllm.detectors.<name>.credentials.<name>.encrypted
Whether to load and decrypt source with LoadCredentialEncrypted=.
Imported credentials are decrypted as needed.
Type: boolean
Default:
false
services.llmhop.vllm.detectors.<name>.credentials.<name>.source
File or socket from which systemd loads the credential. null
imports the credential of the same name with ImportCredential=
from the system credential store, such as /etc/credstore and
/etc/credstore.encrypted, and from the credentials passed to the
system.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.detectors.<name>.environment
Additional environment variables set on this watermark detector’s service.
Merged with services.llmhop.vllm.environment; per-watermark detector entries
take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.detectors.<name>.environmentFile
File in KEY=VALUE format forwarded to this watermark detector’s service.
Loaded after services.llmhop.vllm.environmentFile, so its entries
override global ones. Prefer credentials for any secret the server
can read from a file. Must be readable by the user systemd reads it as.
Type: null or absolute path
Default:
null
services.llmhop.vllm.detectors.<name>.name
Canonical identifier for this watermark detector. Used for the unit name
(vllm-detector-<name>) and as the routing key registered with llmhop,
which clients send in the model field. Shares one namespace with
every other routing key, so a collision fails evaluation.
Defaults to the attribute key, so the key itself must match the required label format.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.detectors.<name>.port
Loopback port on which the detector serves /detect.
null binds the unix socket socket instead, which claims no port
and only llmhop can connect to.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.detectors.<name>.script
Detector server run by the detector’s Python interpreter.
It receives detect, --tokenizer, --watermark-config,
--watermark-key-file, the listener flags (--host and --port,
or --uds) and settings, and must serve GET /health and
POST /detect.
The native default is checked against package at build time, the
Quadlet one only at startup, since the image is opaque to the build.
Type: absolute path
Default:
the script built by mkVllmWatermark
services.llmhop.vllm.detectors.<name>.serviceConfig
Extra [Service] settings merged into this workload’s
vllm-detector-<name> unit after the hardened baseline and
backend-specific relaxations. The module retains ownership of
ExecStart, KillMode, and Type because they implement readiness
supervision as one lifecycle contract.
Type: attribute set of anything
Default:
{ }
Example:
{
MemoryHigh = "64G";
}
services.llmhop.vllm.detectors.<name>.settings
Arguments passed to the detector script.
p-value-threshold is the only detection-side setting.
tokenizer, the watermark flags and the listener (host, port,
uds) are derived from the options and always win over entries set
here.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of anything
Default:
{ }
Example:
{
p-value-threshold = 0.01;
}
services.llmhop.vllm.detectors.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-detector-<name>/http.sock while port is null, else null
services.llmhop.vllm.detectors.<name>.tokenizer
Tokenizer used to encode candidate text. It must exactly match the tokenizer used for watermarked generation.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.vllm.detectors.<name>.unitConfig
Extra [Unit] settings merged into this workload’s
vllm-detector-<name> unit after the shared baseline.
Ordering and dependency directives (After=, Requires=, Wants=)
do not belong here: NixOS renders those from the after, requires
and wants options, so a definition of the same key in unitConfig
conflicts with it instead of merging. Declare them on
systemd.services."vllm-detector-<name>" from your own module,
where the module system concatenates them with what this one sets.
Type: attribute set of anything
Default:
{ }
Example:
{
StartLimitBurst = 10;
}
services.llmhop.vllm.detectors.<name>.watermark
vLLM’s WatermarkConfig without key, validated at startup by the
installed release. A generating worker and its detector must share
it.
The key, an unsigned 64-bit integer, is the credential
vllm.watermark-key. It is imported from the system credential store,
for example the root-only file /etc/credstore/vllm.watermark-key,
unless credentials."vllm.watermark-key" names another source. An
encrypted credential must be created under that name, e.g. with
systemd-creds encrypt --name=vllm.watermark-key.
Type: attribute set of anything
Default:
{ }
Example:
{
algorithm = "dual_key_gumbel";
context_width = 4;
}
services.llmhop.vllm.environment
Environment variables set on every model service.
Merged with services.llmhop.vllm.models.<name>.environment; per-model
entries take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.environmentFile
File in KEY=VALUE format forwarded to every service.
Use only for upstream features that require environment variables,
such as HF_TOKEN for gated Hugging Face repositories. Environment
variables are not systemd credentials and are visible to every model,
so prefer credentials for any secret a server can read from a file.
Loaded before services.llmhop.vllm.models.<name>.environmentFile, so
per-model files override these entries.
Type: null or absolute path
Default:
null
Example:
"/etc/vllm/.env"
services.llmhop.vllm.gid
GID of the declared group. null lets NixOS allocate one.
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.vllm.uid
services.llmhop.vllm.group
Primary group of user. The module declares it while it keeps its
default name, any other group is the deployer’s to declare.
Type: string
Default:
config.services.llmhop.vllm.user
services.llmhop.vllm.modelSettings
CLI flags forwarded to the model server for every model.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.vllm.models.<name>.settings; per-model
entries take precedence.
Type: attribute set of anything
Default:
{ }
services.llmhop.vllm.models
Models to serve.
Each enabled entry produces one systemd service named vllm-<name>;
the attribute name is the routing key surfaced through llmhop as the
OpenAI model field.
Enabled entries are sorted by ascending name.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen2-5-7b" = {
model = "Qwen/Qwen2.5-7B-Instruct";
};
"llama-3-8b" = {
model = "meta-llama/Meta-Llama-3-8B-Instruct";
settings.max-model-len = 8192;
};
}
services.llmhop.vllm.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.models.<name>.package
Package providing this model’s worker, overriding the backend-wide
package. Set it for a model that needs a different vllm
release than the rest — e.g. a nightly wheel for a just-released
architecture — built the same way with mkUvEnv over a per-model
uv workspace. Defaults to the backend-wide package.
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.models.<name>.credentials
Credentials granted exclusively to this service through systemd.
A path outside the Nix store uses LoadCredential=. The attribute form
can select LoadCredentialEncrypted= for a systemd-creds encrypted
source, which must be encrypted under the same name, or omit source
to import the credential of that name from the system credential store.
That store is shared by every service, so prefix an imported name with
its service, as in llmhop.hf-token.
Reference the resulting read-only file from settings as
${cred:<name>}. The module resolves the reference to the native or
container credential path without copying its contents to the Nix store
or command line.
Type: attribute set of ((submodule) or absolute path convertible to it)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
tls-key = {
source = "/run/secrets/tls-key.cred";
encrypted = true;
};
"llmhop.hf-token" = { };
}
services.llmhop.vllm.models.<name>.credentials.<name>.encrypted
Whether to load and decrypt source with LoadCredentialEncrypted=.
Imported credentials are decrypted as needed.
Type: boolean
Default:
false
services.llmhop.vllm.models.<name>.credentials.<name>.source
File or socket from which systemd loads the credential. null
imports the credential of the same name with ImportCredential=
from the system credential store, such as /etc/credstore and
/etc/credstore.encrypted, and from the credentials passed to the
system.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.vllm.models.<name>.environment
Additional environment variables set on this model’s service.
Merged with services.llmhop.vllm.environment; per-model entries
take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s service.
Loaded after services.llmhop.vllm.environmentFile, so its entries
override global ones. Prefer credentials for any secret the server
can read from a file. Must be readable by the user systemd reads it as.
Type: null or absolute path
Default:
null
services.llmhop.vllm.models.<name>.model
Hugging Face repo id (or local path) passed as the vllm serve positional argument.
Type: string
Example:
"Qwen/Qwen2.5-7B-Instruct"
services.llmhop.vllm.models.<name>.name
Canonical identifier for this model. Used for the unit name
(vllm-<name>) and as the routing key registered with llmhop,
which clients send in the model field. Shares one namespace with
every other routing key, so a collision fails evaluation.
Defaults to the attribute key, so the key itself must match the required label format.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.models.<name>.port
Loopback host port vllm binds to (--host 127.0.0.1 --port <port>).
Must be unique per enabled model; llmhop reaches the backend at
http://127.0.0.1:<port>.
null binds the unix socket socket instead, which claims no port
and only llmhop can connect to.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.vllm.models.<name>.serviceConfig
Extra [Service] settings merged into this workload’s
vllm-<name> unit after the hardened baseline and
backend-specific relaxations. The module retains ownership of
ExecStart, KillMode, and Type because they implement readiness
supervision as one lifecycle contract.
Type: attribute set of anything
Default:
{ }
Example:
{
MemoryHigh = "64G";
}
services.llmhop.vllm.models.<name>.settings
CLI flags forwarded to the model server for this model.
Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.vllm.modelSettings; per-model entries
take precedence. The flags llmhop derives from the model options
(its served name and listener) always win over both.
Type: attribute set of anything
Default:
{ }
services.llmhop.vllm.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null
services.llmhop.vllm.models.<name>.unitConfig
Extra [Unit] settings merged into this workload’s
vllm-<name> unit after the shared baseline.
Ordering and dependency directives (After=, Requires=, Wants=)
do not belong here: NixOS renders those from the after, requires
and wants options, so a definition of the same key in unitConfig
conflicts with it instead of merging. Declare them on
systemd.services."vllm-<name>" from your own module,
where the module system concatenates them with what this one sets.
Type: attribute set of anything
Default:
{ }
Example:
{
StartLimitBurst = 10;
}
services.llmhop.vllm.models.<name>.watermark
vLLM’s WatermarkConfig without key, validated at startup by the
installed release. A generating worker and its detector must share
it.
The key, an unsigned 64-bit integer, is the credential
vllm.watermark-key. It is imported from the system credential store,
for example the root-only file /etc/credstore/vllm.watermark-key,
unless credentials."vllm.watermark-key" names another source. An
encrypted credential must be created under that name, e.g. with
systemd-creds encrypt --name=vllm.watermark-key.
Type: null or (attribute set of anything)
Default:
null
Example:
{
algorithm = "dual_key_gumbel";
context_width = 4;
}
services.llmhop.vllm.openFilesLimit
File descriptor limit (LimitNOFILE) applied to every vllm systemd unit.
Increase if the server logs accept: Too many open files under concurrent load.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.vllm.startupOrdering
Whether to chain enabled model services by ascending name during startup.
GPU-memory profiling races otherwise: two workers booting on the same device
each see it as fully free and race to claim their share, leading to OOM.
Disable only when each model pins itself to a dedicated device via environment (the variable is stack-specific: CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ZE_AFFINITY_MASK, …).
Type: boolean
Default:
true
services.llmhop.vllm.uid
UID of the declared user. null lets NixOS allocate one.
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.vllm.user
System user the units run as. The module declares it while it keeps its default name, any other user is the deployer’s to declare.
Type: string
Default:
"vllm"