Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

services.llmhop.vllm.enable

Whether to enable vLLM model serving.

Type: boolean

Default:

false

Example:

true

services.llmhop.vllm.package

Package providing the vllm CLI at bin/vllm, built from a uv workspace with the flake’s mkUvEnv, as in inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./vllm-env; }. Only used if runtime is "native".

Type: package

services.llmhop.vllm.cache.directory

Root of the cache directories of this backend’s workloads, created with mode 0711, each <name> the HF_HOME of one fetcher and its server, created with mode 0700 for user and group. The fetcher writes its hub/, which the server only reads. Point it at a data disk for large models. null keeps each cache in the Podman named volume <unit> instead, which needs no tmpfiles rules but only works for a container backend without fetch.enable.

Type: null or absolute path not in the Nix store

Default:

"/var/cache/vllm"

services.llmhop.vllm.container.config

[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.vllm.container.devices

Devices exposed to every model container as Quadlet AddDevice= lines, CDI references such as nvidia.com/gpu=all or host device paths. Per-model container.devices replaces it. Only used if runtime is "quadlet".

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.vllm.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/vllm/vllm-openai:latest"

services.llmhop.vllm.container.imageUnit

Whether every container runs image through the image unit vllm-image, which pulls it once for all of them and restarts them when restarted. Otherwise each container references image directly and pulls it while it is missing. Only used if runtime is "quadlet".

Type: boolean

Default:

true

services.llmhop.vllm.container.mountOptions.cache

Podman volume options of the cache mounts below cache.directory, such as idmap so a user namespace owns them, or SELinux relabeling. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.vllm.container.mountOptions.credentials

Podman volume options of the credential mount of every container of this backend, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.vllm.detectors

Standalone watermark detection services. The llmhop gateway routes them as unlisted models, while every other gateway leaves them to direct clients.

Type: attribute set of (submodule)

Default:

{ }

services.llmhop.vllm.detectors.<name>.enable

Whether to enable watermark detector ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.detectors.<name>.package

Environment this detector runs in. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.vllm.package

services.llmhop.vllm.detectors.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.detectors.<name>.environment

Environment variables of this watermark detector’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.detectors.<name>.environmentFile

File in KEY=VALUE format forwarded to this watermark detector’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.detectors.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.vllm.detectors.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.vllm.detectors.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.detectors.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.vllm.detectors.<name>.name

Name of this watermark detector in its unit vllm-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm.detectors.<name>.offline

Whether this watermark detector’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.vllm.detectors.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this watermark detector. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.vllm.detectors.<name>.port

Host port the server binds to on host. null binds the unix socket socket instead, which claims no port, only the gateway and services.llmhop.socketClients can connect to, and keeps the server off any network.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.detectors.<name>.profiles

Generation configurations to test, keyed by the name a response reports as profile. Profiles sharing a tokenizer and revision share its encoding of the text, and the reported p-value is that of the most significant profile multiplied by their number.

Type: attribute set of (submodule)

Example:

{
  qwen = {
    tokenizer = "Qwen/Qwen3-8B";
    inherit (config.services.llmhop.vllm.models.qwen) revision watermark;
  };
}

services.llmhop.vllm.detectors.<name>.profiles.<name>.revision

Branch, tag or commit of tokenizer, which must match the revision of the generating model, or its settings.tokenizer-revision. null follows the default branch.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.vllm.detectors.<name>.profiles.<name>.tokenizer

Tokenizer used to encode candidate text, a Hugging Face repository or an absolute path. It must exactly match the tokenizer used for watermarked generation. The fetcher downloads only the files of a repository that a tokenizer may need, not its weights.

Type: string

Example:

"Qwen/Qwen3-8B"

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark

The watermark of the generating model.

Type: submodule

Default:

{ }

Example:

{
  credential = "vllm.watermark-key.2026-10";
  settings = {
    algorithm = "dual_key_gumbel";
  };
}

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.credential

Credential holding the key, an unsigned 64-bit integer, imported from the system credential store unless the service’s credentials names a file for it. A new name per key, such as a date suffix, restarts every service using it when the key is rotated.

Type: string

Default:

"vllm.watermark-key"

Example:

"vllm.watermark-key.2026-10"

services.llmhop.vllm.detectors.<name>.profiles.<name>.watermark.settings

vLLM’s WatermarkConfig without key.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.detectors.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.vllm.detectors.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.vllm.detectors.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.vllm.detectors.<name>.script

Detector server run by the detector’s Python interpreter. It receives detect and settings, the profiles among them as JSON, and must serve GET /health and POST /detect. p-value-threshold is the only detection-side setting of the default.

Type: absolute path

Default: the script of mkVllmWatermark, built for package if the backend’s runtime is "native"

services.llmhop.vllm.detectors.<name>.settings

CLI flags of this watermark detector’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.detectors.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null

services.llmhop.vllm.environment

Environment variables set on every model service, under the environment of each model, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.environmentFile

File in KEY=VALUE format forwarded to every server, loaded before the environmentFile of each model. Prefer credentials for any secret a server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

Example:

"/etc/vllm/.env"

services.llmhop.vllm.fetch.enable

Whether every workload has a fetcher <unit>-fetch downloading its repositories before its server starts, with a refresh unit and vllm-update, after which the server runs offline. Otherwise each server downloads its repositories itself when it starts, which requires offline = false, and none of the other fetch options apply.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.fetch.credentials

Credentials granted to every fetcher, in the format of a workload’s credentials. Reference them from fetch.environment as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.fetch.environment

Environment variables set on every fetcher, such as a proxy. Values may reference fetch.credentials as ${cred:<name>}. The servers never see them.

Type: attribute set of string

Default:

{ }

Example:

{
  HTTPS_PROXY = "http://proxy.example:3128";
}

services.llmhop.vllm.fetch.environmentFile

File in KEY=VALUE format forwarded to every fetcher, for variables that only exist as values.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.fetch.hfTokenCredential

Credential holding a Hugging Face token for gated repositories, imported from the system credential store unless fetch.credentials names a file for it. Only the fetchers receive it, as HF_TOKEN, and skip it while it is missing.

Type: null or string

Default:

null

Example:

"llmhop.hf-token"

services.llmhop.vllm.fetch.schedule

OnCalendar= schedule of vllm-update, which runs every fetcher and restarts each running server after a successful fetch, even if its model did not change. null disables scheduled fetching.

Type: null or string

Default:

"weekly"

Example:

"daily"

services.llmhop.vllm.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.vllm.uid

services.llmhop.vllm.group

Primary group of user. If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.vllm.runtime == "native" then config.services.llmhop.vllm.user else "root"

services.llmhop.vllm.modelSettings

CLI flags forwarded to the server of every model, under the settings of each model. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.models

Models to serve, each as the unit vllm-<name> beside its fetcher vllm-<name>-fetch, natively a hardened system service and otherwise a container, whose units virtualisation.quadlet.containers.<unit> override.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b".model = "Qwen/Qwen2.5-7B-Instruct";
}

services.llmhop.vllm.models.<name>.enable

Whether to enable model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.models.<name>.package

Package providing this model’s server, overriding the backend-wide package, e.g. a nightly build for a just-released architecture. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.vllm.package

services.llmhop.vllm.models.<name>.container.devices

Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".

Type: list of string

Default:

config.services.llmhop.vllm.container.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.vllm.models.<name>.container.shmSize

Size of the container’s private /dev/shm tmpfs, which PyTorch uses for tensor-parallel inference. Only used if the backend’s runtime is "quadlet".

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.vllm.models.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.vllm.models.<name>.environment

Environment variables of this model’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.vllm.models.<name>.extraRepositories

Further repositories the fetcher downloads beside model, such as a speculative draft model or a LoRA adapter that settings reference, since the server cannot download them itself.

Type: list of string

Default:

[ ]

Example:

[
  "Qwen/Qwen3-0.6B"
]

services.llmhop.vllm.models.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.vllm.models.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.vllm.models.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.models.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.vllm.models.<name>.model

Hugging Face repository passed to vllm serve, or an absolute path. A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.

Type: string

Example:

"Qwen/Qwen2.5-7B-Instruct"

services.llmhop.vllm.models.<name>.name

Name of this model in its unit vllm-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm.models.<name>.native.devices

Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".

Type: null or (list of string)

Default:

config.services.llmhop.vllm.native.devices

Example:

[
  "/dev/nvidia0"
]

services.llmhop.vllm.models.<name>.offline

Whether this model’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.vllm.models.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this model. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.vllm.models.<name>.port

Host port the server binds to on host. null binds the unix socket socket instead, which claims no port, only the gateway and services.llmhop.socketClients can connect to, and keeps the server off any network.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.vllm.models.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.vllm.models.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.vllm.models.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.vllm.models.<name>.revision

Branch, tag or commit of model that the fetcher downloads and the server loads. null follows the default branch, so fetch.schedule picks up upstream changes, while a commit pins the model.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.vllm.models.<name>.settings

CLI flags of this model’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false as --no-<key> and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.models.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/vllm-<name>/http.sock while port is null, else null

services.llmhop.vllm.models.<name>.watermark

The settings of vLLM’s WatermarkConfig without key, which a detector profile must repeat, and the credential holding the key.

Type: null or (submodule)

Default:

null

Example:

{
  credential = "vllm.watermark-key.2026-10";
  settings = {
    algorithm = "dual_key_gumbel";
  };
}

services.llmhop.vllm.models.<name>.watermark.credential

Credential holding the key, an unsigned 64-bit integer, imported from the system credential store unless the service’s credentials names a file for it. A new name per key, such as a date suffix, restarts every service using it when the key is rotated.

Type: string

Default:

"vllm.watermark-key"

Example:

"vllm.watermark-key.2026-10"

services.llmhop.vllm.models.<name>.watermark.settings

vLLM’s WatermarkConfig without key.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.vllm.native.devices

Devices every model service may access as systemd DeviceAllow= entries under DevicePolicy=closed, while null allows all. Per-model native.devices replaces it. Only used if runtime is "native".

Type: null or (list of string)

Default:

null

Example:

[
  "/dev/nvidiactl"
  "char-nvidia-uvm"
  "/dev/nvidia0"
]

services.llmhop.vllm.openFilesLimit

File descriptor limit (LimitNOFILE) of every server of this backend. Increase it if a server logs accept: Too many open files.

Type: positive integer, meaning >0

Default:

1048576

services.llmhop.vllm.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.vllm.startupOrdering

Whether to start model services one at a time by ascending name, so two workers profiling memory on the same GPU do not race into an OOM. Disable only when each model pins itself to a dedicated device, natively via its own native.devices or environment, e.g. CUDA_VISIBLE_DEVICES, or in a container via its own container.devices.

Type: boolean

Default:

true

services.llmhop.vllm.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.vllm.user

System user the native units and any refresh units run as, which owns their directories. If runtime is "native", the module declares it while it keeps the name vllm, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.vllm.runtime == "native" then "vllm" else "root"