services.llmhop.vllm.enable
Whether to enable vLLM model serving via systemd (native host process), fronted by llmhop.
Type: boolean
Default:
false
Example:
true
services.llmhop.vllm.package
Package providing the vllm CLI at bin/vllm.
No default on purpose: vLLM has no one-derivation-fits-all (new model architectures routinely need dev snapshots, and the wheels come in per-accelerator variants), so you build the package from a uv workspace and pin / follow upstream there. The flake exposes a helper:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./vllm-env; # your pyproject.toml + uv.lock
}
Individual models may override this with models.<name>.package.
Type: package
Example:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./vllm-env;
}
services.llmhop.vllm.environment
Environment variables set on every model service.
Merged with services.llmhop.vllm.models.<name>.environment; per-model
entries take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.environmentFile
File in KEY=VALUE format forwarded to every service.
Use for secrets managed by sops-nix/agenix, e.g. a file containing
HF_TOKEN=<token> to access gated Hugging Face repositories.
Loaded before services.llmhop.vllm.models.<name>.environmentFile, so
per-model files override these entries.
Type: null or absolute path
Default:
null
Example:
"/etc/vllm/.env"
services.llmhop.vllm.gid
Host GID assigned to services.llmhop.vllm.group.
Defaults to uid.
Type: unsigned integer, meaning >=0
Default:
config.services.llmhop.vllm.uid
services.llmhop.vllm.group
Primary group for services.llmhop.vllm.user.
Defaults to the user name (matching the typical 1:1 user/group layout).
Type: string
Default:
config.services.llmhop.vllm.user
services.llmhop.vllm.modelSettings
CLI flags forwarded to the model server for every model.
true collapses to --<key>, null and empty lists are dropped, and an
attribute set is serialised to JSON.
false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false.
A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.
Merged with services.llmhop.vllm.models.<name>.settings; per-model
entries take precedence.
Type: attribute set of anything
Default:
{ }
services.llmhop.vllm.models
Models to serve.
Each enabled entry produces one systemd service named vllm-<name>;
the attribute name is the routing key surfaced through llmhop as the
OpenAI model field.
Enabled entries are sorted by ascending port.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen2-5-7b" = {
model = "Qwen/Qwen2.5-7B-Instruct";
port = 18001;
};
"llama-3-8b" = {
model = "meta-llama/Meta-Llama-3-8B-Instruct";
port = 18002;
settings.max-model-len = 8192;
};
}
services.llmhop.vllm.models.<name>.enable
Whether to enable serving of model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.vllm.models.<name>.package
Package providing this model’s worker, overriding the backend-wide
package. Set it for a model that needs a different vllm
release than the rest — e.g. a nightly wheel for a just-released
architecture — built the same way with mkUvEnv over a per-model
uv workspace. Defaults to the backend-wide package.
Type: package
Default:
config.services.llmhop.vllm.package
services.llmhop.vllm.models.<name>.environment
Additional environment variables set on this model’s service.
Merged with services.llmhop.vllm.environment; per-model entries
take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.vllm.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s service.
Loaded after services.llmhop.vllm.environmentFile, so its entries
override global ones. Must be readable by the user systemd reads it as.
Type: null or absolute path
Default:
null
services.llmhop.vllm.models.<name>.model
Hugging Face repo id (or local path) passed as the vllm serve positional argument.
Type: string
Example:
"Qwen/Qwen2.5-7B-Instruct"
services.llmhop.vllm.models.<name>.name
Canonical identifier for this model. Used for the unit name
(vllm-<name>) and as the routing key registered with llmhop
(clients select the backend by sending this value in the OpenAI
model field).
Defaults to the attribute key, so the key itself must match the required label format.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.vllm.models.<name>.port
Loopback host port vllm binds to (--host 127.0.0.1 --port <port>).
Must be unique per enabled model; llmhop reaches the backend at
http://127.0.0.1:<port>.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.vllm.models.<name>.serviceConfig
Extra [Service] settings merged into this model’s
vllm-<name> unit after the hardened baseline and
backend-specific relaxations. The module retains ownership of
ExecStart, KillMode, and Type because they implement readiness
supervision as one lifecycle contract.
Type: attribute set of anything
Default:
{ }
Example:
{
MemoryHigh = "64G";
}
services.llmhop.vllm.models.<name>.settings
CLI flags forwarded to the model server for this model.
true collapses to --<key>, null and empty lists are dropped, and an
attribute set is serialised to JSON.
false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false.
A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.
Merged with services.llmhop.vllm.modelSettings; per-model entries
take precedence.
Type: attribute set of anything
Default:
{ }
services.llmhop.vllm.openFilesLimit
File descriptor limit (LimitNOFILE) applied to every vllm systemd unit.
Increase if the server logs accept: Too many open files under concurrent load.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.vllm.startupOrdering
Whether to chain enabled model services by ascending port during startup.
GPU-memory profiling races otherwise: two workers booting on the same device
each see it as fully free and race to claim their share, leading to OOM.
Disable only when each model pins itself to a dedicated device via environment (the variable is stack-specific: CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ZE_AFFINITY_MASK, …).
Type: boolean
Default:
true
services.llmhop.vllm.uid
Host UID assigned to services.llmhop.vllm.user.
Required — pick a value that does not clash with other system users on the
host.
Type: unsigned integer, meaning >=0
Example:
503
services.llmhop.vllm.user
Dedicated system user owning the vllm data and cache
directories. Defaults to the backend name; override to point at a
user the deployer manages externally (in which case the matching
users.users.<name> and users.groups.<name> declarations become the
deployer’s responsibility).
Type: string
Default:
"vllm"