services.llmhop.sglang-quadlet.enable
Whether to enable SGLang model serving via Quadlet, optionally fronted by the SGL Model Gateway.
Type: boolean
Default:
false
Example:
true
services.llmhop.sglang-quadlet.cache.containerDirectory
Path at which the cache is mounted inside every model container.
Type: string
Default:
"/root/.cache/huggingface"
services.llmhop.sglang-quadlet.cache.directory
Host directory bind-mounted as the Hugging Face cache.
Type: absolute path
Default:
"/var/cache/sglang"
services.llmhop.sglang-quadlet.cache.environmentVariable
Environment variable set on every container to point its runtime
at containerDirectory.
Type: string
Default:
"HF_HOME"
services.llmhop.sglang-quadlet.cache.group
Host group used when cache.manage is enabled.
Type: string
Default:
if config.services.llmhop.sglang-quadlet.quadlet.user == null then
"root"
else
config.services.llmhop.sglang-quadlet.quadlet.user.group
services.llmhop.sglang-quadlet.cache.manage
Whether llmhop creates the host cache directory with
systemd-tmpfiles, owned by user/group and mode 0700.
Type: boolean
Default:
true
services.llmhop.sglang-quadlet.cache.mountOptions
Options appended to the cache’s Quadlet Volume= entry. This can
be used for Podman ownership mechanisms such as U, idmap, or
SELinux relabeling.
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.sglang-quadlet.cache.user
Host owner used when cache.manage is enabled. Override it when a
UIDMap/idmap mapping makes the container see a different owner
than the host account running Podman.
Type: string
Default:
if config.services.llmhop.sglang-quadlet.quadlet.user == null then
"root"
else
config.services.llmhop.sglang-quadlet.quadlet.user.name
services.llmhop.sglang-quadlet.devices
Devices exposed to every model container — passed verbatim as Quadlet
AddDevice= lines. Accepts both CDI references (recommended:
nvidia.com/gpu=…, amd.com/gpu=…, intel.com/gpu=…, …) and raw
host device paths (e.g. /dev/dri/renderD128). For CDI, the
corresponding spec must be generated on the host (e.g.
nvidia-ctk cdi generate).
Defaults to [ "nvidia.com/gpu=all" ] when
hardware.nvidia-container-toolkit.enable is set, otherwise empty
(CPU-only). Per-model devices overrides this.
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then
[ "nvidia.com/gpu=all" ]
else
[ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.sglang-quadlet.environment
Environment variables set on every model service.
Merged with services.llmhop.sglang-quadlet.models.<name>.environment; per-model
entries take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang-quadlet.environmentFile
File in KEY=VALUE format forwarded to every service.
Use only for upstream features that require environment variables,
such as HF_TOKEN for gated Hugging Face repositories. Environment
variables are not systemd credentials and are visible to every model,
so prefer credentials for any secret a server can read from a file.
Loaded before services.llmhop.sglang-quadlet.models.<name>.environmentFile, so
per-model files override these entries.
Type: null or absolute path
Default:
null
Example:
"/etc/sglang-quadlet/.env"
services.llmhop.sglang-quadlet.gateway.enable
Whether to enable the SGL Model Gateway in front of the workers. Disabled by default — llmhop already routes between every backend, and the gateway is only needed when you want SGLang’s IGW dispatch features (custom routing, prefix caching across workers, etc.) .
Type: boolean
Default:
false
Example:
true
services.llmhop.sglang-quadlet.gateway.enableMetrics
Whether to enable Prometheus metrics on the gateway.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang-quadlet.gateway.bindAddress
Host address the gateway binds its listeners to. Defaults to the loopback so external clients must go through Caddy / llmhop.
Type: string
Default:
"127.0.0.1"
services.llmhop.sglang-quadlet.gateway.credentials
Credentials granted exclusively to this service through systemd.
A path outside the Nix store uses LoadCredential=. The attribute form
can select LoadCredentialEncrypted= for a systemd-creds encrypted
source, which must be encrypted under the same name, or omit source
to import the credential of that name from the system credential store.
That store is shared by every service, so prefix an imported name with
its service, as in llmhop.hf-token.
Reference the resulting read-only file from settings as
${cred:<name>}. The module resolves the reference to the native or
container credential path without copying its contents to the Nix store
or command line.
Type: attribute set of ((submodule) or absolute path convertible to it)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
tls-key = {
source = "/run/secrets/tls-key.cred";
encrypted = true;
};
"llmhop.hf-token" = { };
}
services.llmhop.sglang-quadlet.gateway.credentials.<name>.encrypted
Whether to load and decrypt source with LoadCredentialEncrypted=.
Imported credentials are decrypted as needed.
Type: boolean
Default:
false
services.llmhop.sglang-quadlet.gateway.credentials.<name>.source
File or socket from which systemd loads the credential. null
imports the credential of the same name with ImportCredential=
from the system credential store, such as /etc/credstore and
/etc/credstore.encrypted, and from the credentials passed to the
system.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang-quadlet.gateway.digest
Immutable digest of the gateway image. Mutually exclusive with tag.
Type: null or string
Default:
null
services.llmhop.sglang-quadlet.gateway.environment
Additional environment variables set on the gateway container.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang-quadlet.gateway.environmentFile
File in KEY=VALUE format forwarded to the gateway via --env-file.
Use only for upstream features that require environment variables,
and prefer credentials for any secret the gateway can read from a
file.
Type: null or absolute path
Default:
null
Example:
"/etc/sglang/gateway.env"
services.llmhop.sglang-quadlet.gateway.image
Container image used for the gateway.
Type: string
Default:
"docker.io/lmsysorg/sgl-model-gateway"
services.llmhop.sglang-quadlet.gateway.metricsPort
Host port the gateway exposes Prometheus metrics on.
Ignored when enableMetrics is false.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
29000
services.llmhop.sglang-quadlet.gateway.port
Host port the gateway listens on.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.sglang-quadlet.gateway.quadlet.containerConfig
Extra [Container] settings applied to the gateway container.
Keys use Quadlet’s native PascalCase names, including User,
UserNS, UIDMap, GIDMap, SubUIDMap, and SubGIDMap.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.gateway.quadlet.extraConfig
Extra unit sections applied to the gateway container after all generated
sections. This is the final escape hatch for settings that do not
fit one of the dedicated *Config options.
Type: attribute set of attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.gateway.quadlet.mountOptions.credentials
Podman volume options appended to the gateway container’s systemd
credential mount. Use an idmap mapping when [Container] User=
selects a non-root identity. The mount is always read-only.
Type: list of string
Default:
[ ]
Example:
[
"idmap=uids=0-1000-1;gids=0-1000-1"
]
services.llmhop.sglang-quadlet.gateway.quadlet.quadletConfig
Extra [Quadlet] settings applied to the gateway container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.gateway.quadlet.serviceConfig
Extra [Service] settings applied to the gateway container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.gateway.quadlet.unitConfig
Extra [Unit] settings applied to the gateway container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.gateway.settings
Additional CLI flags forwarded to sgl-model-gateway.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
--worker-urls is rendered from the enabled models, so setting it here
replaces the generated list. The listener flags (host, port,
prometheus-host, prometheus-port) come from the options of the
same name and always win over entries set here.
Type: attribute set of anything
Default:
{ }
Example:
{
tls-cert-path = "/etc/sglang/tls/server.crt";
tls-key-path = "\${cred:tls-key}";
}
services.llmhop.sglang-quadlet.gateway.tag
Default tag of the gateway image. Mutually exclusive with digest.
Type: null or string
Default:
"latest"
services.llmhop.sglang-quadlet.image
Container image used for every model worker.
Type: string
Default:
"docker.io/lmsysorg/sglang"
services.llmhop.sglang-quadlet.modelSettings
CLI flags forwarded to the model server for every model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.sglang-quadlet.models.<name>.settings; per-model
entries take precedence.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models
Models to serve.
Each entry produces one quadlet container; the attribute name is the routing key
(advertised via --served-model-name and surfaced through both llmhop and the
optional SGL Model Gateway as the OpenAI model field).
Enabled entries are sorted by ascending name.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 19001;
settings = {
reasoning-parser = "qwen3";
tool-call-parser = "qwen3_coder";
mem-fraction-static = 0.6;
cuda-graph-max-bs = 4;
};
};
}
services.llmhop.sglang-quadlet.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang-quadlet.models.<name>.credentials
Credentials granted exclusively to this service through systemd.
A path outside the Nix store uses LoadCredential=. The attribute form
can select LoadCredentialEncrypted= for a systemd-creds encrypted
source, which must be encrypted under the same name, or omit source
to import the credential of that name from the system credential store.
That store is shared by every service, so prefix an imported name with
its service, as in llmhop.hf-token.
Reference the resulting read-only file from settings as
${cred:<name>}. The module resolves the reference to the native or
container credential path without copying its contents to the Nix store
or command line.
Type: attribute set of ((submodule) or absolute path convertible to it)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
tls-key = {
source = "/run/secrets/tls-key.cred";
encrypted = true;
};
"llmhop.hf-token" = { };
}
services.llmhop.sglang-quadlet.models.<name>.credentials.<name>.encrypted
Whether to load and decrypt source with LoadCredentialEncrypted=.
Imported credentials are decrypted as needed.
Type: boolean
Default:
false
services.llmhop.sglang-quadlet.models.<name>.credentials.<name>.source
File or socket from which systemd loads the credential. null
imports the credential of the same name with ImportCredential=
from the system credential store, such as /etc/credstore and
/etc/credstore.encrypted, and from the credentials passed to the
system.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang-quadlet.models.<name>.devices
Devices exposed to this model’s container — passed verbatim as
Quadlet AddDevice= lines. Replaces (does not extend)
services.llmhop.sglang-quadlet.devices for this model.
Use to pin a model to specific device indices
(e.g. [ "nvidia.com/gpu=0" ]).
Type: list of string
Default:
config.services.llmhop.sglang-quadlet.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.sglang-quadlet.models.<name>.digest
Immutable digest of the container image (e.g. sha256:…).
Mutually exclusive with tag.
Type: null or string
Default:
null
Example:
"sha256:a73fb0b9046fee099f7c1829d2548e6cc1740f4c2776a6855fa659ae5d0deb49"
services.llmhop.sglang-quadlet.models.<name>.environment
Additional environment variables set on this model’s service.
Merged with services.llmhop.sglang-quadlet.environment; per-model entries
take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s service.
Loaded after services.llmhop.sglang-quadlet.environmentFile, so its entries
override global ones. Prefer credentials for any secret the server
can read from a file. Must be readable by the user systemd reads it as.
Type: null or absolute path
Default:
null
services.llmhop.sglang-quadlet.models.<name>.model
Hugging Face repo id (or local path) passed to the model server.
Type: string
Example:
"Qwen/Qwen2.5-7B-Instruct"
services.llmhop.sglang-quadlet.models.<name>.name
Canonical identifier for this model. Used for the unit name
(sglang-<name>) and as the routing key registered with llmhop,
which clients send in the model field. Shares one namespace with
every other routing key, so a collision fails evaluation.
Defaults to the attribute key, so the key itself must match the required label format.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.sglang-quadlet.models.<name>.port
Loopback host port forwarded to the container’s SGLang API.
Must be unique per model and must not collide with gateway.port /
gateway.metricsPort when the gateway is enabled.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.sglang-quadlet.models.<name>.quadlet.containerConfig
Extra [Container] settings applied to this model container.
Keys use Quadlet’s native PascalCase names, including User,
UserNS, UIDMap, GIDMap, SubUIDMap, and SubGIDMap.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.quadlet.extraConfig
Extra unit sections applied to this model container after all generated
sections. This is the final escape hatch for settings that do not
fit one of the dedicated *Config options.
Type: attribute set of attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.quadlet.mountOptions.credentials
Podman volume options appended to this model container’s systemd
credential mount. Use an idmap mapping when [Container] User=
selects a non-root identity. The mount is always read-only.
Type: list of string
Default:
[ ]
Example:
[
"idmap=uids=0-1000-1;gids=0-1000-1"
]
services.llmhop.sglang-quadlet.models.<name>.quadlet.quadletConfig
Extra [Quadlet] settings applied to this model container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.quadlet.serviceConfig
Extra [Service] settings applied to this model container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.quadlet.unitConfig
Extra [Unit] settings applied to this model container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.settings
CLI flags forwarded to the model server for this model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.sglang-quadlet.modelSettings; per-model entries
take precedence. The flags llmhop derives from the model options
(its served name and listener) always win over both.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.models.<name>.shmSize
Size of the container’s private /dev/shm tmpfs.
PyTorch and friends use shared memory for NCCL/tensor-parallel inference;
upstream recommends 32g (or --ipc=host). A private tmpfs is preferred for
isolation: raise the value for larger models or higher tensor-parallel sizes.
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.sglang-quadlet.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null
services.llmhop.sglang-quadlet.models.<name>.tag
Tag of the container image used for this model.
Mutually exclusive with digest.
Type: null or string
Default:
null
services.llmhop.sglang-quadlet.openFilesLimit
File descriptor limit (LimitNOFILE) applied to every sglang-quadlet systemd unit.
Increase if the server logs accept: Too many open files under concurrent load.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.sglang-quadlet.quadlet.containerConfig
Extra [Container] settings applied to every generated container.
Keys use Quadlet’s native PascalCase names, including User,
UserNS, UIDMap, GIDMap, SubUIDMap, and SubGIDMap.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.quadlet.extraConfig
Extra unit sections applied to every generated container after all generated
sections. This is the final escape hatch for settings that do not
fit one of the dedicated *Config options.
Type: attribute set of attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.quadlet.quadletConfig
Extra [Quadlet] settings applied to every generated container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.quadlet.serviceConfig
Extra [Service] settings applied to every generated container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.quadlet.unitConfig
Extra [Unit] settings applied to every generated container.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang-quadlet.quadlet.user
Host account whose systemd user manager owns these Quadlets.
null installs system units and runs Podman rootfully. An
attribute set installs user units for its uid and runs Podman
rootlessly. This is independent of [Container] User= and the
container’s user namespace configuration.
A managed account gets NixOS-allocated subordinate ID ranges.
Set users.users.<name>.subUidRanges and subGidRanges (with
autoSubUidGidRange = false) to pick them yourself.
Type: null or (submodule)
Default:
null
services.llmhop.sglang-quadlet.quadlet.user.gid
GID of the managed account’s primary group.
Type: positive integer, meaning >0
Default:
config.uid
services.llmhop.sglang-quadlet.quadlet.user.group
Primary group of the managed account.
Type: string
Default:
config.name
services.llmhop.sglang-quadlet.quadlet.user.home
Home directory used for rootless Podman storage. This must live on a filesystem that supports the selected storage driver.
Type: absolute path
Default:
"/var/lib/sglang"
services.llmhop.sglang-quadlet.quadlet.user.manage
Whether llmhop creates and configures this account. Disable this for an account managed elsewhere, including its home, linger setting, group, and subordinate ID ranges.
Type: boolean
Default:
true
services.llmhop.sglang-quadlet.quadlet.user.name
Host account whose systemd user manager owns the Quadlets.
Type: string
Default:
"sglang"
services.llmhop.sglang-quadlet.quadlet.user.uid
UID of the systemd user manager that owns the Quadlets.
Type: positive integer, meaning >0
Example:
503
services.llmhop.sglang-quadlet.startupOrdering
Whether to chain enabled model services by ascending name during startup.
GPU-memory profiling races otherwise: two workers booting on the same device
each see it as fully free and race to claim their share, leading to OOM.
Disable only when each model pins itself to a dedicated device via its own devices.
Type: boolean
Default:
true
services.llmhop.sglang-quadlet.tag
Default tag of the container image used for models that do not set their own
tag or digest.
Type: string
Example:
"latest"