services.llmhop.sglang.enable
Whether to enable SGLang model serving.
Type: boolean
Default:
false
Example:
true
services.llmhop.sglang.package
Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server, built from
a uv workspace with the flake’s mkUvEnv, as in
inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./sglang-env; }.
Only used if runtime is "native".
Type: package
services.llmhop.sglang.cache.directory
Root of the cache directories of this backend’s workloads, created
with mode 0711, each <name> the HF_HOME of one fetcher and its
server, created with mode 0700 for user and group. The
fetcher writes its hub/, which the server only reads. Point it at
a data disk for large models.
null keeps each cache in the Podman named volume <unit> instead,
which needs no tmpfiles rules but only works for a container backend
without fetch.enable.
Type: null or absolute path not in the Nix store
Default:
"/var/cache/sglang"
services.llmhop.sglang.container.config
[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.sglang.container.devices
Devices exposed to every model container as Quadlet AddDevice=
lines, CDI references such as nvidia.com/gpu=all or host device
paths. Per-model container.devices replaces it.
Only used if runtime is "quadlet".
Type: list of string
Default:
if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]
Example:
[
"amd.com/gpu=all"
]
services.llmhop.sglang.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/lmsysorg/sglang:latest"
services.llmhop.sglang.container.imageUnit
Whether every container runs image through the image unit
sglang-image, which pulls it once for all of them and restarts
them when restarted. Otherwise each container references image
directly and pulls it while it is missing.
Only used if runtime is "quadlet".
Type: boolean
Default:
true
services.llmhop.sglang.container.mountOptions.cache
Podman volume options of the cache mounts below
cache.directory, such as idmap so a user namespace owns
them, or SELinux relabeling.
Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.sglang.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
this backend, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.sglang.environment
Environment variables set on every model service, under the
environment of each model, where ${cred:<name>} expands to the
path of a credential.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.environmentFile
File in KEY=VALUE format forwarded to every server, loaded before
the environmentFile of each model. Prefer credentials for any
secret a server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
Example:
"/etc/sglang/.env"
services.llmhop.sglang.fetch.enable
Whether every workload has a fetcher <unit>-fetch downloading
its repositories before its server starts, with a refresh unit and
sglang-update, after which the server runs offline. Otherwise
each server downloads its repositories itself when it starts, which
requires offline = false, and none of the other fetch options
apply.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang.fetch.credentials
Credentials granted to every fetcher, in the format of a
workload’s credentials. Reference them from fetch.environment
as ${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.sglang.fetch.environment
Environment variables set on every fetcher, such as a proxy. Values
may reference fetch.credentials as ${cred:<name>}.
The servers never see them.
Type: attribute set of string
Default:
{ }
Example:
{
HTTPS_PROXY = "http://proxy.example:3128";
}
services.llmhop.sglang.fetch.environmentFile
File in KEY=VALUE format forwarded to every fetcher, for
variables that only exist as values.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang.fetch.hfTokenCredential
Credential holding a Hugging Face token for gated repositories,
imported from the system credential store unless
fetch.credentials names a file for it. Only the fetchers receive
it, as HF_TOKEN, and skip it while it is missing.
Type: null or string
Default:
null
Example:
"llmhop.hf-token"
services.llmhop.sglang.fetch.schedule
OnCalendar= schedule of sglang-update, which runs every
fetcher and restarts each running server after a successful fetch,
even if its model did not change.
null disables scheduled fetching.
Type: null or string
Default:
"weekly"
Example:
"daily"
services.llmhop.sglang.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.sglang.uid
services.llmhop.sglang.group
Primary group of user.
If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.sglang.runtime == "native" then config.services.llmhop.sglang.user else "root"
services.llmhop.sglang.modelSettings
CLI flags forwarded to the server of every model, under the
settings of each model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.sglang.models
Models to serve, each as the unit sglang-<name> beside its
fetcher sglang-<name>-fetch, natively a hardened system service
and otherwise a container, whose units
virtualisation.quadlet.containers.<unit> override.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b".model = "Qwen/Qwen3-8B";
}
services.llmhop.sglang.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang.models.<name>.package
Package providing this model’s server, overriding the
backend-wide package, e.g. a nightly build for a
just-released architecture.
Only used if the backend’s runtime is "native".
Type: package
Default:
config.services.llmhop.sglang.package
services.llmhop.sglang.models.<name>.container.devices
Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".
Type: list of string
Default:
config.services.llmhop.sglang.container.devices
Example:
[
"nvidia.com/gpu=0"
]
services.llmhop.sglang.models.<name>.container.shmSize
Size of the container’s private /dev/shm tmpfs, which
PyTorch uses for tensor-parallel inference.
Only used if the backend’s runtime is "quadlet".
Type: string
Default:
"32g"
Example:
"64g"
services.llmhop.sglang.models.<name>.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.sglang.models.<name>.environment
Environment variables of this model’s server, over the backend’s
environment, where ${cred:<name>} expands to the path of a
credential.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s server,
loaded after the backend’s environmentFile. Prefer credentials
for any secret the server can read from a file.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang.models.<name>.extraRepositories
Further repositories the fetcher downloads beside model, such as a
speculative draft model or a LoRA adapter that settings
reference, since the server cannot download them itself.
Type: list of string
Default:
[ ]
Example:
[
"Qwen/Qwen3-0.6B"
]
services.llmhop.sglang.models.<name>.host
IP address the server binds port to. Any but 127.0.0.1 or
::1 requires that no gateway routes it and offline = false.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.sglang.models.<name>.listener.host
Address the server binds.
Type: string
services.llmhop.sglang.models.<name>.listener.port
Port the server binds, or null for its socket.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
services.llmhop.sglang.models.<name>.listener.socket
Unix socket the server binds, or null for its port.
Type: null or string
Default:
null
services.llmhop.sglang.models.<name>.model
Hugging Face repository passed as --model-path, or an absolute path.
A repository is downloaded by the fetcher, while an absolute path,
in a container one inside it, is served as is.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.sglang.models.<name>.name
Name of this model in its unit sglang-<name> and in the
model field clients send, unique across every workload.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.sglang.models.<name>.native.devices
Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".
Type: null or (list of string)
Default:
config.services.llmhop.sglang.native.devices
Example:
[
"/dev/nvidia0"
]
services.llmhop.sglang.models.<name>.offline
Whether this model’s server runs without network access. Natively,
a socket gets a network namespace of its own and a port an IP
address filter, which requires a loopback host. A container runs
with Network=none, which requires its socket.
Disable it for features that download at runtime, such as media URLs
in requests.
With the backend’s fetch.enable, the server serves only what its
fetcher downloaded either way, and otherwise downloads its
repositories itself, which requires false.
Type: boolean
Default:
false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback
services.llmhop.sglang.models.<name>.openFirewall
Whether to open port in the host firewall, which requires a
host other than loopback and no gateway routing this model.
A container’s published port bypasses the firewall either way.
Type: boolean
Default:
false
services.llmhop.sglang.models.<name>.port
Host port the server binds to on host.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.sglang.models.<name>.repositories.*.include
Glob patterns of the files to download, or all files if empty.
Type: list of string
Default:
[ ]
services.llmhop.sglang.models.<name>.repositories.*.repo
Hugging Face repository.
Type: string
services.llmhop.sglang.models.<name>.repositories.*.revision
Branch, tag or commit, or null for the default branch.
Type: null or string
Default:
null
services.llmhop.sglang.models.<name>.revision
Branch, tag or commit of model that the fetcher downloads and the
server loads. null follows the default branch, so fetch.schedule
picks up upstream changes, while a commit pins the model.
Type: null or string
Default:
null
Example:
"main"
services.llmhop.sglang.models.<name>.settings
CLI flags of this model’s server, over the backend’s
modelSettings. The listener and the flags derived from the other
options, such as the served name, always win.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Type: attribute set of (JSON value)
Default:
{ }
services.llmhop.sglang.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null
services.llmhop.sglang.native.devices
Devices every model service may access as systemd DeviceAllow=
entries under DevicePolicy=closed, while null allows all.
Per-model native.devices replaces it.
Only used if runtime is "native".
Type: null or (list of string)
Default:
null
Example:
[
"/dev/nvidiactl"
"char-nvidia-uvm"
"/dev/nvidia0"
]
services.llmhop.sglang.openFilesLimit
File descriptor limit (LimitNOFILE) of every server of this
backend. Increase it if a server logs accept: Too many open files.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.sglang.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.sglang.startupOrdering
Whether to start model services one at a time by ascending name,
so two workers profiling memory on the same GPU do not race into an
OOM. Disable only when each model pins itself to a dedicated device,
natively via its own native.devices or environment, e.g.
CUDA_VISIBLE_DEVICES, or in a container via its own
container.devices.
Type: boolean
Default:
true
services.llmhop.sglang.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.sglang.user
System user the native units and any refresh units run as, which owns their directories.
If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the
deployer declares it.
Type: string
Default:
if config.services.llmhop.sglang.runtime == "native" then "sglang" else "root"