services.llmhop.sglang.enable
Whether to enable SGLang model serving via systemd (native host process), fronted by llmhop.
Type: boolean
Default:
false
Example:
true
services.llmhop.sglang.package
Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server.
No default on purpose: SGLang has no one-derivation-fits-all (new model architectures routinely need dev snapshots, and the wheels come in per-accelerator variants), so you build the package from a uv workspace and pin / follow upstream there. The flake exposes a helper:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./sglang-env; # your pyproject.toml + uv.lock
}
Individual models may override this with models.<name>.package.
The native module serves workers only; the SGL Model Gateway remains a sglang-quadlet feature (llmhop already routes between backends).
Type: package
Example:
inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
workspaceRoot = ./sglang-env;
}
services.llmhop.sglang.environment
Environment variables set on every model service.
Merged with services.llmhop.sglang.models.<name>.environment; per-model
entries take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.environmentFile
File in KEY=VALUE format forwarded to every service.
Use only for upstream features that require environment variables,
such as HF_TOKEN for gated Hugging Face repositories. Environment
variables are not systemd credentials and are visible to every model,
so prefer credentials for any secret a server can read from a file.
Loaded before services.llmhop.sglang.models.<name>.environmentFile, so
per-model files override these entries.
Type: null or absolute path
Default:
null
Example:
"/etc/sglang/.env"
services.llmhop.sglang.gid
GID of the declared group. null lets NixOS allocate one.
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.sglang.uid
services.llmhop.sglang.group
Primary group of user. The module declares it while it keeps its
default name, any other group is the deployer’s to declare.
Type: string
Default:
config.services.llmhop.sglang.user
services.llmhop.sglang.modelSettings
CLI flags forwarded to the model server for every model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.sglang.models.<name>.settings; per-model
entries take precedence.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang.models
Models to serve.
Each enabled entry produces one systemd service named sglang-<name>;
the attribute name is the routing key surfaced through llmhop as the
OpenAI model field.
Enabled entries are sorted by ascending name.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
"qwen3-8b" = {
model = "Qwen/Qwen3-8B";
port = 19001;
settings = {
reasoning-parser = "qwen3";
mem-fraction-static = 0.6;
};
};
}
services.llmhop.sglang.models.<name>.enable
Whether to enable model ‹name›.
Type: boolean
Default:
true
Example:
true
services.llmhop.sglang.models.<name>.package
Package providing this model’s worker, overriding the backend-wide
package. Set it for a model that needs a different sglang
release than the rest — e.g. a nightly wheel for a just-released
architecture — built the same way with mkUvEnv over a per-model
uv workspace. Defaults to the backend-wide package.
Type: package
Default:
config.services.llmhop.sglang.package
services.llmhop.sglang.models.<name>.credentials
Credentials granted exclusively to this service through systemd.
A path outside the Nix store uses LoadCredential=. The attribute form
can select LoadCredentialEncrypted= for a systemd-creds encrypted
source, which must be encrypted under the same name, or omit source
to import the credential of that name from the system credential store.
That store is shared by every service, so prefix an imported name with
its service, as in llmhop.hf-token.
Reference the resulting read-only file from settings as
${cred:<name>}. The module resolves the reference to the native or
container credential path without copying its contents to the Nix store
or command line.
Type: attribute set of ((submodule) or absolute path convertible to it)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
tls-key = {
source = "/run/secrets/tls-key.cred";
encrypted = true;
};
"llmhop.hf-token" = { };
}
services.llmhop.sglang.models.<name>.credentials.<name>.encrypted
Whether to load and decrypt source with LoadCredentialEncrypted=.
Imported credentials are decrypted as needed.
Type: boolean
Default:
false
services.llmhop.sglang.models.<name>.credentials.<name>.source
File or socket from which systemd loads the credential. null
imports the credential of the same name with ImportCredential=
from the system credential store, such as /etc/credstore and
/etc/credstore.encrypted, and from the credentials passed to the
system.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.sglang.models.<name>.environment
Additional environment variables set on this model’s service.
Merged with services.llmhop.sglang.environment; per-model entries
take precedence.
Type: attribute set of string
Default:
{ }
services.llmhop.sglang.models.<name>.environmentFile
File in KEY=VALUE format forwarded to this model’s service.
Loaded after services.llmhop.sglang.environmentFile, so its entries
override global ones. Prefer credentials for any secret the server
can read from a file. Must be readable by the user systemd reads it as.
Type: null or absolute path
Default:
null
services.llmhop.sglang.models.<name>.model
Hugging Face repo id (or local path) passed as --model-path.
Type: string
Example:
"Qwen/Qwen3-8B"
services.llmhop.sglang.models.<name>.name
Canonical identifier for this model. Used for the unit name
(sglang-<name>) and as the routing key registered with llmhop,
which clients send in the model field. Shares one namespace with
every other routing key, so a collision fails evaluation.
Defaults to the attribute key, so the key itself must match the required label format.
Type: string matching the pattern [[:alnum:]][[:alnum:].-]*
Default:
"‹name›"
services.llmhop.sglang.models.<name>.port
Loopback host port sglang binds to (--host 127.0.0.1 --port <port>).
Must be unique per enabled model; llmhop reaches the backend at
http://127.0.0.1:<port>.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
services.llmhop.sglang.models.<name>.serviceConfig
Extra [Service] settings merged into this workload’s
sglang-<name> unit after the hardened baseline and
backend-specific relaxations. The module retains ownership of
ExecStart, KillMode, and Type because they implement readiness
supervision as one lifecycle contract.
Type: attribute set of anything
Default:
{ }
Example:
{
MemoryHigh = "64G";
}
services.llmhop.sglang.models.<name>.settings
CLI flags forwarded to the model server for this model.
Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.
Merged with services.llmhop.sglang.modelSettings; per-model entries
take precedence. The flags llmhop derives from the model options
(its served name and listener) always win over both.
Type: attribute set of anything
Default:
{ }
services.llmhop.sglang.models.<name>.socket
Unix socket the server binds, derived from port.
Type: null or string (read only)
Default:
<services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null
services.llmhop.sglang.models.<name>.unitConfig
Extra [Unit] settings merged into this workload’s
sglang-<name> unit after the shared baseline.
Ordering and dependency directives (After=, Requires=, Wants=)
do not belong here: NixOS renders those from the after, requires
and wants options, so a definition of the same key in unitConfig
conflicts with it instead of merging. Declare them on
systemd.services."sglang-<name>" from your own module,
where the module system concatenates them with what this one sets.
Type: attribute set of anything
Default:
{ }
Example:
{
StartLimitBurst = 10;
}
services.llmhop.sglang.openFilesLimit
File descriptor limit (LimitNOFILE) applied to every sglang systemd unit.
Increase if the server logs accept: Too many open files under concurrent load.
Type: positive integer, meaning >0
Default:
1048576
services.llmhop.sglang.startupOrdering
Whether to chain enabled model services by ascending name during startup.
GPU-memory profiling races otherwise: two workers booting on the same device
each see it as fully free and race to claim their share, leading to OOM.
Disable only when each model pins itself to a dedicated device via environment (the variable is stack-specific: CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ZE_AFFINITY_MASK, …).
Type: boolean
Default:
true
services.llmhop.sglang.uid
UID of the declared user. null lets NixOS allocate one.
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.sglang.user
System user the units run as. The module declares it while it keeps its default name, any other user is the deployer’s to declare.
Type: string
Default:
"sglang"