Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

services.llmhop.sglang.enable

Whether to enable SGLang model serving.

Type: boolean

Default:

false

Example:

true

services.llmhop.sglang.package

Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server, built from a uv workspace with the flake’s mkUvEnv, as in inputs.llmhop.legacyPackages.${pkgs.stdenv.hostPlatform.system}.mkUvEnv { workspaceRoot = ./sglang-env; }. Only used if runtime is "native".

Type: package

services.llmhop.sglang.cache.directory

Root of the cache directories of this backend’s workloads, created with mode 0711, each <name> the HF_HOME of one fetcher and its server, created with mode 0700 for user and group. The fetcher writes its hub/, which the server only reads. Point it at a data disk for large models. null keeps each cache in the Podman named volume <unit> instead, which needs no tmpfiles rules but only works for a container backend without fetch.enable.

Type: null or absolute path not in the Nix store

Default:

"/var/cache/sglang"

services.llmhop.sglang.container.config

[Container] keys of every container of this backend, servers, fetchers and detectors alike, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.sglang.container.devices

Devices exposed to every model container as Quadlet AddDevice= lines, CDI references such as nvidia.com/gpu=all or host device paths. Per-model container.devices replaces it. Only used if runtime is "quadlet".

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then [ "nvidia.com/gpu=all" ] else [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.sglang.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/lmsysorg/sglang:latest"

services.llmhop.sglang.container.imageUnit

Whether every container runs image through the image unit sglang-image, which pulls it once for all of them and restarts them when restarted. Otherwise each container references image directly and pulls it while it is missing. Only used if runtime is "quadlet".

Type: boolean

Default:

true

services.llmhop.sglang.container.mountOptions.cache

Podman volume options of the cache mounts below cache.directory, such as idmap so a user namespace owns them, or SELinux relabeling. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.sglang.container.mountOptions.credentials

Podman volume options of the credential mount of every container of this backend, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.sglang.environment

Environment variables set on every model service, under the environment of each model, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.environmentFile

File in KEY=VALUE format forwarded to every server, loaded before the environmentFile of each model. Prefer credentials for any secret a server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

Example:

"/etc/sglang/.env"

services.llmhop.sglang.fetch.enable

Whether every workload has a fetcher <unit>-fetch downloading its repositories before its server starts, with a refresh unit and sglang-update, after which the server runs offline. Otherwise each server downloads its repositories itself when it starts, which requires offline = false, and none of the other fetch options apply.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang.fetch.credentials

Credentials granted to every fetcher, in the format of a workload’s credentials. Reference them from fetch.environment as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.sglang.fetch.environment

Environment variables set on every fetcher, such as a proxy. Values may reference fetch.credentials as ${cred:<name>}. The servers never see them.

Type: attribute set of string

Default:

{ }

Example:

{
  HTTPS_PROXY = "http://proxy.example:3128";
}

services.llmhop.sglang.fetch.environmentFile

File in KEY=VALUE format forwarded to every fetcher, for variables that only exist as values.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.sglang.fetch.hfTokenCredential

Credential holding a Hugging Face token for gated repositories, imported from the system credential store unless fetch.credentials names a file for it. Only the fetchers receive it, as HF_TOKEN, and skip it while it is missing.

Type: null or string

Default:

null

Example:

"llmhop.hf-token"

services.llmhop.sglang.fetch.schedule

OnCalendar= schedule of sglang-update, which runs every fetcher and restarts each running server after a successful fetch, even if its model did not change. null disables scheduled fetching.

Type: null or string

Default:

"weekly"

Example:

"daily"

services.llmhop.sglang.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.sglang.uid

services.llmhop.sglang.group

Primary group of user. If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.sglang.runtime == "native" then config.services.llmhop.sglang.user else "root"

services.llmhop.sglang.modelSettings

CLI flags forwarded to the server of every model, under the settings of each model. Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.sglang.models

Models to serve, each as the unit sglang-<name> beside its fetcher sglang-<name>-fetch, natively a hardened system service and otherwise a container, whose units virtualisation.quadlet.containers.<unit> override.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b".model = "Qwen/Qwen3-8B";
}

services.llmhop.sglang.models.<name>.enable

Whether to enable model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang.models.<name>.package

Package providing this model’s server, overriding the backend-wide package, e.g. a nightly build for a just-released architecture. Only used if the backend’s runtime is "native".

Type: package

Default:

config.services.llmhop.sglang.package

services.llmhop.sglang.models.<name>.container.devices

Devices exposed to this model’s container, replacing the backend’s container.devices. Only used if the backend’s runtime is "quadlet".

Type: list of string

Default:

config.services.llmhop.sglang.container.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.sglang.models.<name>.container.shmSize

Size of the container’s private /dev/shm tmpfs, which PyTorch uses for tensor-parallel inference. Only used if the backend’s runtime is "quadlet".

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.sglang.models.<name>.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.sglang.models.<name>.environment

Environment variables of this model’s server, over the backend’s environment, where ${cred:<name>} expands to the path of a credential.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s server, loaded after the backend’s environmentFile. Prefer credentials for any secret the server can read from a file.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.sglang.models.<name>.extraRepositories

Further repositories the fetcher downloads beside model, such as a speculative draft model or a LoRA adapter that settings reference, since the server cannot download them itself.

Type: list of string

Default:

[ ]

Example:

[
  "Qwen/Qwen3-0.6B"
]

services.llmhop.sglang.models.<name>.host

IP address the server binds port to. Any but 127.0.0.1 or ::1 requires that no gateway routes it and offline = false.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.sglang.models.<name>.listener.host

Address the server binds.

Type: string

services.llmhop.sglang.models.<name>.listener.port

Port the server binds, or null for its socket.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

services.llmhop.sglang.models.<name>.listener.socket

Unix socket the server binds, or null for its port.

Type: null or string

Default:

null

services.llmhop.sglang.models.<name>.model

Hugging Face repository passed as --model-path, or an absolute path. A repository is downloaded by the fetcher, while an absolute path, in a container one inside it, is served as is.

Type: string

Example:

"Qwen/Qwen3-8B"

services.llmhop.sglang.models.<name>.name

Name of this model in its unit sglang-<name> and in the model field clients send, unique across every workload.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.sglang.models.<name>.native.devices

Devices this model’s service may access, replacing the backend’s native.devices. Only used if the backend’s runtime is "native".

Type: null or (list of string)

Default:

config.services.llmhop.sglang.native.devices

Example:

[
  "/dev/nvidia0"
]

services.llmhop.sglang.models.<name>.offline

Whether this model’s server runs without network access. Natively, a socket gets a network namespace of its own and a port an IP address filter, which requires a loopback host. A container runs with Network=none, which requires its socket. Disable it for features that download at runtime, such as media URLs in requests. With the backend’s fetch.enable, the server serves only what its fetcher downloaded either way, and otherwise downloads its repositories itself, which requires false.

Type: boolean

Default: false without the backend’s fetch.enable, else whether socket is set or, if its runtime is "native", host is loopback

services.llmhop.sglang.models.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback and no gateway routing this model. A container’s published port bypasses the firewall either way.

Type: boolean

Default:

false

services.llmhop.sglang.models.<name>.port

Host port the server binds to on host.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.sglang.models.<name>.repositories.*.include

Glob patterns of the files to download, or all files if empty.

Type: list of string

Default:

[ ]

services.llmhop.sglang.models.<name>.repositories.*.repo

Hugging Face repository.

Type: string

services.llmhop.sglang.models.<name>.repositories.*.revision

Branch, tag or commit, or null for the default branch.

Type: null or string

Default:

null

services.llmhop.sglang.models.<name>.revision

Branch, tag or commit of model that the fetcher downloads and the server loads. null follows the default branch, so fetch.schedule picks up upstream changes, while a commit pins the model.

Type: null or string

Default:

null

Example:

"main"

services.llmhop.sglang.models.<name>.settings

CLI flags of this model’s server, over the backend’s modelSettings. The listener and the flags derived from the other options, such as the served name, always win. Rendered as --<key> <value>, with false dropped, since this CLI pairs --enable-X with --disable-X and a list handed to a single flag. See settings rendering for the full rules.

Type: attribute set of (JSON value)

Default:

{ }

services.llmhop.sglang.models.<name>.socket

Unix socket the server binds, derived from port.

Type: null or string (read only)

Default: <services.llmhop.socketDirectory>/sglang-<name>/http.sock while port is null, else null

services.llmhop.sglang.native.devices

Devices every model service may access as systemd DeviceAllow= entries under DevicePolicy=closed, while null allows all. Per-model native.devices replaces it. Only used if runtime is "native".

Type: null or (list of string)

Default:

null

Example:

[
  "/dev/nvidiactl"
  "char-nvidia-uvm"
  "/dev/nvidia0"
]

services.llmhop.sglang.openFilesLimit

File descriptor limit (LimitNOFILE) of every server of this backend. Increase it if a server logs accept: Too many open files.

Type: positive integer, meaning >0

Default:

1048576

services.llmhop.sglang.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.sglang.startupOrdering

Whether to start model services one at a time by ascending name, so two workers profiling memory on the same GPU do not race into an OOM. Disable only when each model pins itself to a dedicated device, natively via its own native.devices or environment, e.g. CUDA_VISIBLE_DEVICES, or in a container via its own container.devices.

Type: boolean

Default:

true

services.llmhop.sglang.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.sglang.user

System user the native units and any refresh units run as, which owns their directories. If runtime is "native", the module declares it while it keeps the name sglang, and otherwise the deployer declares it.

Type: string

Default:

if config.services.llmhop.sglang.runtime == "native" then "sglang" else "root"