services.llmhop.gateway.active
Whether the gateway of type runs, which needs at least one
endpoint, so a host whose backends are all disabled, such as a test
VM without GPUs, keeps its gateway config without running it. Units
relying on the gateway, such as clients joining its socket group,
can follow it.
Type: boolean (read only)
Default:
config.services.llmhop.gateway.type != null && config.services.llmhop.endpoints != { }
services.llmhop.gateway.bifrost.package
Package providing bifrost-http as its main program, taken from
Bifrost’s own flake, since nixpkgs lacks it.
Only used if runtime is "native".
Type: package
services.llmhop.gateway.bifrost.container.config
[Container] keys of every container of Bifrost, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.bifrost.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"docker.io/maximhq/bifrost:v2.3.2"
services.llmhop.gateway.bifrost.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
Bifrost, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.bifrost.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.bifrost.environment
Environment variables of Bifrost. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.bifrost.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to Bifrost.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.bifrost.host
IP address Bifrost listens on.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.gateway.bifrost.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.bifrost.port
TCP port Bifrost listens on.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
8080
services.llmhop.gateway.bifrost.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.bifrost.settings
Bifrost’s config.json, with every listed model as a key of its
vllm or sgl provider. Changes from its web UI or API persist
until the matching section of this file changes, while
source_of_truth = "config.json" prunes them and
config_store.enabled = false disables them. Use upstream env.
references with environmentFile for secrets.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
encryption_key = "env.BIFROST_ENCRYPTION_KEY";
governance = {
virtual_keys = [
{
id = "default";
is_active = true;
name = "default";
value = "env.BIFROST_VIRTUAL_KEY";
}
];
};
}
services.llmhop.gateway.llmhop.package
The llmhop package to use.
Type: package
Default:
config.services.llmhop.package
services.llmhop.gateway.llmhop.container.config
[Container] keys of every container of the llmhop gateway, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.llmhop.container.image
Container image reference, including a tag or digest, or null
for an archive of package, which validated the config at build
time and never auto-updates.
Only used if runtime is "quadlet".
Type: null or string
Default:
null
Example:
"ghcr.io/mirkolenz/llmhop:latest"
services.llmhop.gateway.llmhop.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
the llmhop gateway, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.llmhop.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
llmhop reads its own config, so in settings the reference expands
to the credential’s contents rather than to its path.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.llmhop.environment
Environment variables of the llmhop gateway. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.llmhop.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to the llmhop gateway.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.llmhop.gid
GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
config.services.llmhop.gateway.llmhop.uid
services.llmhop.gateway.llmhop.group
Primary group of user.
The module declares it while it keeps the name llmhop, and otherwise the
deployer declares it. Only used if runtime is "native".
Type: string
Default:
config.services.llmhop.gateway.llmhop.user
services.llmhop.gateway.llmhop.listen
Listeners of the gateway, each a llmhop-<name>.socket unit handing
its socket to it through socket activation. The default one always
exists, on port 8080 unless configured otherwise.
Type: attribute set of (submodule)
Default:
{ }
Example:
{
caddy = {
socketGroup = "caddy";
};
default = {
host = "";
};
}
services.llmhop.gateway.llmhop.listen.<name>.host
IP address to bind port to. An empty string binds every interface,
leaving access control to the firewall. IPv6 literals are written
plain (e.g. ::1).
Type: string
Default:
"127.0.0.1"
Example:
""
services.llmhop.gateway.llmhop.listen.<name>.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.llmhop.listen.<name>.port
TCP port to listen on. null listens on the unix socket socket
instead. The default listener defaults to 8080.
Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
null
Example:
8080
services.llmhop.gateway.llmhop.listen.<name>.socket
Unix socket to listen on while port is null.
Type: string
Default:
"${config.services.llmhop.socketDirectory}/‹name›.sock"
services.llmhop.gateway.llmhop.listen.<name>.socketGroup
Group of socket, typically the one of a reverse proxy in front of llmhop.
Type: string
Default:
"root"
Example:
"caddy"
services.llmhop.gateway.llmhop.listen.<name>.socketMode
Mode of socket. Connecting needs write permission.
Type: string
Default:
"0660"
services.llmhop.gateway.llmhop.listen.<name>.socketUser
Owner of socket.
Type: string
Default:
"root"
services.llmhop.gateway.llmhop.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.llmhop.settings
llmhop’s JSON config, validated at build time by the binary itself,
with every endpoint of the backends as a model. Its listeners come
from listen, so settings must not set it.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
authTokens = {
client = "\${cred:llmhop.client-token}";
};
models = {
"openai/gpt-4" = {
url = "https://api.openai.com";
};
};
}
services.llmhop.gateway.llmhop.uid
UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".
Type: null or (unsigned integer, meaning >=0)
Default:
null
Example:
503
services.llmhop.gateway.llmhop.user
System user the gateway runs as.
The module declares it while it keeps the name llmhop, and otherwise the
deployer declares it. Only used if runtime is "native".
Type: string
Default:
"llmhop"
services.llmhop.gateway.smg.package
Python environment providing the smg package, built with the
flake’s mkUvEnv, since nixpkgs lacks SMG.
Only used if runtime is "native".
Type: package
services.llmhop.gateway.smg.container.config
[Container] keys of every container of SMG, over the generated
ones, such as UserNS= or UIDMap= and GIDMap= to run it without
host root, or User=. Only used if runtime is "quadlet".
Type: attribute set of (systemd option)
Default:
{ }
Example:
{
UserNS = "auto";
}
services.llmhop.gateway.smg.container.image
Container image reference, including a tag or digest. Only used if runtime is "quadlet".
Type: string
Example:
"ghcr.io/smg-project/smg:nightly"
services.llmhop.gateway.smg.container.mountOptions.credentials
Podman volume options of the credential mount of every container of
SMG, which stays read-only. idmap lets a user namespace read the
credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1
a non-root User= in it. Only used if runtime is "quadlet".
Type: list of string
Default:
[ ]
Example:
[
"idmap"
]
services.llmhop.gateway.smg.credentials
Credentials granted exclusively to this service. A path outside the
Nix store is loaded with LoadCredential=, while null imports the
credential of the same name from the system credential store with
ImportCredential=, where a trailing * imports every credential
starting with the rest. Reference the read-only file from settings as
${cred:<name>}.
Type: attribute set of (null or absolute path not in the Nix store)
Default:
{ }
Example:
{
api-keys = "/run/secrets/api-keys";
"llmhop.hf-token" = null;
}
services.llmhop.gateway.smg.environment
Environment variables of SMG. With credentials,
${cred:<name>} expands to the path of one of them.
Type: attribute set of string
Default:
{ }
services.llmhop.gateway.smg.environmentFile
File outside the Nix store in KEY=VALUE format, loaded by
systemd or Podman and forwarded to SMG.
Type: null or absolute path not in the Nix store
Default:
null
services.llmhop.gateway.smg.host
IP address SMG listens on.
Type: string
Default:
"127.0.0.1"
Example:
"0.0.0.0"
services.llmhop.gateway.smg.openFirewall
Whether to open port in the host firewall, which requires a host other than loopback.
Type: boolean
Default:
false
services.llmhop.gateway.smg.port
TCP port SMG listens on.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
8080
services.llmhop.gateway.smg.runtime
Whether the units run natively as hardened services or as Quadlet containers.
Type: one of “native”, “quadlet”
Default:
config.services.llmhop.runtime
services.llmhop.gateway.smg.settings
CLI flags of SMG, rendered as --<key> <value> with false
dropped. The listener and the discovery file of every listed model
always win, while metrics on 127.0.0.1:29000 and disabled
tokenizer downloads are defaults. Credential references
resolve to file paths for upstream options such as api-key-file
and tenant-api-keys-file, which require a supporting SMG build.
Type: attribute set of (JSON value)
Default:
{ }
Example:
{
api-key-file = "\${cred:smg.api-key}";
policy = "cache_aware";
}
services.llmhop.gateway.type
The one gateway in front of every backend, configured by the option
set of the same name, or null for none, so clients connect to each
workload directly. It routes every listed model, which must then
stay on loopback and leave authentication to the gateway.
Type: null or one of “llmhop”, “smg”, “bifrost”
Default:
null
Example:
"llmhop"