Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

services.llmhop.gateway.active

Whether the gateway of type runs, which needs at least one endpoint, so a host whose backends are all disabled, such as a test VM without GPUs, keeps its gateway config without running it. Units relying on the gateway, such as clients joining its socket group, can follow it.

Type: boolean (read only)

Default:

config.services.llmhop.gateway.type != null && config.services.llmhop.endpoints != { }

services.llmhop.gateway.bifrost.package

Package providing bifrost-http as its main program, taken from Bifrost’s own flake, since nixpkgs lacks it. Only used if runtime is "native".

Type: package

services.llmhop.gateway.bifrost.container.config

[Container] keys of every container of Bifrost, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.bifrost.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"docker.io/maximhq/bifrost:v2.3.2"

services.llmhop.gateway.bifrost.container.mountOptions.credentials

Podman volume options of the credential mount of every container of Bifrost, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.bifrost.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.bifrost.environment

Environment variables of Bifrost. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.bifrost.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to Bifrost.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.bifrost.host

IP address Bifrost listens on.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.gateway.bifrost.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.bifrost.port

TCP port Bifrost listens on.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

8080

services.llmhop.gateway.bifrost.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.bifrost.settings

Bifrost’s config.json, with every listed model as a key of its vllm or sgl provider. Changes from its web UI or API persist until the matching section of this file changes, while source_of_truth = "config.json" prunes them and config_store.enabled = false disables them. Use upstream env. references with environmentFile for secrets.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  encryption_key = "env.BIFROST_ENCRYPTION_KEY";
  governance = {
    virtual_keys = [
      {
        id = "default";
        is_active = true;
        name = "default";
        value = "env.BIFROST_VIRTUAL_KEY";
      }
    ];
  };
}

services.llmhop.gateway.llmhop.package

The llmhop package to use.

Type: package

Default:

config.services.llmhop.package

services.llmhop.gateway.llmhop.container.config

[Container] keys of every container of the llmhop gateway, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.llmhop.container.image

Container image reference, including a tag or digest, or null for an archive of package, which validated the config at build time and never auto-updates. Only used if runtime is "quadlet".

Type: null or string

Default:

null

Example:

"ghcr.io/mirkolenz/llmhop:latest"

services.llmhop.gateway.llmhop.container.mountOptions.credentials

Podman volume options of the credential mount of every container of the llmhop gateway, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.llmhop.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

llmhop reads its own config, so in settings the reference expands to the credential’s contents rather than to its path.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.llmhop.environment

Environment variables of the llmhop gateway. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.llmhop.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to the llmhop gateway.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.llmhop.gid

GID of the declared group. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

config.services.llmhop.gateway.llmhop.uid

services.llmhop.gateway.llmhop.group

Primary group of user. The module declares it while it keeps the name llmhop, and otherwise the deployer declares it. Only used if runtime is "native".

Type: string

Default:

config.services.llmhop.gateway.llmhop.user

services.llmhop.gateway.llmhop.listen

Listeners of the gateway, each a llmhop-<name>.socket unit handing its socket to it through socket activation. The default one always exists, on port 8080 unless configured otherwise.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  caddy = {
    socketGroup = "caddy";
  };
  default = {
    host = "";
  };
}

services.llmhop.gateway.llmhop.listen.<name>.host

IP address to bind port to. An empty string binds every interface, leaving access control to the firewall. IPv6 literals are written plain (e.g. ::1).

Type: string

Default:

"127.0.0.1"

Example:

""

services.llmhop.gateway.llmhop.listen.<name>.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.llmhop.listen.<name>.port

TCP port to listen on. null listens on the unix socket socket instead. The default listener defaults to 8080.

Type: null or 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

null

Example:

8080

services.llmhop.gateway.llmhop.listen.<name>.socket

Unix socket to listen on while port is null.

Type: string

Default:

"${config.services.llmhop.socketDirectory}/‹name›.sock"

services.llmhop.gateway.llmhop.listen.<name>.socketGroup

Group of socket, typically the one of a reverse proxy in front of llmhop.

Type: string

Default:

"root"

Example:

"caddy"

services.llmhop.gateway.llmhop.listen.<name>.socketMode

Mode of socket. Connecting needs write permission.

Type: string

Default:

"0660"

services.llmhop.gateway.llmhop.listen.<name>.socketUser

Owner of socket.

Type: string

Default:

"root"

services.llmhop.gateway.llmhop.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.llmhop.settings

llmhop’s JSON config, validated at build time by the binary itself, with every endpoint of the backends as a model. Its listeners come from listen, so settings must not set it.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  authTokens = {
    client = "\${cred:llmhop.client-token}";
  };
  models = {
    "openai/gpt-4" = {
      url = "https://api.openai.com";
    };
  };
}

services.llmhop.gateway.llmhop.uid

UID of the declared user. null lets NixOS allocate one. Only used if runtime is "native".

Type: null or (unsigned integer, meaning >=0)

Default:

null

Example:

503

services.llmhop.gateway.llmhop.user

System user the gateway runs as. The module declares it while it keeps the name llmhop, and otherwise the deployer declares it. Only used if runtime is "native".

Type: string

Default:

"llmhop"

services.llmhop.gateway.smg.package

Python environment providing the smg package, built with the flake’s mkUvEnv, since nixpkgs lacks SMG. Only used if runtime is "native".

Type: package

services.llmhop.gateway.smg.container.config

[Container] keys of every container of SMG, over the generated ones, such as UserNS= or UIDMap= and GIDMap= to run it without host root, or User=. Only used if runtime is "quadlet".

Type: attribute set of (systemd option)

Default:

{ }

Example:

{
  UserNS = "auto";
}

services.llmhop.gateway.smg.container.image

Container image reference, including a tag or digest. Only used if runtime is "quadlet".

Type: string

Example:

"ghcr.io/smg-project/smg:nightly"

services.llmhop.gateway.smg.container.mountOptions.credentials

Podman volume options of the credential mount of every container of SMG, which stays read-only. idmap lets a user namespace read the credentials owned by host root, and idmap=uids=0-<uid>-1;gids=0-<gid>-1 a non-root User= in it. Only used if runtime is "quadlet".

Type: list of string

Default:

[ ]

Example:

[
  "idmap"
]

services.llmhop.gateway.smg.credentials

Credentials granted exclusively to this service. A path outside the Nix store is loaded with LoadCredential=, while null imports the credential of the same name from the system credential store with ImportCredential=, where a trailing * imports every credential starting with the rest. Reference the read-only file from settings as ${cred:<name>}.

Type: attribute set of (null or absolute path not in the Nix store)

Default:

{ }

Example:

{
  api-keys = "/run/secrets/api-keys";
  "llmhop.hf-token" = null;
}

services.llmhop.gateway.smg.environment

Environment variables of SMG. With credentials, ${cred:<name>} expands to the path of one of them.

Type: attribute set of string

Default:

{ }

services.llmhop.gateway.smg.environmentFile

File outside the Nix store in KEY=VALUE format, loaded by systemd or Podman and forwarded to SMG.

Type: null or absolute path not in the Nix store

Default:

null

services.llmhop.gateway.smg.host

IP address SMG listens on.

Type: string

Default:

"127.0.0.1"

Example:

"0.0.0.0"

services.llmhop.gateway.smg.openFirewall

Whether to open port in the host firewall, which requires a host other than loopback.

Type: boolean

Default:

false

services.llmhop.gateway.smg.port

TCP port SMG listens on.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

8080

services.llmhop.gateway.smg.runtime

Whether the units run natively as hardened services or as Quadlet containers.

Type: one of “native”, “quadlet”

Default:

config.services.llmhop.runtime

services.llmhop.gateway.smg.settings

CLI flags of SMG, rendered as --<key> <value> with false dropped. The listener and the discovery file of every listed model always win, while metrics on 127.0.0.1:29000 and disabled tokenizer downloads are defaults. Credential references resolve to file paths for upstream options such as api-key-file and tenant-api-keys-file, which require a supporting SMG build.

Type: attribute set of (JSON value)

Default:

{ }

Example:

{
  api-key-file = "\${cred:smg.api-key}";
  policy = "cache_aware";
}

services.llmhop.gateway.type

The one gateway in front of every backend, configured by the option set of the same name, or null for none, so clients connect to each workload directly. It routes every listed model, which must then stay on loopback and leave authentication to the gateway.

Type: null or one of “llmhop”, “smg”, “bifrost”

Default:

null

Example:

"llmhop"