LLMhop

One port, many models: A tiny, stateless HTTP router for OpenAI-compatible LLM inference backends.

LLMhop peeks at the model field of an incoming OpenAI-compatible request and reverse-proxies it to the matching backend. It is primarily designed for single-model inference servers like vLLM and sglang that serve one model per process and need a thin model-aware gateway in front of them, but it works with any OpenAI-compatible backend (including multi-model servers and hosted providers) whenever you want to consolidate several upstreams behind a single endpoint.

Features

OpenAI-compatible reverse proxy, model router and request dispatcher for self-hosted LLM inference.
Native GET /v1/models and GET /v1/models/{model} endpoints served directly from the config, so clients can discover every backend behind the single endpoint.
Unauthenticated GET /health for load balancers and probes, plus sd_notify readiness so systemd reports the service as started only once the port answers.
Stateless single-binary HTTP service: no database, no cache, no background workers, safe behind any load balancer.
Zero external dependencies: pure Go, no third-party packages, no CGO.
Works with any OpenAI API-compatible backend, self-hosted or remote: vLLM, sglang, TabbyAPI, Aphrodite, Ollama, LocalAI, OpenRouter, together.ai, DeepInfra, etc.
Ships as a static binary, a minimal Docker image and a hardened NixOS module that can optionally spin up llama.cpp, sglang or vLLM workers alongside the router.

How it works

Client sends a request with a JSON body containing {"model": "..."}.
LLMhop reads the model field and looks it up in its config.
The request is forwarded verbatim to the configured backend URL.
Unknown models return 404.

GET /v1/models and GET /v1/models/{model} are answered by LLMhop itself from the configured models, never proxied, so the catalog reflects exactly what is routable. Everything else is dispatched by its model field as above. When authTokens is set, all routes (the models API included) require a valid bearer token.

Health

GET /health is served by LLMhop itself and is the one route that never requires a token, so probes and load balancers do not need a credential:

{ "status": "ok", "models": 3 }

The model count lets a downstream check assert that the proxy came up with the catalog it expects, not merely that the process is listening. Under systemd the same guarantee comes for free: LLMhop sends READY=1 only after the listener is bound, so a Type=notify unit stays in activating until requests are actually served.

Authentication

LLMhop can optionally gate incoming requests with a list of bearer tokens and inject per-model Authorization (or any other) headers when forwarding to the backend. Both sides are opt-in: leave authTokens and models.*.headers unset and headers are forwarded verbatim.

When authTokens is set, the router validates the incoming Authorization: Bearer <token> header (constant-time compare) and then strips it before forwarding, so the client-facing token never leaks upstream. Per-model headers are applied last, so a configured Authorization always wins over whatever the client sent.

Configuration

Create a config.json:

{
  "host": "127.0.0.1",
  "port": 8080,
  "authTokens": ["${file:client_token}"],
  "models": {
    "llama-3-8b": {
      "url": "http://localhost:30000"
    },
    "openai-gpt-4o": {
      "url": "https://api.openai.com",
      "headers": {
        "Authorization": "Bearer ${env:OPENAI_KEY}"
      }
    }
  }
}

host defaults to every interface and port to 8080. IPv6 literals are written plain ("host": "::1") and bracketed internally.

Secret references

String values inside authTokens and models.*.headers are expanded at startup, so no plaintext secret ever has to live in the config file:

${env:NAME}: read from the NAME environment variable.
${file:path}: read from a file. Relative paths are resolved against $CREDENTIALS_DIRECTORY when set (e.g. when launched by systemd with LoadCredential=), otherwise against the current working directory. A single trailing newline is trimmed.
$NAME: shorthand for ${env:NAME}.

Unresolved references are a hard startup error.

Validation

Unknown keys are rejected rather than ignored, so a misspelled maxBodyBytes fails loudly instead of silently falling back to its default, and every model url must be an absolute http(s) URL.

--check runs the full startup path (parsing, validation, router construction) and exits without binding a port:

llmhop --check --config config.json

Secret references are left unexpanded in this mode, so a config can be validated where the referenced files and environment variables do not exist, such as a CI job or a Nix build. The NixOS module uses exactly this to validate the generated config at build time.

Request size limit

LLMhop buffers each request body in memory so it can peek at the model field before forwarding. To keep a single request from exhausting memory, the body is capped at 100 MiB by default; bodies beyond the cap are rejected with 413 Request Entity Too Large. Override it when vision or other multimodal payloads need more:

{ "maxBodyBytes": 524288000 }

Running

# native
llmhop --config config.json

# nix
nix run github:mirkolenz/llmhop -- --config config.json

# docker
docker run --rm -p 8080:8080 -v ./config.json:/config.json ghcr.io/mirkolenz/llmhop --config /config.json

NixOS module

A hardened systemd service is provided out of the box. Add LLMhop to your flake inputs and import the module into your system configuration:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    llmhop = {
      url = "github:mirkolenz/llmhop";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs =
    { nixpkgs, llmhop, ... }:
    {
      nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          llmhop.nixosModules.default
          {
            services.llmhop = {
              enable = true;
              port = 8080;
              openFirewall = true;
              settings.models = {
                "llama-3-8b".url = "http://localhost:30000";
                "qwen-2.5-7b".url = "http://localhost:30001";
              };
            };
          }
        ];
      };
    };
}

The unit runs under DynamicUser with aggressive sandboxing (ProtectSystem, PrivateTmp, restricted syscalls and address families, no new privileges, …) and restarts on failure.

The module and the binary are deliberately coupled in four places:

host and port are module options that map one-to-one onto the binary’s own config fields, so the listener port is a first-class value on both sides, with nothing rendered or re-parsed in between. It joins the same global port registry the inference backends use, so a backend model reusing it fails evaluation instead of leaving one of the two services unable to bind, and openFirewall can act on it directly.
The generated config is validated at build time by the binary itself (llmhop -check), so a typo or a malformed model URL fails nixos-rebuild rather than the service. The schema therefore lives in exactly one place, the Go Config struct, instead of being mirrored in Nix. Validation is skipped when the target platform cannot be executed by the build machine (cross-compiled deployments).
The unit is Type=notify, matching the binary’s readiness signal, so anything ordered after llmhop.service can assume the port answers.
The same package ships llmhop-notify, which the native backends prefix to every model server’s command line. None of them speak sd_notify, so it polls /health and reports readiness on their behalf, letting the worker units be Type=notify too. It stays the unit’s main process and exits with the server’s status, so a model that dies while loading fails its unit immediately instead of being waited out until TimeoutStartSec.

The NixOS module is split into two exports. nixosModules.default ships the reverse proxy and the native systemd backends (llama.cpp, and vLLM and SGLang from prebuilt wheels), with no dependency on quadlet-nix, so it stays compatible with non-NixOS deployers such as system-manager. nixosModules.quadlet includes all of that and additionally provides the container variants of vLLM and SGLang, pulling in the quadlet-nix dependency they require. Import the latter only if you need vllm-quadlet or sglang-quadlet.

Inference backends

The module can also run the inference servers themselves, so you don’t have to wire up llama.cpp, sglang or vLLM by hand. Each backend exposes a models attrset under services.llmhop.<backend> and every entry becomes one isolated worker bound to a loopback port, with the matching route registered automatically with llmhop. All three backends can be enabled side by side and mixed freely in the same configuration.

Every native worker stays in activating until its server answers /health, so systemctl start <backend>-<model> returns only once the model is actually servable rather than merely spawned. Cold starts download weights and profile the GPU, so that wait can be long: TimeoutStartSec allows an hour. The container variants get the same guarantee from their Notify=healthy health check.

llama.cpp runs as a native, hardened systemd system unit under DynamicUser, and the default vllm and sglang backends run the same way from prebuilt wheels, except under a dedicated system user (see below). As a last resort, when the prebuilt wheels cannot be used, vLLM and SGLang can instead run as rootless Podman containers through quadlet-nix, via the suffixed vllm-quadlet and sglang-quadlet options. Each Quadlet backend gets a dedicated, lingering system user (sglang, vllm) that owns its cache directory, sub-UID range and rootless container store. The container units are installed under that user’s per-UID search path and therefore run as systemd user units, not system units. This is a deliberate workaround for NVIDIA/nvidia-container-toolkit#648: nvidia-cdi-hook runs as an OCI createContainer hook inside the container’s user namespace and fails to read the OCI bundle’s config.json whenever Podman uses a UID-mapped namespace (e.g., --userns auto or --userns nomap), which is the mode you end up in when systemd’s system manager launches a rootless container. Running each Quadlet unit under a real, lingering system user’s systemd instance keeps Podman in the keep-id-style mapping where the CDI hook can read the bundle and the GPU is correctly exposed. No worker ever runs as root.

For convenience, the module injects a tiny per-backend helper into environment.systemPackages whenever the backend’s default user is used:

Native workers (llama-cpp, vllm, sglang) are plain system units, so they are managed with the usual systemctl status <backend>-<model> and journalctl -u <backend>-<model>.
For the container variants, sglang-shell and vllm-shell are writeShellApplication wrappers around machinectl shell that drop you into the backend user’s session, where systemctl --user, journalctl --user and podman ps see the worker units directly. Run them with no arguments for an interactive shell, or pass a command to execute it inside the session.

services.llmhop = {
  enable = true;
  llama-cpp = {
    enable = true;
    models."qwen3-8b" = {
      port = 18001;
      settings.hf-repo = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
    };
  };
  sglang = {
    enable = true;
    package = inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv { workspaceRoot = ./sglang-env; };
    models."qwen3-coder" = {
      port = 19001;
      model = "Qwen/Qwen3-8B";
      settings.reasoning-parser = "qwen3";
    };
  };
  vllm = {
    enable = true;
    package = inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv { workspaceRoot = ./vllm-env; };
    models."llama-3-8b" = {
      port = 20001;
      model = "meta-llama/Meta-Llama-3-8B-Instruct";
    };
  };
};

See the options reference for the full list of per-backend options.

Native vLLM and SGLang from prebuilt wheels

The default vLLM and SGLang backends run as native systemd units built from upstream’s prebuilt wheels: no Podman, and the same sandboxing as the llama.cpp backend. They need a dedicated system user rather than DynamicUser, so uid is required: the /var/lib/private layout DynamicUser implies hands the state and cache directories to the unit as noexec ID-mapped mounts, and these runtimes dlopen kernels they compiled into that cache. vLLM and SGLang lean heavily on dev snapshots and architecture-specific builds, so there is no one-derivation-fits-all version, and you pin yours in a tiny uv workspace and build the package with the flake’s mkUvEnv helper.

# vllm-env/pyproject.toml — your single version knob; edit and run `uv lock` to follow upstream.
#   [project]
#   name = "vllm-env"
#   requires-python = "==3.12.*"
#   dependencies = [ "vllm==0.16.2" ]   # or a nightly via [tool.uv.sources] / [[tool.uv.index]]

services.llmhop.vllm = {
  enable = true;
  uid = 503; # required: pick one free on this host
  package = inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
    workspaceRoot = ./vllm-env; # directory holding pyproject.toml + uv.lock
  };
  models."llama-3-8b" = {
    model = "meta-llama/Meta-Llama-3-8B-Instruct";
    port = 20001;
  };
};

mkUvEnv installs the wheels, so no GPU or C++ toolchain runs at build time, and patches them for NixOS by baking the GPU driver runpath into the closure. The driver itself is host state, so enable hardware.graphics and your vendor configuration (hardware.nvidia, the amdgpu kernel driver, …) as usual. services.llmhop.sglang works identically, launched via python -m sglang.launch_server.

GPUs other than NVIDIA

Nothing in the module is CUDA-specific. Every GPU worker joins the render and video groups inside a PrivateUsers = "identity" namespace that keeps those group IDs intact, which is what opening /dev/kfd and /dev/dri/renderD* takes. Runtime kernel caches are redirected into the unit’s cache root for every stack at once, and the NCCL_* defaults cover AMD too, since RCCL reads the same variables.

Only the wheels differ per vendor:

Stack	Native (uv) backends	Notes
NVIDIA CUDA	vLLM, SGLang	The wheels published on PyPI.
AMD ROCm	vLLM	Official ROCm wheels from v0.14.0 onwards, off a separate index. SGLang’s are still landing upstream.
Intel XPU	none	vLLM ships no prebuilt XPU wheels and needs a oneAPI source build, so use `vllm-quadlet` with an Intel image, or llama.cpp built for SYCL or Vulkan.

A ROCm workspace differs from a CUDA one only in where the wheel comes from:

# vllm-env/pyproject.toml — the ROCm version is part of both the pin and the index URL.
[project]
name = "vllm-env"
requires-python = "==3.12.*"
dependencies = [ "vllm==0.15.0+rocm700" ]

[[tool.uv.index]]
name = "vllm-rocm"
url = "https://wheels.vllm.ai/rocm/0.15.0/rocm700"
explicit = true

[tool.uv.sources]
vllm = { index = "vllm-rocm" }

Those wheels carry a matched ROCm and torch build, so the host contributes only the kernel driver. Expect a different set of missing native libraries than a CUDA workspace, and use the ignoreMissingLibs = [ ] triage below to find them.

Missing build systems

Not every dependency ships a wheel. The few that resolve to an sdist are built from source, and pre-PEP-517 projects that assume setuptools is simply present fail the build with No module named 'setuptools' or The build backend returned an error. Declare what they need in the workspace rather than patching the Nix side, so uv and mkUvEnv read it from the same place:

# sglang-env/pyproject.toml — SGLang reaches antlr4 through omegaconf.
[tool.uv.extra-build-dependencies]
antlr4-python3-runtime = ["setuptools"]

Re-run uv lock afterwards. The key is the package name as it appears in uv.lock, and the value is whatever its build backend needs (setuptools, cython, meson-python, …).

Missing native libraries

Wheels are built for manylinux and expect a distro underneath them. Which libraries a workspace needs beyond the driver follows from what it locks, so there are no defaults: you supply them per workspace through buildInputs and runtimePaths, which are merged into every wheel. nativeBuildInputs is accepted alongside them for build-time tooling an sdist needs beyond its Python build backend.

buildInputs covers libraries a wheel names in a DT_NEEDED entry. They are added to the autoPatchelf search path, so a library only lands in the runpath of a wheel that actually links it and listing one nothing needs is harmless. By default an unresolved entry does not fail the build, because most of them are unresolvable on purpose: the host driver, sibling wheels that only meet each other once the venv merges them, and alternative backends where one of several variants is expected to load. To see the whole list, narrow ignoreMissingLibs for one build:

mkUvEnv {
  workspaceRoot = ./vllm-env;
  ignoreMissingLibs = [ ];   # accept nothing; every unresolved entry is now an error
}

Each error names both the library and the wheel that wants it:

auto-patchelf could not satisfy dependency libtbb.so.12 wanted by
  /nix/store/...-numba-0.65.0/lib/python3.12/site-packages/numba/np/ufunc/tbbpool...so

Triage that list, then add the ones that are genuinely missing and drop the tightened setting again:

buildInputs = [
  pkgs.ffmpeg-headless   # torchcodec, PyAV
  pkgs.tbb_2022          # numba's threading layer — plain `tbb` is too old for libtbb.so.12
  pkgs.z3.lib            # tilelang's TVM analyzer
];

runtimePaths covers the other kind, reached by a bare dlopen("libfoo.so") from Python via cffi or ctypes. Nothing announces those in the ELF, so no build ever fails over one and no runpath resolves it; the environment builds cleanly and the import dies:

OSError: cannot load library 'libsndfile.so': cannot open shared object file

Only importing finds them, so run the modules you care about once after a version bump. Entries here are appended to the runpath of every wheel rather than a chosen one, because the object issuing the dlopen is generally not the package that appears in the traceback — soundfile fails, but the call comes from cffi’s _cffi_backend:

runtimePaths = [ "${pkgs.lib.getLib pkgs.libsndfile}/lib" ];   # soundfile, reached through cffi

Each model defaults to the backend’s package but can pin its own with models.<name>.package, so a single model can follow a nightly build for a freshly-released architecture while the rest stay on the stable pin.

Because a unit only goes active once it is healthy, startupOrdering (on by default) is effective here: workers boot one at a time in ascending port order, each finishing its GPU-memory profiling before the next begins, which is what keeps two models sharing a device from racing into an OOM.

The container variants live under services.llmhop.vllm-quadlet and sglang-quadlet. A backend’s native (vllm/sglang) and container (vllm-quadlet/sglang-quadlet) variants emit the same vllm-<model> and sglang-<model> units and are therefore mutually exclusive, so enable at most one per backend.

Secrets

The generated config file lives in the world-readable Nix store, so secrets should never be placed in services.llmhop.settings directly. Instead, reference them via ${file:...} and hand the files to the service through the credentials option, which maps each entry to systemd’s LoadCredential=. The right-hand side is just a file path, so anything that produces a file works: agenix or sops-nix outputs, a manually-managed file under /etc/llmhop/, or a path emitted by your own secret-provisioning tool.

services.llmhop = {
  credentials.client_token = "/etc/llmhop/client-token";
  settings = {
    authTokens = [ "\${file:client_token}" ];
    models."openai-gpt-4o" = {
      url = "https://api.openai.com";
      headers.Authorization = "Bearer \${env:OPENAI_KEY}";
    };
  };
};

systemd.services.llmhop.serviceConfig.EnvironmentFile = [ "/etc/llmhop/openai.env" ];

/etc/llmhop/openai.env is a plain KEY=VALUE file:

OPENAI_KEY=sk-...

${file:...} references are resolved against $CREDENTIALS_DIRECTORY, which systemd exposes as a per-unit tmpfs accessible only to this service, compatible with DynamicUser and the rest of the sandbox. ${env:...} picks up anything the unit inherits, typically via EnvironmentFile=. Pick whichever matches how your secret tooling hands you the data; mixing both in one config is fine.

Core

services.llmhop.enable

Whether to enable llmhop reverse proxy.

Type: boolean

Default:

false

Example:

true

services.llmhop.package

The llmhop package to use.

Type: package

Default:

pkgs.callPackage ./package.nix { }

services.llmhop.credentials

Files handed to the service through systemd’s LoadCredential=, keyed by the name they are exposed under. Reference them from settings as ${file:<name>}: relative paths resolve against $CREDENTIALS_DIRECTORY, a per-unit tmpfs readable only by this service, so secrets never enter the world-readable Nix store.

Any path works, including agenix/sops-nix outputs and manually managed files.

Type: attribute set of absolute path

Default:

{ }

Example:

{ client_token = "/run/secrets/llmhop-token"; }

services.llmhop.host

Interface llmhop binds to. The default binds every interface, leaving access control to the firewall. IPv6 literals are written plain (e.g. ::1) and bracketed by llmhop itself.

Type: string

Default:

""

Example:

"127.0.0.1"

services.llmhop.openFirewall

Whether to open port in the host firewall.

Type: boolean

Default:

false

services.llmhop.port

Port llmhop listens on. Registered in the global port registry, so a backend model reusing it fails evaluation instead of leaving one of the two services unable to bind.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

services.llmhop.settings

Configuration written to the JSON config file passed to llmhop. See the upstream Config struct for available fields; host and port are contributed by the options of the same name.

The generated file is validated at build time by the binary itself, so unknown keys and malformed model URLs fail nixos-rebuild rather than the service.

Type: JSON value

Default:

{ }

Example:

{
  models = {
    gpt-4 = {
      url = "https://api.openai.com";
    };
  };
}

llama-cpp

services.llmhop.llama-cpp.enable

Whether to enable llama.cpp model serving via systemd, fronted by llmhop.

Type: boolean

Default:

false

Example:

true

services.llmhop.llama-cpp.package

The llama-cpp package to use.

Type: package

Default:

pkgs.llama-cpp

services.llmhop.llama-cpp.environment

Environment variables set on every model service. Merged with services.llmhop.llama-cpp.models.<name>.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.llama-cpp.environmentFile

File in KEY=VALUE format forwarded to every service. Use for secrets managed by sops-nix/agenix, e.g. a file containing HF_TOKEN=<token> to access gated Hugging Face repositories. Loaded before services.llmhop.llama-cpp.models.<name>.environmentFile, so per-model files override these entries.

Type: null or absolute path

Default:

null

Example:

"/etc/llama-cpp/.env"

services.llmhop.llama-cpp.modelSettings

CLI flags forwarded to the model server for every model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list repeats the flag once per element (--<key> a --<key> b).

Merged with services.llmhop.llama-cpp.models.<name>.settings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.llama-cpp.models

Models to serve. Each entry produces one systemd service running llama-server; the attribute name is the routing key surfaced through llmhop and the OpenAI model field.

GPU selection is done via build-specific environment variables on environment (top-level or per-model), since llama.cpp runs as a host process — no CDI involved. Common variables: CUDA_VISIBLE_DEVICES (CUDA), HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES (ROCm), GGML_VK_VISIBLE_DEVICES (Vulkan), ZE_AFFINITY_MASK (SYCL).

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b" = {
    port = 18001;
    settings = {
      hf-repo = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
      temperature = 1.0;
      top-k = 20;
    };
    # Pin this model to a specific GPU. The right variable depends on
    # the llama.cpp build: CUDA_VISIBLE_DEVICES for CUDA,
    # HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES for ROCm,
    # GGML_VK_VISIBLE_DEVICES for Vulkan, ZE_AFFINITY_MASK for SYCL.
    environment.CUDA_VISIBLE_DEVICES = "0";
  };
}

services.llmhop.llama-cpp.models.<name>.enable

Whether to enable serving of model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.llama-cpp.models.<name>.environment

Additional environment variables set on this model’s service. Merged with services.llmhop.llama-cpp.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.llama-cpp.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s service. Loaded after services.llmhop.llama-cpp.environmentFile, so its entries override global ones. Must be readable by the user systemd reads it as.

Type: null or absolute path

Default:

null

services.llmhop.llama-cpp.models.<name>.name

Canonical identifier for this model. Used for the unit name (llama-cpp-<name>) and as the routing key registered with llmhop (clients select the backend by sending this value in the OpenAI model field).

Defaults to the attribute key, so the key itself must match the required label format.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.llama-cpp.models.<name>.port

Loopback host port that llama-server binds to. Must be unique per enabled model; the gateway (llmhop) reaches each backend at http://127.0.0.1:<port>.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.llama-cpp.models.<name>.serviceConfig

Extra [Service] settings merged into this model’s llama-cpp-<name> unit after the hardened baseline and backend-specific relaxations. The module retains ownership of ExecStart, KillMode, and Type because they implement readiness supervision as one lifecycle contract.

Type: attribute set of anything

Default:

{ }

Example:

{
  MemoryHigh = "64G";
}

services.llmhop.llama-cpp.models.<name>.settings

CLI flags forwarded to the model server for this model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list repeats the flag once per element (--<key> a --<key> b).

Merged with services.llmhop.llama-cpp.modelSettings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.llama-cpp.openFilesLimit

File descriptor limit (LimitNOFILE) applied to every llama-cpp systemd unit. Increase if the server logs accept: Too many open files under concurrent load.

Type: positive integer, meaning >0

Default:

sglang

services.llmhop.sglang.enable

Whether to enable SGLang model serving via systemd (native host process), fronted by llmhop.

Type: boolean

Default:

false

Example:

true

services.llmhop.sglang.package

Package providing the SGLang Python environment, launched as bin/python -m sglang.launch_server.

No default on purpose: SGLang has no one-derivation-fits-all (new model architectures routinely need dev snapshots, and the wheels come in per-accelerator variants), so you build the package from a uv workspace and pin / follow upstream there. The flake exposes a helper:

inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
  workspaceRoot = ./sglang-env; # your pyproject.toml + uv.lock
}

Individual models may override this with models.<name>.package.

The native module serves workers only; the SGL Model Gateway remains a sglang-quadlet feature (llmhop already routes between backends).

Type: package

Example:

inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
  workspaceRoot = ./sglang-env;
}

services.llmhop.sglang.environment

Environment variables set on every model service. Merged with services.llmhop.sglang.models.<name>.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.environmentFile

File in KEY=VALUE format forwarded to every service. Use for secrets managed by sops-nix/agenix, e.g. a file containing HF_TOKEN=<token> to access gated Hugging Face repositories. Loaded before services.llmhop.sglang.models.<name>.environmentFile, so per-model files override these entries.

Type: null or absolute path

Default:

null

Example:

"/etc/sglang/.env"

services.llmhop.sglang.gid

Host GID assigned to services.llmhop.sglang.group. Defaults to uid.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.sglang.uid

services.llmhop.sglang.group

Primary group for services.llmhop.sglang.user. Defaults to the user name (matching the typical 1:1 user/group layout).

Type: string

Default:

config.services.llmhop.sglang.user

services.llmhop.sglang.modelSettings

CLI flags forwarded to the model server for every model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false is dropped, since the CLI pairs --enable-X with --disable-X instead of auto-negating: write the negated key explicitly, e.g. disable-radix-cache = true;. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.sglang.models.<name>.settings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.sglang.models

Models to serve. Each enabled entry produces one systemd service named sglang-<name>; the attribute name is the routing key surfaced through llmhop as the OpenAI model field. Enabled entries are sorted by ascending port.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b" = {
    model = "Qwen/Qwen3-8B";
    port = 19001;
    settings = {
      reasoning-parser = "qwen3";
      mem-fraction-static = 0.6;
    };
  };
}

services.llmhop.sglang.models.<name>.enable

Whether to enable serving of model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang.models.<name>.package

Package providing this model’s worker, overriding the backend-wide package. Set it for a model that needs a different sglang release than the rest — e.g. a nightly wheel for a just-released architecture — built the same way with mkUvEnv over a per-model uv workspace. Defaults to the backend-wide package.

Type: package

Default:

config.services.llmhop.sglang.package

services.llmhop.sglang.models.<name>.environment

Additional environment variables set on this model’s service. Merged with services.llmhop.sglang.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s service. Loaded after services.llmhop.sglang.environmentFile, so its entries override global ones. Must be readable by the user systemd reads it as.

Type: null or absolute path

Default:

null

services.llmhop.sglang.models.<name>.model

Hugging Face repo id (or local path) passed as --model-path.

Type: string

Example:

"Qwen/Qwen3-8B"

services.llmhop.sglang.models.<name>.name

Canonical identifier for this model. Used for the unit name (sglang-<name>) and as the routing key registered with llmhop (clients select the backend by sending this value in the OpenAI model field).

Defaults to the attribute key, so the key itself must match the required label format.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.sglang.models.<name>.port

Loopback host port sglang binds to (--host 127.0.0.1 --port <port>). Must be unique per enabled model; llmhop reaches the backend at http://127.0.0.1:<port>.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.sglang.models.<name>.serviceConfig

Extra [Service] settings merged into this model’s sglang-<name> unit after the hardened baseline and backend-specific relaxations. The module retains ownership of ExecStart, KillMode, and Type because they implement readiness supervision as one lifecycle contract.

Type: attribute set of anything

Default:

{ }

Example:

{
  MemoryHigh = "64G";
}

services.llmhop.sglang.models.<name>.settings

CLI flags forwarded to the model server for this model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false is dropped, since the CLI pairs --enable-X with --disable-X instead of auto-negating: write the negated key explicitly, e.g. disable-radix-cache = true;. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.sglang.modelSettings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.sglang.openFilesLimit

File descriptor limit (LimitNOFILE) applied to every sglang systemd unit. Increase if the server logs accept: Too many open files under concurrent load.

Type: positive integer, meaning >0

Default:

services.llmhop.sglang.startupOrdering

Whether to chain enabled model services by ascending port during startup. GPU-memory profiling races otherwise: two workers booting on the same device each see it as fully free and race to claim their share, leading to OOM. Disable only when each model pins itself to a dedicated device via environment (the variable is stack-specific: CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ZE_AFFINITY_MASK, …).

Type: boolean

Default:

true

services.llmhop.sglang.uid

Host UID assigned to services.llmhop.sglang.user. Required — pick a value that does not clash with other system users on the host.

Type: unsigned integer, meaning >=0

Example:

services.llmhop.sglang.user

Dedicated system user owning the sglang data and cache directories. Defaults to the backend name; override to point at a user the deployer manages externally (in which case the matching users.users.<name> and users.groups.<name> declarations become the deployer’s responsibility).

Type: string

Default:

"sglang"

sglang-quadlet

services.llmhop.sglang-quadlet.enable

Whether to enable SGLang model serving via Quadlet, optionally fronted by the SGL Model Gateway.

Type: boolean

Default:

false

Example:

true

services.llmhop.sglang-quadlet.cacheDir

Host directory bind-mounted as the Hugging Face cache for every worker.

Type: absolute path

Default:

"/var/cache/sglang"

services.llmhop.sglang-quadlet.dataDir

Home directory of services.llmhop.sglang-quadlet.user. Used by rootless podman for container storage (~/.local/share/containers), so it must live on a filesystem that tolerates overlayfs.

Type: absolute path

Default:

"/var/lib/sglang"

services.llmhop.sglang-quadlet.devices

Devices exposed to every model container — passed verbatim as Quadlet AddDevice= lines. Accepts both CDI references (recommended: nvidia.com/gpu=…, amd.com/gpu=…, intel.com/gpu=…, …) and raw host device paths (e.g. /dev/dri/renderD128). For CDI, the corresponding spec must be generated on the host (e.g. nvidia-ctk cdi generate). Defaults to [ "nvidia.com/gpu=all" ] when hardware.nvidia-container-toolkit.enable is set, otherwise empty (CPU-only). Per-model devices overrides this.

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then
  [ "nvidia.com/gpu=all" ]
else
  [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.sglang-quadlet.environment

Environment variables set on every model service. Merged with services.llmhop.sglang-quadlet.models.<name>.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang-quadlet.environmentFile

File in KEY=VALUE format forwarded to every service. Use for secrets managed by sops-nix/agenix, e.g. a file containing HF_TOKEN=<token> to access gated Hugging Face repositories. Loaded before services.llmhop.sglang-quadlet.models.<name>.environmentFile, so per-model files override these entries.

Type: null or absolute path

Default:

null

Example:

"/etc/sglang-quadlet/.env"

services.llmhop.sglang-quadlet.gateway.enable

Whether to enable the SGL Model Gateway in front of the workers. Disabled by default — llmhop already routes between every backend, and the gateway is only needed when you want SGLang’s IGW dispatch features (custom routing, prefix caching across workers, etc.) .

Type: boolean

Default:

false

Example:

true

services.llmhop.sglang-quadlet.gateway.enableMetrics

Whether to enable Prometheus metrics on the gateway.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang-quadlet.gateway.bindAddress

Host address the gateway binds its listeners to. Defaults to the loopback so external clients must go through Caddy / llmhop.

Type: string

Default:

"127.0.0.1"

services.llmhop.sglang-quadlet.gateway.digest

Immutable digest of the gateway image. Mutually exclusive with tag.

Type: null or string

Default:

null

services.llmhop.sglang-quadlet.gateway.environment

Additional environment variables set on the gateway container.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang-quadlet.gateway.environmentFile

File in KEY=VALUE format forwarded to the gateway via --env-file. Use for secrets like API keys; the gateway’s --api-key flag may also be passed via settings if the value is non-secret.

Type: null or absolute path

Default:

null

Example:

"/etc/sglang/gateway.env"

services.llmhop.sglang-quadlet.gateway.image

Container image used for the gateway.

Type: string

Default:

"docker.io/lmsysorg/sgl-model-gateway"

services.llmhop.sglang-quadlet.gateway.metricsPort

Host port the gateway exposes Prometheus metrics on. Ignored when enableMetrics is false.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

services.llmhop.sglang-quadlet.gateway.port

Host port the gateway listens on.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.sglang-quadlet.gateway.settings

Additional CLI flags forwarded to sgl-model-gateway. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false is dropped, since the CLI pairs --enable-X with --disable-X instead of auto-negating: write the negated key explicitly, e.g. disable-radix-cache = true;. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

--worker-urls is rendered from the enabled models, so setting it here replaces the generated list.

Type: attribute set of anything

Default:

{ }

Example:

{
  api-key = "secret";
  tls-cert-path = "/etc/sglang/tls/server.crt";
}

services.llmhop.sglang-quadlet.gateway.tag

Default tag of the gateway image. Mutually exclusive with digest.

Type: null or string

Default:

"latest"

services.llmhop.sglang-quadlet.gid

Host GID assigned to services.llmhop.sglang-quadlet.group. Defaults to uid. It is also the inner-to-outer target of --gidmap.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.sglang-quadlet.uid

services.llmhop.sglang-quadlet.group

Primary group for services.llmhop.sglang-quadlet.user. Defaults to the user name (matching the typical 1:1 user/group layout).

Type: string

Default:

config.services.llmhop.sglang-quadlet.user

services.llmhop.sglang-quadlet.image

Container image used for every model worker.

Type: string

Default:

"docker.io/lmsysorg/sglang"

services.llmhop.sglang-quadlet.modelSettings

Merged with services.llmhop.sglang-quadlet.models.<name>.settings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.sglang-quadlet.models

Models to serve. Each entry produces one quadlet container; the attribute name is the routing key (advertised via --served-model-name and surfaced through both llmhop and the optional SGL Model Gateway as the OpenAI model field). Enabled entries are sorted by ascending port.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen3-8b" = {
    model = "Qwen/Qwen3-8B";
    port = 19001;
    settings = {
      reasoning-parser = "qwen3";
      tool-call-parser = "qwen3_coder";
      mem-fraction-static = 0.6;
      cuda-graph-max-bs = 4;
    };
  };
}

services.llmhop.sglang-quadlet.models.<name>.enable

Whether to enable serving of model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.sglang-quadlet.models.<name>.devices

Devices exposed to this model’s container — passed verbatim as Quadlet AddDevice= lines. Replaces (does not extend) services.llmhop.sglang-quadlet.devices for this model. Use to pin a model to specific device indices (e.g. [ "nvidia.com/gpu=0" ]).

Type: list of string

Default:

config.services.llmhop.sglang-quadlet.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.sglang-quadlet.models.<name>.digest

Immutable digest of the container image (e.g. sha256:…). Mutually exclusive with tag.

Type: null or string

Default:

null

Example:

"sha256:a73fb0b9046fee099f7c1829d2548e6cc1740f4c2776a6855fa659ae5d0deb49"

services.llmhop.sglang-quadlet.models.<name>.environment

Additional environment variables set on this model’s service. Merged with services.llmhop.sglang-quadlet.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.sglang-quadlet.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s service. Loaded after services.llmhop.sglang-quadlet.environmentFile, so its entries override global ones. Must be readable by the user systemd reads it as.

Type: null or absolute path

Default:

null

services.llmhop.sglang-quadlet.models.<name>.model

Hugging Face repo id (or local path) passed to the model server.

Type: string

Example:

"Qwen/Qwen2.5-7B-Instruct"

services.llmhop.sglang-quadlet.models.<name>.name

Defaults to the attribute key, so the key itself must match the required label format.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.sglang-quadlet.models.<name>.port

Loopback host port forwarded to the container’s SGLang API. Must be unique per model and must not collide with gateway.port / gateway.metricsPort when the gateway is enabled.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.sglang-quadlet.models.<name>.settings

Merged with services.llmhop.sglang-quadlet.modelSettings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.sglang-quadlet.models.<name>.shmSize

Size of the container’s private /dev/shm tmpfs. PyTorch and friends use shared memory for NCCL/tensor-parallel inference; upstream recommends 32g (or --ipc=host). A private tmpfs is preferred for isolation: raise the value for larger models or higher tensor-parallel sizes.

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.sglang-quadlet.models.<name>.tag

Tag of the container image used for this model. Mutually exclusive with digest.

Type: null or string

Default:

null

services.llmhop.sglang-quadlet.openFilesLimit

File descriptor limit (LimitNOFILE) applied to every sglang-quadlet systemd unit. Increase if the server logs accept: Too many open files under concurrent load.

Type: positive integer, meaning >0

Default:

services.llmhop.sglang-quadlet.startupOrdering

Type: boolean

Default:

true

services.llmhop.sglang-quadlet.subGidCount

Size of the subordinate GID range mapped into every container. Defaults to subUidCount.

Type: positive integer, meaning >0

Default:

config.services.llmhop.sglang-quadlet.subUidCount

services.llmhop.sglang-quadlet.subGidStart

First host GID of the subordinate range mapped into every container. Defaults to subUidStart — most setups keep the UID and GID ranges aligned.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.sglang-quadlet.subUidStart

services.llmhop.sglang-quadlet.subUidCount

Size of the subordinate UID range mapped into every container. 65536 covers the full unprivileged ID space inside the namespace.

Type: positive integer, meaning >0

Default:

services.llmhop.sglang-quadlet.subUidStart

First host UID of the subordinate range mapped into every container. Container UIDs ≥1 are mapped to subUidCount consecutive host IDs starting here. Required — pick a value clear of NixOS system users (<1000), regular login UIDs, and other backends’ subordinate ranges on the same host.

Type: unsigned integer, meaning >=0

Example:

services.llmhop.sglang-quadlet.tag

Default tag of the container image used for models that do not set their own tag or digest.

Type: string

Example:

"latest"

services.llmhop.sglang-quadlet.uid

Host UID assigned to services.llmhop.sglang-quadlet.user. Required — pick a value that does not clash with other system users on the host. It is also the inner-to-outer target of --uidmap.

Type: unsigned integer, meaning >=0

Example:

services.llmhop.sglang-quadlet.user

Type: string

Default:

"sglang"

vllm

services.llmhop.vllm.enable

Whether to enable vLLM model serving via systemd (native host process), fronted by llmhop.

Type: boolean

Default:

false

Example:

true

services.llmhop.vllm.package

Package providing the vllm CLI at bin/vllm.

No default on purpose: vLLM has no one-derivation-fits-all (new model architectures routinely need dev snapshots, and the wheels come in per-accelerator variants), so you build the package from a uv workspace and pin / follow upstream there. The flake exposes a helper:

inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
  workspaceRoot = ./vllm-env; # your pyproject.toml + uv.lock
}

Individual models may override this with models.<name>.package.

Type: package

Example:

inputs.llmhop.legacyPackages.${pkgs.system}.mkUvEnv {
  workspaceRoot = ./vllm-env;
}

services.llmhop.vllm.environment

Environment variables set on every model service. Merged with services.llmhop.vllm.models.<name>.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.environmentFile

File in KEY=VALUE format forwarded to every service. Use for secrets managed by sops-nix/agenix, e.g. a file containing HF_TOKEN=<token> to access gated Hugging Face repositories. Loaded before services.llmhop.vllm.models.<name>.environmentFile, so per-model files override these entries.

Type: null or absolute path

Default:

null

Example:

"/etc/vllm/.env"

services.llmhop.vllm.gid

Host GID assigned to services.llmhop.vllm.group. Defaults to uid.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.vllm.uid

services.llmhop.vllm.group

Primary group for services.llmhop.vllm.user. Defaults to the user name (matching the typical 1:1 user/group layout).

Type: string

Default:

config.services.llmhop.vllm.user

services.llmhop.vllm.modelSettings

CLI flags forwarded to the model server for every model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.vllm.models.<name>.settings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.vllm.models

Models to serve. Each enabled entry produces one systemd service named vllm-<name>; the attribute name is the routing key surfaced through llmhop as the OpenAI model field. Enabled entries are sorted by ascending port.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen2-5-7b" = {
    model = "Qwen/Qwen2.5-7B-Instruct";
    port = 18001;
  };
  "llama-3-8b" = {
    model = "meta-llama/Meta-Llama-3-8B-Instruct";
    port = 18002;
    settings.max-model-len = 8192;
  };
}

services.llmhop.vllm.models.<name>.enable

Whether to enable serving of model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm.models.<name>.package

Package providing this model’s worker, overriding the backend-wide package. Set it for a model that needs a different vllm release than the rest — e.g. a nightly wheel for a just-released architecture — built the same way with mkUvEnv over a per-model uv workspace. Defaults to the backend-wide package.

Type: package

Default:

config.services.llmhop.vllm.package

services.llmhop.vllm.models.<name>.environment

Additional environment variables set on this model’s service. Merged with services.llmhop.vllm.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s service. Loaded after services.llmhop.vllm.environmentFile, so its entries override global ones. Must be readable by the user systemd reads it as.

Type: null or absolute path

Default:

null

services.llmhop.vllm.models.<name>.model

Hugging Face repo id (or local path) passed as the vllm serve positional argument.

Type: string

Example:

"Qwen/Qwen2.5-7B-Instruct"

services.llmhop.vllm.models.<name>.name

Canonical identifier for this model. Used for the unit name (vllm-<name>) and as the routing key registered with llmhop (clients select the backend by sending this value in the OpenAI model field).

Defaults to the attribute key, so the key itself must match the required label format.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm.models.<name>.port

Loopback host port vllm binds to (--host 127.0.0.1 --port <port>). Must be unique per enabled model; llmhop reaches the backend at http://127.0.0.1:<port>.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.vllm.models.<name>.serviceConfig

Extra [Service] settings merged into this model’s vllm-<name> unit after the hardened baseline and backend-specific relaxations. The module retains ownership of ExecStart, KillMode, and Type because they implement readiness supervision as one lifecycle contract.

Type: attribute set of anything

Default:

{ }

Example:

{
  MemoryHigh = "64G";
}

services.llmhop.vllm.models.<name>.settings

CLI flags forwarded to the model server for this model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.vllm.modelSettings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.vllm.openFilesLimit

File descriptor limit (LimitNOFILE) applied to every vllm systemd unit. Increase if the server logs accept: Too many open files under concurrent load.

Type: positive integer, meaning >0

Default:

services.llmhop.vllm.startupOrdering

Type: boolean

Default:

true

services.llmhop.vllm.uid

Host UID assigned to services.llmhop.vllm.user. Required — pick a value that does not clash with other system users on the host.

Type: unsigned integer, meaning >=0

Example:

services.llmhop.vllm.user

Dedicated system user owning the vllm data and cache directories. Defaults to the backend name; override to point at a user the deployer manages externally (in which case the matching users.users.<name> and users.groups.<name> declarations become the deployer’s responsibility).

Type: string

Default:

"vllm"

vllm-quadlet

services.llmhop.vllm-quadlet.enable

Whether to enable vLLM model serving via Quadlet, fronted by llmhop.

Type: boolean

Default:

false

Example:

true

services.llmhop.vllm-quadlet.cacheDir

Host directory bind-mounted as the Hugging Face cache for every worker.

Type: absolute path

Default:

"/var/cache/vllm"

services.llmhop.vllm-quadlet.dataDir

Home directory of services.llmhop.vllm-quadlet.user. Used by rootless podman for container storage (~/.local/share/containers), so it must live on a filesystem that tolerates overlayfs.

Type: absolute path

Default:

"/var/lib/vllm"

services.llmhop.vllm-quadlet.devices

Type: list of string

Default:

if config.hardware.nvidia-container-toolkit.enable then
  [ "nvidia.com/gpu=all" ]
else
  [ ]

Example:

[
  "amd.com/gpu=all"
]

services.llmhop.vllm-quadlet.environment

Environment variables set on every model service. Merged with services.llmhop.vllm-quadlet.models.<name>.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm-quadlet.environmentFile

File in KEY=VALUE format forwarded to every service. Use for secrets managed by sops-nix/agenix, e.g. a file containing HF_TOKEN=<token> to access gated Hugging Face repositories. Loaded before services.llmhop.vllm-quadlet.models.<name>.environmentFile, so per-model files override these entries.

Type: null or absolute path

Default:

null

Example:

"/etc/vllm-quadlet/.env"

services.llmhop.vllm-quadlet.gid

Host GID assigned to services.llmhop.vllm-quadlet.group. Defaults to uid. It is also the inner-to-outer target of --gidmap.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.vllm-quadlet.uid

services.llmhop.vllm-quadlet.group

Primary group for services.llmhop.vllm-quadlet.user. Defaults to the user name (matching the typical 1:1 user/group layout).

Type: string

Default:

config.services.llmhop.vllm-quadlet.user

services.llmhop.vllm-quadlet.image

Container image used for every model worker.

Type: string

Default:

"docker.io/vllm/vllm-openai"

services.llmhop.vllm-quadlet.modelSettings

CLI flags forwarded to the model server for every model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.vllm-quadlet.models.<name>.settings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.vllm-quadlet.models

Models to serve. Each entry produces one quadlet container; the attribute name is the routing key. Enabled entries are sorted by ascending port.

Type: attribute set of (submodule)

Default:

{ }

Example:

{
  "qwen2-5-7b" = {
    model = "Qwen/Qwen2.5-7B-Instruct";
    port = 18001;
  };
  "llama-3-8b" = {
    model = "meta-llama/Meta-Llama-3-8B-Instruct";
    port = 18002;
    settings.max-model-len = 8192;
  };
}

services.llmhop.vllm-quadlet.models.<name>.enable

Whether to enable serving of model ‹name›.

Type: boolean

Default:

true

Example:

true

services.llmhop.vllm-quadlet.models.<name>.devices

Devices exposed to this model’s container — passed verbatim as Quadlet AddDevice= lines. Replaces (does not extend) services.llmhop.vllm-quadlet.devices for this model. Use to pin a model to specific device indices (e.g. [ "nvidia.com/gpu=0" ]).

Type: list of string

Default:

config.services.llmhop.vllm-quadlet.devices

Example:

[
  "nvidia.com/gpu=0"
]

services.llmhop.vllm-quadlet.models.<name>.digest

Immutable digest of the container image (e.g. sha256:…). Mutually exclusive with tag.

Type: null or string

Default:

null

Example:

"sha256:a73fb0b9046fee099f7c1829d2548e6cc1740f4c2776a6855fa659ae5d0deb49"

services.llmhop.vllm-quadlet.models.<name>.environment

Additional environment variables set on this model’s service. Merged with services.llmhop.vllm-quadlet.environment; per-model entries take precedence.

Type: attribute set of string

Default:

{ }

services.llmhop.vllm-quadlet.models.<name>.environmentFile

File in KEY=VALUE format forwarded to this model’s service. Loaded after services.llmhop.vllm-quadlet.environmentFile, so its entries override global ones. Must be readable by the user systemd reads it as.

Type: null or absolute path

Default:

null

services.llmhop.vllm-quadlet.models.<name>.model

Hugging Face repo id (or local path) passed to the model server.

Type: string

Example:

"Qwen/Qwen2.5-7B-Instruct"

services.llmhop.vllm-quadlet.models.<name>.name

Defaults to the attribute key, so the key itself must match the required label format.

Type: string matching the pattern [[:alnum:]][[:alnum:].-]*

Default:

"‹name›"

services.llmhop.vllm-quadlet.models.<name>.port

Loopback host port forwarded to the container’s vLLM API. Must be unique per model.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

services.llmhop.vllm-quadlet.models.<name>.settings

CLI flags forwarded to the model server for this model. true collapses to --<key>, null and empty lists are dropped, and an attribute set is serialised to JSON. false renders as --no-<key>, so a flag with no negated twin (an on-only one, or a tri-state one taking on|off|auto) has to be omitted or given its value explicitly rather than set to false. A list hands every element to a single flag (--<key> a b), which is what most multi-value options of this CLI take. The few that instead expect a repeated flag have to be written out one value at a time.

Merged with services.llmhop.vllm-quadlet.modelSettings; per-model entries take precedence.

Type: attribute set of anything

Default:

{ }

services.llmhop.vllm-quadlet.models.<name>.shmSize

Type: string

Default:

"32g"

Example:

"64g"

services.llmhop.vllm-quadlet.models.<name>.tag

Tag of the container image used for this model. Mutually exclusive with digest.

Type: null or string

Default:

null

services.llmhop.vllm-quadlet.openFilesLimit

File descriptor limit (LimitNOFILE) applied to every vllm-quadlet systemd unit. Increase if the server logs accept: Too many open files under concurrent load.

Type: positive integer, meaning >0

Default:

services.llmhop.vllm-quadlet.startupOrdering

Type: boolean

Default:

true

services.llmhop.vllm-quadlet.subGidCount

Size of the subordinate GID range mapped into every container. Defaults to subUidCount.

Type: positive integer, meaning >0

Default:

config.services.llmhop.vllm-quadlet.subUidCount

services.llmhop.vllm-quadlet.subGidStart

First host GID of the subordinate range mapped into every container. Defaults to subUidStart — most setups keep the UID and GID ranges aligned.

Type: unsigned integer, meaning >=0

Default:

config.services.llmhop.vllm-quadlet.subUidStart

services.llmhop.vllm-quadlet.subUidCount

Size of the subordinate UID range mapped into every container. 65536 covers the full unprivileged ID space inside the namespace.

Type: positive integer, meaning >0

Default:

services.llmhop.vllm-quadlet.subUidStart

Type: unsigned integer, meaning >=0

Example:

services.llmhop.vllm-quadlet.tag

Default tag of the container image used for models that do not set their own tag or digest.

Type: string

Example:

"v0.11.0"

services.llmhop.vllm-quadlet.uid

Host UID assigned to services.llmhop.vllm-quadlet.user. Required — pick a value that does not clash with other system users on the host. It is also the inner-to-outer target of --uidmap.

Type: unsigned integer, meaning >=0

Example:

services.llmhop.vllm-quadlet.user

Type: string

Default:

"vllm"

Keyboard shortcuts

llmhop