181 lines
8.4 KiB
Markdown
181 lines
8.4 KiB
Markdown
# Compute nodes and the Node Agent
|
|
|
|
A compute node is a GPU host that serves capabilities. The Node Agent runs there, reports inventory
|
|
and telemetry, and claims work.
|
|
|
|
**The control plane never dials a node.** The agent is outbound-only: it needs to reach the API, and
|
|
the API needs no route back. That is what makes a node behind NAT, on a home network, or on a
|
|
different site work without inbound exposure.
|
|
|
|
## Requirements on the node
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| NVIDIA driver | matching your GPU |
|
|
| NVIDIA container runtime | required for the Runtime Worker |
|
|
| Docker | 24+ |
|
|
| Reachability | outbound HTTPS to the control plane |
|
|
| Storage | artifact root, quarantine and agent state on durable storage |
|
|
|
|
## Enrolling
|
|
|
|
For the v1.3.0 RC, an administrator can use **Console → Hardware → Enroll node**. The wizard offers
|
|
5-minute, 15-minute, 1-hour, and 24-hour token expiry, and locks the node's API origin to the
|
|
Console's current API origin. RC enrollment always sets `production_eligible=false`. The one-time
|
|
token is held only in the open wizard's component memory; closing the wizard, checking readiness,
|
|
revoking the unused token, or reaching its expiry clears the reveal. The generated `.env` contains
|
|
only non-secret settings. Paste the start command into a Bash shell from the node-agent repository;
|
|
it disables shell tracing, reads the token without echo, uses a temporary mode-0600 env file, and
|
|
removes that file on exit, including error and signal paths.
|
|
|
|
Use **Check node readiness** after starting the agent. Readiness is tied to the exact node ID returned
|
|
by the enrollment record; it checks connection, inventory age, protocol compatibility, agent health,
|
|
and telemetry age and includes diagnostic next steps. After enrollment is consumed, use the wizard's
|
|
container recreation command to remove the one-time token from the running container environment
|
|
while preserving its persisted identity and credential. A node is not ready merely because it
|
|
connected.
|
|
|
|
The API flow below remains available for automation and existing installations.
|
|
|
|
The RC wizard defaults to the local four-image release bundle. Verify the release `SHA256SUMS`, load
|
|
`modelforge-1.3.0-rc.1-images.tar` on the node with `docker image load`, and paste
|
|
`images.modelforge-node-agent` from the matching release manifest. The wizard requires the exact
|
|
`modelforge-node-agent:1.3.0-rc.1` tag, RootFS layer list, architecture, and all recorded OCI labels.
|
|
It ignores the build-host image/config ID because Docker may rewrite it during `docker load`; before
|
|
Compose starts it captures the local image ID and verifies the created container uses that same ID.
|
|
Start and cleanup use `--no-build --pull never`, and the generated configuration sets the local tag
|
|
explicitly. A registry image pinned to `@sha256:<digest>` remains available as an alternate mode.
|
|
The source-build Compose workflow below is separate and intended for development.
|
|
|
|
1. An operator mints a single-use token:
|
|
|
|
```bash
|
|
curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
|
|
-d '{"display_name":"GPU Node","expires_in_seconds":900}' \
|
|
http://127.0.0.1:8000/api/v1/admin/node-enrollments
|
|
```
|
|
|
|
2. Configure the agent on the node:
|
|
|
|
```ini
|
|
MODELFORGE_AGENT_CONTROL_PLANE_URL=https://modelforge.example.internal:8000
|
|
MODELFORGE_AGENT_ENROLLMENT_TOKEN=<the token>
|
|
MODELFORGE_AGENT_HOSTNAME=gpu_node
|
|
MODELFORGE_AGENT_ACCELERATOR_MODE=nvidia
|
|
MODELFORGE_AGENT_TLS_VERIFY=true
|
|
MODELFORGE_AGENT_STATE_VOLUME=/mnt/user/appdata/modelforge/agent-state
|
|
```
|
|
|
|
3. Start it:
|
|
|
|
```bash
|
|
export MODELFORGE_NODE_AGENT_IMAGE=modelforge-node-agent:1.1.1
|
|
docker compose -f docker-compose.yml -f docker-compose.node-agent.yml up -d node-agent
|
|
```
|
|
|
|
For production, set `MODELFORGE_NODE_AGENT_IMAGE` to the exact release tag or registry digest. The
|
|
Compose file retains `build:` for local source builds, but its fallback is explicitly `local` and
|
|
never the floating `latest` tag.
|
|
|
|
The token is **single use and atomically claimed**. Sixty concurrent attempts against one token
|
|
produce exactly one identity and one active credential — that is enforced by a conditional update
|
|
whose row count decides the winner, not by timing.
|
|
|
|
Verified on a genuinely fresh control plane before release: the node appeared once, `online`, with
|
|
exactly one active credential.
|
|
|
|
## NVIDIA startup contract
|
|
|
|
The v1.1.1 Node Agent runs on a digest-pinned Debian/glibc base. NVIDIA Container Toolkit injects
|
|
the host's driver-facing `libnvidia-ml.so.1`, dynamic loader bindings and `/dev/nvidia*` devices;
|
|
the image deliberately does not bundle a userspace driver library. The earlier v1.1.0 Alpine/musl
|
|
image could import the pure-Python `pynvml` package but could not relocate GPU Node's glibc-linked
|
|
NVML library, producing `NVMLError_LibraryNotFound`.
|
|
|
|
`MODELFORGE_AGENT_ACCELERATOR_MODE` has three values:
|
|
|
|
- `nvidia`: require NVML, at least one real GPU, and matching telemetry; canonical GPU Compose
|
|
deployments use this value.
|
|
- `cpu`: permit a legitimate CPU-only node without NVML.
|
|
- `auto`: require NVML only when NVIDIA devices or `NVIDIA_VISIBLE_DEVICES` are observed.
|
|
|
|
The NVIDIA preflight runs before enrollment. A missing library, empty device result, or missing
|
|
telemetry exits with code 3 and one of `NVIDIA_NVML_UNAVAILABLE`, `NVIDIA_DEVICE_NOT_FOUND`, or
|
|
`NVIDIA_TELEMETRY_UNAVAILABLE`; no empty-success inventory or fabricated telemetry is sent. A
|
|
bounded diagnostic check that never enrolls is available as:
|
|
|
|
```bash
|
|
docker run --rm --gpus all --network none \
|
|
-e MODELFORGE_AGENT_ACCELERATOR_MODE=nvidia \
|
|
modelforge-node-agent:1.1.1 python3 -m modelforge_node_agent.main --preflight
|
|
```
|
|
|
|
The chiselled runtime intentionally contains no shell or packaging-script directory. Invoke the
|
|
reviewed Python module directly for this bounded diagnostic, just as the image's default command
|
|
does.
|
|
|
|
## Identity
|
|
|
|
A node's identity is persisted and survives restarts. If a node reappears as a *new* identity, its
|
|
state volume is not durable — fix that before anything else, because duplicate identities are the
|
|
one thing the platform cannot untangle for you.
|
|
|
|
`MODELFORGE_AGENT_IDENTITY_MODE=persisted` keeps the identity in
|
|
`MODELFORGE_AGENT_STATE_VOLUME`. On Unraid, point that at `appdata`, not at a container-local path.
|
|
|
|
## Liveness
|
|
|
|
`online → stale → offline` on the configured thresholds
|
|
(`MODELFORGE_NODE_STALE_AFTER_SECONDS`, `MODELFORGE_NODE_OFFLINE_AFTER_SECONDS`; offline must exceed
|
|
stale, and startup refuses a configuration where it does not).
|
|
|
|
No work is assigned to a node that is not online. A disconnected agent stops publishing and the
|
|
control plane says so rather than assuming the node is fine — verified by disconnecting a disposable
|
|
agent for 100 seconds and confirming its heartbeat genuinely stopped advancing.
|
|
|
|
## Private CA
|
|
|
|
If the control plane is behind a private certificate authority:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.yml \
|
|
-f docker-compose.node-agent.yml \
|
|
-f docker-compose.node-agent.private-ca.yml up -d node-agent
|
|
```
|
|
|
|
with `MODELFORGE_AGENT_CA_CERT_PATH` and `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` set. Do not
|
|
turn off `MODELFORGE_AGENT_TLS_VERIFY` to work around a certificate problem — that replaces a
|
|
certificate error with a silent one.
|
|
|
|
## The Runtime Worker
|
|
|
|
The worker executes the model. It is a separate service and a separate image, pinned by digest,
|
|
built on the CUDA runtime base:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.yml -f docker-compose.runtime-worker.yml up -d runtime-worker
|
|
```
|
|
|
|
It reports its own protocol version, and the same four compatibility answers apply.
|
|
|
|
## Diagnosing an agent that will not report
|
|
|
|
A failed publication names the status, the method and the path — never the response body, which can
|
|
echo what the agent was publishing, and never the credential:
|
|
|
|
```text
|
|
ConnectError on POST /api/v1/agent/heartbeat (backoff 1s…16s)
|
|
```
|
|
|
|
That distinguishes a revoked credential from a DNS failure. If the cause is not obvious, check that
|
|
`MODELFORGE_AGENT_CONTROL_PLANE_URL` is a real address and not the example placeholder.
|
|
|
|
## Compatibility
|
|
|
|
The control plane speaks agent protocol version 1 and accepts version 1. An agent reporting anything
|
|
else is told which of `TOO_OLD`, `TOO_NEW` or `UNKNOWN` applies and what to do. There is no silent
|
|
mismatch. See [COMPATIBILITY.md](COMPATIBILITY.md).
|
|
|
|
v1.1.1 keeps protocol 1, so v1.0.0 and v1.1.0 agents remain protocol-compatible. NVIDIA nodes
|
|
should run v1.1.1 to obtain the glibc/NVML fix and fail-closed startup contract.
|