3.1 KiB
Runbook: node loss or credential loss
Trigger
A compute node stops publishing, its credential is lost or believed stolen, or a node must be rebuilt after a control-plane restore.
This runbook is for recoverable loss. If hardware is permanently leaving service, use
RUNBOOK_NODE_DECOMMISSION.md; do not disable or delete the row by hand.
Diagnose
Distinguish the three failures, because they have different recoveries:
| Symptom | Meaning |
|---|---|
| heartbeats 401 | the credential is revoked or does not match a stored hash |
| heartbeats 403 | the credential authenticates but is not authorised for that node |
| no requests at all | the agent is down, or cannot reach the control plane |
Check node_credentials for the node: how many are active, how many revoked, and when the active
one was last used. Only hashes are stored; the plaintext cannot be read back from anywhere.
Act and recover
The node still has its credential. Nothing to do. After a database restore the restored hash matches and the node reconnects on its own.
The credential is lost. Issue a fresh enrolment token
(POST /api/v1/admin/node-enrollments) and restart the agent with it. The agent keeps its persisted
identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
history intact. Enrolment revokes the previous credential automatically.
If the node's identity volume is still present and an administrator can install a replacement
credential directly, open the node detail and use Rotate node credential. Rotation revokes the
old key immediately. Replace /data/state/node-credential with the one-time value and recreate the
agent before expecting heartbeats or inventory to resume. The Console cannot install the value on
the node. If the identity volume is missing, use fresh enrolment instead.
The credential is believed stolen. Revoke it first
(DELETE /api/v1/admin/hardware/nodes/{node_id}/credential), then re-enrol. Revocation is
permanent: the old credential stays refused after re-enrolment. If a replacement can be installed
immediately, the node detail's rotation action combines revocation and replacement; the old key is
still permanently refused.
The identity file is also lost. The agent enrols as a new node, because ModelForge genuinely cannot tell it apart from different hardware. If the old identity is permanently superseded, use the audited decommission preview/execute flow rather than deleting or hand-editing it, and confirm only one enabled node claims that hostname.
One enrolment token mints exactly one node identity. If two nodes appear from one token, the control plane predates the M15 atomic-claim fix.
Confirm
Exactly one enabled node for that hardware, one active credential with the old ones revoked, heartbeats returning 200, inventory and telemetry re-collected, the scheduler reconciled against freshly measured NVML state rather than a restored snapshot, and a capability smoke against the recovered node.
Never revoke a production node credential merely to rehearse this. Use a disposable agent identity;
docker-compose.node-recovery.yml exists for exactly that.