255 lines
14 KiB
Markdown
255 lines
14 KiB
Markdown
# Upgrade boundaries — historical v1.2.3 procedure and current RC source
|
|
|
|
> Status reconciled on 2026-09-23: no v1.2.3 Gitea Release object is published. The observed
|
|
> production API is v1.2.3/schema `20260909_0032`; the current source line targets schema
|
|
> `20260923_0037` and has not been production-qualified. The detailed commands below
|
|
> retain the earlier v1.2.3/v1.2.2 schema-0024 and intermediate-RC instructions as historical
|
|
> guidance; they are not an authorization to deploy current master or to claim migration-0037
|
|
> recovery evidence. Use the current-state reconciliation and RC roadmap for the active candidate.
|
|
|
|
## Source and schema boundary
|
|
|
|
The schema numbers and commands in this section describe an earlier source snapshot. The current
|
|
`1.3.0-rc.1` source targets `20260923_0037`, while the observed production database remains at
|
|
`20260909_0032`. Rehearse that exact upgrade and recovery path before publishing the RC.
|
|
|
|
The previously documented v1.2.3 password hotfix was built from the v1.2.2 line and retains schema
|
|
`20260830_0024`. The RC development branch containing this integration requires
|
|
`20260908_0028`. A matching version string does not make these builds interchangeable:
|
|
use the exact commit, image digests and reported schema contract.
|
|
|
|
For this RC source, the upgrade tool accepts schema 0021 through 0028 and targets 0028.
|
|
In addition to the 0024 role boundary below, it applies migrations 0025 (configuration),
|
|
0026 (project onboarding), 0027 (managed operator identities) and 0028 (platform secret slots).
|
|
Back up and rehearse those migrations in the isolated RC environment before promotion;
|
|
retain the external secret-slot root key for restore. Do not deploy the published schema-0024
|
|
hotfix API against an RC database advanced to 0028. Rollback across that boundary requires
|
|
restoring the matching verified database backup and application images.
|
|
|
|
The remaining release commands document the previously documented v1.2.3 hotfix. They are not an RC
|
|
promotion instruction; use the separately maintained RC roadmap for its outstanding gates.
|
|
|
|
|
|
v1.2.3 retains schema `20260830_0024` from v1.2.2 and adds operator password sign-in.
|
|
An existing v1.2.2/schema-0024 installation needs no schema migration or runtime-worker update.
|
|
Configure the password hash as described in [INSTALLATION.md](INSTALLATION.md), retain the
|
|
bootstrap key, and deploy the verified API and web images. Earlier schema sources still require
|
|
the provisioning and migration procedure below. Publication alone does not promote production.
|
|
|
|
## Required v1.2.1/schema-0022 role provisioning before migration 0024
|
|
|
|
Schema 0024 changes the production authority boundary. The API may no longer run as the table owner
|
|
or the bootstrap superuser. Provisioning is deliberately not hidden inside Alembic because an
|
|
application migration must not mint or retain an administrator credential.
|
|
|
|
1. Take and verify the pre-upgrade backup, then stop `api`, `web`, and every older writer. This is a
|
|
downtime migration; the 0024 migration also takes an advisory lock and `ACCESS EXCLUSIVE` on
|
|
`audit_events`, but the stopped-application contract prevents an old process from resuming with
|
|
its former owner credential.
|
|
2. Generate three distinct URL-safe secrets: retained PostgreSQL administrator, non-superuser
|
|
`modelforge` migration/schema owner, and NOINHERIT `modelforge_runtime` API role. Set
|
|
`MODELFORGE_POSTGRES_ADMIN_USER` to a new retained name such as `modelforge_admin`; set the three
|
|
password variables and both owner/runtime SQLAlchemy URLs documented in `CONFIGURATION.md`.
|
|
3. Determine whether the existing `modelforge` role is PostgreSQL's protected bootstrap
|
|
superuser before changing anything:
|
|
|
|
```sql
|
|
select oid, rolsuper, oid = 10 as protected_bootstrap
|
|
from pg_roles where rolname = 'modelforge';
|
|
```
|
|
|
|
If `protected_bootstrap` is false, recreate only the PostgreSQL container so the new values are
|
|
in its environment. It may report
|
|
unhealthy until the new admin role exists; that is expected during this bounded downtime. While
|
|
still connecting with the existing v1.2.1 `modelforge` bootstrap superuser, run:
|
|
|
|
```bash
|
|
docker compose exec -T postgres psql -v ON_ERROR_STOP=1 \
|
|
-U modelforge -d "$MODELFORGE_POSTGRES_DB" \
|
|
< deploy/postgres/provision-existing-1.2.1.sql
|
|
```
|
|
|
|
Compose exposes the five `MODELFORGE_*` provisioning variables to this bounded `psql` process;
|
|
no value is printed. The script first refuses a protected OID-10 owner before any mutation,
|
|
otherwise creates the retained administrator and preserves the existing `modelforge`
|
|
role OID/table ownership while demoting it to a non-superuser, creates the distinct non-owner
|
|
runtime role, and purges stale memberships that would still permit `SET ROLE`. Passwords are
|
|
read from process environment; none is in the SQL file.
|
|
4. If `protected_bootstrap` is true, do **not** edit `pg_authid`. PostgreSQL intentionally prevents
|
|
demotion of OID 10. Use this recoverable cluster migration instead:
|
|
|
|
1. Keep the verified ModelForge backup and create a second PostgreSQL custom-format logical dump
|
|
with `--no-owner --no-privileges --serializable-deferrable`; prove `pg_restore --list` can
|
|
read it and record its SHA-256.
|
|
2. Stop all application writers. Preserve the old PostgreSQL volume unchanged as the rollback
|
|
source. Bind `postgres-data` to a new, empty, explicitly named volume in a deployment-local
|
|
Compose override.
|
|
3. Start PostgreSQL with `MODELFORGE_POSTGRES_ADMIN_USER` as the cluster bootstrap user. The
|
|
committed init script creates the non-superuser `modelforge` owner and NOINHERIT
|
|
`modelforge_runtime` role.
|
|
4. Restore the logical dump into the empty `modelforge` database while directly authenticated as
|
|
`modelforge`, using `pg_restore --no-owner --no-privileges --exit-on-error
|
|
--single-transaction`. Verify the source schema revision and authoritative row counts before
|
|
migration.
|
|
5. Continue with the one-shot migration below. Retain the old volume until the new deployment,
|
|
state fingerprint and recovery point have passed the operator's rollback-retention window.
|
|
|
|
This is the expected path for older installations originally initialized with
|
|
`POSTGRES_USER=modelforge`; it is not an exceptional permission bypass.
|
|
5. Run only the one-shot migration service. Its environment contains
|
|
`MODELFORGE_MIGRATION_DATABASE_URL` but not the runtime/admin URL:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.yml -f docker-compose.production.yml run --rm migrate
|
|
```
|
|
|
|
Migration 0024 fails before schema mutation unless both roles exist, `modelforge` is the current
|
|
directly authenticated non-superuser owner, `modelforge_runtime` is NOINHERIT/non-admin, and
|
|
neither application role has an explicit role membership.
|
|
The refusal explicitly says to run this v1.2.1-to-schema-0024 provisioning step.
|
|
6. Start the API only after Alembic reaches 0024. Production startup checks live catalog truth:
|
|
runtime role/ownership/admin flags, table/schema/database privileges, canonical function
|
|
owner/SECURITY DEFINER/fixed search path/grants and exact body digest, PUBLIC execution, four
|
|
exact origin-mode protection-trigger shapes, role memberships, and every unexpected executable
|
|
non-system function. Any drift stops startup.
|
|
|
|
The live PostgreSQL privilege/function exercise is a managed-runner release gate. SQLite exercises
|
|
only the defense-in-depth application Engine hooks and is never production proof.
|
|
|
|
Supported direct schema sources for v1.2.3 are `20260827_0021`,
|
|
`20260828_0022`, and `20260830_0023`; the target is `20260830_0024`. Older releases require an
|
|
intermediate upgrade to v1.0.0.
|
|
|
|
## What this upgrade actually changes
|
|
|
|
The migration chain first adds the 0022 terminal-node/decommission state when needed, then 0023
|
|
validates every existing node-enrollment and node-publisher scope before adding exact database
|
|
constraints. Any row outside `node.enroll` or `node.publish` aborts the upgrade before constraint
|
|
DDL; it is never silently rewritten.
|
|
|
|
That has a consequence worth stating plainly: **rolling back across these schema revisions needs
|
|
the verified pre-upgrade database restore.** Older applications do not claim compatibility with
|
|
the newer schema.
|
|
|
|
## 1. Preflight
|
|
|
|
```bash
|
|
python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --plan
|
|
```
|
|
|
|
It changes nothing and reports:
|
|
|
|
| Check | Blocks the upgrade |
|
|
| --- | --- |
|
|
| PostgreSQL major version | yes |
|
|
| Current schema revision is one this release supports | yes |
|
|
| The schema is complete, not half-migrated | yes |
|
|
| A verified backup exists | yes (overridable) |
|
|
| That backup's age | no, advisory |
|
|
| Serving work in flight | no, advisory |
|
|
| Enabled nodes speak a supported agent protocol | yes |
|
|
| A migration cutover sitting in an intermediate stage | no, advisory |
|
|
|
|
A refusal looks like this and is the tool working:
|
|
|
|
```text
|
|
BLOCK current schema 20990101_0099 is TOO_NEW; upgrade the control plane, which is older
|
|
than the component reporting to it
|
|
BLOCK verified backup no verified backup exists; an upgrade without a recovery point cannot be
|
|
undone if the schema turns out to be irreversible
|
|
BLOCK agent protocol enabled nodes speak [2]; this release supports [1]; incompatible: [2]
|
|
|
|
refusing to upgrade: 3 blocker(s)
|
|
```
|
|
|
|
## 2. Take a backup, and verify it
|
|
|
|
The preflight refuses without one. If you have taken a backup by other means, `--allow-without-backup`
|
|
proceeds — that override is explicit and recorded in the report, never a default.
|
|
|
|
See [OPERATIONS.md](OPERATIONS.md#backups-and-recovery).
|
|
|
|
## 3. Stop the application
|
|
|
|
```bash
|
|
docker compose -f docker-compose.yml -f docker-compose.production.yml stop api web
|
|
```
|
|
|
|
Leave PostgreSQL and Redis running. **This upgrade requires downtime**: the control plane is stopped
|
|
while the new version starts. Rolling, zero-downtime upgrade is not supported in v1 and is not
|
|
claimed. Record the measured downtime for your deployment.
|
|
|
|
## 4. Migrate
|
|
|
|
```bash
|
|
python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --apply
|
|
```
|
|
|
|
For v1.2.3 this reports `schema migrated: <source> -> 20260830_0024`. Runtime
|
|
compatibility begins only after the scope validation, audit-chain checkpoint and PostgreSQL role
|
|
boundary reach `0024`.
|
|
|
|
## 5. Start the new version
|
|
|
|
```bash
|
|
MODELFORGE_VERSION=1.2.3 \
|
|
MODELFORGE_API_IMAGE=modelforge-api:1.2.3 \
|
|
MODELFORGE_WEB_IMAGE=modelforge-web:1.2.3 \
|
|
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --no-build api web
|
|
```
|
|
|
|
Pin `MODELFORGE_API_IMAGE` and `MODELFORGE_WEB_IMAGE` to the exact release digests from the release
|
|
manifest when the images are available through a registry. A local source build is an explicit
|
|
alternative for development or rehearsal only: pass `MODELFORGE_COMMIT`, `MODELFORGE_BUILT_AT` and
|
|
`--build` yourself. Never overwrite a release tag with such a local build.
|
|
|
|
## 6. Verify
|
|
|
|
```bash
|
|
curl -s http://127.0.0.1:8000/api/v1/version # expect 1.2.3 and your commit
|
|
curl -s http://127.0.0.1:8000/api/v1/health/ready
|
|
```
|
|
|
|
Then compare the pre/post authoritative-state fingerprint: models, revisions, artifact sets,
|
|
runtime profiles, projects, bindings, deployments, lifecycle and migration state, recovery state,
|
|
policies, SLO state, node identities and credential hashes. Only the expected 0022 structures,
|
|
0023 scope constraints, 0024 audit boundary and legitimate reconciliation/audit activity may differ.
|
|
|
|
Two changes are expected and are not damage: the liveness monitor writes a `NODE_BECAME_OFFLINE`
|
|
event for any node not reporting yet, and the alert engine records the transitions that follow. Both
|
|
are the platform observing its new surroundings correctly.
|
|
|
|
## If the upgrade fails
|
|
|
|
| Symptom | What it means | What to do |
|
|
| --- | --- | --- |
|
|
| Preflight blocks | The upgrade never started; nothing changed | Fix what it names and re-run |
|
|
| The API exits during startup | Configuration was refused before any request was served; the exit code is 3 and the log lists every problem | Fix the settings it names and start again |
|
|
| The API starts but is unhealthy | Something beyond configuration | Roll back, then investigate with the deployment up |
|
|
|
|
### Rolling back across schemas 0022, 0023 and 0024
|
|
|
|
Application rollback alone is not enough. Restore the verified backup taken in step 2, then start
|
|
the previous application version — see
|
|
[RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md). The 0022 → 0021 Alembic
|
|
downgrade is retained and release-tested to prove DDL mechanics on an isolated, tombstone-free
|
|
copy; 0023 → 0022 only removes the two scope constraints. A 0024 downgrade is refused after v2 audit
|
|
events exist because rewriting accepted audit history would destroy the evidence the boundary was
|
|
designed to preserve. These DDL paths are not the production rollback promise;
|
|
the 0022 → 0021 step removes terminal node tombstones and
|
|
decommission-operation evidence. Production rollback therefore restores the verified pre-upgrade
|
|
backup instead of discarding history created after the upgrade.
|
|
|
|
## Agent compatibility
|
|
|
|
The control plane speaks agent protocol version 1 and accepts version 1. An agent reporting anything
|
|
else is answered explicitly — `TOO_OLD`, `TOO_NEW` or `UNKNOWN` — and told what to do, rather than
|
|
being silently accepted. The preflight blocks the upgrade when an enabled node speaks a protocol
|
|
this release does not support, so you find out before the control plane restarts rather than after.
|
|
|
|
Node Agents do not need to be upgraded in lockstep because protocol 1 is unchanged. Pin and deploy
|
|
the matching Node Agent image to align package and OCI provenance. Upgrading only the Node Agent
|
|
does not require re-enrollment; the control plane applies the fixed-scope migration before serving.
|
|
The image deliberately preserves runtime UID 100 and GID 101, so the existing durable agent-state
|
|
volume retains its ownership contract.
|