# Upgrade boundaries — historical v1.2.3 procedure and current RC source > Status reconciled on 2026-09-23: no v1.2.3 Gitea Release object is published. The observed > production API is v1.2.3/schema `20260909_0032`; the current source line targets schema > `20260923_0037` and has not been production-qualified. The detailed commands below > retain the earlier v1.2.3/v1.2.2 schema-0024 and intermediate-RC instructions as historical > guidance; they are not an authorization to deploy current master or to claim migration-0037 > recovery evidence. Use the current-state reconciliation and RC roadmap for the active candidate. ## Source and schema boundary The schema numbers and commands in this section describe an earlier source snapshot. The current `1.3.0-rc.1` source targets `20260923_0037`, while the observed production database remains at `20260909_0032`. Rehearse that exact upgrade and recovery path before publishing the RC. The previously documented v1.2.3 password hotfix was built from the v1.2.2 line and retains schema `20260830_0024`. The RC development branch containing this integration requires `20260908_0028`. A matching version string does not make these builds interchangeable: use the exact commit, image digests and reported schema contract. For this RC source, the upgrade tool accepts schema 0021 through 0028 and targets 0028. In addition to the 0024 role boundary below, it applies migrations 0025 (configuration), 0026 (project onboarding), 0027 (managed operator identities) and 0028 (platform secret slots). Back up and rehearse those migrations in the isolated RC environment before promotion; retain the external secret-slot root key for restore. Do not deploy the published schema-0024 hotfix API against an RC database advanced to 0028. Rollback across that boundary requires restoring the matching verified database backup and application images. The remaining release commands document the previously documented v1.2.3 hotfix. They are not an RC promotion instruction; use the separately maintained RC roadmap for its outstanding gates. v1.2.3 retains schema `20260830_0024` from v1.2.2 and adds operator password sign-in. An existing v1.2.2/schema-0024 installation needs no schema migration or runtime-worker update. Configure the password hash as described in [INSTALLATION.md](INSTALLATION.md), retain the bootstrap key, and deploy the verified API and web images. Earlier schema sources still require the provisioning and migration procedure below. Publication alone does not promote production. ## Required v1.2.1/schema-0022 role provisioning before migration 0024 Schema 0024 changes the production authority boundary. The API may no longer run as the table owner or the bootstrap superuser. Provisioning is deliberately not hidden inside Alembic because an application migration must not mint or retain an administrator credential. 1. Take and verify the pre-upgrade backup, then stop `api`, `web`, and every older writer. This is a downtime migration; the 0024 migration also takes an advisory lock and `ACCESS EXCLUSIVE` on `audit_events`, but the stopped-application contract prevents an old process from resuming with its former owner credential. 2. Generate three distinct URL-safe secrets: retained PostgreSQL administrator, non-superuser `modelforge` migration/schema owner, and NOINHERIT `modelforge_runtime` API role. Set `MODELFORGE_POSTGRES_ADMIN_USER` to a new retained name such as `modelforge_admin`; set the three password variables and both owner/runtime SQLAlchemy URLs documented in `CONFIGURATION.md`. 3. Determine whether the existing `modelforge` role is PostgreSQL's protected bootstrap superuser before changing anything: ```sql select oid, rolsuper, oid = 10 as protected_bootstrap from pg_roles where rolname = 'modelforge'; ``` If `protected_bootstrap` is false, recreate only the PostgreSQL container so the new values are in its environment. It may report unhealthy until the new admin role exists; that is expected during this bounded downtime. While still connecting with the existing v1.2.1 `modelforge` bootstrap superuser, run: ```bash docker compose exec -T postgres psql -v ON_ERROR_STOP=1 \ -U modelforge -d "$MODELFORGE_POSTGRES_DB" \ < deploy/postgres/provision-existing-1.2.1.sql ``` Compose exposes the five `MODELFORGE_*` provisioning variables to this bounded `psql` process; no value is printed. The script first refuses a protected OID-10 owner before any mutation, otherwise creates the retained administrator and preserves the existing `modelforge` role OID/table ownership while demoting it to a non-superuser, creates the distinct non-owner runtime role, and purges stale memberships that would still permit `SET ROLE`. Passwords are read from process environment; none is in the SQL file. 4. If `protected_bootstrap` is true, do **not** edit `pg_authid`. PostgreSQL intentionally prevents demotion of OID 10. Use this recoverable cluster migration instead: 1. Keep the verified ModelForge backup and create a second PostgreSQL custom-format logical dump with `--no-owner --no-privileges --serializable-deferrable`; prove `pg_restore --list` can read it and record its SHA-256. 2. Stop all application writers. Preserve the old PostgreSQL volume unchanged as the rollback source. Bind `postgres-data` to a new, empty, explicitly named volume in a deployment-local Compose override. 3. Start PostgreSQL with `MODELFORGE_POSTGRES_ADMIN_USER` as the cluster bootstrap user. The committed init script creates the non-superuser `modelforge` owner and NOINHERIT `modelforge_runtime` role. 4. Restore the logical dump into the empty `modelforge` database while directly authenticated as `modelforge`, using `pg_restore --no-owner --no-privileges --exit-on-error --single-transaction`. Verify the source schema revision and authoritative row counts before migration. 5. Continue with the one-shot migration below. Retain the old volume until the new deployment, state fingerprint and recovery point have passed the operator's rollback-retention window. This is the expected path for older installations originally initialized with `POSTGRES_USER=modelforge`; it is not an exceptional permission bypass. 5. Run only the one-shot migration service. Its environment contains `MODELFORGE_MIGRATION_DATABASE_URL` but not the runtime/admin URL: ```bash docker compose -f docker-compose.yml -f docker-compose.production.yml run --rm migrate ``` Migration 0024 fails before schema mutation unless both roles exist, `modelforge` is the current directly authenticated non-superuser owner, `modelforge_runtime` is NOINHERIT/non-admin, and neither application role has an explicit role membership. The refusal explicitly says to run this v1.2.1-to-schema-0024 provisioning step. 6. Start the API only after Alembic reaches 0024. Production startup checks live catalog truth: runtime role/ownership/admin flags, table/schema/database privileges, canonical function owner/SECURITY DEFINER/fixed search path/grants and exact body digest, PUBLIC execution, four exact origin-mode protection-trigger shapes, role memberships, and every unexpected executable non-system function. Any drift stops startup. The live PostgreSQL privilege/function exercise is a managed-runner release gate. SQLite exercises only the defense-in-depth application Engine hooks and is never production proof. Supported direct schema sources for v1.2.3 are `20260827_0021`, `20260828_0022`, and `20260830_0023`; the target is `20260830_0024`. Older releases require an intermediate upgrade to v1.0.0. ## What this upgrade actually changes The migration chain first adds the 0022 terminal-node/decommission state when needed, then 0023 validates every existing node-enrollment and node-publisher scope before adding exact database constraints. Any row outside `node.enroll` or `node.publish` aborts the upgrade before constraint DDL; it is never silently rewritten. That has a consequence worth stating plainly: **rolling back across these schema revisions needs the verified pre-upgrade database restore.** Older applications do not claim compatibility with the newer schema. ## 1. Preflight ```bash python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --plan ``` It changes nothing and reports: | Check | Blocks the upgrade | | --- | --- | | PostgreSQL major version | yes | | Current schema revision is one this release supports | yes | | The schema is complete, not half-migrated | yes | | A verified backup exists | yes (overridable) | | That backup's age | no, advisory | | Serving work in flight | no, advisory | | Enabled nodes speak a supported agent protocol | yes | | A migration cutover sitting in an intermediate stage | no, advisory | A refusal looks like this and is the tool working: ```text BLOCK current schema 20990101_0099 is TOO_NEW; upgrade the control plane, which is older than the component reporting to it BLOCK verified backup no verified backup exists; an upgrade without a recovery point cannot be undone if the schema turns out to be irreversible BLOCK agent protocol enabled nodes speak [2]; this release supports [1]; incompatible: [2] refusing to upgrade: 3 blocker(s) ``` ## 2. Take a backup, and verify it The preflight refuses without one. If you have taken a backup by other means, `--allow-without-backup` proceeds — that override is explicit and recorded in the report, never a default. See [OPERATIONS.md](OPERATIONS.md#backups-and-recovery). ## 3. Stop the application ```bash docker compose -f docker-compose.yml -f docker-compose.production.yml stop api web ``` Leave PostgreSQL and Redis running. **This upgrade requires downtime**: the control plane is stopped while the new version starts. Rolling, zero-downtime upgrade is not supported in v1 and is not claimed. Record the measured downtime for your deployment. ## 4. Migrate ```bash python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --apply ``` For v1.2.3 this reports `schema migrated: -> 20260830_0024`. Runtime compatibility begins only after the scope validation, audit-chain checkpoint and PostgreSQL role boundary reach `0024`. ## 5. Start the new version ```bash MODELFORGE_VERSION=1.2.3 \ MODELFORGE_API_IMAGE=modelforge-api:1.2.3 \ MODELFORGE_WEB_IMAGE=modelforge-web:1.2.3 \ docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --no-build api web ``` Pin `MODELFORGE_API_IMAGE` and `MODELFORGE_WEB_IMAGE` to the exact release digests from the release manifest when the images are available through a registry. A local source build is an explicit alternative for development or rehearsal only: pass `MODELFORGE_COMMIT`, `MODELFORGE_BUILT_AT` and `--build` yourself. Never overwrite a release tag with such a local build. ## 6. Verify ```bash curl -s http://127.0.0.1:8000/api/v1/version # expect 1.2.3 and your commit curl -s http://127.0.0.1:8000/api/v1/health/ready ``` Then compare the pre/post authoritative-state fingerprint: models, revisions, artifact sets, runtime profiles, projects, bindings, deployments, lifecycle and migration state, recovery state, policies, SLO state, node identities and credential hashes. Only the expected 0022 structures, 0023 scope constraints, 0024 audit boundary and legitimate reconciliation/audit activity may differ. Two changes are expected and are not damage: the liveness monitor writes a `NODE_BECAME_OFFLINE` event for any node not reporting yet, and the alert engine records the transitions that follow. Both are the platform observing its new surroundings correctly. ## If the upgrade fails | Symptom | What it means | What to do | | --- | --- | --- | | Preflight blocks | The upgrade never started; nothing changed | Fix what it names and re-run | | The API exits during startup | Configuration was refused before any request was served; the exit code is 3 and the log lists every problem | Fix the settings it names and start again | | The API starts but is unhealthy | Something beyond configuration | Roll back, then investigate with the deployment up | ### Rolling back across schemas 0022, 0023 and 0024 Application rollback alone is not enough. Restore the verified backup taken in step 2, then start the previous application version — see [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md). The 0022 → 0021 Alembic downgrade is retained and release-tested to prove DDL mechanics on an isolated, tombstone-free copy; 0023 → 0022 only removes the two scope constraints. A 0024 downgrade is refused after v2 audit events exist because rewriting accepted audit history would destroy the evidence the boundary was designed to preserve. These DDL paths are not the production rollback promise; the 0022 → 0021 step removes terminal node tombstones and decommission-operation evidence. Production rollback therefore restores the verified pre-upgrade backup instead of discarding history created after the upgrade. ## Agent compatibility The control plane speaks agent protocol version 1 and accepts version 1. An agent reporting anything else is answered explicitly — `TOO_OLD`, `TOO_NEW` or `UNKNOWN` — and told what to do, rather than being silently accepted. The preflight blocks the upgrade when an enabled node speaks a protocol this release does not support, so you find out before the control plane restarts rather than after. Node Agents do not need to be upgraded in lockstep because protocol 1 is unchanged. Pin and deploy the matching Node Agent image to align package and OCI provenance. Upgrading only the Node Agent does not require re-enrollment; the control plane applies the fixed-scope migration before serving. The image deliberately preserves runtime UID 100 and GID 101, so the existing durable agent-state volume retains its ownership contract.