itworx-knowledge-hub-Public/docs/17-leafwiki-migration.md

# LeafWiki migration

## Objective

Migrate the owner's existing LeafWiki knowledge into ItWorx Knowledge Hub without
modifying the source and without silently losing or overwriting content.

LeafWiki versions and deployments may differ. The importer must be discovery-driven and
adapter-versioned rather than assuming one undocumented fixed schema.

## Safety contract

- Source mount is read-only at the operating-system/container level.
- SQLite files are opened with read-only/immutable semantics.
- No source file is renamed, normalized, repaired or touched.
- The importer never writes cache, lock, journal or temporary files beside the source.
- A complete source fingerprint is captured before and after.
- Import writes only to an isolated target workspace/database.
- Every target record is attributable to an import ID and source item.
- Re-running an accepted plan is idempotent.
- Rollback removes only that import's creations/changes according to the recorded plan.
- Conflicts are explicit; no silent overwrite.

## Supported source discovery

Inventory recursively:

- Markdown files;
- SQLite databases, detected by header rather than extension only;
- images and other assets;
- JSON/YAML/config manifests;
- directory hierarchy;
- symlinks, which are recorded but not followed outside the source root;
- unknown files.

For SQLite, inspect read-only:

- SQLite version and pragmas safe to read;
- tables/views/indexes;
- columns and foreign keys;
- row counts;
- candidate content, tree, tag, revision and attachment tables;
- application version markers where present.

The bundled `tools/leafwiki-inspector` provides a neutral first inventory. It is not the
final importer.

## Adapter architecture

```text
IImportSource
  Discover()
  Fingerprint()
  SelectAdapter()

ILeafWikiAdapter
  Analyze()
  BuildPlan()
  ReadItem()

LeafWikiFilesystemMarkdownAdapter
LeafWikiSqliteAdapter_<known-version>
LeafWikiHybridAdapter_<known-version>
```

A known adapter must validate its expected signature before reading semantic columns. An
unknown signature produces a preflight blocker, not guessed mappings.

A filesystem-first adapter can import Markdown hierarchy even when DB enrichment is
unknown, but the preflight must clearly state which tags/revisions/metadata would be
omitted and require explicit acceptance.

### Implemented adapters

| Adapter name | Applies when | Recovers |
| --- | --- | --- |
| `synthetic-sqlite` | a SQLite file with the recognized synthetic schema | pages and hierarchy from the database |
| `leafwiki-filesystem-v5` | `schema.json` declares version 5 (or, absent that file, Markdown carries `leafwiki_id` frontmatter) | titles from `leafwiki_title` (YAML-escape decoded), directory hierarchy, the author's sibling order from each directory's `.order.json`, and assets |
| `filesystem-markdown` | any source containing Markdown | hierarchy from directory structure only; titles from `title:` frontmatter, first heading, or filename |

The catalog tries them in that order and takes the first that reports it applies, so a
version-validated adapter is always preferred over the signature-free fallback, and an
unrecognized LeafWiki version falls back rather than being guessed at
(`docs/adr/ADR-032-leafwiki-v5-import-fidelity.md`).

### Target-side transformations

An import writes Markdown that must work in *this* application, so two changes are made
to the imported copy — never to the source:

- an asset link whose bytes were genuinely imported (`](/assets/<page-id>/<file>)`)
  becomes `](/api/v1/attachments/<reference-id>/content)`; an asset that was rejected or
  unreadable keeps its original link and is reported as a diagnostic;
- a leading level-1 heading is dropped when it exactly repeats the page title, because
  this application renders the title above the body and would otherwise show it twice.
  A heading that differs from the title is content and is kept.

## Source fingerprint

A deterministic manifest includes:

- normalized relative path;
- item type;
- byte length;
- SHA-256;
- symlink target descriptor without traversal;
- for SQLite: schema fingerprint and selected row-count summary.

Sort entries ordinally and hash the manifest. Do not include source absolute paths in
user-facing logs unless administrator-only and redacted.

## Preflight plan

Preflight produces:

- adapter and version;
- source fingerprint;
- discovered counts;
- proposed target hierarchy;
- title/slug mapping;
- Markdown encoding/line-ending diagnostics;
- link-resolution plan;
- attachment mapping and hashes;
- tags/revision metadata availability;
- duplicate/conflict classifications;
- unsupported/ambiguous items;
- estimated target changes;
- deterministic plan hash.

No target mutation occurs during preflight.

## Content mapping

### Pages and hierarchy

Preference order:

1. authoritative LeafWiki tree metadata from a validated adapter;
2. Markdown directory hierarchy;
3. explicit operator mapping.

Title sources:

1. validated metadata;
2. frontmatter title;
3. first H1;
4. filename.

Keep original relative path and legacy ID as provenance.

### Markdown

- Preserve original Markdown bytes as an import artifact/hash.
- Decode according to detected/declared encoding; report replacements.
- Convert only syntax with a deterministic documented rule.
- Preserve unknown constructs as fenced/raw text rather than discard them.
- Convert internal links after all page identities are mapped.
- Record unresolved/ambiguous links.

### Tags

Import normalized tags when the adapter can prove their relation. Preserve original
display value.

### Revisions

When reliable legacy revisions exist:

- import them in chronological order;
- preserve source timestamp/author label as provenance;
- mark source attribution;
- never assign legacy author labels to current user identities without explicit mapping.

If revisions cannot be safely reconstructed, import current content only and report it.

### Attachments

- Hash and validate each asset.
- Map references after page identities exist.
- Reject path traversal.
- Preserve safe original filename and source path provenance.
- Deduplicate through normal attachment storage.
- Report missing references and unreferenced assets.

## Conflict policy

Classifications:

- no target conflict;
- exact duplicate by source provenance/hash;
- same title/path, different content;
- same attachment hash;
- ambiguous link;
- unsupported source item.

Default actions:

- exact rerun duplicate: skip idempotently;
- same hash attachment: reuse blob, create intended logical reference;
- title/path conflict: create under an explicit import namespace or require plan choice;
- existing user page: never overwrite;
- ambiguous link: leave unresolved and report.

## Execution

- Persist the accepted immutable plan.
- Use bounded transactions/checkpoints to avoid one giant lock.
- Ensure each item is idempotent.
- Create provenance with the target row.
- Defer cross-links until page mapping exists.
- Allow cancellation between batches.
- Resume only when source and plan fingerprints still match.
- On failure, keep a diagnosable job state; do not mark complete.

## Validation

At minimum compare:

- discovered vs planned vs imported/skipped/failed counts;
- hierarchy/root/subtree mapping;
- current page content hashes after deterministic conversion;
- link resolved/unresolved counts;
- tags;
- attachment bytes/hashes and references;
- revision counts when imported;
- target searchability;
- representative rendered pages;
- source fingerprint before/after.

A validation mismatch blocks completion.

## Rollback

Rollback uses import provenance and the accepted plan:

- remove imported relations;
- remove imported current entities when they were created by the job;
- preserve any subsequent user-edited target by moving it to a review/conflict state
  rather than deleting it;
- release attachment references and clean blobs only through normal retention;
- append audit records;
- never touch pre-existing target content;
- never touch source.

Test full rollback and partial-failure rollback.

## Real migration procedure

1. Update and back up the existing LeafWiki.
2. Create an immutable/read-only snapshot.
3. Keep LeafWiki running normally during dry runs.
4. Run inspector and preflight.
5. Resolve all blockers.
6. Import into a non-production target.
7. Validate representative school pages, links, files and search with the user.
8. Rehearse rollback.
9. Schedule cutover.
10. Place LeafWiki read-only.
11. Capture a final snapshot/fingerprint.
12. Run final import or delta strategy supported by the adapter.
13. Validate and take a Knowledge Hub backup.
14. Keep LeafWiki read-only through the rollback window.
15. Retire only after explicit user sign-off.

## Synthetic fixtures

Tests must cover:

- filesystem-only Markdown tree;
- duplicate titles;
- nested pages;
- UTF-8 and special characters;
- wiki links and unresolved links;
- attachments and missing attachments;
- SQLite database with known synthetic schema;
- unknown SQLite schema;
- symlink escape attempt;
- corrupted database/file;
- rerun idempotency;
- rollback after user edit;
- source hash immutability.