Initial curated itworx-knowledge-hub source from 3b1751d0d9dee551aa6994eee0093704dc334817
This commit is contained in:
Public Source Publisher committed
commit
ad8ad252a9
825 files changed
+124216
No files matched your search
@@ -0,0 +1,281 @@
|
||||
# LeafWiki migration
|
||||
|
||||
## Objective
|
||||
|
||||
Migrate the owner's existing LeafWiki knowledge into ItWorx Knowledge Hub without
|
||||
modifying the source and without silently losing or overwriting content.
|
||||
|
||||
LeafWiki versions and deployments may differ. The importer must be discovery-driven and
|
||||
adapter-versioned rather than assuming one undocumented fixed schema.
|
||||
|
||||
## Safety contract
|
||||
|
||||
- Source mount is read-only at the operating-system/container level.
|
||||
- SQLite files are opened with read-only/immutable semantics.
|
||||
- No source file is renamed, normalized, repaired or touched.
|
||||
- The importer never writes cache, lock, journal or temporary files beside the source.
|
||||
- A complete source fingerprint is captured before and after.
|
||||
- Import writes only to an isolated target workspace/database.
|
||||
- Every target record is attributable to an import ID and source item.
|
||||
- Re-running an accepted plan is idempotent.
|
||||
- Rollback removes only that import's creations/changes according to the recorded plan.
|
||||
- Conflicts are explicit; no silent overwrite.
|
||||
|
||||
## Supported source discovery
|
||||
|
||||
Inventory recursively:
|
||||
|
||||
- Markdown files;
|
||||
- SQLite databases, detected by header rather than extension only;
|
||||
- images and other assets;
|
||||
- JSON/YAML/config manifests;
|
||||
- directory hierarchy;
|
||||
- symlinks, which are recorded but not followed outside the source root;
|
||||
- unknown files.
|
||||
|
||||
For SQLite, inspect read-only:
|
||||
|
||||
- SQLite version and pragmas safe to read;
|
||||
- tables/views/indexes;
|
||||
- columns and foreign keys;
|
||||
- row counts;
|
||||
- candidate content, tree, tag, revision and attachment tables;
|
||||
- application version markers where present.
|
||||
|
||||
The bundled `tools/leafwiki-inspector` provides a neutral first inventory. It is not the
|
||||
final importer.
|
||||
|
||||
## Adapter architecture
|
||||
|
||||
```text
|
||||
IImportSource
|
||||
Discover()
|
||||
Fingerprint()
|
||||
SelectAdapter()
|
||||
|
||||
ILeafWikiAdapter
|
||||
Analyze()
|
||||
BuildPlan()
|
||||
ReadItem()
|
||||
|
||||
LeafWikiFilesystemMarkdownAdapter
|
||||
LeafWikiSqliteAdapter_<known-version>
|
||||
LeafWikiHybridAdapter_<known-version>
|
||||
```
|
||||
|
||||
A known adapter must validate its expected signature before reading semantic columns. An
|
||||
unknown signature produces a preflight blocker, not guessed mappings.
|
||||
|
||||
A filesystem-first adapter can import Markdown hierarchy even when DB enrichment is
|
||||
unknown, but the preflight must clearly state which tags/revisions/metadata would be
|
||||
omitted and require explicit acceptance.
|
||||
|
||||
### Implemented adapters
|
||||
|
||||
| Adapter name | Applies when | Recovers |
|
||||
| --- | --- | --- |
|
||||
| `synthetic-sqlite` | a SQLite file with the recognized synthetic schema | pages and hierarchy from the database |
|
||||
| `leafwiki-filesystem-v5` | `schema.json` declares version 5 (or, absent that file, Markdown carries `leafwiki_id` frontmatter) | titles from `leafwiki_title` (YAML-escape decoded), directory hierarchy, the author's sibling order from each directory's `.order.json`, and assets |
|
||||
| `filesystem-markdown` | any source containing Markdown | hierarchy from directory structure only; titles from `title:` frontmatter, first heading, or filename |
|
||||
|
||||
The catalog tries them in that order and takes the first that reports it applies, so a
|
||||
version-validated adapter is always preferred over the signature-free fallback, and an
|
||||
unrecognized LeafWiki version falls back rather than being guessed at
|
||||
(`docs/adr/ADR-032-leafwiki-v5-import-fidelity.md`).
|
||||
|
||||
### Target-side transformations
|
||||
|
||||
An import writes Markdown that must work in *this* application, so two changes are made
|
||||
to the imported copy — never to the source:
|
||||
|
||||
- an asset link whose bytes were genuinely imported (`](/assets/<page-id>/<file>)`)
|
||||
becomes `](/api/v1/attachments/<reference-id>/content)`; an asset that was rejected or
|
||||
unreadable keeps its original link and is reported as a diagnostic;
|
||||
- a leading level-1 heading is dropped when it exactly repeats the page title, because
|
||||
this application renders the title above the body and would otherwise show it twice.
|
||||
A heading that differs from the title is content and is kept.
|
||||
|
||||
## Source fingerprint
|
||||
|
||||
A deterministic manifest includes:
|
||||
|
||||
- normalized relative path;
|
||||
- item type;
|
||||
- byte length;
|
||||
- SHA-256;
|
||||
- symlink target descriptor without traversal;
|
||||
- for SQLite: schema fingerprint and selected row-count summary.
|
||||
|
||||
Sort entries ordinally and hash the manifest. Do not include source absolute paths in
|
||||
user-facing logs unless administrator-only and redacted.
|
||||
|
||||
## Preflight plan
|
||||
|
||||
Preflight produces:
|
||||
|
||||
- adapter and version;
|
||||
- source fingerprint;
|
||||
- discovered counts;
|
||||
- proposed target hierarchy;
|
||||
- title/slug mapping;
|
||||
- Markdown encoding/line-ending diagnostics;
|
||||
- link-resolution plan;
|
||||
- attachment mapping and hashes;
|
||||
- tags/revision metadata availability;
|
||||
- duplicate/conflict classifications;
|
||||
- unsupported/ambiguous items;
|
||||
- estimated target changes;
|
||||
- deterministic plan hash.
|
||||
|
||||
No target mutation occurs during preflight.
|
||||
|
||||
## Content mapping
|
||||
|
||||
### Pages and hierarchy
|
||||
|
||||
Preference order:
|
||||
|
||||
1. authoritative LeafWiki tree metadata from a validated adapter;
|
||||
2. Markdown directory hierarchy;
|
||||
3. explicit operator mapping.
|
||||
|
||||
Title sources:
|
||||
|
||||
1. validated metadata;
|
||||
2. frontmatter title;
|
||||
3. first H1;
|
||||
4. filename.
|
||||
|
||||
Keep original relative path and legacy ID as provenance.
|
||||
|
||||
### Markdown
|
||||
|
||||
- Preserve original Markdown bytes as an import artifact/hash.
|
||||
- Decode according to detected/declared encoding; report replacements.
|
||||
- Convert only syntax with a deterministic documented rule.
|
||||
- Preserve unknown constructs as fenced/raw text rather than discard them.
|
||||
- Convert internal links after all page identities are mapped.
|
||||
- Record unresolved/ambiguous links.
|
||||
|
||||
### Tags
|
||||
|
||||
Import normalized tags when the adapter can prove their relation. Preserve original
|
||||
display value.
|
||||
|
||||
### Revisions
|
||||
|
||||
When reliable legacy revisions exist:
|
||||
|
||||
- import them in chronological order;
|
||||
- preserve source timestamp/author label as provenance;
|
||||
- mark source attribution;
|
||||
- never assign legacy author labels to current user identities without explicit mapping.
|
||||
|
||||
If revisions cannot be safely reconstructed, import current content only and report it.
|
||||
|
||||
### Attachments
|
||||
|
||||
- Hash and validate each asset.
|
||||
- Map references after page identities exist.
|
||||
- Reject path traversal.
|
||||
- Preserve safe original filename and source path provenance.
|
||||
- Deduplicate through normal attachment storage.
|
||||
- Report missing references and unreferenced assets.
|
||||
|
||||
## Conflict policy
|
||||
|
||||
Classifications:
|
||||
|
||||
- no target conflict;
|
||||
- exact duplicate by source provenance/hash;
|
||||
- same title/path, different content;
|
||||
- same attachment hash;
|
||||
- ambiguous link;
|
||||
- unsupported source item.
|
||||
|
||||
Default actions:
|
||||
|
||||
- exact rerun duplicate: skip idempotently;
|
||||
- same hash attachment: reuse blob, create intended logical reference;
|
||||
- title/path conflict: create under an explicit import namespace or require plan choice;
|
||||
- existing user page: never overwrite;
|
||||
- ambiguous link: leave unresolved and report.
|
||||
|
||||
## Execution
|
||||
|
||||
- Persist the accepted immutable plan.
|
||||
- Use bounded transactions/checkpoints to avoid one giant lock.
|
||||
- Ensure each item is idempotent.
|
||||
- Create provenance with the target row.
|
||||
- Defer cross-links until page mapping exists.
|
||||
- Allow cancellation between batches.
|
||||
- Resume only when source and plan fingerprints still match.
|
||||
- On failure, keep a diagnosable job state; do not mark complete.
|
||||
|
||||
## Validation
|
||||
|
||||
At minimum compare:
|
||||
|
||||
- discovered vs planned vs imported/skipped/failed counts;
|
||||
- hierarchy/root/subtree mapping;
|
||||
- current page content hashes after deterministic conversion;
|
||||
- link resolved/unresolved counts;
|
||||
- tags;
|
||||
- attachment bytes/hashes and references;
|
||||
- revision counts when imported;
|
||||
- target searchability;
|
||||
- representative rendered pages;
|
||||
- source fingerprint before/after.
|
||||
|
||||
A validation mismatch blocks completion.
|
||||
|
||||
## Rollback
|
||||
|
||||
Rollback uses import provenance and the accepted plan:
|
||||
|
||||
- remove imported relations;
|
||||
- remove imported current entities when they were created by the job;
|
||||
- preserve any subsequent user-edited target by moving it to a review/conflict state
|
||||
rather than deleting it;
|
||||
- release attachment references and clean blobs only through normal retention;
|
||||
- append audit records;
|
||||
- never touch pre-existing target content;
|
||||
- never touch source.
|
||||
|
||||
Test full rollback and partial-failure rollback.
|
||||
|
||||
## Real migration procedure
|
||||
|
||||
1. Update and back up the existing LeafWiki.
|
||||
2. Create an immutable/read-only snapshot.
|
||||
3. Keep LeafWiki running normally during dry runs.
|
||||
4. Run inspector and preflight.
|
||||
5. Resolve all blockers.
|
||||
6. Import into a non-production target.
|
||||
7. Validate representative school pages, links, files and search with the user.
|
||||
8. Rehearse rollback.
|
||||
9. Schedule cutover.
|
||||
10. Place LeafWiki read-only.
|
||||
11. Capture a final snapshot/fingerprint.
|
||||
12. Run final import or delta strategy supported by the adapter.
|
||||
13. Validate and take a Knowledge Hub backup.
|
||||
14. Keep LeafWiki read-only through the rollback window.
|
||||
15. Retire only after explicit user sign-off.
|
||||
|
||||
## Synthetic fixtures
|
||||
|
||||
Tests must cover:
|
||||
|
||||
- filesystem-only Markdown tree;
|
||||
- duplicate titles;
|
||||
- nested pages;
|
||||
- UTF-8 and special characters;
|
||||
- wiki links and unresolved links;
|
||||
- attachments and missing attachments;
|
||||
- SQLite database with known synthetic schema;
|
||||
- unknown SQLite schema;
|
||||
- symlink escape attempt;
|
||||
- corrupted database/file;
|
||||
- rerun idempotency;
|
||||
- rollback after user edit;
|
||||
- source hash immutability.
|
||||
Reference in new issue
Block a user