# Real Backend Architecture Decision Status: Proposed decision Date: 2026-06-06 Scope: Move ProjectBrain from Vite mock middleware to a real backend architecture without implementing that backend yet. ## Decision ProjectBrain should move to a dedicated TypeScript backend using Fastify, PostgreSQL, Prisma, Zod, and a background worker backed by Redis/BullMQ. The current Vite middleware should remain as a dev/demo contract surface until the frontend is switched behind a configuration flag. This keeps the product API-first, self-hostable, Docker-friendly, and aligned with the existing TypeScript contract layer while leaving room for later AI-assisted imports, codebase analysis jobs, file uploads, MCP integration, and ChatGPT/custom app connectors. ## Current Backend-Mock State The current backend surface is implemented as Vite dev middleware in `src/shared/api/b1ReadBackend.ts` and mounted from `vite.config.ts` under `/api/*`. B1-B6 endpoint families now covered as mock contract surface: - B1: Projects, Workspace, DNA, context-pack summary. - B2: Feature groups, Features, Evidence reads. - B3: Context Packs and prompt generation mock. - B4: Checks and review-board mutations. - B5: Builds, Regressions, Snapshots. - B6: Insights, Relationships, Knowledge, Memory Rules. Current mock limitations: - In-memory state only; data resets when the Vite process restarts. - No database, migrations, transactions, row-level integrity, or durable idempotency store. - No auth, permissions, tenancy, or user ownership model. - No real AI calls; prompt/current-state style operations are deterministic mock outputs. - No file upload storage, parsing pipeline, codebase scan worker, or long-running job runner. - Vite middleware is development infrastructure, not a production backend boundary. - Idempotency is process-local and keyed by method, path, and `Idempotency-Key`. Where state lives today: - `src/shared/lib/mock-data.ts`: central fixtures used by the UI and mock clients. - `src/shared/api/mockClient.ts`: frontend in-memory store used by `projectBrainClient`. - `src/shared/api/b1ReadBackend.ts`: separate Vite middleware in-memory state seeded from mock data. Contracts and docs currently present: - `docs/API_CONTRACT_PREP.md` - `docs/BACKEND_B1_READ_API.md` - `docs/BACKEND_B2_FEATURES_EVIDENCE_READ_API.md` - `docs/BACKEND_B3_CONTEXT_PACKS_PROMPTS_API.md` - `docs/BACKEND_B4_CHECKS_REVIEW_API.md` - `docs/BACKEND_B5_BUILDS_REGRESSIONS_SNAPSHOTS_API.md` - `docs/BACKEND_B6_INSIGHTS_RELATIONSHIPS_KNOWLEDGE_MEMORY_API.md` - `src/shared/api/types.ts` - `src/shared/api/contracts.ts` - `src/shared/api/projectBrainClient.ts` Frontend consumption today: - UI pages should consume through `projectBrainClient` or higher-level selectors/actions. - `projectBrainClient` currently delegates to `mockClient`. - The Vite middleware is not yet the frontend source of truth. - Future switching should happen inside the client boundary, not by changing pages directly. ## Backend Options ### A) Node/Fastify Backend Pros: - Best fit with the existing TypeScript contracts and model types. - Shared Zod schemas can validate both API requests and frontend clients. - Fastify has strong performance, structured plugins, JSON schema support, and good Docker ergonomics. - Natural fit for API-first design, MCP servers, ChatGPT app connectors, and local self-hosting. - Background jobs can use BullMQ/Redis with a separate worker process. - File upload/import endpoints can stream to local disk or S3-compatible storage. Cons: - Codebase analysis and ML-heavy pipelines may eventually want Python helpers. - Requires discipline around service/repository boundaries to avoid a large Node monolith. - Worker deployment adds another runtime process. Migration effort: - Moderate. Existing TypeScript DTOs, contract docs, and mock behavior can be ported incrementally. Fit: - AI/codebase analysis later: good for orchestration; Python sidecars can be added for specialist analysis. - File uploads/imports later: good. - Self-hosting: excellent. - Docker deployment: excellent. - Future MCP/ChatGPT connector/API: excellent. ### B) FastAPI Backend Pros: - Excellent for Python-native AI/code analysis, embeddings, parsing, and background processing. - Strong OpenAPI generation and Pydantic validation. - Good for future ML-heavy worker code. Cons: - Introduces a second primary language while the app contracts are already TypeScript. - DTO drift risk between frontend TypeScript and backend Python models. - More migration work to preserve existing TypeScript contract shape. - Frontend and backend type sharing becomes generated rather than native. Migration effort: - Moderate to high. Requires translating contracts into Pydantic or generating schemas carefully. Fit: - AI/codebase analysis later: excellent. - File uploads/imports later: excellent. - Self-hosting: good. - Docker deployment: good. - Future MCP/ChatGPT connector/API: good, but TypeScript app connector work may be less direct. ### C) Next.js-Style Fullstack Pros: - Single deployment surface if the product were already Next.js. - API routes and UI can live together. Cons: - Not applicable without migrating the existing Vite SPA. - Would create a framework migration before the backend architecture is settled. - Long-running imports and codebase analysis jobs still need separate worker infrastructure. Migration effort: - High and not justified now. Fit: - AI/codebase analysis later: mixed. - File uploads/imports later: good with extra worker setup. - Self-hosting: good. - Docker deployment: good. - Future MCP/ChatGPT connector/API: good, but not worth the app migration cost. ### D) Keep Vite Middleware Only Pros: - Lowest immediate effort. - Good demo and contract-testing surface. - Useful during frontend prototyping. Cons: - Not a production backend. - No durable persistence, job runner, upload pipeline, auth boundary, or deployment separation. - Dev server lifecycle makes idempotency and state unreliable. - Encourages backend growth inside frontend tooling. Migration effort: - None short-term, but high long-term cleanup cost if it keeps expanding. Fit: - AI/codebase analysis later: poor. - File uploads/imports later: poor. - Self-hosting: poor for production. - Docker deployment: demo only. - Future MCP/ChatGPT connector/API: poor as a durable API target. ## Recommended Architecture Use a dedicated Fastify backend in TypeScript. Recommended stack: - Runtime/framework: Node.js with Fastify. - Database: PostgreSQL. - ORM/query layer: Prisma initially, with raw SQL escape hatches for graph/query-heavy paths. - Validation/schema: Zod as source-level request/response validation, optionally exported to OpenAPI. - Background jobs: BullMQ with Redis, split into `api` and `worker` processes. - File storage: local filesystem volume for self-hosting first, with S3-compatible adapter later. - API structure: versioned REST under `/api/v1`, preserving current envelopes. - Logging: Pino structured logs with request IDs. - Error handling: centralized Fastify error handler returning the existing error envelope. - Config/env: `.env` for local, `.env.example`, environment-specific Docker Compose values. - Docker: `frontend`, `api`, `worker`, `postgres`, `redis`, and optional `storage` volume. Why this choice: - Keeps TypeScript contracts close to the frontend. - Avoids a framework migration. - Supports self-hosting and Docker from the beginning. - Leaves room for Python analysis workers later without making Python the main API boundary. - Fits future MCP/ChatGPT/custom app API use because the backend becomes a stable HTTP service. ## Proposed Folder Structure ```text apps/ web/ # current Vite React app, moved later or referenced in place api/ src/ server.ts app.ts config/ plugins/ routes/ v1/ projects.routes.ts workspace.routes.ts features.routes.ts context-packs.routes.ts checks.routes.ts builds.routes.ts knowledge.routes.ts services/ repositories/ schemas/ errors/ idempotency/ logging/ worker/ src/ worker.ts jobs/ processors/ packages/ contracts/ src/ api-types.ts schemas.ts envelopes.ts ``` This repo does not need to be reorganized immediately. R1 can start with a smaller `backend/` folder if a monorepo move is too disruptive. ## API Versioning - New backend routes should use `/api/v1`. - Current Vite middleware may remain under `/api` for dev/demo until switch-over. - Response envelope remains: ```json { "data": {}, "meta": { "requestId": "...", "timestamp": "..." } } ``` - Error envelope remains: ```json { "error": { "code": "...", "message": "...", "validation": [] }, "meta": { "requestId": "...", "timestamp": "..." } } ``` - Idempotency remains required for write endpoints using `Idempotency-Key`, but should move to a durable table: - method - path - key - request hash - response body - status code - expiresAt ## Migration Strategy ### R1: Real backend skeleton - Add Fastify API skeleton. - Add health endpoint. - Add shared envelope helpers. - Add request ID middleware. - Add config/env validation. - Add Docker Compose with Postgres and Redis. - Do not switch frontend yet. ### R2: Projects / Workspace / DNA reads - Implement B1 against database-backed repositories. - Seed database from `mock-data.ts`. - Keep Vite middleware as fallback/demo. ### R3: Features / Evidence reads - Implement B2. - Add feature/evidence schema migrations. - Preserve pagination and filter response shape. ### R4: Context Packs / Prompts mock generation - Implement context-pack reads/writes. - Keep prompt generation deterministic mock-only. - Store generated prompt records in the database. ### R5: Checks / Review mutations - Implement checks, review board grouping, review actions, conversions. - Add durable idempotency. ### R6: Builds / Regressions / Snapshots - Implement build journal, regression memory, snapshot compare. - Keep compare deterministic until richer diffing exists. ### R7: Knowledge / Memory / Relationships / Insights - Implement B6 endpoints against the database. - Preserve include-in-context-pack behavior. - Add relationship integrity checks. ### R8: Frontend API switch - Update `projectBrainClient` to choose real HTTP client or `mockClient` via config flag. - Pages must continue to use `projectBrainClient`, selectors, and actions only. - No page-level direct fetch calls. ### R9: Persistence hardening - Add migrations, seed scripts, backup/restore guidance, indexes, idempotency TTL cleanup, and integration tests. ### R10: Import/codebase analysis jobs - Add upload endpoints and job creation. - Add worker processors for ZIP/codebase/document analysis. - Add optional Python sidecar only for analysis tasks that genuinely need Python libraries. ## Source of Truth Transition - `mock-data.ts` remains a fixture and demo seed source. - The real PostgreSQL database becomes source of truth after R8/R9. - Seed scripts should transform fixture data into database rows. - Migration scripts should be repeatable for local self-hosting. - The frontend must keep using `projectBrainClient`; switching source should be a client configuration concern. - Vite middleware should be retired only after the real backend covers B1-B6 and the frontend switch is stable. ## Docker / Self-Hosting Notes Recommended services: - `web`: Vite-built static frontend served by Nginx or a small Node static server. - `api`: Fastify HTTP API. - `worker`: BullMQ worker for imports and analysis. - `postgres`: ProjectBrain database. - `redis`: queue and job coordination. - `storage`: mounted local volume for uploads and generated artifacts. Self-hosting defaults: - No paid API required for core CRUD/project memory. - AI providers configured through optional environment variables later. - Local file storage first, S3-compatible storage optional. - Health checks for `web`, `api`, `worker`, `postgres`, and `redis`. ## Alternatives Rejected - FastAPI as primary API: strong for analysis, but higher contract drift risk and more translation work from existing TypeScript contracts. - Next.js fullstack: would require a frontend framework migration before backend needs justify it. - Vite middleware as production backend: useful contract scaffold, but not durable or deployable as the product backend. ## Open Questions - Should the repo become a formal monorepo now, or should R1 start with a `backend/` folder? - Should Prisma be the long-term ORM, or should graph-heavy relationship queries move to Kysely/raw SQL later? - Should file storage stay local-only for v1 self-hosting, or ship an S3-compatible adapter in the first backend release? - What auth model is desired later: local admin, single-user self-hosted, or multi-user teams? - Should Python analysis workers be planned from R10 or deferred until codebase import requirements are precise? ## Final Recommendation Proceed with R1 as a dedicated Fastify backend skeleton, keeping the Vite middleware untouched as a contract/demo reference. Do not add B7 to the middleware. The next implementation phase should create the real backend foundation and implement only health/config/envelope plumbing before porting B1.