projectbrain-Public/docs/REAL_BACKEND_ARCHITECTURE_DECISION.md

Real Backend Architecture Decision

Status: Proposed decision
Date: 2026-06-06
Scope: Move ProjectBrain from Vite mock middleware to a real backend architecture without implementing that backend yet.

Decision

ProjectBrain should move to a dedicated TypeScript backend using Fastify, PostgreSQL, Prisma, Zod, and a background worker backed by Redis/BullMQ. The current Vite middleware should remain as a dev/demo contract surface until the frontend is switched behind a configuration flag.

This keeps the product API-first, self-hostable, Docker-friendly, and aligned with the existing TypeScript contract layer while leaving room for later AI-assisted imports, codebase analysis jobs, file uploads, MCP integration, and ChatGPT/custom app connectors.

Current Backend-Mock State

The current backend surface is implemented as Vite dev middleware in src/shared/api/b1ReadBackend.ts and mounted from vite.config.ts under /api/*.

B1-B6 endpoint families now covered as mock contract surface:

• B1: Projects, Workspace, DNA, context-pack summary.
• B2: Feature groups, Features, Evidence reads.
• B3: Context Packs and prompt generation mock.
• B4: Checks and review-board mutations.
• B5: Builds, Regressions, Snapshots.
• B6: Insights, Relationships, Knowledge, Memory Rules.

Current mock limitations:

• In-memory state only; data resets when the Vite process restarts.
• No database, migrations, transactions, row-level integrity, or durable idempotency store.
• No auth, permissions, tenancy, or user ownership model.
• No real AI calls; prompt/current-state style operations are deterministic mock outputs.
• No file upload storage, parsing pipeline, codebase scan worker, or long-running job runner.
• Vite middleware is development infrastructure, not a production backend boundary.
• Idempotency is process-local and keyed by method, path, and Idempotency-Key.

Where state lives today:

• src/shared/lib/mock-data.ts: central fixtures used by the UI and mock clients.
• src/shared/api/mockClient.ts: frontend in-memory store used by projectBrainClient.
• src/shared/api/b1ReadBackend.ts: separate Vite middleware in-memory state seeded from mock data.

Contracts and docs currently present:

• docs/API_CONTRACT_PREP.md
• docs/BACKEND_B1_READ_API.md
• docs/BACKEND_B2_FEATURES_EVIDENCE_READ_API.md
• docs/BACKEND_B3_CONTEXT_PACKS_PROMPTS_API.md
• docs/BACKEND_B4_CHECKS_REVIEW_API.md
• docs/BACKEND_B5_BUILDS_REGRESSIONS_SNAPSHOTS_API.md
• docs/BACKEND_B6_INSIGHTS_RELATIONSHIPS_KNOWLEDGE_MEMORY_API.md
• src/shared/api/types.ts
• src/shared/api/contracts.ts
• src/shared/api/projectBrainClient.ts

Frontend consumption today:

• UI pages should consume through projectBrainClient or higher-level selectors/actions.
• projectBrainClient currently delegates to mockClient.
• The Vite middleware is not yet the frontend source of truth.
• Future switching should happen inside the client boundary, not by changing pages directly.

Backend Options

A) Node/Fastify Backend

Pros:

• Best fit with the existing TypeScript contracts and model types.
• Shared Zod schemas can validate both API requests and frontend clients.
• Fastify has strong performance, structured plugins, JSON schema support, and good Docker ergonomics.
• Natural fit for API-first design, MCP servers, ChatGPT app connectors, and local self-hosting.
• Background jobs can use BullMQ/Redis with a separate worker process.
• File upload/import endpoints can stream to local disk or S3-compatible storage.

Cons:

• Codebase analysis and ML-heavy pipelines may eventually want Python helpers.
• Requires discipline around service/repository boundaries to avoid a large Node monolith.
• Worker deployment adds another runtime process.

Migration effort:

• Moderate. Existing TypeScript DTOs, contract docs, and mock behavior can be ported incrementally.

Fit:

• AI/codebase analysis later: good for orchestration; Python sidecars can be added for specialist analysis.
• File uploads/imports later: good.
• Self-hosting: excellent.
• Docker deployment: excellent.
• Future MCP/ChatGPT connector/API: excellent.

B) FastAPI Backend

Pros:

• Excellent for Python-native AI/code analysis, embeddings, parsing, and background processing.
• Strong OpenAPI generation and Pydantic validation.
• Good for future ML-heavy worker code.

Cons:

• Introduces a second primary language while the app contracts are already TypeScript.
• DTO drift risk between frontend TypeScript and backend Python models.
• More migration work to preserve existing TypeScript contract shape.
• Frontend and backend type sharing becomes generated rather than native.

Migration effort:

• Moderate to high. Requires translating contracts into Pydantic or generating schemas carefully.

Fit:

• AI/codebase analysis later: excellent.
• File uploads/imports later: excellent.
• Self-hosting: good.
• Docker deployment: good.
• Future MCP/ChatGPT connector/API: good, but TypeScript app connector work may be less direct.

C) Next.js-Style Fullstack

Pros:

• Single deployment surface if the product were already Next.js.
• API routes and UI can live together.

Cons:

• Not applicable without migrating the existing Vite SPA.
• Would create a framework migration before the backend architecture is settled.
• Long-running imports and codebase analysis jobs still need separate worker infrastructure.

Migration effort:

• High and not justified now.

Fit:

• AI/codebase analysis later: mixed.
• File uploads/imports later: good with extra worker setup.
• Self-hosting: good.
• Docker deployment: good.
• Future MCP/ChatGPT connector/API: good, but not worth the app migration cost.

D) Keep Vite Middleware Only

Pros:

• Lowest immediate effort.
• Good demo and contract-testing surface.
• Useful during frontend prototyping.

Cons:

• Not a production backend.
• No durable persistence, job runner, upload pipeline, auth boundary, or deployment separation.
• Dev server lifecycle makes idempotency and state unreliable.
• Encourages backend growth inside frontend tooling.

Migration effort:

• None short-term, but high long-term cleanup cost if it keeps expanding.

Fit:

• AI/codebase analysis later: poor.
• File uploads/imports later: poor.
• Self-hosting: poor for production.
• Docker deployment: demo only.
• Future MCP/ChatGPT connector/API: poor as a durable API target.

Recommended Architecture

Use a dedicated Fastify backend in TypeScript.

Recommended stack:

• Runtime/framework: Node.js with Fastify.
• Database: PostgreSQL.
• ORM/query layer: Prisma initially, with raw SQL escape hatches for graph/query-heavy paths.
• Validation/schema: Zod as source-level request/response validation, optionally exported to OpenAPI.
• Background jobs: BullMQ with Redis, split into api and worker processes.
• File storage: local filesystem volume for self-hosting first, with S3-compatible adapter later.
• API structure: versioned REST under /api/v1, preserving current envelopes.
• Logging: Pino structured logs with request IDs.
• Error handling: centralized Fastify error handler returning the existing error envelope.
• Config/env: .env for local, .env.example, environment-specific Docker Compose values.
• Docker: frontend, api, worker, postgres, redis, and optional storage volume.

Why this choice:

• Keeps TypeScript contracts close to the frontend.
• Avoids a framework migration.
• Supports self-hosting and Docker from the beginning.
• Leaves room for Python analysis workers later without making Python the main API boundary.
• Fits future MCP/ChatGPT/custom app API use because the backend becomes a stable HTTP service.

Proposed Folder Structure

apps/
  web/
    # current Vite React app, moved later or referenced in place
  api/
    src/
      server.ts
      app.ts
      config/
      plugins/
      routes/
        v1/
          projects.routes.ts
          workspace.routes.ts
          features.routes.ts
          context-packs.routes.ts
          checks.routes.ts
          builds.routes.ts
          knowledge.routes.ts
      services/
      repositories/
      schemas/
      errors/
      idempotency/
      logging/
  worker/
    src/
      worker.ts
      jobs/
      processors/
packages/
  contracts/
    src/
      api-types.ts
      schemas.ts
      envelopes.ts

This repo does not need to be reorganized immediately. R1 can start with a smaller backend/ folder if a monorepo move is too disruptive.

API Versioning

• New backend routes should use /api/v1.
• Current Vite middleware may remain under /api for dev/demo until switch-over.
• Response envelope remains:

{ "data": {}, "meta": { "requestId": "...", "timestamp": "..." } }

• Error envelope remains:

{ "error": { "code": "...", "message": "...", "validation": [] }, "meta": { "requestId": "...", "timestamp": "..." } }

• Idempotency remains required for write endpoints using Idempotency-Key, but should move to a durable table:
  • method
  • path
  • key
  • request hash
  • response body
  • status code
  • expiresAt

Migration Strategy

R1: Real backend skeleton

• Add Fastify API skeleton.
• Add health endpoint.
• Add shared envelope helpers.
• Add request ID middleware.
• Add config/env validation.
• Add Docker Compose with Postgres and Redis.
• Do not switch frontend yet.

R2: Projects / Workspace / DNA reads

• Implement B1 against database-backed repositories.
• Seed database from mock-data.ts.
• Keep Vite middleware as fallback/demo.

R3: Features / Evidence reads

• Implement B2.
• Add feature/evidence schema migrations.
• Preserve pagination and filter response shape.

R4: Context Packs / Prompts mock generation

• Implement context-pack reads/writes.
• Keep prompt generation deterministic mock-only.
• Store generated prompt records in the database.

R5: Checks / Review mutations

• Implement checks, review board grouping, review actions, conversions.
• Add durable idempotency.

R6: Builds / Regressions / Snapshots

• Implement build journal, regression memory, snapshot compare.
• Keep compare deterministic until richer diffing exists.

R7: Knowledge / Memory / Relationships / Insights

• Implement B6 endpoints against the database.
• Preserve include-in-context-pack behavior.
• Add relationship integrity checks.

R8: Frontend API switch

• Update projectBrainClient to choose real HTTP client or mockClient via config flag.
• Pages must continue to use projectBrainClient, selectors, and actions only.
• No page-level direct fetch calls.

R9: Persistence hardening

• Add migrations, seed scripts, backup/restore guidance, indexes, idempotency TTL cleanup, and integration tests.

R10: Import/codebase analysis jobs

• Add upload endpoints and job creation.
• Add worker processors for ZIP/codebase/document analysis.
• Add optional Python sidecar only for analysis tasks that genuinely need Python libraries.

Source of Truth Transition

• mock-data.ts remains a fixture and demo seed source.
• The real PostgreSQL database becomes source of truth after R8/R9.
• Seed scripts should transform fixture data into database rows.
• Migration scripts should be repeatable for local self-hosting.
• The frontend must keep using projectBrainClient; switching source should be a client configuration concern.
• Vite middleware should be retired only after the real backend covers B1-B6 and the frontend switch is stable.

Docker / Self-Hosting Notes

Recommended services:

• web: Vite-built static frontend served by Nginx or a small Node static server.
• api: Fastify HTTP API.
• worker: BullMQ worker for imports and analysis.
• postgres: ProjectBrain database.
• redis: queue and job coordination.
• storage: mounted local volume for uploads and generated artifacts.

Self-hosting defaults:

• No paid API required for core CRUD/project memory.
• AI providers configured through optional environment variables later.
• Local file storage first, S3-compatible storage optional.
• Health checks for web, api, worker, postgres, and redis.

Alternatives Rejected

• FastAPI as primary API: strong for analysis, but higher contract drift risk and more translation work from existing TypeScript contracts.
• Next.js fullstack: would require a frontend framework migration before backend needs justify it.
• Vite middleware as production backend: useful contract scaffold, but not durable or deployable as the product backend.

Open Questions

• Should the repo become a formal monorepo now, or should R1 start with a backend/ folder?
• Should Prisma be the long-term ORM, or should graph-heavy relationship queries move to Kysely/raw SQL later?
• Should file storage stay local-only for v1 self-hosting, or ship an S3-compatible adapter in the first backend release?
• What auth model is desired later: local admin, single-user self-hosted, or multi-user teams?
• Should Python analysis workers be planned from R10 or deferred until codebase import requirements are precise?

Final Recommendation

Proceed with R1 as a dedicated Fastify backend skeleton, keeping the Vite middleware untouched as a contract/demo reference. Do not add B7 to the middleware. The next implementation phase should create the real backend foundation and implement only health/config/envelope plumbing before porting B1.