projectbrain-Public/docs/REAL_BACKEND_ARCHITECTURE_DECISION.md

# Real Backend Architecture Decision

Status: Proposed decision  
Date: 2026-06-06  
Scope: Move ProjectBrain from Vite mock middleware to a real backend architecture without implementing that backend yet.

## Decision

ProjectBrain should move to a dedicated TypeScript backend using Fastify, PostgreSQL, Prisma, Zod, and a background worker backed by Redis/BullMQ. The current Vite middleware should remain as a dev/demo contract surface until the frontend is switched behind a configuration flag.

This keeps the product API-first, self-hostable, Docker-friendly, and aligned with the existing TypeScript contract layer while leaving room for later AI-assisted imports, codebase analysis jobs, file uploads, MCP integration, and ChatGPT/custom app connectors.

## Current Backend-Mock State

The current backend surface is implemented as Vite dev middleware in `src/shared/api/b1ReadBackend.ts` and mounted from `vite.config.ts` under `/api/*`.

B1-B6 endpoint families now covered as mock contract surface:

- B1: Projects, Workspace, DNA, context-pack summary.
- B2: Feature groups, Features, Evidence reads.
- B3: Context Packs and prompt generation mock.
- B4: Checks and review-board mutations.
- B5: Builds, Regressions, Snapshots.
- B6: Insights, Relationships, Knowledge, Memory Rules.

Current mock limitations:

- In-memory state only; data resets when the Vite process restarts.
- No database, migrations, transactions, row-level integrity, or durable idempotency store.
- No auth, permissions, tenancy, or user ownership model.
- No real AI calls; prompt/current-state style operations are deterministic mock outputs.
- No file upload storage, parsing pipeline, codebase scan worker, or long-running job runner.
- Vite middleware is development infrastructure, not a production backend boundary.
- Idempotency is process-local and keyed by method, path, and `Idempotency-Key`.

Where state lives today:

- `src/shared/lib/mock-data.ts`: central fixtures used by the UI and mock clients.
- `src/shared/api/mockClient.ts`: frontend in-memory store used by `projectBrainClient`.
- `src/shared/api/b1ReadBackend.ts`: separate Vite middleware in-memory state seeded from mock data.

Contracts and docs currently present:

- `docs/API_CONTRACT_PREP.md`
- `docs/BACKEND_B1_READ_API.md`
- `docs/BACKEND_B2_FEATURES_EVIDENCE_READ_API.md`
- `docs/BACKEND_B3_CONTEXT_PACKS_PROMPTS_API.md`
- `docs/BACKEND_B4_CHECKS_REVIEW_API.md`
- `docs/BACKEND_B5_BUILDS_REGRESSIONS_SNAPSHOTS_API.md`
- `docs/BACKEND_B6_INSIGHTS_RELATIONSHIPS_KNOWLEDGE_MEMORY_API.md`
- `src/shared/api/types.ts`
- `src/shared/api/contracts.ts`
- `src/shared/api/projectBrainClient.ts`

Frontend consumption today:

- UI pages should consume through `projectBrainClient` or higher-level selectors/actions.
- `projectBrainClient` currently delegates to `mockClient`.
- The Vite middleware is not yet the frontend source of truth.
- Future switching should happen inside the client boundary, not by changing pages directly.

## Backend Options

### A) Node/Fastify Backend

Pros:

- Best fit with the existing TypeScript contracts and model types.
- Shared Zod schemas can validate both API requests and frontend clients.
- Fastify has strong performance, structured plugins, JSON schema support, and good Docker ergonomics.
- Natural fit for API-first design, MCP servers, ChatGPT app connectors, and local self-hosting.
- Background jobs can use BullMQ/Redis with a separate worker process.
- File upload/import endpoints can stream to local disk or S3-compatible storage.

Cons:

- Codebase analysis and ML-heavy pipelines may eventually want Python helpers.
- Requires discipline around service/repository boundaries to avoid a large Node monolith.
- Worker deployment adds another runtime process.

Migration effort:

- Moderate. Existing TypeScript DTOs, contract docs, and mock behavior can be ported incrementally.

Fit:

- AI/codebase analysis later: good for orchestration; Python sidecars can be added for specialist analysis.
- File uploads/imports later: good.
- Self-hosting: excellent.
- Docker deployment: excellent.
- Future MCP/ChatGPT connector/API: excellent.

### B) FastAPI Backend

Pros:

- Excellent for Python-native AI/code analysis, embeddings, parsing, and background processing.
- Strong OpenAPI generation and Pydantic validation.
- Good for future ML-heavy worker code.

Cons:

- Introduces a second primary language while the app contracts are already TypeScript.
- DTO drift risk between frontend TypeScript and backend Python models.
- More migration work to preserve existing TypeScript contract shape.
- Frontend and backend type sharing becomes generated rather than native.

Migration effort:

- Moderate to high. Requires translating contracts into Pydantic or generating schemas carefully.

Fit:

- AI/codebase analysis later: excellent.
- File uploads/imports later: excellent.
- Self-hosting: good.
- Docker deployment: good.
- Future MCP/ChatGPT connector/API: good, but TypeScript app connector work may be less direct.

### C) Next.js-Style Fullstack

Pros:

- Single deployment surface if the product were already Next.js.
- API routes and UI can live together.

Cons:

- Not applicable without migrating the existing Vite SPA.
- Would create a framework migration before the backend architecture is settled.
- Long-running imports and codebase analysis jobs still need separate worker infrastructure.

Migration effort:

- High and not justified now.

Fit:

- AI/codebase analysis later: mixed.
- File uploads/imports later: good with extra worker setup.
- Self-hosting: good.
- Docker deployment: good.
- Future MCP/ChatGPT connector/API: good, but not worth the app migration cost.

### D) Keep Vite Middleware Only

Pros:

- Lowest immediate effort.
- Good demo and contract-testing surface.
- Useful during frontend prototyping.

Cons:

- Not a production backend.
- No durable persistence, job runner, upload pipeline, auth boundary, or deployment separation.
- Dev server lifecycle makes idempotency and state unreliable.
- Encourages backend growth inside frontend tooling.

Migration effort:

- None short-term, but high long-term cleanup cost if it keeps expanding.

Fit:

- AI/codebase analysis later: poor.
- File uploads/imports later: poor.
- Self-hosting: poor for production.
- Docker deployment: demo only.
- Future MCP/ChatGPT connector/API: poor as a durable API target.

## Recommended Architecture

Use a dedicated Fastify backend in TypeScript.

Recommended stack:

- Runtime/framework: Node.js with Fastify.
- Database: PostgreSQL.
- ORM/query layer: Prisma initially, with raw SQL escape hatches for graph/query-heavy paths.
- Validation/schema: Zod as source-level request/response validation, optionally exported to OpenAPI.
- Background jobs: BullMQ with Redis, split into `api` and `worker` processes.
- File storage: local filesystem volume for self-hosting first, with S3-compatible adapter later.
- API structure: versioned REST under `/api/v1`, preserving current envelopes.
- Logging: Pino structured logs with request IDs.
- Error handling: centralized Fastify error handler returning the existing error envelope.
- Config/env: `.env` for local, `.env.example`, environment-specific Docker Compose values.
- Docker: `frontend`, `api`, `worker`, `postgres`, `redis`, and optional `storage` volume.

Why this choice:

- Keeps TypeScript contracts close to the frontend.
- Avoids a framework migration.
- Supports self-hosting and Docker from the beginning.
- Leaves room for Python analysis workers later without making Python the main API boundary.
- Fits future MCP/ChatGPT/custom app API use because the backend becomes a stable HTTP service.

## Proposed Folder Structure

```text
apps/
  web/
    # current Vite React app, moved later or referenced in place
  api/
    src/
      server.ts
      app.ts
      config/
      plugins/
      routes/
        v1/
          projects.routes.ts
          workspace.routes.ts
          features.routes.ts
          context-packs.routes.ts
          checks.routes.ts
          builds.routes.ts
          knowledge.routes.ts
      services/
      repositories/
      schemas/
      errors/
      idempotency/
      logging/
  worker/
    src/
      worker.ts
      jobs/
      processors/
packages/
  contracts/
    src/
      api-types.ts
      schemas.ts
      envelopes.ts
```

This repo does not need to be reorganized immediately. R1 can start with a smaller `backend/` folder if a monorepo move is too disruptive.

## API Versioning

- New backend routes should use `/api/v1`.
- Current Vite middleware may remain under `/api` for dev/demo until switch-over.
- Response envelope remains:

```json
{ "data": {}, "meta": { "requestId": "...", "timestamp": "..." } }
```

- Error envelope remains:

```json
{ "error": { "code": "...", "message": "...", "validation": [] }, "meta": { "requestId": "...", "timestamp": "..." } }
```

- Idempotency remains required for write endpoints using `Idempotency-Key`, but should move to a durable table:
  - method
  - path
  - key
  - request hash
  - response body
  - status code
  - expiresAt

## Migration Strategy

### R1: Real backend skeleton

- Add Fastify API skeleton.
- Add health endpoint.
- Add shared envelope helpers.
- Add request ID middleware.
- Add config/env validation.
- Add Docker Compose with Postgres and Redis.
- Do not switch frontend yet.

### R2: Projects / Workspace / DNA reads

- Implement B1 against database-backed repositories.
- Seed database from `mock-data.ts`.
- Keep Vite middleware as fallback/demo.

### R3: Features / Evidence reads

- Implement B2.
- Add feature/evidence schema migrations.
- Preserve pagination and filter response shape.

### R4: Context Packs / Prompts mock generation

- Implement context-pack reads/writes.
- Keep prompt generation deterministic mock-only.
- Store generated prompt records in the database.

### R5: Checks / Review mutations

- Implement checks, review board grouping, review actions, conversions.
- Add durable idempotency.

### R6: Builds / Regressions / Snapshots

- Implement build journal, regression memory, snapshot compare.
- Keep compare deterministic until richer diffing exists.

### R7: Knowledge / Memory / Relationships / Insights

- Implement B6 endpoints against the database.
- Preserve include-in-context-pack behavior.
- Add relationship integrity checks.

### R8: Frontend API switch

- Update `projectBrainClient` to choose real HTTP client or `mockClient` via config flag.
- Pages must continue to use `projectBrainClient`, selectors, and actions only.
- No page-level direct fetch calls.

### R9: Persistence hardening

- Add migrations, seed scripts, backup/restore guidance, indexes, idempotency TTL cleanup, and integration tests.

### R10: Import/codebase analysis jobs

- Add upload endpoints and job creation.
- Add worker processors for ZIP/codebase/document analysis.
- Add optional Python sidecar only for analysis tasks that genuinely need Python libraries.

## Source of Truth Transition

- `mock-data.ts` remains a fixture and demo seed source.
- The real PostgreSQL database becomes source of truth after R8/R9.
- Seed scripts should transform fixture data into database rows.
- Migration scripts should be repeatable for local self-hosting.
- The frontend must keep using `projectBrainClient`; switching source should be a client configuration concern.
- Vite middleware should be retired only after the real backend covers B1-B6 and the frontend switch is stable.

## Docker / Self-Hosting Notes

Recommended services:

- `web`: Vite-built static frontend served by Nginx or a small Node static server.
- `api`: Fastify HTTP API.
- `worker`: BullMQ worker for imports and analysis.
- `postgres`: ProjectBrain database.
- `redis`: queue and job coordination.
- `storage`: mounted local volume for uploads and generated artifacts.

Self-hosting defaults:

- No paid API required for core CRUD/project memory.
- AI providers configured through optional environment variables later.
- Local file storage first, S3-compatible storage optional.
- Health checks for `web`, `api`, `worker`, `postgres`, and `redis`.

## Alternatives Rejected

- FastAPI as primary API: strong for analysis, but higher contract drift risk and more translation work from existing TypeScript contracts.
- Next.js fullstack: would require a frontend framework migration before backend needs justify it.
- Vite middleware as production backend: useful contract scaffold, but not durable or deployable as the product backend.

## Open Questions

- Should the repo become a formal monorepo now, or should R1 start with a `backend/` folder?
- Should Prisma be the long-term ORM, or should graph-heavy relationship queries move to Kysely/raw SQL later?
- Should file storage stay local-only for v1 self-hosting, or ship an S3-compatible adapter in the first backend release?
- What auth model is desired later: local admin, single-user self-hosted, or multi-user teams?
- Should Python analysis workers be planned from R10 or deferred until codebase import requirements are precise?

## Final Recommendation

Proceed with R1 as a dedicated Fastify backend skeleton, keeping the Vite middleware untouched as a contract/demo reference. Do not add B7 to the middleware. The next implementation phase should create the real backend foundation and implement only health/config/envelope plumbing before porting B1.