Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
3.0 KiB
3.0 KiB
Developer & Contributor Guide
Welcome to the DataStack Developer Documentation. This section contains in-depth architectural specifications, development workflows, compilation instructions, and contribution conventions for engineers working on the DataStack codebase.
Developer Documentation Index
Explore the core technical guides:
1. Architecture Blueprint
Detailed breakdown of DataStack's decoupled dual-runtime model:
- Port 3000 (
web): Next.js 15 App Router, React 19, serverless REST API, streaming CSV/JSON export. - Port 4000 (
ingestion): Dedicated Express 5 + Socket.IO real-time ingestion server. - Internal Service Bridge: Authenticated
/internal/*communication viaDATASTACK_INTERNAL_SECRET. - Storage Subsystem: In-memory and Redis-backed buffering, cooperative partition file locking, and streaming row counters.
- PostgreSQL Control-Plane: Schema provisioning and metadata storage rules (
pg.Pool+prisma/schema.prisma).
2. Local Development Setup
Instructions for configuring your local machine:
- Prerequisites (Node.js 22, npm, PostgreSQL 16, Redis 7).
- Environment variable configuration (
.env.example). - Dual-terminal local workflow (
npm run dev+npm run server). - Database schema initialization.
3. Build, Compilation & Testing
Standard build and testing commands:
- Compiling Next.js standalone output (
npm run build). - Code quality and linting (
npm run lint). - Running the comprehensive TypeScript test suite with
node:testandtsx. - Executing Python SDK unit tests.
- Docker multi-stage container builds.
4. Contribution Rules & Conventions
Guidelines for submitting pull requests and modifying code:
- Universal Multi-IDE Memory Rule (
MEMORY.md&.ai-rules). - Single-source-of-truth UI primitives in
components/ui/(0remradius, zero shadows). - Architectural guardrails (zero telemetry rows in PostgreSQL, no Socket.IO in Next.js).
- Pull Request and code review checklist.
High-Level Repository Layout
datastack/
├── app/ # Next.js 15 App Router (Frontend, /vcon/*, and /api/*)
├── components/ # UI primitives (ui/), layout shells (layout/), and sections (sections/)
├── core/ # Backend TypeScript modules (auth, Database, data, Storage, Security, sdk)
├── server/ # Dedicated Port 4000 Express + Socket.IO service & CSV batch worker
├── sdk/python/datastack/ # Official pip-installable Python SDK
├── sdk-python/ # Backward-compatible Python SDK wrapper
├── storage/data/ # Partitioned CSV telemetry storage root
├── docker/ # Production Dockerfiles, Nginx proxy configs, and docker-compose.yml
├── nginx/ # Local Nginx development configuration
├── tests/ # Automated test suites (phase01 - phase07)
└── docs/ # User and developer documentation