chore: establish cross-IDE synchronization protocol

Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
This commit is contained in:
Team Dark Devil committed 2026-11-18 07:11:11 +06:00
1 parent 05407fd338
commit 4279da45bb
14 files changed
+1162

No files matched your search

+58
View File
@@ -0,0 +1,58 @@
# Developer & Contributor Guide
Welcome to the **DataStack Developer Documentation**. This section contains in-depth architectural specifications, development workflows, compilation instructions, and contribution conventions for engineers working on the DataStack codebase.
---
## Developer Documentation Index
Explore the core technical guides:
### 1. [Architecture Blueprint](./architecture.md)
Detailed breakdown of DataStack's decoupled dual-runtime model:
- **Port 3000 (`web`)**: Next.js 15 App Router, React 19, serverless REST API, streaming CSV/JSON export.
- **Port 4000 (`ingestion`)**: Dedicated Express 5 + Socket.IO real-time ingestion server.
- **Internal Service Bridge**: Authenticated `/internal/*` communication via `DATASTACK_INTERNAL_SECRET`.
- **Storage Subsystem**: In-memory and Redis-backed buffering, cooperative partition file locking, and streaming row counters.
- **PostgreSQL Control-Plane**: Schema provisioning and metadata storage rules (`pg.Pool` + `prisma/schema.prisma`).
### 2. [Local Development Setup](./setup.md)
Instructions for configuring your local machine:
- Prerequisites (Node.js 22, npm, PostgreSQL 16, Redis 7).
- Environment variable configuration (`.env.example`).
- Dual-terminal local workflow (`npm run dev` + `npm run server`).
- Database schema initialization.
### 3. [Build, Compilation & Testing](./build.md)
Standard build and testing commands:
- Compiling Next.js standalone output (`npm run build`).
- Code quality and linting (`npm run lint`).
- Running the comprehensive TypeScript test suite with `node:test` and `tsx`.
- Executing Python SDK unit tests.
- Docker multi-stage container builds.
### 4. [Contribution Rules & Conventions](./contributing.md)
Guidelines for submitting pull requests and modifying code:
- **Universal Multi-IDE Memory Rule** (`MEMORY.md` & `.ai-rules`).
- Single-source-of-truth UI primitives in `components/ui/` (`0rem` radius, zero shadows).
- Architectural guardrails (zero telemetry rows in PostgreSQL, no Socket.IO in Next.js).
- Pull Request and code review checklist.
---
## High-Level Repository Layout
```text
datastack/
├── app/ # Next.js 15 App Router (Frontend, /vcon/*, and /api/*)
├── components/ # UI primitives (ui/), layout shells (layout/), and sections (sections/)
├── core/ # Backend TypeScript modules (auth, Database, data, Storage, Security, sdk)
├── server/ # Dedicated Port 4000 Express + Socket.IO service & CSV batch worker
├── sdk/python/datastack/ # Official pip-installable Python SDK
├── sdk-python/ # Backward-compatible Python SDK wrapper
├── storage/data/ # Partitioned CSV telemetry storage root
├── docker/ # Production Dockerfiles, Nginx proxy configs, and docker-compose.yml
├── nginx/ # Local Nginx development configuration
├── tests/ # Automated test suites (phase01 - phase07)
└── docs/ # User and developer documentation
```
+105
View File
@@ -0,0 +1,105 @@
# Architecture Blueprint
This document details the architectural principles, runtime boundaries, and subsystems powering DataStack.
---
## 1. Core Architectural Principles
DataStack is designed around three foundational engineering constraints:
1. **Decoupled Runtimes**: Long-lived WebSocket connections require stateful, persistent event loops, while modern administrative consoles benefit from serverless, SSR App Router paradigms. DataStack strictly decouples these into **Port 3000** (Web & REST API) and **Port 4000** (Real-Time Ingestion).
2. **Zero Per-Row Database Writes**: Telemetry streams are never inserted per-row into relational tables. Events are buffered in memory and Redis, then flushed in sequential batches directly to daily partitioned CSV files on disk.
3. **Control-Plane vs Data-Plane Isolation**: PostgreSQL is strictly reserved for the control plane (user authentication, client token registry, rate limits, and platform configuration). The data plane resides entirely on disk in partitioned CSV files.
---
## 2. Dual-Runtime Architecture
```text
┌────────────────────────────────────────────────────────┐
│ Nginx Reverse Proxy (Port 80) │
└───────────────┬────────────────────────┬───────────────┘
│ │
/socket.io/ │ │ / & /api/*
▼ ▼
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
│ Ingestion Service (Port 4000) │ │ Web & Admin Console (Port 3000) │
│ Express 5 + Socket.IO 4 │ │ Next.js 15 App Router + React 19 │
├───────────────────────────────────┤ ├───────────────────────────────────┤
│ - WebSocket Handshake Auth (JWT) │ │ - Public Landing & PWA Lifecycle │
│ - Real-time Rate Limiting │ │ - /vcon/* Admin Console │
│ - 5-min Sliding Window Idempotency│ │ - Authenticated HTTP Ingestion │
│ - Memory & Redis Batch Buffer │ │ - Streaming CSV / JSON Export │
│ - CsvWriterWorker Flusher │ │ - Session Auth & Brute-Force Guard│
└─────────────────┬─────────────────┘ └─────────────────┬─────────────────┘
│ │
│ Authenticated /internal/* Bridge │
└────────────────◄═══════════════════════┘
(DATASTACK_INTERNAL_SECRET)
```
### Port 3000 (`web`): Next.js 15 App Router
- **Role**: Serves the user interface and HTTP REST endpoints.
- **Key Modules**:
- `middleware.ts`: Authenticates `/vcon/*` routes using signed WebCrypto session cookies, and enforces CSRF protection by validating `Origin`/`Referer` headers against `Host` and `X-Forwarded-Host`.
- `app/api/events/route.ts`: Accepts HTTP ingestion for single events, arrays (`{ batch: [...] }`), or structured file imports (`{ fileName, records: [...] }`).
- `app/api/export/route.ts`: Streams filtered CSV or JSON files directly from disk partitions to the client browser without buffering entire datasets in server RAM.
### Port 4000 (`ingestion`): Dedicated Ingestion Server
- **Role**: High-throughput Socket.IO WebSocket server (`server/index.js`).
- **Key Modules**:
- `server/socket/handlers.js`: Enforces client JWT authentication, sliding-window rate limits (per-client and global), payload size validation, and idempotency deduplication.
- `server/workers/csvWriter.js`: Flushes buffered batches to daily partitioned CSV files with file rotation and cooperative locking.
---
## 3. Internal Service Bridge (`/internal/*`)
Because Port 3000 and Port 4000 operate as separate processes, administrative actions in `/vcon/` must synchronize with the real-time ingestion service instantly:
- **Security**: All `/internal/*` routes require a shared, timing-safe authentication header: `x-datastack-internal-secret`.
- **Secret Derivation**: `DATASTACK_INTERNAL_SECRET` can be explicitly defined in `.env` or is automatically derived from the active JWT secret via SHA-256.
### Bridge Endpoints:
1. `GET /internal/metrics`: Aggregates active Socket.IO connections, buffer depths, and security stats for the `/vcon/dashboard`.
2. `POST /internal/settings-sync`: Pushes updated rate limits, payload sizes, flush thresholds, or rotated JWT secrets instantly to Port 4000.
3. `POST /internal/client-sync`: Pushes registered, rotated, revoked, or deleted client updates. If a client is revoked, Port 4000 immediately kicks all active sockets for that client.
4. `POST /internal/flush`: Triggers an immediate write of Port 4000's buffer to disk when an admin initiates a manual flush or export.
5. `GET /internal/recent`: Fetches recently buffered events from Port 4000's queue so `/vcon/data` displays zero-latency activity.
---
## 4. Storage Subsystem & Durability
### Data Ingestion Flow:
1. **Push**: Events enter via WebSocket (`sync_data`) or HTTP (`POST /api/events`).
2. **Buffer Queue**: Events are written to the Redis queue (`datastack:buffer:events`) with local RAM fallback.
3. **Batch Flush Condition**: A flush triggers when:
- Buffer size reaches `batchFlushThreshold` (default: 500 events), OR
- Time elapsed reaches `batchFlushIntervalMs` (default: 2000 ms), OR
- A manual flush or export is requested.
4. **Distributed Flush Lock**: In multi-process deployments, workers acquire `datastack:flush:lock` in Redis (`SET lockToken PX 15000 NX`) to prevent concurrent writes.
5. **Safe Trim**: Redis queue items are trimmed (`LTRIM`) **strictly after** the batch append operation confirms success on disk.
### Cooperative Partition File Locking
When writing to `/storage/data/{clientId}/{YYYY-MM-DD}.csv`:
- The writer creates a cooperative `.lock` file alongside the partition (`{file}.lock`).
- Stale locks (older than 10 seconds) created by crashed processes are automatically cleared.
- This prevents race conditions when both Port 3000 and Port 4000 flush to the same client partition simultaneously.
---
## 5. PostgreSQL Metadata Subsystem
PostgreSQL persistence is managed via raw `pg.Pool` connections (`core/Database/database.ts`), mirrored by the reference schema in `prisma/schema.prisma`.
### Managed Tables:
1. `admin_users`: Stores admin credentials with `crypto.scrypt` password hashing and a `token_version` column for instant session invalidation.
2. `admin_login_attempts`: Tracks failed login attempts per composite IP + Email hash with 15-minute brute-force lockout.
3. `client_sdks`: Stores registered client metadata, hashed secrets, signed tokens, rate limits, status (`active`/`revoked`), and total event counters.
4. `deleted_clients`: Tombstone table ensuring deleted client IDs and tokens can never be revived or accepted.
5. `admin_settings`: Single-row configuration storing platform limits, ports, thresholds, and secrets.
### Non-Destructive Migrations
On startup, `DatabaseEngine.ensureSchema()` runs non-destructive `CREATE TABLE IF NOT EXISTS` and additive `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` DDL statements, guaranteeing seamless upgrades on existing production databases without data loss.
+94
View File
@@ -0,0 +1,94 @@
# Build, Compilation & Testing Guide
This guide outlines compilation commands, linter checks, test suite execution, and Docker container packaging.
---
## 1. Production Web Compilation
DataStack uses Next.js 15 App Router configured for **Standalone Output** (`output: "standalone"` in `next.config.ts`), which bundles only the required Node.js runtime files into `.next/standalone`.
### Compiling the Application:
```bash
npm run build
```
### Build Verification Checklist:
- Compiles TypeScript without type errors (`tsc --noEmit`).
- Generates serverless routes and static assets under `.next/`.
- Prepares standalone bundle under `.next/standalone/server.js`.
---
## 2. Code Quality & Linting
DataStack uses ESLint 9 with `eslint-config-next`:
```bash
npm run lint
```
Ensure this command exits cleanly (`code: 0`) with zero warnings or errors prior to committing code or opening a pull request.
---
## 3. Automated Test Suites
DataStack includes comprehensive, fast-executing automated test suites verifying security, synchronization, storage durability, PWA compliance, and deployment health.
### Running TypeScript Integration & Unit Tests
Run the test runner using `npx --yes tsx --test`:
```bash
# Run all core test suites
npx --yes tsx --test tests/phase01-auth.test.ts \
tests/phase02-sync.test.ts \
tests/phase03-storage-buffer.test.ts \
tests/phase06-pwa-landing.test.ts \
tests/phase07-deployment.test.ts
```
#### What These Tests Verify:
- **`phase01-auth.test.ts`**: Secret fail-closed checks, `tokenVersion` session invalidation, brute-force IP throttling, client token redaction, and CSRF Origin validation.
- **`phase02-sync.test.ts`**: Port 3000 ↔ Port 4000 internal secret authentication, settings propagation, instant client revocation, and JWT rotation disconnection.
- **`phase03-storage-buffer.test.ts`**: Redis AOF durability, distributed flush locking, cooperative partition `.lock` files, streaming row counting, and filtered CSV export.
- **`phase06-pwa-landing.test.ts`**: Web App Manifest validity, Service Worker registration, and navigation integrity.
- **`phase07-deployment.test.ts`**: Docker Compose configuration, Redis AOF persistence, Nginx `/health` proxying, deterministic lockfile parity, startup ownership checks, and unhydrated-DB auto-prune guards.
### Running Python SDK Unit Tests
```bash
python3 -m unittest tests/phase05_sdk_test.py
```
Verifies Python SDK WebSocket streaming, Heartbeat HTTP fallback, disk spooling (`.datastack_spool/`), and `.csv`/`.json` file streaming.
---
## 4. Docker Container Builds
To test production containerization locally:
### 1. Build the Web Container (Port 3000)
```bash
docker build -f docker/Dockerfile.web -t datastack-web:latest .
```
### 2. Build the Ingestion Container (Port 4000)
```bash
docker build -f docker/Dockerfile.server -t datastack-ingestion:latest .
```
### 3. Build the Nginx Proxy Container (Port 80)
```bash
docker build -f docker/Dockerfile.proxy -t datastack-proxy:latest .
```
### 4. Build & Run the Entire Stack with Compose
```bash
docker compose -f docker/docker-compose.yml build
docker compose -f docker/docker-compose.yml up -d
```
Verify container health with:
```bash
docker compose -f docker/docker-compose.yml ps
```
All containers (`web`, `ingestion`, `proxy`, `postgres`, `redis`) should display `(healthy)`.
+64
View File
@@ -0,0 +1,64 @@
# Contribution Rules & Guidelines
Thank you for contributing to **DataStack**! To maintain architectural purity, production reliability, and seamless collaboration across multiple development environments, all contributors must adhere to these guidelines.
---
## 1. Universal Multi-IDE Persistent Memory (`MEMORY.md`)
> **CRITICAL RULE**: DataStack is developed across multiple IDEs and AI toolchains (such as VS Code, Google Antigravity, Cursor, and terminal environments). Whenever any modification, update, or bug fix is completed in your session, you must automatically update the `MEMORY.md` file (or provide the exact markdown blocks) so the project context remains perfectly synchronized across all IDEs (see `.ai-rules` and `AGENTS.md`).
When updating `MEMORY.md`:
- Note any new files, directories, routes, or modules added to the repository.
- Record any updated environment variables, runtime parameters, or architectural adjustments.
- Update the status of active features or resolved bugs.
---
## 2. Architectural Guardrails (Strict Enforcement)
1. **Dual-Runtime Separation**:
- Next.js 15 App Router (`app/`) runs on **Port 3000**.
- Express 5 + Socket.IO (`server/index.js`) runs on **Port 4000**.
- **Never** initialize Socket.IO server instances inside Next.js serverless route handlers.
2. **Zero Telemetry Writes to PostgreSQL**:
- Incoming telemetry events **must never** be inserted as rows into PostgreSQL.
- Events are buffered in memory and Redis, then appended to daily CSV partitions in `/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
- PostgreSQL is strictly reserved for `admin_users`, `admin_login_attempts`, `client_sdks`, `deleted_clients`, and `admin_settings`.
3. **Non-Destructive Database Migrations**:
- Never write code that executes `DROP TABLE`, `DROP COLUMN`, or `TRUNCATE` against production databases.
- All schema upgrades must use defensive `CREATE TABLE IF NOT EXISTS` and `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` statements.
4. **Cooperative CSV Locking**:
- All disk write operations must acquire cooperative file locks (`.lock`) with automated stale-lock recovery to prevent dual-writer corruption.
---
## 3. UI Component & Design System Standards
- **Single Source of Truth**: All UI primitives must reside in `components/ui/` (`Button`, `Input`, `Modal`, `Table`, `Panel`, `StatCard`, etc.) and layout shells in `components/layout/`. Do not introduce ad-hoc isolated CSS or duplicate button components.
- **Design Tokens**:
- Border radius: `--radius: 0rem` across all components.
- Shadows: None (`--shadow: none`). Crisp, clean 1px borders with dark and light mode support.
- Font stack: `"Segoe UI", "Segoe UI Web", system-ui, -apple-system, "Helvetica Neue", sans-serif`.
- **Icons**: Import all icons exclusively from `lucide-react`.
---
## 4. Pull Request & Pre-Submission Checklist
Before submitting a Pull Request:
1. **Lint Check**: Run `npm run lint` and verify zero errors and warnings.
2. **Build Check**: Run `npm run build` and ensure the Next.js standalone application compiles cleanly.
3. **Automated Tests**:
- Run the TypeScript integration tests:
```bash
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts
```
- Run the Python SDK tests if modifying SDK code:
```bash
python3 -m unittest tests/phase05_sdk_test.py
```
4. **Documentation**:
- If adding or changing features, update the relevant files in `docs/users/` or `docs/dev/`.
- Update `docs/MANIFEST.md` if new documentation files were introduced.
5. **Memory Synchronization**: Ensure `MEMORY.md` reflects all changes made during your session.
+96
View File
@@ -0,0 +1,96 @@
# Local Development Setup
This guide walks contributors through configuring and running DataStack locally on your development workstation.
---
## 1. Prerequisites
Ensure the following tools are installed:
- **Node.js**: Version `22.x` (LTS recommended)
- **Package Manager**: `npm` (v10+)
- **Python**: Version `3.10+` (optional, for developing and testing the Python client SDK)
- **Docker & Docker Compose**: Optional, useful for running local PostgreSQL and Redis instances.
---
## 2. Clone & Install Dependencies
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
# Install root dependencies deterministically using the committed lockfile
npm ci
```
---
## 3. Environment Configuration
Copy the example environment configuration:
```bash
cp .env.example .env
```
### Local Development Defaults:
In local development (`NODE_ENV=development`), DataStack includes built-in fallbacks:
- If `DATABASE_URL` is omitted, DataStack operates seamlessly in **In-Memory Mock Mode**, allowing you to test the full UI, client registration, and REST endpoints without running PostgreSQL.
- If `REDIS_URL` is omitted, DataStack uses an **in-memory RAM buffer queue**, persisting directly to `./storage/data`.
For full production parity during development, you can spin up local PostgreSQL and Redis containers:
```bash
docker run --name datastack-postgres -e POSTGRES_USER=datastack -e POSTGRES_PASSWORD=password -e POSTGRES_DB=datastack_db -p 5432:5432 -d postgres:16-alpine
docker run --name datastack-redis -p 6379:6379 -d redis:7-alpine redis-server --appendonly yes
```
Then update your `.env`:
```dotenv
DATABASE_URL=postgresql://datastack:password@127.0.0.1:5432/datastack_db
REDIS_URL=redis://127.0.0.1:6379
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
DATASTACK_ADMIN_EMAIL=admin@example.com
```
---
## 4. Running the Development Servers
DataStack requires two concurrent runtime processes. Open two terminal tabs in the project root:
### Terminal 1: Web Console & REST API (Port 3000)
```bash
npm run dev
```
Starts the Next.js 15 development server with Fast Refresh at `http://localhost:3000`.
### Terminal 2: Ingestion Server (Port 4000)
```bash
npm run server
```
Starts the dedicated Express 5 + Socket.IO ingestion service at `http://localhost:4000`.
---
## 5. First-Time Setup in Development
1. Open `http://localhost:3000/vcon/setup` in your browser.
2. Complete the setup wizard using your `DATASTACK_ADMIN_SETUP_KEY` from `.env`.
3. In `/vcon/clients`, register a test client application to generate your first JWT token.
4. In `/vcon/simulator`, test sending live events and observe the buffer counter in `/vcon/dashboard`.
---
## 6. Python SDK Development
If you are modifying the Python SDK in `sdk/python/datastack/`:
```bash
cd sdk/python/datastack
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r ../../../tests/requirements.txt # If running SDK test suites
```