Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
13 KiB
13 KiB
MEMORY.md — DataStack Cross-IDE Persistent Project Memory
Last Synchronized: 2026-11-17
Status: Production-Ready (Phase-01throughPhase-07Completed & Verified)
Scope Rule: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see.ai-rulesandAGENTS.md).
1. Project Identity & Core Principles
- Project: DataStack — Self-hosted, real-time telemetry ingestion, admin governance (
/vcon/*), and partitioned CSV storage/export platform. - Runtime Split (CRITICAL):
- Port
3000(web): Next.js 15 App Router — public landing (/), PWA lifecycle,/vcon/*Admin Console, REST ingestion (POST /api/events), metrics/settings/client APIs, and streaming exports (GET /api/export). - Port
4000(ingestion): Dedicated Node.js Express 5 + Socket.IO server (server/index.js) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker. - Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port
4000.
- Port
- Storage & Database Boundary (CRITICAL):
- PostgreSQL (
pg+prisma/schema.prisma): Stores strictly control-plane metadata (admin_users,admin_login_attempts,client_sdks,deleted_clients,admin_settings). Never insert per-row telemetry events into PostgreSQL. - Partitioned CSV Storage (
/storage/data/{clientId}/{YYYY-MM-DD}.csv): All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (fileRotationMb, e.g.,-part2.csv), and retention pruning (autoPruneDays). - Redis (
ioredis): Shared cross-process buffer queue (datastack:buffer:events), recent event cache (datastack:recent:events), and distributed flush lock (datastack:flush:lock). Persisted in Docker via--appendonly yes(redisdata:/data). Falls back gracefully to local RAM whenREDIS_URLis omitted.
- PostgreSQL (
2. Tech Stack & Local Setup
Stack
- Frontend / Web API: Next.js
15.4.9(App Router,output: "standalone"), React19.2.1, TypeScript5.9.3, Tailwind CSS v4 (@tailwindcss/postcss),lucide-react,motion. - Ingestion Server: Node.js
22/ Express5.3.0, Socket.IO4.8.4. - Persistence & Caching:
pg8.23.1(PostgreSQL 16 in Docker),ioredis6.0.0(Redis 7 AOF in Docker),csv-parse7.0.3,json2csv6.0.0-alpha.2. - Auth & Security:
jsonwebtoken9.0.3, Nodecrypto.scryptpassword hashing, WebCrypto HMAC-SHA256 admin session cookies (middleware.ts), CSRFOrigin/ReferervsHost/X-Forwarded-Hostchecks,DATASTACK_INTERNAL_SECRETtiming-safe bridge auth between Port3000↔4000. - Client SDKs: Official pip-installable Python SDK in
sdk/python/datastack/(datastack) + backward-compatible wrappersdk-python/(datastack-sdk).
Environment & Setup Commands
- Configure Environment: Copy
.env.exampleto.env(in productionNODE_ENV=production, all 4 secrets MUST be 32+ chars:DATASTACK_ADMIN_SESSION_SECRET,DATASTACK_JWT_SECRET,DATASTACK_ADMIN_SETUP_KEY,DATASTACK_INTERNAL_SECRET; optionalDATABASE_URLandREDIS_URL). - Install Dependencies:
npm ci(ornpm install) using rootpackage.json+package-lock.json. - Run Locally (Two Terminals):
- Terminal 1 (Port
3000):npm run dev - Terminal 2 (Port
4000):npm run server
- Terminal 1 (Port
- Build, Lint & Test Verification:
- Build:
npm run build - Lint:
npm run lint(eslint .) - TS Test Suite (Phases 01–03, 06, 07):
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts - Python SDK Tests:
python3 -m unittest tests/phase05_sdk_test.py
- Build:
3. Folder Structure & File Architecture
datastack/
├── .ai-rules # Universal cross-IDE rules (enforces MEMORY.md sync)
├── AGENTS.md # Engineering rules, UI single-source-of-truth & memory sync
├── MEMORY.md # Persistent cross-IDE project memory (this file)
├── package.json / package-lock.json
├── middleware.ts # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
├── app/
│ ├── (landing)/page.tsx # Public landing page
│ ├── globals.css # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
│ ├── layout.tsx / not-found.tsx / error.tsx
│ ├── api/
│ │ ├── auth/ # Admin login, logout, setup (/vcon/setup), profile update
│ │ ├── clients/ # CRUD clients, token regeneration, revocation
│ │ ├── data/ # Paginated CSV search/query across partitions
│ │ ├── events/ # Authenticated HTTP ingestion (single/batch/file records) & recent list
│ │ ├── export/ # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
│ │ ├── health/ # Web health + client Bearer token probe + async buffer depth
│ │ ├── metrics/ # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
│ │ ├── settings/ # Allowlisted runtime config get/post + live Port 4000 sync
│ │ └── storage/ # CSV partition listing & manual cross-process buffer flush
│ └── vcon/ # Protected Admin Console pages:
│ ├── dashboard/ clients/ data/ export/ login/ profile/ sdk/ settings/ setup/ simulator/ storage/
├── components/
│ ├── ui/ # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, etc.)
│ ├── layout/ # MainLayout, Navbar, Sidebar, PageShell, AdminPageHeader, Footer
│ ├── sections/ # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, SettingsWorkspace, etc.)
│ └── pwa/ # PwaLifecycle.tsx service worker registration
├── core/
│ ├── adminSettings/ # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
│ ├── apiEngine/ # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
│ ├── auth/ # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
│ ├── data/ # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
│ ├── Database/ # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
│ ├── sdk/ # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
│ ├── Security/ # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
│ └── Storage/ # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
├── server/
│ ├── index.js # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, graceful exit
│ ├── socket/handlers.js # Handshake JWT auth, ping_verify, sync_data, batch_sync
│ └── workers/csvWriter.js # Partitioned CSV batch writer & rotation/pruning worker
├── sdk/python/datastack/ # Official pip-installable Python SDK (`datastack`)
│ └── datastack/ # client.py, config.py, heartbeat.py, spool.py, exceptions.py
├── sdk-python/ # Backward-compatible re-export package (`datastack-sdk`)
├── docker/
│ ├── docker-compose.yml # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
│ ├── Dockerfile.web # Multi-stage standalone Next.js build + safe /app/storage ownership check
│ ├── Dockerfile.server # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
│ ├── Dockerfile.proxy # Nginx reverse proxy container
│ └── coolify-nginx.conf # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
├── nginx/default.conf # Local Nginx proxy config matching coolify-nginx.conf
├── prisma/schema.prisma # Reference PostgreSQL metadata schema
├── public/ # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
├── storage/data/ # Runtime partitioned CSV root ({clientId}/{YYYY-MM-DD}.csv)
├── docs/ # Official documentation suite:
│ ├── MANIFEST.md # Documentation index, breakdown, and governance rules
│ ├── users/ # End-user & operator guides (installation, features, tutorials, faq)
│ └── dev/ # Contributor & developer guides (architecture, setup, build, contributing)
├── notes/ # Audit logs & phase implementation plans (gitignored)
└── tests/ # Automated verification suites:
├── phase01-auth.test.ts # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
├── phase02-sync.test.ts # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
├── phase05_sdk_test.py # Python SDK WebSocket/HTTP fallback, offline spool, file streaming
├── phase06-pwa-landing.test.ts # PWA manifest/icons/sw + navigation routes
└── phase07-deployment.test.ts # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush
4. Active Features & Development Status
All planned hardening & deployment phases (Phase-01 through Phase-07) are Complete & Verified:
- Admin Authentication & CSRF Hardening (Completed — Phase-01)
- Fail-closed secret enforcement in
NODE_ENV=production(/vcon/setupbootstrap viaDATASTACK_ADMIN_SETUP_KEY). - Password hashing via
crypto.scrypt, session invalidation via DB-backedtokenVersion, composite IP+email login lockout (admin_login_attempts), and strictOrigin/ReferervsHost/X-Forwarded-HostCSRF verification.
- Fail-closed secret enforcement in
- Real-Time Dual-Runtime Sync (Completed — Phase-02)
- Authenticated
/internal/*bridge between Port3000and Port4000(DATASTACK_INTERNAL_SECRET, backward-compatible Coolify fallback). - Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
- Authenticated
- Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)
- Shared Redis queue (
datastack:buffer:events) with distributed flush lock (NX PX) whereLTRIMexecutes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects. - Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (
GET /api/export).
- Shared Redis queue (
- Admin Console UI & Data/Export Workspace (Completed — Phase-04)
- Built with shared
components/ui/andcomponents/layout/primitives (0remradius, zero shadows, Fluent/Segoe dark & light tokens).
- Built with shared
- Production Python Client SDK (Completed — Phase-05)
- Supports single events (
send()/sync()), auto-splitting/rate-limit-paced batches (send_batch()/batch_sync()), and direct file ingestion (send_file()for.csv,.json,.jsonl,.ndjsonwith SHA-256 deterministiceventId). - Automatic
config.datastack.pydiscovery, WebSocket primary transport -> Heartbeat HTTP fallback ->.datastack_spool/spool.jsonloffline spool & auto-drain.
- Supports single events (
- PWA & Landing Integrity (Completed — Phase-06)
- Complete
public/manifest.json, icons,public/sw.js, and root-relative navigation links.
- Complete
- Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)
- Redis AOF persistence (
redisdata:/data), deterministicpackage-lock.jsonfornpm ci, Nginxlocation = /health→ingestion:4000,/api/healthcontainer probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DBautoPruneDayssafety guard inSettingsManager, DB JWT hydration inSDKManager, gracefulSIGTERM/SIGINTflush inDataBufferEngine, and non-destructiveADD COLUMN IF NOT EXISTSDDL guards.
- Redis AOF persistence (
5. Known Constraints & Operational Notes for Future Contributors
- Never add per-row telemetry
INSERTqueries to PostgreSQL or addDROP/TRUNCATEauto-migrations. - Never move Socket.IO handlers into Next.js
app/api/*routes. - When deploying updates to an existing Coolify VPS, always preserve existing
SERVICE_*environment variables and avoiddocker compose down -vsopgdata,datastack-data, andredisdatavolumes remain intact. - After any code change, verify with
npm run build,npm run lint, the TStests/phase*.test.tssuite (npx --yes tsx --test), andpython3 -m unittest tests/phase05_sdk_test.py, then synchronize thisMEMORY.md.