Files
DataStack/MEMORY.md
T
Team Dark Devil 935f12a1e9 docs: update memory status and prune obsolete notes
Update the synchronization timestamp and status, and remove the deprecated notes directory structure from the project architecture overview.
2026-11-19 01:07:12 +06:00

18 KiB
Raw Blame History

MEMORY.md — DataStack Cross-IDE Persistent Project Memory

Last Synchronized: 2026-11-18 11:00:00 UTC
Status: Fresh Source-Code Forensic Audit Complete (notes/forensic-audit-report.md — 27 verified findings across 5 categories; awaiting approval to plan incremental remediation phases)
Scope Rule: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see .ai-rules and AGENTS.md).


1. Project Identity & Core Principles

  • Project: DataStack — Self-hosted, real-time telemetry ingestion, admin governance (/vcon/*), and partitioned CSV storage/export platform.
  • Runtime Split (CRITICAL):
    • Port 3000 (web): Next.js 15 App Router — public landing (/), PWA lifecycle, /vcon/* Admin Console, REST ingestion (POST /api/events), metrics/settings/client APIs, and streaming exports (GET /api/export).
    • Port 4000 (ingestion): Dedicated Node.js Express 5 + Socket.IO server (server/index.js) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker.
    • Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port 4000.
  • Storage & Database Boundary (CRITICAL):
    • PostgreSQL (pg + prisma/schema.prisma): Stores strictly control-plane metadata (admin_users, admin_login_attempts, client_sdks, deleted_clients, admin_settings). Never insert per-row telemetry events into PostgreSQL.
    • Partitioned CSV Storage (/storage/data/{clientId}/{YYYY-MM-DD}.csv): All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (fileRotationMb, e.g., -part2.csv), and retention pruning (autoPruneDays).
    • Redis (ioredis): Shared cross-process buffer queue (datastack:buffer:events), recent event cache (datastack:recent:events), and distributed flush lock (datastack:flush:lock). Persisted in Docker via --appendonly yes (redisdata:/data). Falls back gracefully to local RAM when REDIS_URL is omitted.

2. Tech Stack & Local Setup

Stack

  • Frontend / Web API: Next.js 15.4.9 (App Router, output: "standalone"), React 19.2.1, TypeScript 5.9.3, Tailwind CSS v4 (@tailwindcss/postcss), lucide-react, motion.
  • Ingestion Server: Node.js 22 / Express 5.3.0, Socket.IO 4.8.4.
  • Persistence & Caching: pg 8.23.1 (PostgreSQL 16 in Docker), ioredis 6.0.0 (Redis 7 AOF in Docker), csv-parse 7.0.3, json2csv 6.0.0-alpha.2.
  • Auth & Security: jsonwebtoken 9.0.3, Node crypto.scrypt password hashing, WebCrypto HMAC-SHA256 admin session cookies (middleware.ts), CSRF Origin/Referer vs Host/X-Forwarded-Host checks, DATASTACK_INTERNAL_SECRET timing-safe bridge auth between Port 3000 ↔ 4000.
  • Client SDKs: Official pip-installable Python SDK in sdk/python/datastack/ (datastack) + backward-compatible wrapper sdk-python/ (datastack-sdk).

Environment & Setup Commands

  1. Configure Environment: Copy .env.example to .env (in production NODE_ENV=production, all 4 secrets MUST be 32+ chars: DATASTACK_ADMIN_SESSION_SECRET, DATASTACK_JWT_SECRET, DATASTACK_ADMIN_SETUP_KEY, DATASTACK_INTERNAL_SECRET; optional DATABASE_URL and REDIS_URL).
  2. Install Dependencies: npm ci (or npm install) using root package.json + package-lock.json.
  3. Run Locally (Two Terminals):
    • Terminal 1 (Port 3000): npm run dev
    • Terminal 2 (Port 4000): npm run server
  4. Build, Lint & Test Verification:
    • Build: npm run build
    • Lint: npm run lint (eslint .)
    • TS Test Suite (Phases 01–03, 06, 07): npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts
    • Python SDK Tests: python3 -m unittest tests/phase05_sdk_test.py

3. Folder Structure & File Architecture

datastack/
├── .ai-rules                      # Universal cross-IDE rules (enforces MEMORY.md sync)
├── AGENTS.md                      # Engineering rules, UI single-source-of-truth & memory sync
├── MEMORY.md                      # Persistent cross-IDE project memory (this file)
├── package.json / package-lock.json
├── middleware.ts                  # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
├── app/
│   ├── (landing)/page.tsx         # Public landing page
│   ├── globals.css                # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
│   ├── layout.tsx / not-found.tsx / error.tsx
│   ├── api/
│   │   ├── auth/                  # Admin login, logout, setup (/vcon/setup), profile update
│   │   ├── clients/               # CRUD clients, token regeneration, revocation
│   │   ├── data/                  # Paginated CSV search/query across partitions
│   │   ├── events/                # Authenticated HTTP ingestion (single/batch/file records) & recent list
│   │   ├── export/                # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
│   │   ├── health/                # Web health + client Bearer token probe + async buffer depth
│   │   ├── metrics/               # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
│   │   ├── settings/              # Allowlisted runtime config get/post + live Port 4000 sync
│   │   └── storage/               # CSV partition listing & manual cross-process buffer flush
│   └── vcon/                      # Protected Admin Console pages:
│       ├── dashboard/ clients/ data/ export/ login/ profile/ sdk/ settings/ setup/ simulator/ storage/
├── components/
│   ├── ui/                        # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, etc.)
│   ├── layout/                    # MainLayout, Navbar, Sidebar, PageShell, AdminPageHeader, Footer
│   ├── sections/                  # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, SettingsWorkspace, etc.)
│   └── pwa/                       # PwaLifecycle.tsx service worker registration
├── core/
│   ├── adminSettings/             # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
│   ├── apiEngine/                 # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
│   ├── auth/                      # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
│   ├── data/                      # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
│   ├── Database/                  # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
│   ├── sdk/                       # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
│   ├── Security/                  # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
│   └── Storage/                   # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
├── server/
│   ├── index.js                   # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, graceful exit
│   ├── socket/handlers.js         # Handshake JWT auth, ping_verify, sync_data, batch_sync
│   └── workers/csvWriter.js       # Partitioned CSV batch writer & rotation/pruning worker
├── sdk/python/datastack/          # Official pip-installable Python SDK (`datastack` v2.2.0)
│   ├── datastack/                 # client.py, config.py, heartbeat.py, spool.py, exceptions.py
│   ├── config.datastack.example.py
│   ├── pyproject.toml / setup.py
│   ├── README.md                  # SDK overview, installation, quickstart, docs index
│   └── docs/                      # Production SDK Documentation Suite (LicenX Standard):
│       ├── README.md              # SDK Documentation Hub & architecture overview
│       ├── INSTALLATION.md        # Python runtimes, pip commands, and verification
│       ├── QUICKSTART.md          # 5-minute zero-boilerplate onboarding guide
│       ├── API_REFERENCE.md       # Complete DataStack class & method specifications
│       ├── CONFIGURATION.md       # config.datastack.py & DATASTACK_* environment reference
│       ├── EXAMPLES.md            # IoT streaming, FastAPI, batching, and file ingestion
│       ├── ERROR_HANDLING.md      # Exception hierarchy, 401/403/413/429 codes & retries
│       └── LIFECYCLE.md           # 5-state machine, zero-CPU IDLE mode & offline spooling
├── sdk-python/                    # Backward-compatible re-export package (`datastack-sdk`)
├── docker/
│   ├── docker-compose.yml         # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
│   ├── Dockerfile.web             # Multi-stage standalone Next.js build + safe /app/storage ownership check
│   ├── Dockerfile.server          # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
│   ├── Dockerfile.proxy           # Nginx reverse proxy container
│   └── coolify-nginx.conf         # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
├── nginx/default.conf             # Local Nginx proxy config matching coolify-nginx.conf
├── prisma/schema.prisma           # Reference PostgreSQL metadata schema
├── public/                        # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
├── docs/                          # Comprehensive Production Documentation Suite:
│   ├── index.md                   # Primary documentation hub and portal navigation index
│   ├── MANIFEST.md                # Master documentation inventory & authoring governance
│   ├── audit-coverage-report.md   # Forensic codebase audit matrix & coverage report
│   ├── users/                     # End-User & Operator Guides:
│   │   ├── index.md               # User portal overview & reading pathways
│   │   ├── getting-started.md     # 5-minute fast-track operator onboarding guide
│   │   ├── installation.md        # Production Docker Compose, Coolify & volume guide
│   │   ├── configuration.md       # Environment variables & platform runtime settings
│   │   ├── features/              # Deep-dive guides: admin-console, client-management,
│   │   │                          # data-simulator, partitioned-storage, data-export
│   │   ├── workflows.md           # End-to-end operator journeys & disaster recovery
│   │   ├── troubleshooting.md     # Forensic error recovery, status codes & diagnostics
│   │   └── faq.md                 # Storage architecture & operational questions
│   ├── dev/                       # Developer & Contributor Guides:
│   │   ├── index.md               # Developer hub index & architectural principles
│   │   ├── architecture.md        # Dual-runtime blueprint, service bridge & storage
│   │   ├── codebase-structure.md  # File-by-file module, route & component inventory
│   │   ├── local-setup.md         # Local workstation setup & dual-terminal workflow
│   │   ├── api-reference.md       # Exhaustive REST API & Socket.IO specification
│   │   ├── database-schema.md     # PostgreSQL control plane & DDL migrations
│   │   ├── security.md            # Security architecture, scrypt, sessions & CSRF
│   │   ├── building-packaging.md  # Next.js standalone build & multi-stage Dockerfiles
│   │   ├── testing.md             # Automated test suites guide (node:test + unittest)
│   │   └── contributing.md        # Architectural rules, memory sync & PR checklist
│   └── reference/                 # Technical Reference Portal:
│       ├── index.md               # Technical reference index
│       ├── configuration-reference.md # Complete environment variable & setting bounds
│       ├── dependencies.md        # Runtime & dev dependencies audit with rationales
│       └── glossary.md            # Technical domain terminology & concept glossary
├── notes/                         # Fresh forensic audit output only (legacy reports/plans cleared):
│   └── forensic-audit-report.md   # 100% source-verified audit report (27 issues: 5C, 11H, 10M, 1L)
└── tests/                         # Automated verification suites:
    ├── phase01-auth.test.ts       # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
    ├── phase02-sync.test.ts       # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
    ├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
    ├── phase05_sdk_test.py        # Python SDK WebSocket/HTTP fallback, offline spool, file streaming
    ├── phase06-pwa-landing.test.ts # PWA manifest/icons/sw + navigation routes
    └── phase07-deployment.test.ts # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush

4. Active Features & Development Status

All planned hardening & deployment phases (Phase-01 through Phase-07) are Complete & Verified:

  1. Admin Authentication & CSRF Hardening (Completed — Phase-01)
    • Fail-closed secret enforcement in NODE_ENV=production (/vcon/setup bootstrap via DATASTACK_ADMIN_SETUP_KEY).
    • Password hashing via crypto.scrypt, session invalidation via DB-backed tokenVersion, composite IP+email login lockout (admin_login_attempts), and strict Origin/Referer vs Host/X-Forwarded-Host CSRF verification.
  2. Real-Time Dual-Runtime Sync (Completed — Phase-02)
    • Authenticated /internal/* bridge between Port 3000 and Port 4000 (DATASTACK_INTERNAL_SECRET, backward-compatible Coolify fallback).
    • Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
  3. Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)
    • Shared Redis queue (datastack:buffer:events) with distributed flush lock (NX PX) where LTRIM executes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects.
    • Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (GET /api/export).
  4. Admin Console UI & Data/Export Workspace (Completed — Phase-04)
    • Built with shared components/ui/ and components/layout/ primitives (0rem radius, zero shadows, Fluent/Segoe dark & light tokens).
  5. Production Python Client SDK (Completed — Phase-05)
    • Supports single events (send()/sync()), auto-splitting/rate-limit-paced batches (send_batch()/batch_sync()), and direct file ingestion (send_file() for .csv, .json, .jsonl, .ndjson with SHA-256 deterministic eventId).
    • Automatic config.datastack.py discovery, WebSocket primary transport -> Heartbeat HTTP fallback -> .datastack_spool/spool.jsonl offline spool & auto-drain.
  6. PWA & Landing Integrity (Completed — Phase-06)
    • Complete public/manifest.json, icons, public/sw.js, and root-relative navigation links.
  7. Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)
    • Redis AOF persistence (redisdata:/data), deterministic package-lock.json for npm ci, Nginx location = /health → ingestion:4000, /api/health container probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DB autoPruneDays safety guard in SettingsManager, DB JWT hydration in SDKManager, graceful SIGTERM/SIGINT flush in DataBufferEngine, and non-destructive ADD COLUMN IF NOT EXISTS DDL guards.
  • Current Forensic Audit Status (2026-11-18):
    • All prior notes in notes/ were cleared and replaced exclusively with notes/forensic-audit-report.md backed strictly by source-code citations and verbatim proofs (zero markdown extrapolation).
    • Baseline checks currently pass (24/24 TS tests, eslint . clean, 3/3 Python unit tests), while deep source inspection uncovered 27 verified defects/gaps (5 Critical, 11 High, 10 Medium, 1 Low) across Auth/Middleware (A01–A05), Demo-vs-Prod Split-Brain (B01–B05), Buffer/CSV Storage (C01–C06), Admin UI vs Backend Reality (D01–D07), and SDK/Packaging/PWA (E01–E04).
    • Next step pending user approval: organize remediation into an incremental phased plan based strictly on notes/forensic-audit-report.md.

5. Known Constraints & Operational Notes for Future Contributors

  • Never add per-row telemetry INSERT queries to PostgreSQL or add DROP/TRUNCATE auto-migrations.
  • Never move Socket.IO handlers into Next.js app/api/* routes.
  • When deploying updates to an existing Coolify VPS, always preserve existing SERVICE_* environment variables and avoid docker compose down -v so pgdata, datastack-data, and redisdata volumes remain intact.
  • After any code change, verify with npm run build, npm run lint, the TS tests/phase*.test.ts suite (npx --yes tsx --test), and python3 -m unittest tests/phase05_sdk_test.py, then synchronize this MEMORY.md.