Files
DataStack/MEMORY.md
T
Md Najmul 725d3c5096 feat: add MCP server configuration management
Integrate MCP server settings into the admin console, including token management, tool permissions, and Coolify build log configuration. Update UI components to support standardized action groups.
2026-11-21 11:26:01 -08:00

89 KiB
Raw Blame History

MEMORY.md — DataStack Cross-IDE Persistent Project Memory

Last Synchronized: 2026-11-21
Status: R5-MCP (BASELINE-2026-11-21-R5-MCP: 3 / 3 Phases Complete — R5-Phase-03 Complete & Verified; All R5-MCP, R4 4/4 & PREPUSH-FIX-01 5/5 Complete).
Scope Rule: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see .ai-rules and AGENTS.md).


1. Project Identity & Core Principles

  • Project: DataStack — Self-hosted, real-time telemetry ingestion, admin governance (/vcon/*), and partitioned CSV storage/export platform.
  • Runtime Split (CRITICAL):
    • Port 3000 (web): Next.js 15 App Router — public landing (/), PWA lifecycle, /vcon/* Admin Console, REST ingestion (POST /api/events), metrics/settings/client APIs, and streaming exports (GET /api/export).
    • Port 4000 (ingestion): Dedicated Node.js Express 5 + Socket.IO server (server/index.js) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker.
    • Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port 4000.
  • Storage & Database Boundary (CRITICAL):
    • PostgreSQL (pg + prisma/schema.prisma): Stores strictly control-plane metadata (admin_users, admin_login_attempts, client_sdks, deleted_clients, admin_settings). Never insert per-row telemetry events into PostgreSQL.
    • Partitioned CSV Storage (/storage/data/{clientId}/{YYYY-MM-DD}.csv): All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (fileRotationMb, e.g., -part2.csv), and retention pruning (autoPruneDays).
    • Redis (ioredis): Shared cross-process buffer queue (datastack:buffer:events), recent event cache (datastack:recent:events), and distributed flush lock (datastack:flush:lock). Persisted in Docker via --appendonly yes (redisdata:/data). Falls back gracefully to local RAM when REDIS_URL is omitted.

2. Tech Stack & Local Setup

Stack

  • Frontend / Web API: Next.js 15.4.9 (App Router, output: "standalone"), React 19.2.1, TypeScript 5.9.3, Tailwind CSS v4 (@tailwindcss/postcss), lucide-react, motion.
  • Ingestion Server: Node.js 22 / Express 5.3.0, Socket.IO 4.8.4.
  • Persistence & Caching: pg 8.23.1 (PostgreSQL 16 in Docker), ioredis 6.0.0 (Redis 7 AOF in Docker), csv-parse 7.0.3, json2csv 6.0.0-alpha.2.
  • Auth & Security: jsonwebtoken 9.0.3, Node crypto.scrypt password hashing, WebCrypto HMAC-SHA256 admin session cookies (middleware.ts), CSRF Origin/Referer vs Host/X-Forwarded-Host checks, DATASTACK_INTERNAL_SECRET timing-safe bridge auth between Port 3000 ↔ 4000.
  • Client SDKs: Official pip-installable Python SDK in sdk/python/datastack/ (datastack) + backward-compatible wrapper sdk-python/ (datastack-sdk).

Environment & Setup Commands

  1. Configure Environment: Copy .env.example to .env (in production NODE_ENV=production, all 4 secrets MUST be 32+ chars: DATASTACK_ADMIN_SESSION_SECRET, DATASTACK_JWT_SECRET, DATASTACK_ADMIN_SETUP_KEY, DATASTACK_INTERNAL_SECRET; optional DATABASE_URL and REDIS_URL).
  2. Install Dependencies: npm ci (or npm install) using root package.json + package-lock.json.
  3. Run Locally (Two Terminals):
    • Terminal 1 (Port 3000): npm run dev
    • Terminal 2 (Port 4000): npm run server
  4. Build, Lint & Test Verification:
    • Build: npm run build
    • Lint: npm run lint (eslint .)
    • TS Test Suite (Phases 01–03, 06, 07): npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts
    • Python SDK Tests: python3 -m unittest tests/phase05_sdk_test.py

3. Folder Structure & File Architecture

datastack/
├── .ai-rules                      # Universal cross-IDE rules (enforces MEMORY.md sync)
├── AGENTS.md                      # Engineering rules, UI single-source-of-truth & memory sync
├── MEMORY.md                      # Persistent cross-IDE project memory (this file)
├── package.json / package-lock.json
├── middleware.ts                  # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
├── app/
│   ├── (landing)/page.tsx         # Public landing page
│   ├── globals.css                # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
│   ├── layout.tsx / not-found.tsx / error.tsx
│   ├── api/
│   │   ├── auth/                  # Admin login, logout, setup (/vcon/setup), profile update
│   │   ├── clients/               # CRUD clients, token regeneration, revocation
│   │   ├── data/                  # Paginated CSV search/query across partitions
│   │   ├── events/                # Authenticated HTTP ingestion (single/batch/file records) & recent list
│   │   ├── export/                # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
│   │   ├── files/status/          # Safe server-side Port 4000 Files Manager probe (catches ECONNREFUSED cleanly)
│   │   ├── health/                # Web health + client Bearer token probe + async buffer depth
│   │   ├── metrics/               # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
│   │   ├── mcp/                   # Owner-only Streamable HTTP MCP settings, token rotation/revocation, & permissions API
│   │   ├── settings/              # Allowlisted runtime config get/post + live Port 4000 sync
│   │   └── storage/               # CSV partition listing & manual cross-process buffer flush
│   └── vcon/                      # Protected Admin Console pages:
│       └── dashboard/ clients/ data/ export/ files/ login/ mcp/ profile/ sdk/ settings/ setup/ simulator/ storage/ team/ audit/
├── components/
│   ├── ui/                        # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, SettingsLayout, etc.)
│   ├── layout/                    # MainLayout, Navbar, Sidebar, VconShell, PageShell, AdminPageHeader, Footer
│   ├── sections/                  # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, McpWorkspace, SettingsWorkspace, etc.)
│   └── pwa/                       # PwaLifecycle.tsx service worker registration
├── lib/
│   ├── adminAuth.ts               # Uniform API session guard (requireAdminApiSession) + proxy-safe extractClientIp()
│   ├── mcpUiHelpers.ts            # Browser-safe MCP endpoint URL, AI client JSON snippet, preset & diff helpers
│   ├── vconCache.ts               # In-memory SWR cache + clearAuthCaches() for logout/SW cleanup
│   └── vconUiHelpers.ts           # Pure Admin UI sync helpers (settings bounds, bridge sync, UTC presets, timestamps)
├── core/
│   ├── adminSettings/             # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
│   ├── apiEngine/                 # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
│   ├── auth/                      # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
│   ├── data/                      # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
│   ├── Database/                  # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
│   ├── sdk/                       # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
│   ├── Security/                  # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
│   └── Storage/                   # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
├── scripts/
│   ├── dev-all.js                 # Local concurrent Next.js (3000) + Ingestion (4000) dev runner
│   └── start-all.js               # Production concurrent Next.js (`next start`) + Ingestion (`server/index.js`) runner
├── server/
│   ├── index.js                   # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, filesManager mount
│   ├── filesManager.js            # Hard-locked cloudcmd@19.23.2 (/vcon/files-engine), isolated Socket.IO, Admin Cookie/DB/RBAC/CSRF/*.lock guard
│   ├── socket/handlers.js         # Handshake JWT auth, ping_verify, sync_data, batch_sync
│   └── workers/csvWriter.js       # Partitioned CSV batch writer & rotation/pruning worker
├── sdk/python/datastack/          # Official pip-installable Python SDK (`datastack` v2.2.0)
│   ├── datastack/                 # client.py, config.py, heartbeat.py, spool.py, exceptions.py
│   ├── config.datastack.example.py
│   ├── pyproject.toml / setup.py
│   ├── README.md                  # SDK overview, installation, quickstart, docs index
│   └── docs/                      # Production SDK Documentation Suite (LicenX Standard):
│       ├── README.md              # SDK Documentation Hub & architecture overview
│       ├── INSTALLATION.md        # Python runtimes, pip commands, and verification
│       ├── QUICKSTART.md          # 5-minute zero-boilerplate onboarding guide
│       ├── API_REFERENCE.md       # Complete DataStack class & method specifications
│       ├── CONFIGURATION.md       # config.datastack.py & DATASTACK_* environment reference
│       ├── EXAMPLES.md            # IoT streaming, FastAPI, batching, and file ingestion
│       ├── ERROR_HANDLING.md      # Exception hierarchy, 401/403/413/429 codes & retries
│       └── LIFECYCLE.md           # 5-state machine, zero-CPU IDLE mode & offline spooling
├── sdk-python/                    # Backward-compatible re-export package (`datastack-sdk`)
├── docker/
│   ├── docker-compose.yml         # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
│   ├── Dockerfile.web             # Multi-stage standalone Next.js build + safe /app/storage ownership check
│   ├── Dockerfile.server          # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
│   ├── Dockerfile.proxy           # Nginx reverse proxy container
│   └── coolify-nginx.conf         # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
├── nginx/default.conf             # Local Nginx proxy config matching coolify-nginx.conf
├── prisma/schema.prisma           # Reference PostgreSQL metadata schema
├── public/                        # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
├── docs/                          # Comprehensive Production Documentation Suite:
│   ├── index.md                   # Primary documentation hub and portal navigation index
│   ├── MANIFEST.md                # Master documentation inventory & authoring governance
│   ├── audit-coverage-report.md   # Forensic codebase audit matrix & coverage report
│   ├── users/                     # End-User & Operator Guides:
│   │   ├── index.md               # User portal overview & reading pathways
│   │   ├── getting-started.md     # 5-minute fast-track operator onboarding guide
│   │   ├── installation.md        # Production Docker Compose, Coolify & volume guide
│   │   ├── configuration.md       # Environment variables & platform runtime settings
│   │   ├── features/              # Deep-dive guides: admin-console, client-management,
│   │   │                          # data-simulator, partitioned-storage, data-export
│   │   ├── workflows.md           # End-to-end operator journeys & disaster recovery
│   │   ├── troubleshooting.md     # Forensic error recovery, status codes & diagnostics
│   │   └── faq.md                 # Storage architecture & operational questions
│   ├── dev/                       # Developer & Contributor Guides:
│   │   ├── index.md               # Developer hub index & architectural principles
│   │   ├── architecture.md        # Dual-runtime blueprint, service bridge & storage
│   │   ├── codebase-structure.md  # File-by-file module, route & component inventory
│   │   ├── local-setup.md         # Local workstation setup & dual-terminal workflow
│   │   ├── api-reference.md       # Exhaustive REST API & Socket.IO specification
│   │   ├── database-schema.md     # PostgreSQL control plane & DDL migrations
│   │   ├── security.md            # Security architecture, scrypt, sessions & CSRF
│   │   ├── building-packaging.md  # Next.js standalone build & multi-stage Dockerfiles
│   │   ├── testing.md             # Automated test suites guide (node:test + unittest)
│   │   └── contributing.md        # Architectural rules, memory sync & PR checklist
│   └── reference/                 # Technical Reference Portal:
│       ├── index.md               # Technical reference index
│       ├── configuration-reference.md # Complete environment variable & setting bounds
│       ├── dependencies.md        # Runtime & dev dependencies audit with rationales
│       └── glossary.md            # Technical domain terminology & concept glossary
├── notes/                         # `R4` Production Hardening Governance Suite (`BASELINE-2026-11-20-R4`):
│   ├── README.md                  # Master governance index, 6-step zero-drift protocol & approval triggers (`R4-Phase-01`..`04`)
│   ├── FULL_PROJECT_FORENSIC_AUDIT.md # 100% raw-source-verified deep scan (20 findings: 3 Critical, 6 High, 7 Medium, 4 Low)
│   ├── BASELINE_FREEZE.md         # Frozen baseline snapshot (`BASELINE-2026-11-20-R4`) & 1:1 audit alignment matrix
│   ├── PRODUCTION_READINESS_SCOPE_LOCK.md # (Doc 001) All 20 defect fix contracts & exact per-phase Scope Locks
│   ├── PRODUCTION_ROADMAP.md      # (Doc 002) Dependency-ordered A–Z 4-phase roadmap & verification gates
│   ├── PHASE_EXECUTION_LOG.md     # (Doc 003) Live phase completion tracker (`4 / 4` complete) & changelog template
│   ├── ERROR_HANDLING_MATRIX.md   # (Doc 004) Verified existing error contracts + 10 planned `R4` contracts (`ERR-R4-01`..`10`)
│   └── ACTUAL_IMPLEMENTATION_STATUS.md # Ground-truth working vs phase-delivered vs pending ledger (Demo & Production)
└── tests/                         # Automated verification suites:
    ├── files-manager.test.ts      # CloudCmd RCE lockdown, CRUD/ZIP/UNZIP, RBAC, CSRF, *.lock, DB revocation, proxy & UI wiring
    ├── phase01-auth.test.ts       # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
    ├── phase02-sync.test.ts       # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
    ├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
    ├── phase04-ui-sync.test.ts      # UI/API settings bounds, bridge status, UTC presets, timestamp & folder matching
    ├── phase05-packaging.test.ts    # Python/Node packaging parity, PEP 561 py.typed, Node engines & start-all exit
    ├── phase05_sdk_test.py          # Python SDK WebSocket/HTTP fallback, offline spool, file streaming, edge hardening
    ├── phase06-pwa-landing.test.ts  # PWA manifest/icons/sw + navigation routes
    └── phase07-deployment.test.ts   # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush

4. Active Features & Development Status

All planned hardening & deployment phases (Phase-01 through Phase-07) are Complete & Verified:

  1. Admin Authentication & CSRF Hardening (Completed — Phase-01)
    • Fail-closed secret enforcement in NODE_ENV=production (/vcon/setup bootstrap via DATASTACK_ADMIN_SETUP_KEY).
    • Password hashing via crypto.scrypt, session invalidation via DB-backed tokenVersion, composite IP+email login lockout (admin_login_attempts), and strict Origin/Referer vs Host/X-Forwarded-Host CSRF verification.
  2. Real-Time Dual-Runtime Sync (Completed — Phase-02)
    • Authenticated /internal/* bridge between Port 3000 and Port 4000 (DATASTACK_INTERNAL_SECRET, backward-compatible Coolify fallback).
    • Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
  3. Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)
    • Shared Redis queue (datastack:buffer:events) with distributed flush lock (NX PX) where LTRIM executes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects.
    • Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (GET /api/export).
  4. Admin Console UI & Data/Export Workspace (Completed — Phase-04)
    • Built with shared components/ui/ and components/layout/ primitives (0rem radius, zero shadows, Fluent/Segoe dark & light tokens).
  5. Production Python Client SDK (Completed — Phase-05)
    • Supports single events (send()/sync()), auto-splitting/rate-limit-paced batches (send_batch()/batch_sync()), and direct file ingestion (send_file() for .csv, .json, .jsonl, .ndjson with SHA-256 deterministic eventId).
    • Automatic config.datastack.py discovery, WebSocket primary transport -> Heartbeat HTTP fallback -> .datastack_spool/spool.jsonl offline spool & auto-drain.
  6. PWA & Landing Integrity (Completed — Phase-06)
    • Complete public/manifest.json, icons, public/sw.js, and root-relative navigation links.
  7. Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)
    • Redis AOF persistence (redisdata:/data), deterministic package-lock.json for npm ci, Nginx location = /health → ingestion:4000, /api/health container probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DB autoPruneDays safety guard in SettingsManager, DB JWT hydration in SDKManager, graceful SIGTERM/SIGINT flush in DataBufferEngine, and non-destructive ADD COLUMN IF NOT EXISTS DDL guards.
  • Current Remediation Status (2026-11-18):
    • Phase-01 Complete & Verified (BUG-A01, BUG-A02, BUG-A03, BUG-A04, BUG-A05, BUG-E02):
      • Removed spoofable header/UA CSRF bypasses in middleware.ts and rejected sec-fetch-site: cross-site.
      • Enforced DB-backed tokenVersion session validation across /vcon/* layout (app/vcon/layout.tsx + components/layout/VconShell.tsx) and read/export APIs (/api/data, GET /api/events, /api/export, /api/metrics, /api/storage).
      • Prevented transient DB errors from triggering false /vcon/setup redirects and prevented swallowed NEXT_REDIRECT errors on /, /vcon/login, and /vcon/setup.
      • Standardized SDKManager.normalizeClientId, blocked deleted/unregistered clients on /api/health and Admin POST /api/events, hardened POST /api/clients & PATCH /api/clients/[id] input validation, and excluded /vcon/* from Service Worker caching with CLEAR_AUTH_CACHE on sign-out.
      • Verified: 26/26 TS tests, 3/3 Python tests, npm run build clean, eslint . clean, Playwright browser auth/logout E2E clean.
    • Phase-02 Complete & Verified (BUG-B01–BUG-B05):
      • Enforced fail-closed client verification in server/index.js (verifyClientInDb) backed by PG, deletion tombstones, Redis datastack:clients:* mirrors, and normalized clientId keys across both runtimes.
      • Added cross-runtime Redis idempotency (datastack:idem:*) and fixed-window rate limiting (datastack:rl:*) in core/Security/security.ts (validateEventAsync) and server/socket/handlers.js (validateSingleEventAsync) with automatic local RAM fallback.
      • Surfaced bridge sync telemetry (BridgeSyncResult, SettingsBridgeSyncStatus, runtime.ingestionBridgeStatus, POST /api/settings bridgeSync) and normalized SDKManager.incrementEventCountAsync().
      • Fixed Socket sync_data flat payload preservation (extractSingleEventData), reconciled PORT_SOCKET/PORT precedence and live /internal/metrics ports, and added concurrent production scripts/start-all.js for npm start.
      • Verified: 29/29 TS tests, 3/3 Python tests, npm run build clean, eslint . clean, Playwright browser check (/vcon/dashboard, /vcon/settings, /api/metrics) clean with 0 console errors.
    • Phase-03 Complete & Verified (BUG-C01–BUG-C06):
      • Hardened Redis flush lock TTL (30000ms), added in-process activeFlushPromise coalescing with a flushDone identity guard (core/data/buffer.ts, server/index.js), isolated redis.rpush from best-effort REDIS_RECENT_KEY, and logged malformed buffered JSON warnings.
      • Enforced sanitizeClientId() path containment and lstat symlink exclusion, unique lock owner tokens (${pid}:${ts}:${uuid}), atomic rename stale-lock takeover, owner-verified lock unlock, and structured JSON.stringify([clientId, date]) partition grouping in core/Storage/csvStorage.ts and server/workers/csvWriter.js.
      • Fixed trailing newline blank-row prevention (fileEndsWithNewline), proactive size-boundary rotation (-partN.csv), StringDecoder('utf8') streaming row counting, cooperatively locked pruneExpiredPartitions, and 1-hour PRUNE_COOLDOWN_MS in core/adminSettings/settingsManager.ts.
      • Enforced parser/fileStream destruction in finally blocks across /api/data, GET /api/events, and /api/export, error propagation inside makeExportStream, 502 surfacing on Port 4000 flush failure in /api/export, normalized clientId filtering, numeric timestamp sorting (parseTimestampMs), and ?format=json / ?includeHeader=false parity for ?filename= exports.
      • Verified: 32/32 TS tests, 3/3 Python tests, npm run build clean, eslint . clean, Playwright browser check (/vcon/dashboard, /vcon/data, /vcon/storage, /vcon/export, /api/data, /api/storage, /api/export) clean with 0 console errors.
    • Phase-04 Complete & Verified (BUG-D01–BUG-D07):
      • Extracted pure UI synchronization helpers in lib/vconUiHelpers.ts (SETTINGS_LIMITS, normalizeEditableSettings, describeSettingsBridgeSync, mapBridgeStatusToSocketUiState, parseEventTimestamp, formatEventTimeLabel, getUtcDateRangeForPreset, matchesSelectedClientFolder, extractContentDispositionFilename).
      • Updated components/sections/SettingsWorkspace.tsx (BUG-D01) to probe /api/metrics honestly (removing false /api/health fallback), surface bridgeSync warning/error toasts on save, align integer bounds with /api/settings (1..100000, 1..10240, 0..3650), and eliminate the startup serverPort double-fetch.
      • Updated app/vcon/storage/page.tsx (BUG-D02) and components/sections/ExportWorkspace.tsx (BUG-D03) with cache: 'no-store', 401 redirects, UTC preset ranges, normalized client folder filters, and fetch+Blob downloads with 502/JSON error surfacing.
      • Updated components/sections/DashboardOverview.tsx (BUG-D04) and components/sections/DataBrowser.tsx (BUG-D05) with ingestionBridgeStatus display, auth_error alert banner, safe ISO/epoch-ms timestamp formatting, invalid date-range guard, and Enter-key filter submission.
      • Updated components/sections/NewClientModal.tsx, components/sections/ClientsTable.tsx, app/vcon/simulator/page.tsx, app/vcon/sdk/page.tsx, components/sections/ProfileWorkspace.tsx, and components/layout/Sidebar.tsx (BUG-D06, BUG-D07) with 1..100000 rate limit parity, live runtime.serverPort sync, real HTTP batch ingestion in Simulator, password bounds check, and datastack:profile-updated live Sidebar sync.
      • Verified: 37/37 TS tests (tests/phase04-ui-sync.test.ts), 3/3 Python tests, npm run build clean, eslint . clean, Playwright browser check across all Admin Console routes clean with 0 console errors.
    • Phase-05 Complete & Verified (BUG-E01, BUG-E03, BUG-E04):
      • Hardened canonical Python SDK (sdk/python/datastack/datastack/*): normalized CLIENT_ID comparison (_normalize_client_id), sanitized ragged CSV extra columns (restkey="_extra"), accurate send_file() success + unprocessedItems, partial-batch rateLimitedEventIds retry exception fallback, DataStackConfigError wrapping for malformed numeric env/config values (config.py), concurrent stop() vs set_idle_connected() race fix + clean thread join (heartbeat.py), os.fsync + orphan .tmp unlink + warning logs (spool.py), and PEP 561 py.typed marker.
      • Aligned packaging metadata across sdk/python/datastack/pyproject.toml, sdk/python/datastack/setup.py, and sdk-python/setup.py, coordinated child-exit shutdown in scripts/start-all.js, and added engines.node >=20.0.0 in package.json.
      • Verified: 39/39 TS tests (tests/phase05-packaging.test.ts), 6/6 Python tests (tests/phase05_sdk_test.py), py_compile, npm run build, and eslint . clean.
    • Previous Remediation Cycle (2026-11-18): Completed Phases 01–05 (39/39 TS tests, 6/6 Python tests).
    • New Production Hardening Cycle (BASELINE-2026-11-19):
      • Re-audited entire codebase from raw source code (notes/FORENSIC_AUDIT_REPORT.md): 29 verified bugs/mismatches (4 Critical, 8 High, 13 Medium, 4 Low).

      • Phase-01 Complete & Verified (2026-11-19) (BUG-01, BUG-04, BUG-05, BUG-06):

        • Blocked volatile RAM admin creation in production without DATABASE_URL (AdminDatabaseRequiredError), surfaced production DB warning on /vcon/setup, and mapped pre-setup login to 409 SETUP_REQUIRED.
        • Added brute-force attempt throttling (429 + Retry-After) and completion-first ordering to POST /api/auth/setup.
        • Added server-side token_version revocation (revokeAdminSessions()) on POST /api/auth/logout, removed client Origin trust from cookie Secure flag, and added sec-fetch-site: same-origin CSRF fallback.
        • Prevented empty production jwt_secret persistence in PG and replaced silent DB write catch {} swallows in SettingsManager and SDKManager with DATABASE_WRITE_FAILED propagation and safe RAM rollback.
        • Verified: 43/43 TS tests, 6/6 Python tests, npm run build, eslint ., and Playwright login/logout revocation check clean.
      • Phase-02 Complete & Verified (2026-11-19) (BUG-02, BUG-03, BUG-17, BUG-22):

        • Protected Port 4000's inMemoryClientRegistry from empty Port 3000 bulk_sync wipes (!forceReplace), exposed inMemoryClients on /internal/metrics, added non-destructive hydration in SDKManager (listClientsAsync, getClientAsync, syncAllClientsToIngestionAsync), and auto-reconciled Port 4000 cold starts in ApiEngine.getDashboardMetrics().
        • Ordered SettingsManager.updateSettingsAsync() JWT rotation to sync Port 4000 before committing Port 3000 RAM/PG/re-sign, aborting fail-closed (JWT_ROTATION_BRIDGE_FAILED) on bridge rejection/prod unreachable, rolling back Port 4000 on PG/re-sign error, and adding a 60s dual-secret grace window in server/index.js requireInternalAuth().
        • Added authenticated POST /internal/security-check on Port 4000 (backed by zero-copy validateSingleEventLocal) so Port 3000 HTTP SecurityEngine.validateEventAsync() shares rate limits & idempotency with Port 4000 WS even without Redis, and exposed explicit securityMode (redis_shared | bridge_coordinated | degraded_local) in /api/metrics.
        • Replaced per-event PG UPDATE client_sdks queries on both Port 3000 (SDKManager) and Port 4000 (server/index.js) with 1s coalescing in-memory deltas flushed via batched UNNEST SQL updates (flushed on reads and graceful shutdown).
        • Verified: 47/47 TS tests, 6/6 Python tests, npm run build, and eslint . clean.
      • Phase-03 Complete & Verified (2026-11-19) (BUG-18, BUG-19, BUG-20, BUG-29):

        • Added bounded Redis flush lock wait/retry, periodic Lua PEXPIRE TTL renewal with ownership fencing before LTRIM, and explicit FLUSH_LOCK_TIMEOUT (503 on /internal/flush) across both core/data/buffer.ts and server/index.js.
        • Added full deleted-client buffer purge (DataBufferEngine.purgeClientEventsAsync, Port 4000 purgeClientFromLocalBuffer) plus flush-time and recent-read tombstone filtering so deleted client directories never resurrect on subsequent flushes; ordered SDKManager.deleteClientAsync() to purge buffers and sync Port 4000 before deleting disk files.
        • Converted server/index.js onBufferPush to async and awaited it before firing Socket.IO ACK callbacks in server/socket/handlers.js (sync_data and batch_sync), returning BUFFER_PERSIST_FAILED on persistence error.
        • Unified canonical CSV storage root resolution via DATASTACK_STORAGE_DIR (resolveDefaultStorageDir()) across core/Storage/csvStorage.ts, server/workers/csvWriter.js, core/sdk/sdkManager.ts, and .env.example.
        • Verified: 51/51 TS tests, 6/6 Python tests, npm run build, and eslint . clean.
      • Phase-04 Complete & Verified (2026-11-19) (BUG-21, BUG-23, BUG-24):

        • Aligned HTTP POST /api/events single-event envelope stripping 1:1 with Port 4000 WS sync_data and eliminated fabricated { sample, rand } telemetry on empty {}.
        • Converted GET /api/data directory/stat scans to non-blocking async node:fs/promises with newest-first date-partition pruning, and scaled REDIS_RECENT_LIMIT boundedly (500..5000) via resolveRedisRecentLimit() / DATASTACK_RECENT_LIMIT across Port 3000 & Port 4000.
        • Replaced all Math.random() client secrets (crypto.randomBytes(16)) and fallback eventIds (crypto.randomBytes(6)) with CSPRNG across core/sdk/sdkManager.ts, app/api/events/route.ts, and server/socket/handlers.js.
        • Unified canonical PostgreSQL metadata DDL (CORE_SCHEMA_TABLE_DDLS + CORE_SCHEMA_ALTER_DDLS + ensureCoreSchemaOnPool()) in core/Database/database.ts, delegated core/auth/adminUsers.ts to it, mirrored all 5 tables + 3 additive ALTER TABLE guards 1:1 in standalone server/index.js, and documented canonical pg ownership in prisma/schema.prisma.
        • Verified: 55/55 TS tests (tests/phase04-ingestion-hygiene.test.ts added), 6/6 Python tests, npm run build, and eslint . clean.
      • Phase-05 Complete & Verified (2026-11-19) (BUG-07, BUG-08, BUG-09, BUG-14):

        • Fixed Simulator HTTP Burst 10 to send canonical { clientId, batch } (with events batch alias & explicit accepted count in POST /api/events) and summarize exact server-confirmed counts via summarizeHttpBatchResult() without res.ok ? count : 0 falsification.
        • Fixed parseEventTimestamp() & added getEventUtcDate() in lib/vconUiHelpers.ts for integer and fractional epoch-seconds (< 1e11 * 1000) 1:1 with backend CSV parsers.
        • Added pending-queue-only getPendingBufferedSummary() (?bufferedOnly=true / ?includeBufferedSummary=true) and fair multi-client partition scanning (selectRecentPartitionFiles(), lifting the global 5-file cap) in GET /api/events.
        • Updated ExportWorkspace.tsx to match buffered events strictly against active client/date filters via countMatchingBufferedEvents() (preventing empty filtered header downloads) and refresh /api/storage, /api/metrics, and vconCache immediately post-export; updated DashboardOverview.tsx to attribute strictly UTC-today buffered events to Events today and call await refresh() after Quick Export.
        • Verified: 58/58 TS tests, 6/6 Python tests, npm run build, eslint ., and Playwright E2E browser check (/vcon/simulator, /vcon/export, /vcon/dashboard, 0 console errors/warnings) clean.
      • Phase-06 Complete & Verified (2026-11-19) (BUG-10, BUG-11, BUG-12, BUG-13, BUG-15, BUG-16):

        • Returned explicit SESSION_EXPIRED / AUTH_UNAVAILABLE vs INVALID_PASSWORD codes in /api/auth/profile, replaced dead 401 && !currentPassword redirect with shouldRedirectProfileUnauthorized(), and formatted Session Expires with calendar date + 24h time (formatSessionExpiryLabel()).
        • Added opt-in ?purgeFiles=true checkbox in ClientsTable.tsx client deletion modal and removed the misleading unused Secret key row from CredentialsDisplay.
        • Refactored SettingsWorkspace.tsx numeric inputs to string drafts (NumericSettingDrafts) validated inline via validateEditableSettingsDraft() (noValidate submit + tab jump to first invalid field, zero snap-to-0, zero silent clamping) and clarified Socket port (PORT_SOCKET) as read-only runtime config.
        • Unified public endpoint resolution (resolvePublicEndpoints()) across SettingsWorkspace, /vcon/sdk, and /vcon/simulator (consistent on port-80 localhost proxies) and updated SdkSection.tsx + /vcon/sdk with external path/wheel pip install guidance.
        • Added pre-hydration datastack-theme bootstrap script + suppressHydrationWarning in app/layout.tsx (zero light-mode FOUC) and theme restoration/toggle in AdminSetup.tsx.
        • Upgraded public/sw.js (datastack-cache-v3) with an uncached 503 HTML offline navigation fallback shell (buildOfflineShellResponse(), Cache-Control: no-store) for /vcon/dashboard and public navigations.
        • Verified: 64/64 TS tests, 6/6 Python tests, npm run build, eslint ., and Playwright Chromium E2E check (/vcon/profile, /vcon/clients, /vcon/settings, /vcon/sdk, Light Mode reload, 0 console errors/warnings) clean.
      • Fresh Post-Phase-07 Forensic Audit & Baseline Freeze (BASELINE-2026-11-19-R2):

        • Previous Phase-07 hardening (BUG-25–BUG-28) landed in Python SDK (client.py), legacy sdk-python, scripts/start-all.js / dev-all.js, nginx/default.conf, server/index.js, docker/docker-compose.yml, and .env.example.
        • Conducted a brand-new 100% raw-source-code audit (notes/FORENSIC_AUDIT_REPORT.md) identifying 18 verified remaining bugs (0 Critical, 6 High, 8 Medium, 4 Low).
        • Frozen baseline (BASELINE-2026-11-19-R2) and created the complete 7-document governance & Scope-Lock suite in notes/ (README.md, BASELINE_FREEZE.md, PRODUCTION_READINESS_SCOPE_LOCK.md, PRODUCTION_ROADMAP.md, PHASE_EXECUTION_LOG.md, ERROR_HANDLING_MATRIX.md, ACTUAL_IMPLEMENTATION_STATUS.md).
        • R2 Phase-01 Complete & Verified (2026-11-19) (BUG-01, BUG-02):
          • Moved same-origin CSRF enforcement in middleware.ts ahead of the public-route early return so POST /api/auth/login and POST /api/auth/setup reject cross-origin/missing-origin requests with 403 while preserving same-origin browser submissions, sec-fetch-site: same-origin fallback, safe GET routes, and Bearer POST /api/events.
          • Extracted shared reverse-proxy-safe extractClientIp() in lib/adminAuth.ts (trusting x-real-ip first, then rightmost non-empty x-forwarded-for hop) and wired it into both app/api/auth/login/route.ts and app/api/auth/setup/route.ts to block spoofed-XFF brute-force lockout bypasses.
          • Verified: 14/14 tests/phase01-auth.test.ts pass, npm run build, npm run lint, live HTTP 403/401 CSRF probes, and Chromium /vcon/login → /vcon/dashboard E2E (0 errors).
        • R2 Phase-02 Complete & Verified (2026-11-19) (BUG-05, BUG-06, BUG-07, BUG-08, BUG-09, BUG-10):
          • Attached .on('error', ...) listeners to DatabaseEngine.getPool() (max: 10) and server/index.js dbPool, and consolidated core/auth/adminUsers.ts onto the shared DatabaseEngine.getPool().
          • Replaced non-atomic Redis LTRIM+LPUSH client deletion purging in core/data/buffer.ts with atomic Lua script PURGE_CLIENT_LIST_LUA_SCRIPT (cjson.decode + chunked RPUSH), and replaced non-atomic Redis INCR+PEXPIRE rate limiting with atomic Lua script REDIS_RATE_LIMIT_LUA_SCRIPT across both core/Security/security.ts and server/socket/handlers.js.
          • Enforced throwOnLockTimeout: true locally and remotely on POST /internal/flush in GET /api/export and POST /api/storage, surfacing explicit 503 FLUSH_LOCK_TIMEOUT.
          • Added DataBufferEngine.registerShutdownHook() and registered SDKManager.flushPendingClientActivityAsync() so Port 3000 flushes coalesced total_events deltas on SIGTERM/SIGINT.
          • Wrapped GET, PATCH (regenerate_token & update), and DELETE in app/api/clients/[id]/route.ts with structured JSON try/catch error responses.
          • Verified: 14/14 tests/phase02-sync.test.ts pass, 3/3 R2 PHASE-02 buffer/storage tests pass, npm run build pass, npm run lint pass (0 errors).
        • R2 Phase-03 Complete & Verified (2026-11-19) (BUG-03, BUG-04, BUG-11, BUG-12, BUG-13, BUG-15):
          • Unified SettingsBridgeSyncStatus & POST /api/settings bridgeSync response (attempted, ok, synced, status, error, checkedAt) and updated describeSettingsBridgeSync() + SettingsWorkspace.tsx so Port 4000 warnings/online state work reliably.
          • Counted regular non-symlink .csv files in SDKManager.purgeClientStorageDir() (deleteClientWithStatsAsync()) and returned exact purgedFiles from DELETE /api/clients/[id]?purgeFiles=true.
          • Aligned flat batch item metadata stripping (eventId, timestamp, clientId) 1:1 with single-event stripping across HTTP POST /api/events, WS batch_sync, and Python SDK send_batch()/send_file().
          • Exposed DataBufferEngine.getLocalBufferSnapshot() for exact Port 3000 RAM pending summary in getPendingBufferedSummary().
          • Added non-blocking pruneExpiredPartitionsAsync() off live event loops in core/Storage/csvStorage.ts & server/workers/csvWriter.js (wired into settingsManager.ts & server/index.js) and deduplicated GET /api/storage to a single listAllFilesAsync() scan with accurate client-scoped stats.
          • Cleared selectedClientToken on /vcon/sdk client switch (with loading placeholder + disabled Copy Config), added 400 INVALID_DATE_RANGE for from > to in GET /api/export, and normalized clientId filtering in ApiEngine.getExportFiles* & Port 4000 /internal/recent.
          • Verified: 21/21 phase04-* tests pass, 2/2 R2 PHASE-03 storage tests pass, 9/9 Python SDK tests pass, Chromium E2E across /vcon/settings, /vcon/clients, /vcon/sdk, /vcon/storage (0 console errors), npm run build & npm run lint (0 errors).
        • R2 Phase-04 Complete & Verified (2026-11-19) (BUG-14, BUG-16, BUG-17, BUG-18):
          • Added from __future__ import annotations, imported Tuple, and replaced PEP 585 tuple[int, Dict[str, Any]] with Python 3.8-safe Tuple[int, Dict[str, Any]] in sdk/python/datastack/datastack/client.py, verified by a full-package Python 3.8 AST scan test in tests/phase05_sdk_test.py.
          • Added unbuffered streaming location = /api/export (proxy_buffering off;, proxy_read_timeout 300s;) targeting http://nextjs_upstream in both nginx/default.conf and docker/coolify-nginx.conf.
          • Removed eslint: { ignoreDuringBuilds: true } from next.config.ts, set package.json "name": "datastack", pruned unused template packages (@google/genai, firebase-tools), and synchronized package-lock.json.
          • Isolated DATASTACK_DISABLE_LISTEN=1 and live Port 4000 coupling across tests/phase01-auth.test.ts, tests/phase02-sync.test.ts, tests/phase03-storage-buffer.test.ts, tests/phase04-ingestion-hygiene.test.ts, and tests/phase07-deployment.test.ts.
          • Verified: 83/83 full TS suite pass (with live Port 4000 running, without manual env flags), 10/10 Python SDK tests + py_compile pass, npm run build (with ESLint active) pass, npm run lint (0 errors, 0 warnings).
        • Progress: 4 / 4 phases complete (18 / 18 bugs resolved — 100% Production Ready).
      • Pre-Release Docker & Coolify Deployment Hardening Complete & Verified (2026-11-19) (DEPLOY-BUG-01–05, audited in notes/DEPLOYMENT_COOLIFY_AUDIT_REPORT.md):

        • DEPLOY-BUG-01: Replaced static upstream blocks in docker/coolify-nginx.conf with Docker embedded DNS resolver 127.0.0.11 valid=10s ipv6=off; and variable proxy_pass targets ($nextjs_upstream = http://web:3000, $socketio_upstream = http://ingestion:4000), preventing 502 Bad Gateway stale container IP outages after container restarts/redeploys.
        • DEPLOY-BUG-02: Refactored Port 4000 startup in server/index.js into startIngestionServer() awaiting bounded syncSettingsFromDatabase() before server.listen(PORT) so rotated PG jwt_secret and rate-limit settings hydrate prior to /health: 200 and /internal/* bridge traffic.
        • DEPLOY-BUG-03: Memoized ensureDatabaseSchema() per active dbPool instance in server/index.js (resetting only on pool swap or DDL failure) so 3-second syncSettingsFromDatabase() ticks no longer issue 8 CREATE/ALTER TABLE DDLs repeatedly.
        • DEPLOY-BUG-04: Reordered gracefulShutdown() in server/index.js to close Socket.IO/HTTP ingress first (closeServerGracefully()), drain pendingRedisWrites and flushBuffer()/flushPendingClientActivity() with bounded timeouts (2000ms/6000ms), and close Redis (quit() with 2000ms timeout + .disconnect() fallback) and PostgreSQL (dbPool.end() bounded 2000ms); added stop_grace_period: 20s to both web and ingestion in docker/docker-compose.yml.
        • DEPLOY-BUG-05: Clarified Coolify Base Directory: / and Docker Compose Location: /docker/docker-compose.yml in docs/users/installation.md and README.md.
        • Verified: 85/85 full TS suite pass (11/11 in tests/phase07-deployment.test.ts), 10/10 Python SDK tests pass, npm run build clean, npm run lint (0 errors, 0 warnings).
      • V2 Baseline Freeze & Governance Planning Complete (BASELINE-2026-11-19-V2):

        • Froze the clean, Deployment-Hardened runtime state (85/85 TS tests, 10/10 Python tests, clean npm run build & eslint ., zero runtime code changes) as BASELINE-2026-11-19-V2.
        • Created raw-source verified V2 planning & contract documents in notes/:
          • notes/V2_FEATURE_GAP_AUDIT.md (V2-FEAT-01–V2-FEAT-07 gap analysis + full /vcon/* control mapping).
          • notes/V2_DATABASE_AND_SETTINGS_CONTRACT.md (zero-loss additive PostgreSQL DDL rules preserving admin_users id = 1 CHECK (id = 1), AES-256-GCM DB secret encryption + masked GET /api/settings, Port 3000↔4000 live sync).
          • notes/BASELINE_FREEZE.md, notes/PRODUCTION_READINESS_SCOPE_LOCK.md (Doc 001 Scope Locks for V2-Phase-01–04), notes/PRODUCTION_ROADMAP.md (Doc 002), notes/PHASE_EXECUTION_LOG.md (Doc 003, 0 / 4 V2 phases done), notes/ERROR_HANDLING_MATRIX.md (Doc 004, including V2-E01–V2-E12), notes/ACTUAL_IMPLEMENTATION_STATUS.md, and notes/README.md.
        • Roadmap grouped into 4 Scope-Locked V2 Phases (V2-Phase-01 Archival/Gzip + Quota/Disk Guard; V2-Phase-02 Webhooks/Alerts + IP Allowlist/Expiry + Prometheus/NDJSON; V2-Phase-03 Multi-Admin RBAC + Audit Log; V2-Phase-04 Node.js/TypeScript SDK + /vcon/sdk guide).
        • V2-Phase-01 Complete & Verified (2026-11-19) (V2-FEAT-01, V2-FEAT-03, V2-E01–V2-E05):
          • AES-256-GCM Secret Vault (core/Security/secretVault.ts): Deterministic HKDF-SHA256 key derivation (encryptSecret, decryptSecret, getMaskedHintFromEncrypted, SecretVaultError) so s3_secret_access_key_enc is encrypted at rest in PostgreSQL and never leaked in plaintext by GET /api/settings (hasS3SecretKey + s3SecretKeyMask: "••••xxxx").
          • S3 / Cloudflare R2 / MinIO SigV4 Client (core/Storage/s3Archive.ts): Pure Node node:crypto + fetch AWS Signature v4 client (testS3Connection, uploadFileToS3, buildS3ObjectKey, normalizeS3EndpointUrl) with POST /api/storage/s3-test and POST /api/storage/archive.
          • Transparent .csv.gz Auto-Compression & Safe Archive-Before-Prune (core/Storage/csvStorage.ts, server/workers/csvWriter.js): Cooperatively locked atomic .csv → .csv.gz compression (compressOldPartitionsAsync / compressOldPartitions) excluding today's active UTC partition; transparent .csv + .csv.gz reading/streaming across GET /api/data (surfacing 500 Corrupt .csv.gz partition on corrupt gzip — V2-E03), GET /api/events, and GET /api/export (application/gzip for raw single .csv.gz download); fail-safe beforeDeleteFile S3 upload hook (V2-E02) that never deletes local partitions if S3 upload fails (502 S3_ARCHIVE_FAILED).
          • Per-Client Storage Quota (storageQuotaMb) & VPS Disk Pause Guard (diskPauseThresholdPct): Additive PostgreSQL columns (client_sdks.storage_quota_mb + 9 admin_settings columns in core/Database/database.ts, synced in server/index.js while preserving ddlCount === 8), 30s TTL cached statfs + directory byte counters, and HTTP 507 Insufficient Storage (DISK_SPACE_LOW / STORAGE_QUOTA_EXCEEDED — V2-E04, V2-E05) on POST /api/events and Port 4000 sync_data/batch_sync while keeping all Admin read/export/settings routes online.
          • Admin UI Controls (SettingsWorkspace.tsx, NewClientModal.tsx, ClientsTable.tsx, lib/vconUiHelpers.ts): Added live VPS disk usage bar, Auto-compress (.csv → .csv.gz), Disk pause threshold (0 or 50–99%), full S3 / R2 off-site archive configuration with Test S3 Connection & Archive Now, and per-client Storage quota (MB, 0 = unlimited) in modal and table.
          • Verified: 89/89 TS tests (4/4 in tests/v2-phase01-storage-quotas.test.ts), 10/10 Python SDK tests, npm run build clean, npm run lint (0 errors / 0 warnings), and Chromium E2E across /vcon/settings (Storage tab) and /vcon/clients (0 console errors).
        • V2-Phase-02 Complete & Verified (2026-11-19) (V2-FEAT-02, V2-FEAT-04, V2-FEAT-06, V2-E06–V2-E09):
          • Per-Client IP / CIDR Allowlist & Token Expiration (core/Security/ipAllowlist.ts, core/sdk/sdkManager.ts, app/api/clients/**, app/api/events/route.ts, server/index.js, server/socket/handlers.js): Validates and normalizes comma-separated IPv4, IPv4 CIDR (/0..32), IPv6, IPv6 CIDR (/0..128), IPv4-mapped IPv6 (::ffff:), loopback equivalence (127.0.0.1 ↔ ::1), and wildcard *; validates ISO expiresAt timestamps. Enforced across both Port 3000 POST /api/events and Port 4000 Socket.IO handshakes/events (403 CLIENT_IP_FORBIDDEN — V2-E07; 401 CLIENT_TOKEN_EXPIRED — V2-E08), synced live via /internal/client-sync and additive PostgreSQL columns (client_sdks.allowed_ips, client_sdks.expires_at).
          • Outbound Webhook Alerting (core/alerts/webhookNotifier.ts, core/adminSettings/settingsManager.ts, app/api/settings/webhook-test/route.ts): Sends Slack (text), Discord (content), and structured JSON webhook payloads signed with optional HMAC-SHA256 (X-DataStack-Signature: sha256=<hex>) and deduplicated per event:clientId via configurable webhookCooldownSec (10..86400s, default 300s). Triggers non-blocking alerts on DISK_SPACE_LOW, STORAGE_QUOTA_EXCEEDED, S3_ARCHIVE_FAILED, CLIENT_IP_FORBIDDEN, and CLIENT_TOKEN_EXPIRED, plus on-demand POST /api/settings/webhook-test (200 on delivery, 502 WEBHOOK_DELIVERY_FAILED on upstream error — V2-E06). webhook_secret_enc is encrypted via AES-256-GCM Secret Vault and masked in GET /api/settings (hasWebhookSecret, webhookSecretMask).
          • Prometheus Scrape Endpoint (app/api/prometheus/route.ts): GET /api/prometheus outputs Prometheus 0.0.4 text exposition format (datastack_up, datastack_ingestion_bridge_up, datastack_active_sockets, datastack_events_total, datastack_buffer_events_total, datastack_storage_files_total, datastack_storage_bytes_total, datastack_storage_rows_total, datastack_disk_used_percent, datastack_disk_free_bytes, datastack_disk_total_bytes, datastack_clients_total, datastack_client_events_total, datastack_client_storage_bytes). Protected by constant-time SHA-256 Bearer token comparison (prometheus_metrics_token_enc encrypted at rest) or active Admin Console session cookie (401 PROMETHEUS_UNAUTHORIZED — V2-E09).
          • Streaming .ndjson Export (app/api/export/route.ts, components/sections/ExportWorkspace.tsx): Added format=ndjson (Content-Type: application/x-ndjson; charset=utf-8, .ndjson extension) streaming newline-delimited JSON objects across both .csv and .csv.gz partitions, with NDJSON format radio selector in /vcon/export.
          • Admin UI Controls (SettingsWorkspace.tsx, NewClientModal.tsx, ClientsTable.tsx, lib/vconUiHelpers.ts): Added Outbound webhook alerts and Prometheus metrics (/api/prometheus) groups under /vcon/settings → Security tab, and Allowed IPs / CIDR + Token expiration controls in /vcon/clients New/Edit modals and table badges.
          • Verified: 93/93 TS tests (4/4 in tests/v2-phase02-alerts-ip-prometheus.test.ts), 10/10 Python SDK tests, npm run build clean, npm run lint (0 errors / 0 warnings), and Chromium E2E across /vcon/settings (0 console errors).
        • V2-Phase-03 Complete & Verified (2026-11-19) (V2-FEAT-05, V2-E10–V2-E11):
          • Multi-Admin Role-Based Access Control (core/Database/database.ts, core/auth/adminSession.ts, core/auth/adminUsers.ts, lib/adminAuth.ts, app/api/team/route.ts, app/api/team/[id]/route.ts): Added canonical admin_team_members table in PostgreSQL (plus in-memory fallback) while preserving admin_users (id = 1 CHECK (id = 1)) as the immutable Primary Owner (owner-1, protected by 400 CANNOT_REVOKE_PRIMARY_OWNER — V2-E11). Supports roles owner (rank 3 — full platform settings, secrets & team management), admin / operator (rank 2 — manage SDK clients, rotate tokens, manual buffer flushes, S3 test/archive), and viewer (rank 1 — read-only dashboards, events, clients, settings & exports). Enforced centrally via requireAdminApiSession(req, { minRole }) returning 403 INSUFFICIENT_ROLE_PERMISSION (V2-E10). Role/status/password changes on any team member immediately bump token_version to revoke existing sessions.
          • Immutable Admin Audit Log (core/auth/auditLog.ts, app/api/audit/route.ts): Added canonical admin_audit_logs table in PostgreSQL (plus bounded 1,000-entry in-memory fallback) with automatic secret key redaction. Records governance events (ADMIN_LOGIN, ADMIN_PROFILE_UPDATED, ADMIN_PASSWORD_UPDATED, TEAM_MEMBER_CREATED, TEAM_MEMBER_UPDATED, TEAM_MEMBER_REVOKED, TEAM_MEMBER_DELETED, CLIENT_CREATED, CLIENT_UPDATED, CLIENT_TOKEN_ROTATED, CLIENT_DELETED, SETTINGS_UPDATED, S3_ARCHIVE_TRIGGERED, WEBHOOK_TEST_TRIGGERED) without ever inserting telemetry events into PostgreSQL. Exposed filterable GET /api/audit (?action=, ?actor=, ?q=, ?limit=).
          • Admin Console UI (components/sections/TeamWorkspace.tsx, components/sections/AuditLogViewer.tsx, app/vcon/team/page.tsx, app/vcon/audit/page.tsx, components/layout/Sidebar.tsx): Added /vcon/team (Team & RBAC directory, add member modal, role selector, session revocation & deletion) and /vcon/audit (filterable governance trail viewer) linked from Sidebar.tsx.
          • Verified: 97/97 TS tests (4/4 in tests/v2-phase03-rbac-audit.test.ts), 10/10 Python SDK tests, npm run build clean, npm run lint (0 errors / 0 warnings), and Chromium E2E across /vcon/team and /vcon/audit (0 console errors).
        • V2-Phase-04 Complete & Verified (2026-11-20) (V2-FEAT-07, V2-E12):
          • Official Node.js / TypeScript Client SDK (sdk/node/datastack/ — @datastack/sdk v2.2.0): Created sdk/node/datastack/package.json, types.ts, spool.ts, index.ts, and README.md. Supports datastack.config.json / DATASTACK_* env / constructor options with strict normalizeClientId vs JWT clientId/sub verification (DataStackConfigError — V2-E12), Socket.IO primary streaming (connect(), send()/sync(), sendBatch()/batchSync()), automatic HTTP POST /api/events fallback, recursive 413 PAYLOAD_TOO_LARGE batch splitting, partial-batch 429 RATE_LIMIT_EXCEEDED retry, typed terminal error mapping (401 CLIENT_TOKEN_EXPIRED & 403 CLIENT_IP_FORBIDDEN -> DataStackAuthError, 507 STORAGE_QUOTA_EXCEEDED & DISK_SPACE_LOW -> DataStackQuotaExceededError), durable .datastack_spool/spool.jsonl local disk spooling (DiskSpooler) & flushSpool(), direct .csv/.json/.jsonl/.ndjson file ingestion (sendFile() with deterministic SHA-256 eventId via makeDeterministicFileEventId), and graceful await client.shutdown(timeoutMs) drain.
          • Multi-Language SDK Guide (app/vcon/sdk/page.tsx): Added interactive language tabs (Python SDK (v2.2.0), Node.js / TypeScript SDK (v2.2.0), HTTP / cURL) with live client JWT token population and copy-ready config.datastack.py / datastack.config.json snippets.
          • Verified: 101/101 TS tests (4/4 in tests/v2-phase04-node-sdk.test.ts), 10/10 Python SDK tests, npm run build clean, and npm run lint (0 errors / 0 warnings).
        • Post-V2 Full-Codebase Forensic Audit & Baseline Freeze (BASELINE-2026-11-20-V2-R2):
          • Conducted a 100% raw-source-code forensic audit across all Port 3000 APIs/engines, Port 4000 Socket.IO workers, /vcon/* UI pages, and Python/Node SDKs (notes/FORENSIC_AUDIT_REPORT.md), identifying 10 verified bugs (BUG-V2-AUDIT-01 .. BUG-V2-AUDIT-10: 5 High, 4 Medium, 1 Low).
          • Froze the current verified codebase (102/102 TS tests, 10/10 Python tests, 0 build/lint errors, 0 source code changes) as BASELINE-2026-11-20-V2-R2 in notes/BASELINE_FREEZE.md.
          • Prepared the complete 7-document governance & Scope-Lock suite in notes/:
            • notes/FORENSIC_AUDIT_REPORT.md (10 verified bugs with verbatim code blocks & line numbers)
            • notes/BASELINE_FREEZE.md (1:1 baseline-to-audit mapping)
            • notes/PRODUCTION_READINESS_SCOPE_LOCK.md (Doc 001 — exact fix contracts & strict per-phase Scope Locks)
            • notes/PRODUCTION_ROADMAP.md (Doc 002 — A–Z 3-phase execution roadmap: V2-Audit-Phase-01 Auth/Email/Audit, V2-Audit-Phase-02 Port 4000 Webhooks/IPv6 CIDR/Health/Python SDK, V2-Audit-Phase-03 UI Disk Usage/Client Expiry/CSV.GZ Badges)
            • notes/PHASE_EXECUTION_LOG.md (Doc 003 — 3 / 3 phases completed, 0 / 3 remaining, 10 / 10 bugs resolved)
            • notes/ERROR_HANDLING_MATRIX.md (Doc 004 — active error catalog + ERR-V2-AUDIT-01..05 all 5 / 5 completed)
            • notes/ACTUAL_IMPLEMENTATION_STATUS.md (ground-truth working vs per-phase delta — 0 remaining items)
            • notes/README.md (governance index & 6-step execution protocol)
          • V2-Audit-Phase-01 Complete & Verified (2026-11-20) (BUG-V2-AUDIT-02, BUG-V2-AUDIT-03, BUG-V2-AUDIT-04, ERR-V2-AUDIT-01):
            • BUG-V2-AUDIT-04: Updated revokeAdminSessions(target) in core/auth/adminUsers.ts to accept explicit { memberId, email } and passed { memberId: check.session.memberId, email: check.session.email } from POST /api/auth/logout (app/api/auth/logout/route.ts), eliminating the multi-admin logout global race condition (globals.__dataStackLastVerifiedMemberId).
            • BUG-V2-AUDIT-03 (ERR-V2-AUDIT-01): Enforced cross-table (admin_users ↔ admin_team_members) and in-memory peer duplicate email checks in updateAdminAccount() (core/auth/adminUsers.ts) returning code: 'ADMIN_EMAIL_CONFLICT', mapped to HTTP 409 Conflict in PUT /api/auth/profile (app/api/auth/profile/route.ts).
            • BUG-V2-AUDIT-02: Wired all 4 missing audit actions (ADMIN_LOGOUT in POST /api/auth/logout, ADMIN_SETUP in POST /api/auth/setup, STORAGE_FLUSH_TRIGGERED in POST /api/storage, and CLIENT_REVOKED when updates.status === 'revoked' in PATCH /api/clients/[id]) and synchronized ACTION_FILTER_OPTIONS in components/sections/AuditLogViewer.tsx 1:1 with AuditActionType in core/auth/auditLog.ts (all 18/18 governance actions active).
            • Verified: 103/103 TS tests (5/5 in tests/v2-phase03-rbac-audit.test.ts), 10/10 Python SDK tests, npm run build clean, and npm run lint (0 errors / 0 warnings).
          • V2-Audit-Phase-02 Complete & Verified (2026-11-20) (BUG-V2-AUDIT-05, BUG-V2-AUDIT-06, BUG-V2-AUDIT-08, BUG-V2-AUDIT-09, ERR-V2-AUDIT-02..05):
            • BUG-V2-AUDIT-05 (ERR-V2-AUDIT-02): Wired outbound HMAC-SHA256 webhook alert dispatching (triggerWebhookAlert + dispatchRejectionWebhook) on Port 4000 Socket.IO (server/index.js, server/socket/handlers.js) for CLIENT_TOKEN_EXPIRED, CLIENT_IP_FORBIDDEN, DISK_SPACE_LOW, and STORAGE_QUOTA_EXCEEDED across handshake, sync_data, batch_sync, and ping_verify, with AES-256-GCM decryptVaultSecret in syncSettingsFromDatabase() and live webhookSecret propagation in core/adminSettings/settingsManager.ts.
            • BUG-V2-AUDIT-06 (ERR-V2-AUDIT-03): Upgraded isClientIpAllowed() and stripIpv6MappedPrefix() in both core/Security/ipAllowlist.ts and server/index.js to use Node's built-in net.BlockList (addSubnet) for all IPv4 (/0..32) and IPv6 (/0..128) CIDR subnets and normalize [ipv6]:port, ipv4:port, %zone, and ::ffff:.
            • BUG-V2-AUDIT-08 (ERR-V2-AUDIT-04): Enforced isClientTokenExpired(registeredClient.expiresAt) (401 CLIENT_TOKEN_EXPIRED) and isClientIpAllowed(extractClientIp(req), registeredClient.allowedIps) (403 CLIENT_IP_FORBIDDEN) in GET /api/health (app/api/health/route.ts).
            • BUG-V2-AUDIT-09 (ERR-V2-AUDIT-05): Added "CLIENT_TOKEN_EXPIRED" and "CLIENT_IP_FORBIDDEN" to _is_non_retryable_auth_error() and _extract_auth_error_code() in sdk/python/datastack/datastack/client.py, immediately raising DataStackAuthError(msg, code=...) matching the Node.js SDK 1:1.
            • Verified: 104/104 TS tests (5/5 in tests/v2-phase02-alerts-ip-prometheus.test.ts), 11/11 Python SDK tests (tests/phase05_sdk_test.py), npm run build clean, and npm run lint (0 errors / 0 warnings).
          • V2-Audit-Phase-03 Complete & Verified (2026-11-20) (BUG-V2-AUDIT-01, BUG-V2-AUDIT-07, BUG-V2-AUDIT-10):
            • BUG-V2-AUDIT-01: Aligned DiskUsageSnapshot (core/Storage/csvStorage.ts), GET /api/storage (app/api/storage/route.ts), and SettingsWorkspace.tsx (components/sections/SettingsWorkspace.tsx) with available: boolean (totalBytes > 0) and freeBytes: number (availableBytes) so /vcon/settings -> Storage -> "VPS disk usage" renders real disk utilization percentage, progress bar, and free/total GiB instead of "Statfs unavailable".
            • BUG-V2-AUDIT-07: Added isClientExpired(expiresAt, nowMs) in app/vcon/clients/page.tsx and excluded expired tokens from onlineCount, pendingCount, and "online"/"pending" filter tabs (classifying non-revoked expired clients under "offline"), matching ClientsTable.tsx "expired" badges 1:1.
            • BUG-V2-AUDIT-10: Updated components/sections/ExportWorkspace.tsx and app/vcon/storage/page.tsx to render a CSV.GZ format badge whenever file.compressed || file.filename.toLowerCase().endsWith('.csv.gz') is true (CSV otherwise).
            • Verified: 105/105 TS tests (4/4 in tests/v2-phase01-storage-quotas.test.ts), 11/11 Python SDK tests (tests/phase05_sdk_test.py), npm run build clean, and npm run lint (0 errors / 0 warnings). All 3 / 3 phases (10 / 10 bugs) resolved.
        • R3 Full-Stack Production Remediation (BASELINE-2026-11-20-R3 — 16 Verified Bugs across 4 Scope-Locked Phases):
          • R3-Phase-01 Complete & Verified (2026-11-20) (BUG-01, BUG-03, BUG-04, BUG-06, BUG-10, BUG-13, BUG-15; ERR-R3-01..ERR-R3-04):
            • BUG-01: Aligned Port 4000 decryptVaultSecret() (server/index.js) 1:1 with core/Security/secretVault.ts (v1: prefix, DATASTACK_ADMIN_SESSION_SECRET, base64url decoding, fail-closed "" + warning) and injected DATASTACK_ADMIN_SESSION_SECRET (SERVICE_REALBASE64_64_ADMINSESSION) into docker/docker-compose.yml services.ingestion.environment.
            • BUG-03: Updated AuthService.generateClientToken(payload, expiresIn?) (core/auth/jwt.ts) to omit JWT exp when expiresIn is null/undefined/"", and passed null across all 6 minting sites in core/sdk/sdkManager.ts so DB client_sdks.expires_at (Never expires, >30d, live extension) is authoritative.
            • BUG-04: Enforced requireAdminApiSession(req, { minRole: 'operator' }) on admin-session calls to POST /api/events (app/api/events/route.ts), returning 403 INSUFFICIENT_ROLE_PERMISSION for read-only viewer accounts.
            • BUG-06: Wrapped decryptSecret(settings.webhookSecretEnc) in SettingsManager.notifyIngestionSettingsSync() (core/adminSettings/settingsManager.ts) in try/catch ("" fallback + warning) and deferred decryptSecret in POST /api/settings/webhook-test (app/api/settings/webhook-test/route.ts) until neither draft webhookSecret nor clearWebhookSecret is provided.
            • BUG-10: Wired non-blocking ADMIN_LOGIN_LOCKOUT webhook alert dispatch inside the 429 TOO_MANY_ATTEMPTS lockout branch of POST /api/auth/login (app/api/auth/login/route.ts).
            • BUG-13: Unified Port 4000 formatOutboundWebhookBody() (server/index.js) with Port 3000 core/alerts/webhookNotifier.ts (instance, event, severity, title, message, clientId, details, timestamp, text, content).
            • BUG-15: Tightened POST /api/storage/s3-test (app/api/storage/s3-test/route.ts) from operator to owner-only RBAC.
            • Verified: 6/6 in tests/r3-phase01-security-vault-webhooks.test.ts, 15/15 in tests/v2-phase01..03, 40/40 in tests/phase01/02/07, npm run build clean, npm run lint (0 errors). 0 UI/SDK/CSV files modified.
          • R3-Phase-02 Complete & Verified (2026-11-20) (BUG-05, BUG-09):
            • BUG-05: Added singleton unref'd periodic maintenance scheduler (ensureMaintenanceScheduler, runScheduledMaintenanceTick, stopMaintenanceScheduler, stopMaintenanceSchedulerForTests, isMaintenanceSchedulerActive) in core/adminSettings/settingsManager.ts with overlap locks (__dataStackMaintenanceRunning, __dataStackArchivePruneInFlight, __dataStackCompressInFlight), cooldown guards, and unhydrated-DB protection so S3 off-site archival + retention pruning and partition compression run unattended on schedule when enableS3Backup === true.
            • BUG-09: Updated extractDateFromFilename() in both core/Storage/csvStorage.ts and server/workers/csvWriter.js to strip .csv(.gz) and trailing -(?:part|gz)\d+ suffixes so -gzN.csv.gz and -partN-gzM.csv.gz collision files work identically across Port 3000/4000 pruning, S3 archiving, /api/data, and /api/export.
            • Verified: 3/3 in tests/r3-phase02-storage-scheduler-gzn.test.ts, 40/40 in tests/phase03/v2-phase01/phase07/r3-phase01, npm run build clean, npm run lint (0 errors). 0 UI/SDK/Auth/Docker files modified.
          • R3-Phase-03 Complete & Verified (2026-11-20) (BUG-02, BUG-07, BUG-08; ERR-R3-05, ERR-R3-06):
            • BUG-02: Fixed Node.js SDK (sdk/node/datastack/index.ts) WS batch_sync to emit a raw Array directly (1:1 with Port 4000 server/socket/handlers.js:595), parse ack.processed ?? ack.processedCount ?? ack.accepted and ack.duplicates ?? 0 across WS & HTTP batch responses, and propagate duplicates through sendBatch(), batchSync(), and sendFile().
            • BUG-07 (ERR-R3-05): Implemented real heartbeatIntervalMs monitor in Node.js SDK (startHeartbeatMonitor with .unref?.(), zero-CPU IDLE_WS pause, automatic ping_verify recovery & spool drain, clean timer teardown, status().heartbeatActive); fixed single HTTP/WS DUPLICATE_DROPPED reporting (accepted: 0, duplicates: 1); wired TOKEN_ROTATED (401) and CLIENT_REVOKED (403) fast-fail DataStackAuthError across WS connect_error, ping_verify, sync_data, batch_sync, and HTTP fallback; compiled CommonJS sdk/node/datastack/dist/{index,types,spool}.{js,d.ts} via tsconfig.json and pointed package.json main/types/exports to ./dist/index.js & ./dist/index.d.ts.
            • BUG-08 (ERR-R3-06): Added non-retryable DataStackQuotaExceededError(DataStackError, retryable=False) in sdk/python/datastack/datastack/exceptions.py, re-exported it in both datastack and datastack_sdk, and wired immediate fail-fast handling for HTTP 507 and WS STORAGE_QUOTA_EXCEEDED / DISK_SPACE_LOW in sdk/python/datastack/datastack/client.py with 0 retries, 0 HTTP fallback after WS quota rejection, and 0 offline spooling.
            • Verified: 13/13 Node SDK/packaging tests (5/5 tests/r3-phase03-sdk-parity.test.ts against live Socket.IO handlers + 4/4 v2-phase04-node-sdk.test.ts + 4/4 phase05-packaging.test.ts), 14/14 Python SDK tests (3/3 tests/r3_phase03_python_sdk_test.py + 11/11 phase05_sdk_test.py), py_compile, tsc, npm run build clean, npm run lint (0 errors, 0 warnings). 0 Port 3000/4000/UI/Docker files modified.
          • R3-Phase-04 Complete & Verified (2026-11-20) (BUG-11, BUG-12, BUG-14, BUG-16; ERR-R3-07):
            • BUG-11 (ERR-R3-07): Added shouldRedirectSimulatorUnauthorized(status, body) in lib/vconUiHelpers.ts and wired it post-JSON-parse across both single & batch HTTP sends in app/vcon/simulator/page.tsx so expired/rotated/revoked client 401 responses display inline without logging out the authenticated admin.
            • BUG-12: Added browser-local formatDateForLocalInput(iso) in lib/vconUiHelpers.ts and replaced UTC .toISOString().slice(0, 16) inside ClientsTable.tsx openEditModal(), eliminating local timezone drift on client edits.
            • BUG-14: Added isUiClientExpired(), classifyUiClientStatus(), and summarizeUiClientCounts() in lib/vconUiHelpers.ts; excluded expired clients from activeClients and exposed expiredClients in core/apiEngine/apiService.ts and DashboardOverview.tsx; emitted datastack_clients_total{status="expired"} and excluded expired clients from status="active" in app/api/prometheus/route.ts; expanded /vcon/clients with a 6-column StatStrip + 6-pill status filter (Total, Online, Offline, Pending, Expired, Revoked).
            • BUG-16: Propagated verified session role from app/vcon/layout.tsx through components/layout/VconShell.tsx (VconRoleContext / useVconRole()) + getVconRoleCapabilities() (lib/vconUiHelpers.ts) to gate all role-restricted UI controls across SettingsWorkspace, ClientsTable, /vcon/clients, DashboardOverview, /vcon/storage, and /vcon/simulator.
            • Verified: 29/29 TS unit/integration tests (4/4 r3-phase04-ui-metrics-rbac.test.ts + 15/15 phase04-ui-sync.test.ts + 5/5 v2-phase02-alerts-ip-prometheus.test.ts + 5/5 v2-phase03-rbac-audit.test.ts), live headless Chromium (playwright-cli, 1440x900) verification across /vcon/dashboard, /vcon/clients, /vcon/simulator, /vcon/storage, and /vcon/settings (0 JS/React pageerrors), npm run build clean, npm run lint (0 errors, 0 warnings). 0 Port 4000/SDK/Auth/Storage/Docker files modified. All 4 / 4 R3 phases (16 / 16 bugs) complete.
        • Full-Project Production Forensic Audit & R4 Baseline Freeze (BASELINE-2026-11-20-R4):
          • Cleaned all 8 prior Files Manager files from notes/ and created the 100% raw-source-verified notes/FULL_PROJECT_FORENSIC_AUDIT.md (20 verified findings: 3 Critical, 6 High, 7 Medium, 4 Low — 19 actionable + 1 verified intentional preview AUDIT-17).

          • Froze the untouched codebase as BASELINE-2026-11-20-R4 and created the complete 8-file R4 governance & Scope-Lock suite in notes/ (README.md, FULL_PROJECT_FORENSIC_AUDIT.md, BASELINE_FREEZE.md, PRODUCTION_READINESS_SCOPE_LOCK.md, PRODUCTION_ROADMAP.md, PHASE_EXECUTION_LOG.md, ERROR_HANDLING_MATRIX.md, ACTUAL_IMPLEMENTATION_STATUS.md) across 4 Scope-Locked phases (R4-Phase-01..R4-Phase-04).

          • R4-Phase-01 Complete & Verified (2026-11-21) (AUDIT-01, AUDIT-03, AUDIT-04, AUDIT-09; ERR-R4-01–ERR-R4-05):

            • AUDIT-01 (ERR-R4-01): Created shared SSRF validator core/Security/ssrfGuard.ts blocking non-HTTP(S) schemes, embedded credentials, loopback, RFC1918/CGNAT/ULA, 0.0.0.0/8, cloud metadata (169.254.0.0/16, fe80::/10, metadata.google.internal), bare internal hostnames, and DNS-rebinding targets, plus redirect: "manual" 3xx rejection; enforced uniformly across core/alerts/webhookNotifier.ts, core/Storage/s3Archive.ts, app/api/settings/route.ts, app/api/settings/webhook-test/route.ts, app/api/storage/s3-test/route.ts, and server/index.js triggerWebhookAlert, with DATASTACK_ALLOW_PRIVATE_URLS=true documented in .env.example (metadata still unconditionally blocked).
            • AUDIT-03 (ERR-R4-02): Mirrored admin_team_members and admin_audit_logs DDL 1:1 into server/index.js ensureDatabaseSchema() for full 7-table parity across Port 3000 and Port 4000 on fresh PostgreSQL boots.
            • AUDIT-04 (ERR-R4-03): Added Next.js instrumentation.ts (register() on NEXT_RUNTIME === "nodejs") invoking SettingsManager.bootstrapMaintenanceSchedulerOnBoot() to hydrate settings and start the periodic storage maintenance scheduler automatically on Port 3000 startup.
            • AUDIT-09 (ERR-R4-04, ERR-R4-05): Added console.warn + storageMode, degraded: true, code: "AUDIT_STORAGE_DEGRADED", warning, and X-DataStack-Audit-Degraded: 1 header in core/auth/auditLog.ts & GET /api/audit; wired POST /internal/admin-session-sync in server/index.js, syncInMemoryAdminSessionState / checkInMemorySessionRevocation in server/filesManager.js, and notifyIngestionAdminSessionSync in core/auth/adminUsers.ts so !dbPool Files Manager sessions revoke immediately on logout/disable/delete/credential rotation.
            • Verified: 5/5 tests/r4-phase01-security-boot.test.ts + 85/85 regression tests (phase01..04, phase07, files-manager, r3-phase01, r3-phase02, v2-phase03, v2-phase04 = 90/90 PASS), npm run build, npm run lint (0 errors). 0 forbidden UI/SDK/Docker/ingestion-batch files modified.
          • R4-Phase-02 Complete & Verified (2026-11-21) (AUDIT-05, AUDIT-06, AUDIT-07, AUDIT-10, AUDIT-11, AUDIT-12; ERR-R4-06–ERR-R4-08):

            • AUDIT-05 (ERR-R4-06): Enforced DATASTACK_MAX_BATCH_ITEMS (default 1000, clamped 1..10000, documented in .env.example) across POST /api/events (400 BATCH_LIMIT_EXCEEDED, evaluated before storage admission scan) and Socket.IO batch_sync (server/socket/handlers.js).
            • AUDIT-06: Added FILES_MANAGER_MUTATION and DATA_EXPORT_DOWNLOADED to core/auth/auditLog.ts and ACTION_FILTER_OPTIONS in components/sections/AuditLogViewer.tsx; wired export download audit logs in app/api/export/route.ts and CloudCmd HTTP/Socket mutation audit logs (admin_audit_logs PG + RAM fallback) in server/filesManager.js.
            • AUDIT-07 (ERR-R4-07): Isolated corrupt .csv.gz decompression failures inside GET /api/data (app/api/data/route.ts) so healthy client partitions return HTTP 200 alongside corruptPartitions (CORRUPT_GZIP_PARTITION) and warnings.
            • AUDIT-10: Added serializeClientResponse() in app/api/clients/[id]/route.ts stripping secret and non-rotation token from PATCH /api/clients/[id] responses while exposing tokenPreview (and full token only on action === "regenerate_token").
            • AUDIT-11: Made POST /internal/client-sync (action === "updated") in server/index.js disconnect live sockets only on security-relevant updates (status === "revoked", expired expiresAt, token rotation, allowedIps change, or live socket credential mismatch), keeping metadata-only updates (name, rateLimit, storageQuotaMb) connected without disruption.
            • AUDIT-12 (ERR-R4-08): Converted uploadFileToS3 (core/Storage/s3Archive.ts) to streaming SHA-256 pre-pass + Readable.toWeb(fs.createReadStream(...)) (duplex: "half") with .destroy() teardown, and hardened CSVStorageManager.openPartitionTextStream + countCsvDataRowsStreamAsync (public static) with for await + .destroy() cleanup.
            • Verified: 5/5 tests/r4-phase02-ingestion-audit-streaming.test.ts PASS, regression suites (r4-phase01, files-manager, phase02-04, v2-phase02-03, r3-phase02, r3-phase04) PASS, Playwright /vcon/audit (0 console errors, new action filter options visible), npm run build, and npm run lint (0 errors). 0 forbidden SDK/Docker/Nginx/middleware/auth/unrelated UI files touched.
          • R4-Phase-03 Complete & Verified (2026-11-21) (AUDIT-02, AUDIT-08, AUDIT-13, AUDIT-20; ERR-R4-09):

            • AUDIT-02: Added "prepare": "npm run build" and "devDependencies": { "typescript": "^5.9.3" } to sdk/node/datastack/package.json and compiled sdk/node/datastack/dist/{index,spool,types}.{js,d.ts} so CommonJS require('./sdk/node/datastack') works cleanly out of the box.
            • AUDIT-20: Added atomic DiskSpooler.ackAndRequeueFront(ackedCount, unprocessed) in sdk/node/datastack/spool.ts and wired deliverChunkWithSplitAndRetry() + flushSpool() in sdk/node/datastack/index.ts (with dist/ rebuilt) so partial-batch rate-limit recoveries commit acknowledged events without dropping or reordering unprocessed envelopes ahead of tail items.
            • AUDIT-08 (ERR-R4-09): Added DATASTACK_ADMIN_EMAIL: "${DATASTACK_ADMIN_EMAIL:-admin@datastack.dev}" to services.ingestion.environment in docker/docker-compose.yml; added proxyMode/wsSupported/wsHint detection in GET /api/files/status (app/api/files/status/route.ts) and rendered a non-blocking [data-testid="files-manager-proxy-hint"] strip in components/sections/FilesManagerWorkspace.tsx when running standalone without Nginx WebSocket proxying.
            • AUDIT-13: Removed dead /api/auth/setup/status whitelist line from middleware.ts and passed explicit role: "owner", memberId: "owner-1" arguments to createAdminSessionToken() in app/api/auth/setup/route.ts.
            • Verified: 4/4 tests/r4-phase03-sdk-deploy-auth.test.ts + 43/43 regression checks (47/47 total PASS across v2-phase04-node-sdk, r3-phase03-sdk-parity, phase05-packaging, phase07-deployment, phase01-auth, files-manager), node -e "require('./sdk/node/datastack')" PASS (2.2.0), Playwright /vcon/files (0 console errors), npm run build, and npm run lint (0 errors). 0 forbidden Storage/Webhook/Client/Team/Profile/Landing files modified.
          • R4-Phase-04 Complete & Verified (2026-11-21) (AUDIT-14, AUDIT-15, AUDIT-16, AUDIT-18, AUDIT-19; ERR-R4-10, plus AUDIT-07 non-blocking corruptPartitions UI notice):

            • AUDIT-14: Updated components/sections/TeamWorkspace.tsx to initialize role from useVconRole() (shellRole || 'viewer'), preserve distinct operator and admin <Select> options and row values without coercing operator to admin, label StatStrip 'Operators / Admins', and require an explicit <Modal> ("Confirm Team Member Removal", [data-testid="confirm-remove-member"]) before DELETE /api/team/${id}.
            • AUDIT-15 (ERR-R4-10): Wired useVconRole() + getVconRoleCapabilities(shellRole) (canOperate) in app/vcon/sdk/page.tsx so viewer sessions skip ?includeToken=true requests cleanly (0 HTTP 403s) and display inline [data-testid="sdk-viewer-rbac-banner"] RBAC guidance while preserving existing loadingToken disabled guards.
            • AUDIT-16: Seeded components/layout/Sidebar.tsx role badge from useVconRole() (formatShellRoleLabel()), renamed 'Python SDK' to 'Client SDKs', and added components/layout/Navbar.tsx mobile VCON_LABELS for '/vcon/files' ('Files Manager'), '/vcon/sdk' ('Client SDKs'), '/vcon/team' ('Team & RBAC'), and '/vcon/audit' ('Audit Log').
            • AUDIT-19: Replaced fake Demo fallback constants in components/sections/ProfileWorkspace.tsx with neutral EMPTY_PROFILE placeholders ('Loading...' / '—') until /api/auth/profile resolves.
            • AUDIT-07 UI Hook: Surfaced non-blocking [data-testid="data-corrupt-partitions-warning"] banners inside components/sections/DataBrowser.tsx when GET /api/data reports corruptPartitions.
            • AUDIT-18: Aligned components/sections/Hero.tsx, components/sections/SdkSection.tsx, and components/sections/LandingAbout.tsx around 'Client SDKs' with side-by-side Python (datastack v2.2.0) and Node.js / TypeScript (@datastack/sdk v2.2.0) quick-start cards.
            • Verified: 5/5 tests/r4-phase04-ui-rbac-copy.test.ts + 46/46 regression checks (51/51 total PASS), live headless Chromium desktop (1440x900) & mobile (390x844) verification across /, /vcon/team, /vcon/profile, /vcon/data, /vcon/files, /vcon/audit, /vcon/sdk (0 console/page errors, 0 viewer 403s), npm run build, and npm run lint (0 errors). 0 forbidden backend/SDK/Docker/Nginx files modified. All 4 / 4 R4 phases (19 / 19 actionable findings) complete.
          • PREPUSH-FIX-01 Complete & Verified (2026-11-21) (DEPLOY-BLK-01, DEPLOY-BUG-02–05):

            • DEPLOY-BLK-01: Resynchronized package-lock.json (lockfileVersion: 3) with package.json (cloudcmd ^19.23.2 + full dependency tree) and verified npm ci --include=dev --dry-run (Dockerfile.web) and npm ci --omit=dev --dry-run (Dockerfile.server) both exit 0.
            • DEPLOY-BUG-02: Hardened docker/coolify-nginx.conf and nginx/default.conf with ngx_http_realip_module (set_real_ip_from loopback/RFC1918/Docker bridges + IPv6 ULA, real_ip_header X-Forwarded-For, real_ip_recursive on) so proxy_set_header X-Real-IP $remote_addr preserves real external client IPs behind Coolify Traefik without permitting untrusted public IP spoofing.
            • DEPLOY-BUG-03 & DEPLOY-BUG-04: Added ARG INGESTION_INTERNAL_URL=http://ingestion:4000 + ENV INGESTION_INTERNAL_URL in docker/Dockerfile.web builder stage for standalone Next.js rewrites, and forwarded optional DATASTACK_ALLOW_PRIVATE_URLS, DATASTACK_MAX_BATCH_ITEMS, DATASTACK_RECENT_LIMIT, DATASTACK_MAINTENANCE_INTERVAL_MS, and DATASTACK_PROXY_MODE env vars in docker/docker-compose.yml.
            • DEPLOY-BUG-05: Updated sdk/node/datastack/index.ts (emitSocketSingle and sendBatchChunkInternal) to call this.disconnect() on terminal DataStackAuthError rejections (TOKEN_ROTATED, CLIENT_REVOKED, etc.) and recompiled sdk/node/datastack/dist/* so subsequent client.connect() performs a fresh handshake (tests/r3-phase03-sdk-parity.test.ts 5/5 PASS).
        • R5-MCP Planning, Foundation & Server Track (BASELINE-2026-11-21-R5-MCP):
          • Froze post-R4 / post-PREPUSH-FIX-01 codebase and documented the full 8-item raw-source-verified MCP forensic audit (MCP-AUDIT-01–MCP-AUDIT-08), Scope Locks, A–Z 3-phase roadmap (R5-Phase-01–03), Phase Execution Log, ERR-R5-01–09 error contracts, and Actual Status Ledger across /notes/.
          • R5-Phase-01 Complete & Verified (2026-11-21) (MCP-AUDIT-02 Complete; MCP-AUDIT-03/04/05/07 Foundation; ERR-R5-01 & ERR-R5-02):
            • Added 8th PostgreSQL table mcp_settings identically in core/Database/database.ts and server/index.js (ensureDatabaseSchema()).
            • Created TS (core/logs/runtimeLogCapture.ts) and CJS (server/runtimeLogCapture.js) bounded rotating runtime/build log writers & readers (.runtime-logs/{web,ingestion}.log, max 2MB + .1 backup, secret redaction for Bearer/dsmcp_/JWT/JSON keys, graceful LOG_CAPTURE_UNAVAILABLE fallback), wired into instrumentation.ts, server/index.js, scripts/start-all.js, and scripts/dev-all.js (DATASTACK_LOG_CAPTURED_BY_RUNNER=1 prevents duplicate tee writes).
            • Created scripts/capture-build-log.js (package.json "build") and updated docker/Dockerfile.web so standalone Next.js builds emit .next/standalone/build.log, .next/standalone/build-meta.json, and .runtime-logs/last-build*.
            • Created core/mcp/mcpSettings.ts (McpSettingsManager with default enabled: false, 17 permission definitions across 4 categories, dsmcp_ token generation + SHA-256 digest + AES-256-GCM vault encryption, IP CIDR validation, SSRF-guarded coolifyUrl, PG/RAM roundtrip, and POST /internal/mcp-sync notifier) and added MCP_* action types + secret redactions in core/auth/auditLog.ts.
            • Added middleware.ts internal secret pass + timing-safe /api/internal/mcp bridge (app/api/internal/mcp/route.ts) supporting metrics, runtime/build logs, audit queries, client CRUD, safe settings updates, CSV+buffer data queries, storage stats/flush, and S3 archive.
            • Verified: 5/5 tests/r5-phase01-mcp-foundation-logs.test.ts + 78/78 regression checks across 8 suites (83/83 PASS), compile_applet (npm run build -> .next/standalone/build.log + build-meta.json generated), lint_applet (eslint . -> 0 errors), and npm ci --include=dev --dry-run / --omit=dev --dry-run clean.
          • R5-Phase-02 Complete & Verified (2026-11-21) (MCP-AUDIT-01, MCP-AUDIT-03/04/05/06/07 Server & Proxy Delivery; ERR-R5-03–ERR-R5-06):
            • Added @modelcontextprotocol/sdk (^1.32.2) + zod (^4.6.7) dependencies and resynchronized package-lock.json (lockfileVersion: 3, removing bun.lock).
            • Created Owner-only Admin MCP REST API (app/api/mcp/route.ts: GET masked status + 17 permission definitions; POST strict allowlist supporting enabled, allowedIps, permissions, rotateToken, revokeToken, and SSRF-guarded coolifyUrl/coolifyToken/coolifyAppUuid, with MCP_TOKEN_ROTATED & MCP_SETTINGS_UPDATED audit logging and live /internal/mcp-sync).
            • Created Port 4000 Streamable HTTP MCP Engine (server/mcp/mcpEngine.js) using official McpServer + stateless StreamableHTTPServerTransport, enforcing disabled gate (503 MCP_DISABLED), CIDR allowlist (403 MCP_IP_FORBIDDEN), timing-safe SHA-256 dsmcp_... Bearer auth (401 MCP_UNAUTHORIZED), per-tool permission gate (MCP_PERMISSION_DENIED), symlink/../*.lock/.runtime-logs write-protected storage sandbox, SSRF-guarded optional Coolify log proxy, admin_audit_logs telemetry (MCP_TOOL_CALLED / MCP_TOOL_DENIED), and all 17 registered datastack_* tools.
            • Wired POST /internal/mcp-sync, app.all('/mcp'), and DB mcp_settings hydration (syncSettingsFromDatabase()) in server/index.js; added /mcp reverse proxy blocks in docker/coolify-nginx.conf, nginx/default.conf, and next.config.ts; added WEB_INTERNAL_URL in docker/docker-compose.yml & .env.example.
            • Verified: 3/3 tests/r5-phase02-mcp-server.test.ts + 50/50 regression checks across 8 suites (53/53 PASS), live Port 3000 & Port 4000 @modelcontextprotocol/sdk E2E test (login -> /api/mcp -> /mcp direct + rewritten tools/list [17] & datastack_metrics_health -> revoke 503), npm ci --include=dev --dry-run / --omit=dev --dry-run exit 0, compile_applet clean, and lint_applet clean. Ready for Proceed to R5-Phase-03.
          • R5-Phase-03 Complete & Verified (2026-11-21) (MCP-AUDIT-08; /vcon/mcp Admin UI, Navigation & E2E):
            • Added browser-safe lib/mcpUiHelpers.ts (MCP_CLIENT_TARGETS, MCP_PERMISSION_CATEGORIES, DEFAULT_PUBLIC_MCP_SETTINGS, buildMcpEndpointUrl, buildMcpClientConfigSnippet, groupMcpPermissionsByCategory, countEnabledMcpPermissions, applyMcpPermissionPreset, applyCategoryPermissionToggle, describeMcpBridgeSync, buildMcpSavePayload, isMcpDraftDirty).
            • Added optional additive action?: React.ReactNode header slot on SettingsGroup (components/ui/SettingsLayout.tsx) while preserving all existing callers unchanged.
            • Created Owner-only components/sections/McpWorkspace.tsx and /vcon/mcp (app/vcon/mcp/page.tsx, exported via components/sections/index.ts) using shared UI primitives (PageShell, StatStrip, SettingsTabs, SettingsPanel, SettingsGroup, SettingsField, SettingsFooter, Panel, Button, Input, FormSwitch, CodeBlock, Modal, useToast): Connection & Token tab (Master Switch, Endpoint URL copy, CIDR allowlist, Mint/Rotate/Revoke & Disable confirmation modals, one-time dsmcp_... component-state reveal banner with auto-scrollIntoView & --accent-text contrast), Permissions tab (17 ON/OFF switches grouped across 4 categories + Read-Only/Defaults/Disable All presets + aligned per-category Enable All/Disable), AI Client Setup tab (copyable JSON for Claude Desktop / Code, Cursor, VS Code Copilot, Windsurf), Build Logs (Coolify) tab (coolifyUrl, masked/clearable coolifyToken, coolifyAppUuid), and Non-owner RBAC banner (mcp-non-owner-banner) skipping /api/mcp fetches cleanly (0 403s).
            • Wired MCP Server (href: '/vcon/mcp', icon: 'Cpu') into components/layout/Sidebar.tsx (System group) and '/vcon/mcp': 'MCP Server' into components/layout/Navbar.tsx, and added tests/r5-phase03-mcp-ui.test.ts.
            • Verified: 30/30 PASS across tests/r5-phase03-mcp-ui.test.ts (2/2), r5-phase02-mcp-server.test.ts (3/3), r5-phase01-mcp-foundation-logs.test.ts (5/5), phase04-ui-sync.test.ts (15/15), and r4-phase04-ui-rbac-copy.test.ts (5/5); two rounds of live headless Chromium (playwright-cli) verification covering Dark & Light themes, desktop (1440x900) & mobile (390x844), enable/mint/reload/rotate/revoke lifecycle, Permissions header alignment, auto-scroll reveal, and viewer RBAC (0 console errors, 0 HTTP 403s); compile_applet & lint_applet clean. All 3 / 3 R5-MCP phases complete.
          • FINAL-DEPLOY-FIX-01 Complete & Verified (2026-11-21) (Pre-GitHub-Push Docker / Coolify Deployment Audit Remediation):
            • BLOCKER-01: Rebuilt @datastack/sdk TypeScript outputs (sdk/node/datastack/dist/{index,spool,types}.{js,d.ts}) via tsc -p sdk/node/datastack/tsconfig.json, fixing AUDIT-02 (require('./sdk/node/datastack') -> 2.2.0).
            • ISSUE-02: Removed stale /bun.lock so /package-lock.json (lockfileVersion: 3) is the sole lockfile across Coolify/Docker npm ci builds.
            • BUG-03: Added PostgreSQL SettingsManager.getSettingsAsync() hydration guards prior to reading resolveInternalIngestionSecret() / getInternalIngestionHeaders() in app/api/internal/mcp/route.ts (POST) and core/mcp/mcpSettings.ts (notifyIngestionMcpSync), preventing cold-boot bridge secret desyncs when DATASTACK_INTERNAL_SECRET is empty and jwt_secret was rotated in DB.
            • ISSUE-04 & ISSUE-05: Pre-created /app/storage/.runtime-logs and enforced chown -R node:node /app/storage/.runtime-logs on boot in both docker/Dockerfile.web and docker/Dockerfile.server without recursing /app/storage/data when UID/GID already matches; synchronized prisma/schema.prisma (model McpSettings @@map("mcp_settings")) with all 8 canonical runtime PostgreSQL tables.
            • Verified: 49/49 PASS across 9 TypeScript suites (r4-phase01, r4-phase02, r4-phase03, phase07-deployment, phase04-ingestion-hygiene, r3-phase03-sdk-parity, r5-phase01, r5-phase02, r5-phase03), 11/11 PASS (tests/phase05_sdk_test.py), npm ci --dry-run + npm ci --omit=dev --dry-run PASS, compile_applet + lint_applet clean.
        • AI Studio Migration Sync:
          • Removed /bun.lock to standardize package management exclusively on npm (package-lock.json).
          • Expanded .env.example to document all 20 runtime environment variables used across the Next.js web app (port 3000) and Express/Socket.IO ingestion engine (port 4000), preserving built-in in-memory fallbacks when DATABASE_URL and REDIS_URL are omitted.

5. Known Constraints & Operational Notes for Future Contributors

  • Never add per-row telemetry INSERT queries to PostgreSQL or add DROP/TRUNCATE auto-migrations.
  • Never move Socket.IO handlers into Next.js app/api/* routes.
  • When deploying updates to an existing Coolify VPS, always preserve existing SERVICE_* environment variables and avoid docker compose down -v so pgdata, datastack-data, and redisdata volumes remain intact.
  • After any code change, verify with npm run build, npm run lint, npm ci --dry-run, and the test suites, then synchronize this MEMORY.md.