- Implement server-side session revocation on logout - Add IP extraction utility and admin setup error handling - Improve error codes and response consistency for auth endpoints - Refine landing page redirect logic to handle transient DB outages - Update deployment documentation and configuration defaults
42 KiB
42 KiB
MEMORY.md — DataStack Cross-IDE Persistent Project Memory
Last Synchronized: 2026-11-19
Status: Production & Deployment Hardening Complete (BASELINE-2026-11-19-R2+ Deployment Hardening) —4 / 4R2 Phases (18/18bugs) +5/5Docker/Coolify Redeploy Fixes (DEPLOY-BUG-01–05) Verified (85/85TS tests,10/10Python tests, ESLint-enabled build & lint0errors /0warnings).
Scope Rule: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see.ai-rulesandAGENTS.md).
1. Project Identity & Core Principles
- Project: DataStack — Self-hosted, real-time telemetry ingestion, admin governance (
/vcon/*), and partitioned CSV storage/export platform. - Runtime Split (CRITICAL):
- Port
3000(web): Next.js 15 App Router — public landing (/), PWA lifecycle,/vcon/*Admin Console, REST ingestion (POST /api/events), metrics/settings/client APIs, and streaming exports (GET /api/export). - Port
4000(ingestion): Dedicated Node.js Express 5 + Socket.IO server (server/index.js) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker. - Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port
4000.
- Port
- Storage & Database Boundary (CRITICAL):
- PostgreSQL (
pg+prisma/schema.prisma): Stores strictly control-plane metadata (admin_users,admin_login_attempts,client_sdks,deleted_clients,admin_settings). Never insert per-row telemetry events into PostgreSQL. - Partitioned CSV Storage (
/storage/data/{clientId}/{YYYY-MM-DD}.csv): All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (fileRotationMb, e.g.,-part2.csv), and retention pruning (autoPruneDays). - Redis (
ioredis): Shared cross-process buffer queue (datastack:buffer:events), recent event cache (datastack:recent:events), and distributed flush lock (datastack:flush:lock). Persisted in Docker via--appendonly yes(redisdata:/data). Falls back gracefully to local RAM whenREDIS_URLis omitted.
- PostgreSQL (
2. Tech Stack & Local Setup
Stack
- Frontend / Web API: Next.js
15.4.9(App Router,output: "standalone"), React19.2.1, TypeScript5.9.3, Tailwind CSS v4 (@tailwindcss/postcss),lucide-react,motion. - Ingestion Server: Node.js
22/ Express5.3.0, Socket.IO4.8.4. - Persistence & Caching:
pg8.23.1(PostgreSQL 16 in Docker),ioredis6.0.0(Redis 7 AOF in Docker),csv-parse7.0.3,json2csv6.0.0-alpha.2. - Auth & Security:
jsonwebtoken9.0.3, Nodecrypto.scryptpassword hashing, WebCrypto HMAC-SHA256 admin session cookies (middleware.ts), CSRFOrigin/ReferervsHost/X-Forwarded-Hostchecks,DATASTACK_INTERNAL_SECRETtiming-safe bridge auth between Port3000↔4000. - Client SDKs: Official pip-installable Python SDK in
sdk/python/datastack/(datastack) + backward-compatible wrappersdk-python/(datastack-sdk).
Environment & Setup Commands
- Configure Environment: Copy
.env.exampleto.env(in productionNODE_ENV=production, all 4 secrets MUST be 32+ chars:DATASTACK_ADMIN_SESSION_SECRET,DATASTACK_JWT_SECRET,DATASTACK_ADMIN_SETUP_KEY,DATASTACK_INTERNAL_SECRET; optionalDATABASE_URLandREDIS_URL). - Install Dependencies:
npm ci(ornpm install) using rootpackage.json+package-lock.json. - Run Locally (Two Terminals):
- Terminal 1 (Port
3000):npm run dev - Terminal 2 (Port
4000):npm run server
- Terminal 1 (Port
- Build, Lint & Test Verification:
- Build:
npm run build - Lint:
npm run lint(eslint .) - TS Test Suite (Phases 01–03, 06, 07):
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts - Python SDK Tests:
python3 -m unittest tests/phase05_sdk_test.py
- Build:
3. Folder Structure & File Architecture
datastack/
├── .ai-rules # Universal cross-IDE rules (enforces MEMORY.md sync)
├── AGENTS.md # Engineering rules, UI single-source-of-truth & memory sync
├── MEMORY.md # Persistent cross-IDE project memory (this file)
├── package.json / package-lock.json
├── middleware.ts # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
├── app/
│ ├── (landing)/page.tsx # Public landing page
│ ├── globals.css # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
│ ├── layout.tsx / not-found.tsx / error.tsx
│ ├── api/
│ │ ├── auth/ # Admin login, logout, setup (/vcon/setup), profile update
│ │ ├── clients/ # CRUD clients, token regeneration, revocation
│ │ ├── data/ # Paginated CSV search/query across partitions
│ │ ├── events/ # Authenticated HTTP ingestion (single/batch/file records) & recent list
│ │ ├── export/ # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
│ │ ├── health/ # Web health + client Bearer token probe + async buffer depth
│ │ ├── metrics/ # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
│ │ ├── settings/ # Allowlisted runtime config get/post + live Port 4000 sync
│ │ └── storage/ # CSV partition listing & manual cross-process buffer flush
│ └── vcon/ # Protected Admin Console pages:
│ ├── dashboard/ clients/ data/ export/ login/ profile/ sdk/ settings/ setup/ simulator/ storage/
├── components/
│ ├── ui/ # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, etc.)
│ ├── layout/ # MainLayout, Navbar, Sidebar, VconShell, PageShell, AdminPageHeader, Footer
│ ├── sections/ # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, SettingsWorkspace, etc.)
│ └── pwa/ # PwaLifecycle.tsx service worker registration
├── lib/
│ ├── adminAuth.ts # Uniform API session guard (requireAdminApiSession) + proxy-safe extractClientIp()
│ ├── vconCache.ts # In-memory SWR cache + clearAuthCaches() for logout/SW cleanup
│ └── vconUiHelpers.ts # Pure Admin UI sync helpers (settings bounds, bridge sync, UTC presets, timestamps)
├── core/
│ ├── adminSettings/ # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
│ ├── apiEngine/ # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
│ ├── auth/ # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
│ ├── data/ # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
│ ├── Database/ # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
│ ├── sdk/ # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
│ ├── Security/ # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
│ └── Storage/ # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
├── scripts/
│ ├── dev-all.js # Local concurrent Next.js (3000) + Ingestion (4000) dev runner
│ └── start-all.js # Production concurrent Next.js (`next start`) + Ingestion (`server/index.js`) runner
├── server/
│ ├── index.js # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, graceful exit
│ ├── socket/handlers.js # Handshake JWT auth, ping_verify, sync_data, batch_sync
│ └── workers/csvWriter.js # Partitioned CSV batch writer & rotation/pruning worker
├── sdk/python/datastack/ # Official pip-installable Python SDK (`datastack` v2.2.0)
│ ├── datastack/ # client.py, config.py, heartbeat.py, spool.py, exceptions.py
│ ├── config.datastack.example.py
│ ├── pyproject.toml / setup.py
│ ├── README.md # SDK overview, installation, quickstart, docs index
│ └── docs/ # Production SDK Documentation Suite (LicenX Standard):
│ ├── README.md # SDK Documentation Hub & architecture overview
│ ├── INSTALLATION.md # Python runtimes, pip commands, and verification
│ ├── QUICKSTART.md # 5-minute zero-boilerplate onboarding guide
│ ├── API_REFERENCE.md # Complete DataStack class & method specifications
│ ├── CONFIGURATION.md # config.datastack.py & DATASTACK_* environment reference
│ ├── EXAMPLES.md # IoT streaming, FastAPI, batching, and file ingestion
│ ├── ERROR_HANDLING.md # Exception hierarchy, 401/403/413/429 codes & retries
│ └── LIFECYCLE.md # 5-state machine, zero-CPU IDLE mode & offline spooling
├── sdk-python/ # Backward-compatible re-export package (`datastack-sdk`)
├── docker/
│ ├── docker-compose.yml # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
│ ├── Dockerfile.web # Multi-stage standalone Next.js build + safe /app/storage ownership check
│ ├── Dockerfile.server # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
│ ├── Dockerfile.proxy # Nginx reverse proxy container
│ └── coolify-nginx.conf # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
├── nginx/default.conf # Local Nginx proxy config matching coolify-nginx.conf
├── prisma/schema.prisma # Reference PostgreSQL metadata schema
├── public/ # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
├── docs/ # Comprehensive Production Documentation Suite:
│ ├── index.md # Primary documentation hub and portal navigation index
│ ├── MANIFEST.md # Master documentation inventory & authoring governance
│ ├── audit-coverage-report.md # Forensic codebase audit matrix & coverage report
│ ├── users/ # End-User & Operator Guides:
│ │ ├── index.md # User portal overview & reading pathways
│ │ ├── getting-started.md # 5-minute fast-track operator onboarding guide
│ │ ├── installation.md # Production Docker Compose, Coolify & volume guide
│ │ ├── configuration.md # Environment variables & platform runtime settings
│ │ ├── features/ # Deep-dive guides: admin-console, client-management,
│ │ │ # data-simulator, partitioned-storage, data-export
│ │ ├── workflows.md # End-to-end operator journeys & disaster recovery
│ │ ├── troubleshooting.md # Forensic error recovery, status codes & diagnostics
│ │ └── faq.md # Storage architecture & operational questions
│ ├── dev/ # Developer & Contributor Guides:
│ │ ├── index.md # Developer hub index & architectural principles
│ │ ├── architecture.md # Dual-runtime blueprint, service bridge & storage
│ │ ├── codebase-structure.md # File-by-file module, route & component inventory
│ │ ├── local-setup.md # Local workstation setup & dual-terminal workflow
│ │ ├── api-reference.md # Exhaustive REST API & Socket.IO specification
│ │ ├── database-schema.md # PostgreSQL control plane & DDL migrations
│ │ ├── security.md # Security architecture, scrypt, sessions & CSRF
│ │ ├── building-packaging.md # Next.js standalone build & multi-stage Dockerfiles
│ │ ├── testing.md # Automated test suites guide (node:test + unittest)
│ │ └── contributing.md # Architectural rules, memory sync & PR checklist
│ └── reference/ # Technical Reference Portal:
│ ├── index.md # Technical reference index
│ ├── configuration-reference.md # Complete environment variable & setting bounds
│ ├── dependencies.md # Runtime & dev dependencies audit with rationales
│ └── glossary.md # Technical domain terminology & concept glossary
├── notes/ # Production Hardening Governance & Scope-Lock Suite (`BASELINE-2026-11-19-R2`):
│ ├── README.md # Master governance index & 6-step scope-locked execution protocol
│ ├── FORENSIC_AUDIT_REPORT.md # 100% raw-code verified audit (18 bugs: 0C/6H/8M/4L), Demo vs Prod & UI-vs-Reality matrices
│ ├── BASELINE_FREEZE.md # Frozen baseline snapshot (`BASELINE-2026-11-19-R2`) & 1:1 bug alignment
│ ├── PRODUCTION_READINESS_SCOPE_LOCK.md # (Doc 001) Exact bug fix contracts & per-phase Scope Locks (`Phase-01`..`Phase-04`)
│ ├── PRODUCTION_ROADMAP.md # (Doc 002) Dependency-ordered A-Z 4-phase roadmap & burn-down tracker
│ ├── PHASE_EXECUTION_LOG.md # (Doc 003) Live phase completion log (`4/4` complete) & zero-regression guardrails
│ ├── ERROR_HANDLING_MATRIX.md # (Doc 004) Active error-handling catalog & 9 planned hardening additions
│ └── ACTUAL_IMPLEMENTATION_STATUS.md # Ground-truth working vs completed-per-phase vs remaining ledger
└── tests/ # Automated verification suites:
├── phase01-auth.test.ts # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
├── phase02-sync.test.ts # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
├── phase04-ui-sync.test.ts # UI/API settings bounds, bridge status, UTC presets, timestamp & folder matching
├── phase05-packaging.test.ts # Python/Node packaging parity, PEP 561 py.typed, Node engines & start-all exit
├── phase05_sdk_test.py # Python SDK WebSocket/HTTP fallback, offline spool, file streaming, edge hardening
├── phase06-pwa-landing.test.ts # PWA manifest/icons/sw + navigation routes
└── phase07-deployment.test.ts # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush
4. Active Features & Development Status
All planned hardening & deployment phases (Phase-01 through Phase-07) are Complete & Verified:
- Admin Authentication & CSRF Hardening (Completed — Phase-01)
- Fail-closed secret enforcement in
NODE_ENV=production(/vcon/setupbootstrap viaDATASTACK_ADMIN_SETUP_KEY). - Password hashing via
crypto.scrypt, session invalidation via DB-backedtokenVersion, composite IP+email login lockout (admin_login_attempts), and strictOrigin/ReferervsHost/X-Forwarded-HostCSRF verification.
- Fail-closed secret enforcement in
- Real-Time Dual-Runtime Sync (Completed — Phase-02)
- Authenticated
/internal/*bridge between Port3000and Port4000(DATASTACK_INTERNAL_SECRET, backward-compatible Coolify fallback). - Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
- Authenticated
- Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)
- Shared Redis queue (
datastack:buffer:events) with distributed flush lock (NX PX) whereLTRIMexecutes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects. - Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (
GET /api/export).
- Shared Redis queue (
- Admin Console UI & Data/Export Workspace (Completed — Phase-04)
- Built with shared
components/ui/andcomponents/layout/primitives (0remradius, zero shadows, Fluent/Segoe dark & light tokens).
- Built with shared
- Production Python Client SDK (Completed — Phase-05)
- Supports single events (
send()/sync()), auto-splitting/rate-limit-paced batches (send_batch()/batch_sync()), and direct file ingestion (send_file()for.csv,.json,.jsonl,.ndjsonwith SHA-256 deterministiceventId). - Automatic
config.datastack.pydiscovery, WebSocket primary transport -> Heartbeat HTTP fallback ->.datastack_spool/spool.jsonloffline spool & auto-drain.
- Supports single events (
- PWA & Landing Integrity (Completed — Phase-06)
- Complete
public/manifest.json, icons,public/sw.js, and root-relative navigation links.
- Complete
- Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)
- Redis AOF persistence (
redisdata:/data), deterministicpackage-lock.jsonfornpm ci, Nginxlocation = /health→ingestion:4000,/api/healthcontainer probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DBautoPruneDayssafety guard inSettingsManager, DB JWT hydration inSDKManager, gracefulSIGTERM/SIGINTflush inDataBufferEngine, and non-destructiveADD COLUMN IF NOT EXISTSDDL guards.
- Redis AOF persistence (
- Current Remediation Status (2026-11-18):
- Phase-01 Complete & Verified (
BUG-A01,BUG-A02,BUG-A03,BUG-A04,BUG-A05,BUG-E02):- Removed spoofable header/UA CSRF bypasses in
middleware.tsand rejectedsec-fetch-site: cross-site. - Enforced DB-backed
tokenVersionsession validation across/vcon/*layout (app/vcon/layout.tsx+components/layout/VconShell.tsx) and read/export APIs (/api/data,GET /api/events,/api/export,/api/metrics,/api/storage). - Prevented transient DB errors from triggering false
/vcon/setupredirects and prevented swallowedNEXT_REDIRECTerrors on/,/vcon/login, and/vcon/setup. - Standardized
SDKManager.normalizeClientId, blocked deleted/unregistered clients on/api/healthand AdminPOST /api/events, hardenedPOST /api/clients&PATCH /api/clients/[id]input validation, and excluded/vcon/*from Service Worker caching withCLEAR_AUTH_CACHEon sign-out. - Verified:
26/26TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser auth/logout E2E clean.
- Removed spoofable header/UA CSRF bypasses in
- Phase-02 Complete & Verified (
BUG-B01–BUG-B05):- Enforced fail-closed client verification in
server/index.js(verifyClientInDb) backed by PG, deletion tombstones, Redisdatastack:clients:*mirrors, and normalizedclientIdkeys across both runtimes. - Added cross-runtime Redis idempotency (
datastack:idem:*) and fixed-window rate limiting (datastack:rl:*) incore/Security/security.ts(validateEventAsync) andserver/socket/handlers.js(validateSingleEventAsync) with automatic local RAM fallback. - Surfaced bridge sync telemetry (
BridgeSyncResult,SettingsBridgeSyncStatus,runtime.ingestionBridgeStatus,POST /api/settingsbridgeSync) and normalizedSDKManager.incrementEventCountAsync(). - Fixed Socket
sync_dataflat payload preservation (extractSingleEventData), reconciledPORT_SOCKET/PORTprecedence and live/internal/metricsports, and added concurrent productionscripts/start-all.jsfornpm start. - Verified:
29/29TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check (/vcon/dashboard,/vcon/settings,/api/metrics) clean with0console errors.
- Enforced fail-closed client verification in
- Phase-03 Complete & Verified (
BUG-C01–BUG-C06):- Hardened Redis flush lock TTL (
30000ms), added in-processactiveFlushPromisecoalescing with aflushDoneidentity guard (core/data/buffer.ts,server/index.js), isolatedredis.rpushfrom best-effortREDIS_RECENT_KEY, and logged malformed buffered JSON warnings. - Enforced
sanitizeClientId()path containment andlstatsymlink exclusion, unique lock owner tokens (${pid}:${ts}:${uuid}), atomicrenamestale-lock takeover, owner-verified lock unlock, and structuredJSON.stringify([clientId, date])partition grouping incore/Storage/csvStorage.tsandserver/workers/csvWriter.js. - Fixed trailing newline blank-row prevention (
fileEndsWithNewline), proactive size-boundary rotation (-partN.csv),StringDecoder('utf8')streaming row counting, cooperatively lockedpruneExpiredPartitions, and 1-hourPRUNE_COOLDOWN_MSincore/adminSettings/settingsManager.ts. - Enforced
parser/fileStreamdestruction infinallyblocks across/api/data,GET /api/events, and/api/export, error propagation insidemakeExportStream,502surfacing on Port4000flush failure in/api/export, normalizedclientIdfiltering, numeric timestamp sorting (parseTimestampMs), and?format=json/?includeHeader=falseparity for?filename=exports. - Verified:
32/32TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check (/vcon/dashboard,/vcon/data,/vcon/storage,/vcon/export,/api/data,/api/storage,/api/export) clean with0console errors.
- Hardened Redis flush lock TTL (
- Phase-04 Complete & Verified (
BUG-D01–BUG-D07):- Extracted pure UI synchronization helpers in
lib/vconUiHelpers.ts(SETTINGS_LIMITS,normalizeEditableSettings,describeSettingsBridgeSync,mapBridgeStatusToSocketUiState,parseEventTimestamp,formatEventTimeLabel,getUtcDateRangeForPreset,matchesSelectedClientFolder,extractContentDispositionFilename). - Updated
components/sections/SettingsWorkspace.tsx(BUG-D01) to probe/api/metricshonestly (removing false/api/healthfallback), surfacebridgeSyncwarning/error toasts on save, align integer bounds with/api/settings(1..100000,1..10240,0..3650), and eliminate the startupserverPortdouble-fetch. - Updated
app/vcon/storage/page.tsx(BUG-D02) andcomponents/sections/ExportWorkspace.tsx(BUG-D03) withcache: 'no-store',401redirects, UTC preset ranges, normalized client folder filters, and fetch+Blob downloads with502/JSON error surfacing. - Updated
components/sections/DashboardOverview.tsx(BUG-D04) andcomponents/sections/DataBrowser.tsx(BUG-D05) withingestionBridgeStatusdisplay,auth_erroralert banner, safe ISO/epoch-ms timestamp formatting, invalid date-range guard, and Enter-key filter submission. - Updated
components/sections/NewClientModal.tsx,components/sections/ClientsTable.tsx,app/vcon/simulator/page.tsx,app/vcon/sdk/page.tsx,components/sections/ProfileWorkspace.tsx, andcomponents/layout/Sidebar.tsx(BUG-D06,BUG-D07) with1..100000rate limit parity, liveruntime.serverPortsync, real HTTP batch ingestion in Simulator, password bounds check, anddatastack:profile-updatedlive Sidebar sync. - Verified:
37/37TS tests (tests/phase04-ui-sync.test.ts),3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check across all Admin Console routes clean with0console errors.
- Extracted pure UI synchronization helpers in
- Phase-05 Complete & Verified (
BUG-E01,BUG-E03,BUG-E04):- Hardened canonical Python SDK (
sdk/python/datastack/datastack/*): normalizedCLIENT_IDcomparison (_normalize_client_id), sanitized ragged CSV extra columns (restkey="_extra"), accuratesend_file()success+unprocessedItems, partial-batchrateLimitedEventIdsretry exception fallback,DataStackConfigErrorwrapping for malformed numeric env/config values (config.py), concurrentstop()vsset_idle_connected()race fix + clean thread join (heartbeat.py),os.fsync+ orphan.tmpunlink + warning logs (spool.py), and PEP 561py.typedmarker. - Aligned packaging metadata across
sdk/python/datastack/pyproject.toml,sdk/python/datastack/setup.py, andsdk-python/setup.py, coordinated child-exit shutdown inscripts/start-all.js, and addedengines.node >=20.0.0inpackage.json. - Verified:
39/39TS tests (tests/phase05-packaging.test.ts),6/6Python tests (tests/phase05_sdk_test.py),py_compile,npm run build, andeslint .clean.
- Hardened canonical Python SDK (
- Previous Remediation Cycle (2026-11-18): Completed Phases 01–05 (
39/39TS tests,6/6Python tests). - New Production Hardening Cycle (
BASELINE-2026-11-19):- Re-audited entire codebase from raw source code (
notes/FORENSIC_AUDIT_REPORT.md): 29 verified bugs/mismatches (4Critical,8High,13Medium,4Low). - Phase-01 Complete & Verified (
2026-11-19) (BUG-01,BUG-04,BUG-05,BUG-06):- Blocked volatile RAM admin creation in production without
DATABASE_URL(AdminDatabaseRequiredError), surfaced production DB warning on/vcon/setup, and mapped pre-setup login to409 SETUP_REQUIRED. - Added brute-force attempt throttling (
429+Retry-After) and completion-first ordering toPOST /api/auth/setup. - Added server-side
token_versionrevocation (revokeAdminSessions()) onPOST /api/auth/logout, removed clientOrigintrust from cookieSecureflag, and addedsec-fetch-site: same-originCSRF fallback. - Prevented empty production
jwt_secretpersistence in PG and replaced silent DB writecatch {}swallows inSettingsManagerandSDKManagerwithDATABASE_WRITE_FAILEDpropagation and safe RAM rollback. - Verified:
43/43TS tests,6/6Python tests,npm run build,eslint ., and Playwright login/logout revocation check clean.
- Blocked volatile RAM admin creation in production without
- Phase-02 Complete & Verified (
2026-11-19) (BUG-02,BUG-03,BUG-17,BUG-22):- Protected Port 4000's
inMemoryClientRegistryfrom empty Port 3000bulk_syncwipes (!forceReplace), exposedinMemoryClientson/internal/metrics, added non-destructive hydration inSDKManager(listClientsAsync,getClientAsync,syncAllClientsToIngestionAsync), and auto-reconciled Port 4000 cold starts inApiEngine.getDashboardMetrics(). - Ordered
SettingsManager.updateSettingsAsync()JWT rotation to sync Port 4000 before committing Port 3000 RAM/PG/re-sign, aborting fail-closed (JWT_ROTATION_BRIDGE_FAILED) on bridge rejection/prod unreachable, rolling back Port 4000 on PG/re-sign error, and adding a 60s dual-secret grace window inserver/index.jsrequireInternalAuth(). - Added authenticated
POST /internal/security-checkon Port 4000 (backed by zero-copyvalidateSingleEventLocal) so Port 3000 HTTPSecurityEngine.validateEventAsync()shares rate limits & idempotency with Port 4000 WS even without Redis, and exposed explicitsecurityMode(redis_shared|bridge_coordinated|degraded_local) in/api/metrics. - Replaced per-event PG
UPDATE client_sdksqueries on both Port 3000 (SDKManager) and Port 4000 (server/index.js) with 1s coalescing in-memory deltas flushed via batchedUNNESTSQL updates (flushed on reads and graceful shutdown). - Verified:
47/47TS tests,6/6Python tests,npm run build, andeslint .clean.
- Protected Port 4000's
- Phase-03 Complete & Verified (
2026-11-19) (BUG-18,BUG-19,BUG-20,BUG-29):- Added bounded Redis flush lock wait/retry, periodic Lua
PEXPIRETTL renewal with ownership fencing beforeLTRIM, and explicitFLUSH_LOCK_TIMEOUT(503on/internal/flush) across bothcore/data/buffer.tsandserver/index.js. - Added full deleted-client buffer purge (
DataBufferEngine.purgeClientEventsAsync, Port 4000purgeClientFromLocalBuffer) plus flush-time and recent-read tombstone filtering so deleted client directories never resurrect on subsequent flushes; orderedSDKManager.deleteClientAsync()to purge buffers and sync Port 4000 before deleting disk files. - Converted
server/index.jsonBufferPushtoasyncand awaited it before firing Socket.IO ACK callbacks inserver/socket/handlers.js(sync_dataandbatch_sync), returningBUFFER_PERSIST_FAILEDon persistence error. - Unified canonical CSV storage root resolution via
DATASTACK_STORAGE_DIR(resolveDefaultStorageDir()) acrosscore/Storage/csvStorage.ts,server/workers/csvWriter.js,core/sdk/sdkManager.ts, and.env.example. - Verified:
51/51TS tests,6/6Python tests,npm run build, andeslint .clean.
- Added bounded Redis flush lock wait/retry, periodic Lua
- Phase-04 Complete & Verified (
2026-11-19) (BUG-21,BUG-23,BUG-24):- Aligned HTTP
POST /api/eventssingle-event envelope stripping 1:1 with Port 4000 WSsync_dataand eliminated fabricated{ sample, rand }telemetry on empty{}. - Converted
GET /api/datadirectory/stat scans to non-blocking asyncnode:fs/promiseswith newest-first date-partition pruning, and scaledREDIS_RECENT_LIMITboundedly (500..5000) viaresolveRedisRecentLimit()/DATASTACK_RECENT_LIMITacross Port 3000 & Port 4000. - Replaced all
Math.random()client secrets (crypto.randomBytes(16)) and fallbackeventIds (crypto.randomBytes(6)) with CSPRNG acrosscore/sdk/sdkManager.ts,app/api/events/route.ts, andserver/socket/handlers.js. - Unified canonical PostgreSQL metadata DDL (
CORE_SCHEMA_TABLE_DDLS+CORE_SCHEMA_ALTER_DDLS+ensureCoreSchemaOnPool()) incore/Database/database.ts, delegatedcore/auth/adminUsers.tsto it, mirrored all 5 tables + 3 additiveALTER TABLEguards 1:1 in standaloneserver/index.js, and documented canonicalpgownership inprisma/schema.prisma. - Verified:
55/55TS tests (tests/phase04-ingestion-hygiene.test.tsadded),6/6Python tests,npm run build, andeslint .clean.
- Aligned HTTP
- Phase-05 Complete & Verified (
2026-11-19) (BUG-07,BUG-08,BUG-09,BUG-14):- Fixed Simulator HTTP
Burst 10to send canonical{ clientId, batch }(witheventsbatch alias & explicitacceptedcount inPOST /api/events) and summarize exact server-confirmed counts viasummarizeHttpBatchResult()withoutres.ok ? count : 0falsification. - Fixed
parseEventTimestamp()& addedgetEventUtcDate()inlib/vconUiHelpers.tsfor integer and fractional epoch-seconds (< 1e11 * 1000) 1:1 with backend CSV parsers. - Added pending-queue-only
getPendingBufferedSummary()(?bufferedOnly=true/?includeBufferedSummary=true) and fair multi-client partition scanning (selectRecentPartitionFiles(), lifting the global 5-file cap) inGET /api/events. - Updated
ExportWorkspace.tsxto match buffered events strictly against active client/date filters viacountMatchingBufferedEvents()(preventing empty filtered header downloads) and refresh/api/storage,/api/metrics, andvconCacheimmediately post-export; updatedDashboardOverview.tsxto attribute strictly UTC-today buffered events toEvents todayand callawait refresh()after Quick Export. - Verified:
58/58TS tests,6/6Python tests,npm run build,eslint ., and Playwright E2E browser check (/vcon/simulator,/vcon/export,/vcon/dashboard,0console errors/warnings) clean.
- Fixed Simulator HTTP
- Phase-06 Complete & Verified (
2026-11-19) (BUG-10,BUG-11,BUG-12,BUG-13,BUG-15,BUG-16):- Returned explicit
SESSION_EXPIRED/AUTH_UNAVAILABLEvsINVALID_PASSWORDcodes in/api/auth/profile, replaced dead401 && !currentPasswordredirect withshouldRedirectProfileUnauthorized(), and formattedSession Expireswith calendar date + 24h time (formatSessionExpiryLabel()). - Added opt-in
?purgeFiles=truecheckbox inClientsTable.tsxclient deletion modal and removed the misleading unusedSecret keyrow fromCredentialsDisplay. - Refactored
SettingsWorkspace.tsxnumeric inputs to string drafts (NumericSettingDrafts) validated inline viavalidateEditableSettingsDraft()(noValidatesubmit + tab jump to first invalid field, zero snap-to-0, zero silent clamping) and clarifiedSocket port (PORT_SOCKET)as read-only runtime config. - Unified public endpoint resolution (
resolvePublicEndpoints()) acrossSettingsWorkspace,/vcon/sdk, and/vcon/simulator(consistent on port-80localhostproxies) and updatedSdkSection.tsx+/vcon/sdkwith external path/wheelpip installguidance. - Added pre-hydration
datastack-themebootstrap script +suppressHydrationWarninginapp/layout.tsx(zero light-mode FOUC) and theme restoration/toggle inAdminSetup.tsx. - Upgraded
public/sw.js(datastack-cache-v3) with an uncached503HTML offline navigation fallback shell (buildOfflineShellResponse(),Cache-Control: no-store) for/vcon/dashboardand public navigations. - Verified:
64/64TS tests,6/6Python tests,npm run build,eslint ., and Playwright Chromium E2E check (/vcon/profile,/vcon/clients,/vcon/settings,/vcon/sdk, Light Mode reload,0console errors/warnings) clean.
- Returned explicit
- Fresh Post-Phase-07 Forensic Audit & Baseline Freeze (
BASELINE-2026-11-19-R2):- Previous
Phase-07hardening (BUG-25–BUG-28) landed in Python SDK (client.py), legacysdk-python,scripts/start-all.js/dev-all.js,nginx/default.conf,server/index.js,docker/docker-compose.yml, and.env.example. - Conducted a brand-new 100% raw-source-code audit (
notes/FORENSIC_AUDIT_REPORT.md) identifying 18 verified remaining bugs (0Critical,6High,8Medium,4Low). - Frozen baseline (
BASELINE-2026-11-19-R2) and created the complete 7-document governance & Scope-Lock suite innotes/(README.md,BASELINE_FREEZE.md,PRODUCTION_READINESS_SCOPE_LOCK.md,PRODUCTION_ROADMAP.md,PHASE_EXECUTION_LOG.md,ERROR_HANDLING_MATRIX.md,ACTUAL_IMPLEMENTATION_STATUS.md). - R2 Phase-01 Complete & Verified (
2026-11-19) (BUG-01,BUG-02):- Moved same-origin CSRF enforcement in
middleware.tsahead of the public-route early return soPOST /api/auth/loginandPOST /api/auth/setupreject cross-origin/missing-origin requests with403while preserving same-origin browser submissions,sec-fetch-site: same-originfallback, safeGETroutes, and BearerPOST /api/events. - Extracted shared reverse-proxy-safe
extractClientIp()inlib/adminAuth.ts(trustingx-real-ipfirst, then rightmost non-emptyx-forwarded-forhop) and wired it into bothapp/api/auth/login/route.tsandapp/api/auth/setup/route.tsto block spoofed-XFF brute-force lockout bypasses. - Verified:
14/14tests/phase01-auth.test.tspass,npm run build,npm run lint, live HTTP403/401CSRF probes, and Chromium/vcon/login→/vcon/dashboardE2E (0errors).
- Moved same-origin CSRF enforcement in
- R2 Phase-02 Complete & Verified (
2026-11-19) (BUG-05,BUG-06,BUG-07,BUG-08,BUG-09,BUG-10):- Attached
.on('error', ...)listeners toDatabaseEngine.getPool()(max: 10) andserver/index.jsdbPool, and consolidatedcore/auth/adminUsers.tsonto the sharedDatabaseEngine.getPool(). - Replaced non-atomic Redis
LTRIM+LPUSHclient deletion purging incore/data/buffer.tswith atomic Lua scriptPURGE_CLIENT_LIST_LUA_SCRIPT(cjson.decode+ chunkedRPUSH), and replaced non-atomic RedisINCR+PEXPIRErate limiting with atomic Lua scriptREDIS_RATE_LIMIT_LUA_SCRIPTacross bothcore/Security/security.tsandserver/socket/handlers.js. - Enforced
throwOnLockTimeout: truelocally and remotely onPOST /internal/flushinGET /api/exportandPOST /api/storage, surfacing explicit503 FLUSH_LOCK_TIMEOUT. - Added
DataBufferEngine.registerShutdownHook()and registeredSDKManager.flushPendingClientActivityAsync()so Port3000flushes coalescedtotal_eventsdeltas onSIGTERM/SIGINT. - Wrapped
GET,PATCH(regenerate_token& update), andDELETEinapp/api/clients/[id]/route.tswith structured JSONtry/catcherror responses. - Verified:
14/14tests/phase02-sync.test.tspass,3/3R2 PHASE-02buffer/storage tests pass,npm run buildpass,npm run lintpass (0errors).
- Attached
- R2 Phase-03 Complete & Verified (
2026-11-19) (BUG-03,BUG-04,BUG-11,BUG-12,BUG-13,BUG-15):- Unified
SettingsBridgeSyncStatus&POST /api/settingsbridgeSyncresponse (attempted,ok,synced,status,error,checkedAt) and updateddescribeSettingsBridgeSync()+SettingsWorkspace.tsxso Port4000warnings/online state work reliably. - Counted regular non-symlink
.csvfiles inSDKManager.purgeClientStorageDir()(deleteClientWithStatsAsync()) and returned exactpurgedFilesfromDELETE /api/clients/[id]?purgeFiles=true. - Aligned flat batch item metadata stripping (
eventId,timestamp,clientId) 1:1 with single-event stripping across HTTPPOST /api/events, WSbatch_sync, and Python SDKsend_batch()/send_file(). - Exposed
DataBufferEngine.getLocalBufferSnapshot()for exact Port3000RAM pending summary ingetPendingBufferedSummary(). - Added non-blocking
pruneExpiredPartitionsAsync()off live event loops incore/Storage/csvStorage.ts&server/workers/csvWriter.js(wired intosettingsManager.ts&server/index.js) and deduplicatedGET /api/storageto a singlelistAllFilesAsync()scan with accurate client-scopedstats. - Cleared
selectedClientTokenon/vcon/sdkclient switch (with loading placeholder + disabledCopy Config), added400 INVALID_DATE_RANGEforfrom > toinGET /api/export, and normalizedclientIdfiltering inApiEngine.getExportFiles*& Port4000/internal/recent. - Verified:
21/21phase04-*tests pass,2/2R2 PHASE-03storage tests pass,9/9Python SDK tests pass, Chromium E2E across/vcon/settings,/vcon/clients,/vcon/sdk,/vcon/storage(0console errors),npm run build&npm run lint(0errors).
- Unified
- R2 Phase-04 Complete & Verified (
2026-11-19) (BUG-14,BUG-16,BUG-17,BUG-18):- Added
from __future__ import annotations, importedTuple, and replaced PEP 585tuple[int, Dict[str, Any]]with Python3.8-safeTuple[int, Dict[str, Any]]insdk/python/datastack/datastack/client.py, verified by a full-package Python3.8AST scan test intests/phase05_sdk_test.py. - Added unbuffered streaming
location = /api/export(proxy_buffering off;,proxy_read_timeout 300s;) targetinghttp://nextjs_upstreamin bothnginx/default.confanddocker/coolify-nginx.conf. - Removed
eslint: { ignoreDuringBuilds: true }fromnext.config.ts, setpackage.json"name": "datastack", pruned unused template packages (@google/genai,firebase-tools), and synchronizedpackage-lock.json. - Isolated
DATASTACK_DISABLE_LISTEN=1and live Port4000coupling acrosstests/phase01-auth.test.ts,tests/phase02-sync.test.ts,tests/phase03-storage-buffer.test.ts,tests/phase04-ingestion-hygiene.test.ts, andtests/phase07-deployment.test.ts. - Verified:
83/83full TS suite pass (with live Port4000running, without manual env flags),10/10Python SDK tests +py_compilepass,npm run build(with ESLint active) pass,npm run lint(0errors,0warnings).
- Added
- Progress:
4 / 4phases complete (18 / 18bugs resolved — 100% Production Ready).
- Previous
- Pre-Release Docker & Coolify Deployment Hardening Complete & Verified (
2026-11-19) (DEPLOY-BUG-01–05, audited innotes/DEPLOYMENT_COOLIFY_AUDIT_REPORT.md):DEPLOY-BUG-01: Replaced staticupstreamblocks indocker/coolify-nginx.confwith Docker embedded DNSresolver 127.0.0.11 valid=10s ipv6=off;and variableproxy_passtargets ($nextjs_upstream=http://web:3000,$socketio_upstream=http://ingestion:4000), preventing502 Bad Gatewaystale container IP outages after container restarts/redeploys.DEPLOY-BUG-02: Refactored Port4000startup inserver/index.jsintostartIngestionServer()awaiting boundedsyncSettingsFromDatabase()beforeserver.listen(PORT)so rotated PGjwt_secretand rate-limit settings hydrate prior to/health: 200and/internal/*bridge traffic.DEPLOY-BUG-03: MemoizedensureDatabaseSchema()per activedbPoolinstance inserver/index.js(resetting only on pool swap or DDL failure) so 3-secondsyncSettingsFromDatabase()ticks no longer issue 8CREATE/ALTER TABLEDDLs repeatedly.DEPLOY-BUG-04: ReorderedgracefulShutdown()inserver/index.jsto close Socket.IO/HTTP ingress first (closeServerGracefully()), drainpendingRedisWritesandflushBuffer()/flushPendingClientActivity()with bounded timeouts (2000ms/6000ms), and close Redis (quit()with2000mstimeout +.disconnect()fallback) and PostgreSQL (dbPool.end()bounded2000ms); addedstop_grace_period: 20sto bothwebandingestionindocker/docker-compose.yml.DEPLOY-BUG-05: Clarified CoolifyBase Directory: /andDocker Compose Location: /docker/docker-compose.ymlindocs/users/installation.mdandREADME.md.- Verified:
85/85full TS suite pass (11/11intests/phase07-deployment.test.ts),10/10Python SDK tests pass,npm run buildclean,npm run lint(0errors,0warnings).
- Re-audited entire codebase from raw source code (
- Phase-01 Complete & Verified (
5. Known Constraints & Operational Notes for Future Contributors
- Never add per-row telemetry
INSERTqueries to PostgreSQL or addDROP/TRUNCATEauto-migrations. - Never move Socket.IO handlers into Next.js
app/api/*routes. - When deploying updates to an existing Coolify VPS, always preserve existing
SERVICE_*environment variables and avoiddocker compose down -vsopgdata,datastack-data, andredisdatavolumes remain intact. - After any code change, verify with
npm run build,npm run lint, the TStests/phase*.test.tssuite (npx --yes tsx --test), andpython3 -m unittest tests/phase05_sdk_test.py, then synchronize thisMEMORY.md.