- Add configurable ingestion batch item limits - Improve client serialization and sanitization - Add graceful handling for corrupt GZIP partitions in data API - Expand admin session token scope - Update project documentation for R4 status
80 KiB
MEMORY.md — DataStack Cross-IDE Persistent Project Memory
Last Synchronized: 2026-11-21
Status:R4Production Hardening (BASELINE-2026-11-20-R4:4 / 4Phases Complete — AllR4-Phase-01..04Complete & Verified (AUDIT-01–20,19 / 19actionable findings resolved)).
Scope Rule: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see.ai-rulesandAGENTS.md).
1. Project Identity & Core Principles
- Project: DataStack — Self-hosted, real-time telemetry ingestion, admin governance (
/vcon/*), and partitioned CSV storage/export platform. - Runtime Split (CRITICAL):
- Port
3000(web): Next.js 15 App Router — public landing (/), PWA lifecycle,/vcon/*Admin Console, REST ingestion (POST /api/events), metrics/settings/client APIs, and streaming exports (GET /api/export). - Port
4000(ingestion): Dedicated Node.js Express 5 + Socket.IO server (server/index.js) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker. - Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port
4000.
- Port
- Storage & Database Boundary (CRITICAL):
- PostgreSQL (
pg+prisma/schema.prisma): Stores strictly control-plane metadata (admin_users,admin_login_attempts,client_sdks,deleted_clients,admin_settings). Never insert per-row telemetry events into PostgreSQL. - Partitioned CSV Storage (
/storage/data/{clientId}/{YYYY-MM-DD}.csv): All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (fileRotationMb, e.g.,-part2.csv), and retention pruning (autoPruneDays). - Redis (
ioredis): Shared cross-process buffer queue (datastack:buffer:events), recent event cache (datastack:recent:events), and distributed flush lock (datastack:flush:lock). Persisted in Docker via--appendonly yes(redisdata:/data). Falls back gracefully to local RAM whenREDIS_URLis omitted.
- PostgreSQL (
2. Tech Stack & Local Setup
Stack
- Frontend / Web API: Next.js
15.4.9(App Router,output: "standalone"), React19.2.1, TypeScript5.9.3, Tailwind CSS v4 (@tailwindcss/postcss),lucide-react,motion. - Ingestion Server: Node.js
22/ Express5.3.0, Socket.IO4.8.4. - Persistence & Caching:
pg8.23.1(PostgreSQL 16 in Docker),ioredis6.0.0(Redis 7 AOF in Docker),csv-parse7.0.3,json2csv6.0.0-alpha.2. - Auth & Security:
jsonwebtoken9.0.3, Nodecrypto.scryptpassword hashing, WebCrypto HMAC-SHA256 admin session cookies (middleware.ts), CSRFOrigin/ReferervsHost/X-Forwarded-Hostchecks,DATASTACK_INTERNAL_SECRETtiming-safe bridge auth between Port3000↔4000. - Client SDKs: Official pip-installable Python SDK in
sdk/python/datastack/(datastack) + backward-compatible wrappersdk-python/(datastack-sdk).
Environment & Setup Commands
- Configure Environment: Copy
.env.exampleto.env(in productionNODE_ENV=production, all 4 secrets MUST be 32+ chars:DATASTACK_ADMIN_SESSION_SECRET,DATASTACK_JWT_SECRET,DATASTACK_ADMIN_SETUP_KEY,DATASTACK_INTERNAL_SECRET; optionalDATABASE_URLandREDIS_URL). - Install Dependencies:
npm ci(ornpm install) using rootpackage.json+package-lock.json. - Run Locally (Two Terminals):
- Terminal 1 (Port
3000):npm run dev - Terminal 2 (Port
4000):npm run server
- Terminal 1 (Port
- Build, Lint & Test Verification:
- Build:
npm run build - Lint:
npm run lint(eslint .) - TS Test Suite (Phases 01–03, 06, 07):
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts - Python SDK Tests:
python3 -m unittest tests/phase05_sdk_test.py
- Build:
3. Folder Structure & File Architecture
datastack/
├── .ai-rules # Universal cross-IDE rules (enforces MEMORY.md sync)
├── AGENTS.md # Engineering rules, UI single-source-of-truth & memory sync
├── MEMORY.md # Persistent cross-IDE project memory (this file)
├── package.json / package-lock.json
├── middleware.ts # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
├── app/
│ ├── (landing)/page.tsx # Public landing page
│ ├── globals.css # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
│ ├── layout.tsx / not-found.tsx / error.tsx
│ ├── api/
│ │ ├── auth/ # Admin login, logout, setup (/vcon/setup), profile update
│ │ ├── clients/ # CRUD clients, token regeneration, revocation
│ │ ├── data/ # Paginated CSV search/query across partitions
│ │ ├── events/ # Authenticated HTTP ingestion (single/batch/file records) & recent list
│ │ ├── export/ # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
│ │ ├── files/status/ # Safe server-side Port 4000 Files Manager probe (catches ECONNREFUSED cleanly)
│ │ ├── health/ # Web health + client Bearer token probe + async buffer depth
│ │ ├── metrics/ # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
│ │ ├── settings/ # Allowlisted runtime config get/post + live Port 4000 sync
│ │ └── storage/ # CSV partition listing & manual cross-process buffer flush
│ └── vcon/ # Protected Admin Console pages:
│ ├── dashboard/ clients/ data/ export/ files/ login/ profile/ sdk/ settings/ setup/ simulator/ storage/ team/ audit/
├── components/
│ ├── ui/ # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, etc.)
│ ├── layout/ # MainLayout, Navbar, Sidebar, VconShell, PageShell, AdminPageHeader, Footer
│ ├── sections/ # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, SettingsWorkspace, etc.)
│ └── pwa/ # PwaLifecycle.tsx service worker registration
├── lib/
│ ├── adminAuth.ts # Uniform API session guard (requireAdminApiSession) + proxy-safe extractClientIp()
│ ├── vconCache.ts # In-memory SWR cache + clearAuthCaches() for logout/SW cleanup
│ └── vconUiHelpers.ts # Pure Admin UI sync helpers (settings bounds, bridge sync, UTC presets, timestamps)
├── core/
│ ├── adminSettings/ # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
│ ├── apiEngine/ # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
│ ├── auth/ # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
│ ├── data/ # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
│ ├── Database/ # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
│ ├── sdk/ # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
│ ├── Security/ # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
│ └── Storage/ # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
├── scripts/
│ ├── dev-all.js # Local concurrent Next.js (3000) + Ingestion (4000) dev runner
│ └── start-all.js # Production concurrent Next.js (`next start`) + Ingestion (`server/index.js`) runner
├── server/
│ ├── index.js # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, filesManager mount
│ ├── filesManager.js # Hard-locked cloudcmd@19.23.2 (/vcon/files-engine), isolated Socket.IO, Admin Cookie/DB/RBAC/CSRF/*.lock guard
│ ├── socket/handlers.js # Handshake JWT auth, ping_verify, sync_data, batch_sync
│ └── workers/csvWriter.js # Partitioned CSV batch writer & rotation/pruning worker
├── sdk/python/datastack/ # Official pip-installable Python SDK (`datastack` v2.2.0)
│ ├── datastack/ # client.py, config.py, heartbeat.py, spool.py, exceptions.py
│ ├── config.datastack.example.py
│ ├── pyproject.toml / setup.py
│ ├── README.md # SDK overview, installation, quickstart, docs index
│ └── docs/ # Production SDK Documentation Suite (LicenX Standard):
│ ├── README.md # SDK Documentation Hub & architecture overview
│ ├── INSTALLATION.md # Python runtimes, pip commands, and verification
│ ├── QUICKSTART.md # 5-minute zero-boilerplate onboarding guide
│ ├── API_REFERENCE.md # Complete DataStack class & method specifications
│ ├── CONFIGURATION.md # config.datastack.py & DATASTACK_* environment reference
│ ├── EXAMPLES.md # IoT streaming, FastAPI, batching, and file ingestion
│ ├── ERROR_HANDLING.md # Exception hierarchy, 401/403/413/429 codes & retries
│ └── LIFECYCLE.md # 5-state machine, zero-CPU IDLE mode & offline spooling
├── sdk-python/ # Backward-compatible re-export package (`datastack-sdk`)
├── docker/
│ ├── docker-compose.yml # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
│ ├── Dockerfile.web # Multi-stage standalone Next.js build + safe /app/storage ownership check
│ ├── Dockerfile.server # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
│ ├── Dockerfile.proxy # Nginx reverse proxy container
│ └── coolify-nginx.conf # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
├── nginx/default.conf # Local Nginx proxy config matching coolify-nginx.conf
├── prisma/schema.prisma # Reference PostgreSQL metadata schema
├── public/ # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
├── docs/ # Comprehensive Production Documentation Suite:
│ ├── index.md # Primary documentation hub and portal navigation index
│ ├── MANIFEST.md # Master documentation inventory & authoring governance
│ ├── audit-coverage-report.md # Forensic codebase audit matrix & coverage report
│ ├── users/ # End-User & Operator Guides:
│ │ ├── index.md # User portal overview & reading pathways
│ │ ├── getting-started.md # 5-minute fast-track operator onboarding guide
│ │ ├── installation.md # Production Docker Compose, Coolify & volume guide
│ │ ├── configuration.md # Environment variables & platform runtime settings
│ │ ├── features/ # Deep-dive guides: admin-console, client-management,
│ │ │ # data-simulator, partitioned-storage, data-export
│ │ ├── workflows.md # End-to-end operator journeys & disaster recovery
│ │ ├── troubleshooting.md # Forensic error recovery, status codes & diagnostics
│ │ └── faq.md # Storage architecture & operational questions
│ ├── dev/ # Developer & Contributor Guides:
│ │ ├── index.md # Developer hub index & architectural principles
│ │ ├── architecture.md # Dual-runtime blueprint, service bridge & storage
│ │ ├── codebase-structure.md # File-by-file module, route & component inventory
│ │ ├── local-setup.md # Local workstation setup & dual-terminal workflow
│ │ ├── api-reference.md # Exhaustive REST API & Socket.IO specification
│ │ ├── database-schema.md # PostgreSQL control plane & DDL migrations
│ │ ├── security.md # Security architecture, scrypt, sessions & CSRF
│ │ ├── building-packaging.md # Next.js standalone build & multi-stage Dockerfiles
│ │ ├── testing.md # Automated test suites guide (node:test + unittest)
│ │ └── contributing.md # Architectural rules, memory sync & PR checklist
│ └── reference/ # Technical Reference Portal:
│ ├── index.md # Technical reference index
│ ├── configuration-reference.md # Complete environment variable & setting bounds
│ ├── dependencies.md # Runtime & dev dependencies audit with rationales
│ └── glossary.md # Technical domain terminology & concept glossary
├── notes/ # `R4` Production Hardening Governance Suite (`BASELINE-2026-11-20-R4`):
│ ├── README.md # Master governance index, 6-step zero-drift protocol & approval triggers (`R4-Phase-01`..`04`)
│ ├── FULL_PROJECT_FORENSIC_AUDIT.md # 100% raw-source-verified deep scan (20 findings: 3 Critical, 6 High, 7 Medium, 4 Low)
│ ├── BASELINE_FREEZE.md # Frozen baseline snapshot (`BASELINE-2026-11-20-R4`) & 1:1 audit alignment matrix
│ ├── PRODUCTION_READINESS_SCOPE_LOCK.md # (Doc 001) All 20 defect fix contracts & exact per-phase Scope Locks
│ ├── PRODUCTION_ROADMAP.md # (Doc 002) Dependency-ordered A–Z 4-phase roadmap & verification gates
│ ├── PHASE_EXECUTION_LOG.md # (Doc 003) Live phase completion tracker (`4 / 4` complete) & changelog template
│ ├── ERROR_HANDLING_MATRIX.md # (Doc 004) Verified existing error contracts + 10 planned `R4` contracts (`ERR-R4-01`..`10`)
│ └── ACTUAL_IMPLEMENTATION_STATUS.md # Ground-truth working vs phase-delivered vs pending ledger (Demo & Production)
└── tests/ # Automated verification suites:
├── files-manager.test.ts # CloudCmd RCE lockdown, CRUD/ZIP/UNZIP, RBAC, CSRF, *.lock, DB revocation, proxy & UI wiring
├── phase01-auth.test.ts # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
├── phase02-sync.test.ts # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
├── phase04-ui-sync.test.ts # UI/API settings bounds, bridge status, UTC presets, timestamp & folder matching
├── phase05-packaging.test.ts # Python/Node packaging parity, PEP 561 py.typed, Node engines & start-all exit
├── phase05_sdk_test.py # Python SDK WebSocket/HTTP fallback, offline spool, file streaming, edge hardening
├── phase06-pwa-landing.test.ts # PWA manifest/icons/sw + navigation routes
└── phase07-deployment.test.ts # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush
4. Active Features & Development Status
All planned hardening & deployment phases (Phase-01 through Phase-07) are Complete & Verified:
- Admin Authentication & CSRF Hardening (Completed — Phase-01)
- Fail-closed secret enforcement in
NODE_ENV=production(/vcon/setupbootstrap viaDATASTACK_ADMIN_SETUP_KEY). - Password hashing via
crypto.scrypt, session invalidation via DB-backedtokenVersion, composite IP+email login lockout (admin_login_attempts), and strictOrigin/ReferervsHost/X-Forwarded-HostCSRF verification.
- Fail-closed secret enforcement in
- Real-Time Dual-Runtime Sync (Completed — Phase-02)
- Authenticated
/internal/*bridge between Port3000and Port4000(DATASTACK_INTERNAL_SECRET, backward-compatible Coolify fallback). - Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
- Authenticated
- Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)
- Shared Redis queue (
datastack:buffer:events) with distributed flush lock (NX PX) whereLTRIMexecutes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects. - Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (
GET /api/export).
- Shared Redis queue (
- Admin Console UI & Data/Export Workspace (Completed — Phase-04)
- Built with shared
components/ui/andcomponents/layout/primitives (0remradius, zero shadows, Fluent/Segoe dark & light tokens).
- Built with shared
- Production Python Client SDK (Completed — Phase-05)
- Supports single events (
send()/sync()), auto-splitting/rate-limit-paced batches (send_batch()/batch_sync()), and direct file ingestion (send_file()for.csv,.json,.jsonl,.ndjsonwith SHA-256 deterministiceventId). - Automatic
config.datastack.pydiscovery, WebSocket primary transport -> Heartbeat HTTP fallback ->.datastack_spool/spool.jsonloffline spool & auto-drain.
- Supports single events (
- PWA & Landing Integrity (Completed — Phase-06)
- Complete
public/manifest.json, icons,public/sw.js, and root-relative navigation links.
- Complete
- Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)
- Redis AOF persistence (
redisdata:/data), deterministicpackage-lock.jsonfornpm ci, Nginxlocation = /health→ingestion:4000,/api/healthcontainer probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DBautoPruneDayssafety guard inSettingsManager, DB JWT hydration inSDKManager, gracefulSIGTERM/SIGINTflush inDataBufferEngine, and non-destructiveADD COLUMN IF NOT EXISTSDDL guards.
- Redis AOF persistence (
- Current Remediation Status (2026-11-18):
- Phase-01 Complete & Verified (
BUG-A01,BUG-A02,BUG-A03,BUG-A04,BUG-A05,BUG-E02):- Removed spoofable header/UA CSRF bypasses in
middleware.tsand rejectedsec-fetch-site: cross-site. - Enforced DB-backed
tokenVersionsession validation across/vcon/*layout (app/vcon/layout.tsx+components/layout/VconShell.tsx) and read/export APIs (/api/data,GET /api/events,/api/export,/api/metrics,/api/storage). - Prevented transient DB errors from triggering false
/vcon/setupredirects and prevented swallowedNEXT_REDIRECTerrors on/,/vcon/login, and/vcon/setup. - Standardized
SDKManager.normalizeClientId, blocked deleted/unregistered clients on/api/healthand AdminPOST /api/events, hardenedPOST /api/clients&PATCH /api/clients/[id]input validation, and excluded/vcon/*from Service Worker caching withCLEAR_AUTH_CACHEon sign-out. - Verified:
26/26TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser auth/logout E2E clean.
- Removed spoofable header/UA CSRF bypasses in
- Phase-02 Complete & Verified (
BUG-B01–BUG-B05):- Enforced fail-closed client verification in
server/index.js(verifyClientInDb) backed by PG, deletion tombstones, Redisdatastack:clients:*mirrors, and normalizedclientIdkeys across both runtimes. - Added cross-runtime Redis idempotency (
datastack:idem:*) and fixed-window rate limiting (datastack:rl:*) incore/Security/security.ts(validateEventAsync) andserver/socket/handlers.js(validateSingleEventAsync) with automatic local RAM fallback. - Surfaced bridge sync telemetry (
BridgeSyncResult,SettingsBridgeSyncStatus,runtime.ingestionBridgeStatus,POST /api/settingsbridgeSync) and normalizedSDKManager.incrementEventCountAsync(). - Fixed Socket
sync_dataflat payload preservation (extractSingleEventData), reconciledPORT_SOCKET/PORTprecedence and live/internal/metricsports, and added concurrent productionscripts/start-all.jsfornpm start. - Verified:
29/29TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check (/vcon/dashboard,/vcon/settings,/api/metrics) clean with0console errors.
- Enforced fail-closed client verification in
- Phase-03 Complete & Verified (
BUG-C01–BUG-C06):- Hardened Redis flush lock TTL (
30000ms), added in-processactiveFlushPromisecoalescing with aflushDoneidentity guard (core/data/buffer.ts,server/index.js), isolatedredis.rpushfrom best-effortREDIS_RECENT_KEY, and logged malformed buffered JSON warnings. - Enforced
sanitizeClientId()path containment andlstatsymlink exclusion, unique lock owner tokens (${pid}:${ts}:${uuid}), atomicrenamestale-lock takeover, owner-verified lock unlock, and structuredJSON.stringify([clientId, date])partition grouping incore/Storage/csvStorage.tsandserver/workers/csvWriter.js. - Fixed trailing newline blank-row prevention (
fileEndsWithNewline), proactive size-boundary rotation (-partN.csv),StringDecoder('utf8')streaming row counting, cooperatively lockedpruneExpiredPartitions, and 1-hourPRUNE_COOLDOWN_MSincore/adminSettings/settingsManager.ts. - Enforced
parser/fileStreamdestruction infinallyblocks across/api/data,GET /api/events, and/api/export, error propagation insidemakeExportStream,502surfacing on Port4000flush failure in/api/export, normalizedclientIdfiltering, numeric timestamp sorting (parseTimestampMs), and?format=json/?includeHeader=falseparity for?filename=exports. - Verified:
32/32TS tests,3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check (/vcon/dashboard,/vcon/data,/vcon/storage,/vcon/export,/api/data,/api/storage,/api/export) clean with0console errors.
- Hardened Redis flush lock TTL (
- Phase-04 Complete & Verified (
BUG-D01–BUG-D07):- Extracted pure UI synchronization helpers in
lib/vconUiHelpers.ts(SETTINGS_LIMITS,normalizeEditableSettings,describeSettingsBridgeSync,mapBridgeStatusToSocketUiState,parseEventTimestamp,formatEventTimeLabel,getUtcDateRangeForPreset,matchesSelectedClientFolder,extractContentDispositionFilename). - Updated
components/sections/SettingsWorkspace.tsx(BUG-D01) to probe/api/metricshonestly (removing false/api/healthfallback), surfacebridgeSyncwarning/error toasts on save, align integer bounds with/api/settings(1..100000,1..10240,0..3650), and eliminate the startupserverPortdouble-fetch. - Updated
app/vcon/storage/page.tsx(BUG-D02) andcomponents/sections/ExportWorkspace.tsx(BUG-D03) withcache: 'no-store',401redirects, UTC preset ranges, normalized client folder filters, and fetch+Blob downloads with502/JSON error surfacing. - Updated
components/sections/DashboardOverview.tsx(BUG-D04) andcomponents/sections/DataBrowser.tsx(BUG-D05) withingestionBridgeStatusdisplay,auth_erroralert banner, safe ISO/epoch-ms timestamp formatting, invalid date-range guard, and Enter-key filter submission. - Updated
components/sections/NewClientModal.tsx,components/sections/ClientsTable.tsx,app/vcon/simulator/page.tsx,app/vcon/sdk/page.tsx,components/sections/ProfileWorkspace.tsx, andcomponents/layout/Sidebar.tsx(BUG-D06,BUG-D07) with1..100000rate limit parity, liveruntime.serverPortsync, real HTTP batch ingestion in Simulator, password bounds check, anddatastack:profile-updatedlive Sidebar sync. - Verified:
37/37TS tests (tests/phase04-ui-sync.test.ts),3/3Python tests,npm run buildclean,eslint .clean, Playwright browser check across all Admin Console routes clean with0console errors.
- Extracted pure UI synchronization helpers in
- Phase-05 Complete & Verified (
BUG-E01,BUG-E03,BUG-E04):- Hardened canonical Python SDK (
sdk/python/datastack/datastack/*): normalizedCLIENT_IDcomparison (_normalize_client_id), sanitized ragged CSV extra columns (restkey="_extra"), accuratesend_file()success+unprocessedItems, partial-batchrateLimitedEventIdsretry exception fallback,DataStackConfigErrorwrapping for malformed numeric env/config values (config.py), concurrentstop()vsset_idle_connected()race fix + clean thread join (heartbeat.py),os.fsync+ orphan.tmpunlink + warning logs (spool.py), and PEP 561py.typedmarker. - Aligned packaging metadata across
sdk/python/datastack/pyproject.toml,sdk/python/datastack/setup.py, andsdk-python/setup.py, coordinated child-exit shutdown inscripts/start-all.js, and addedengines.node >=20.0.0inpackage.json. - Verified:
39/39TS tests (tests/phase05-packaging.test.ts),6/6Python tests (tests/phase05_sdk_test.py),py_compile,npm run build, andeslint .clean.
- Hardened canonical Python SDK (
- Previous Remediation Cycle (2026-11-18): Completed Phases 01–05 (
39/39TS tests,6/6Python tests). - New Production Hardening Cycle (
BASELINE-2026-11-19):-
Re-audited entire codebase from raw source code (
notes/FORENSIC_AUDIT_REPORT.md): 29 verified bugs/mismatches (4Critical,8High,13Medium,4Low). -
Phase-01 Complete & Verified (
2026-11-19) (BUG-01,BUG-04,BUG-05,BUG-06):- Blocked volatile RAM admin creation in production without
DATABASE_URL(AdminDatabaseRequiredError), surfaced production DB warning on/vcon/setup, and mapped pre-setup login to409 SETUP_REQUIRED. - Added brute-force attempt throttling (
429+Retry-After) and completion-first ordering toPOST /api/auth/setup. - Added server-side
token_versionrevocation (revokeAdminSessions()) onPOST /api/auth/logout, removed clientOrigintrust from cookieSecureflag, and addedsec-fetch-site: same-originCSRF fallback. - Prevented empty production
jwt_secretpersistence in PG and replaced silent DB writecatch {}swallows inSettingsManagerandSDKManagerwithDATABASE_WRITE_FAILEDpropagation and safe RAM rollback. - Verified:
43/43TS tests,6/6Python tests,npm run build,eslint ., and Playwright login/logout revocation check clean.
- Blocked volatile RAM admin creation in production without
-
Phase-02 Complete & Verified (
2026-11-19) (BUG-02,BUG-03,BUG-17,BUG-22):- Protected Port 4000's
inMemoryClientRegistryfrom empty Port 3000bulk_syncwipes (!forceReplace), exposedinMemoryClientson/internal/metrics, added non-destructive hydration inSDKManager(listClientsAsync,getClientAsync,syncAllClientsToIngestionAsync), and auto-reconciled Port 4000 cold starts inApiEngine.getDashboardMetrics(). - Ordered
SettingsManager.updateSettingsAsync()JWT rotation to sync Port 4000 before committing Port 3000 RAM/PG/re-sign, aborting fail-closed (JWT_ROTATION_BRIDGE_FAILED) on bridge rejection/prod unreachable, rolling back Port 4000 on PG/re-sign error, and adding a 60s dual-secret grace window inserver/index.jsrequireInternalAuth(). - Added authenticated
POST /internal/security-checkon Port 4000 (backed by zero-copyvalidateSingleEventLocal) so Port 3000 HTTPSecurityEngine.validateEventAsync()shares rate limits & idempotency with Port 4000 WS even without Redis, and exposed explicitsecurityMode(redis_shared|bridge_coordinated|degraded_local) in/api/metrics. - Replaced per-event PG
UPDATE client_sdksqueries on both Port 3000 (SDKManager) and Port 4000 (server/index.js) with 1s coalescing in-memory deltas flushed via batchedUNNESTSQL updates (flushed on reads and graceful shutdown). - Verified:
47/47TS tests,6/6Python tests,npm run build, andeslint .clean.
- Protected Port 4000's
-
Phase-03 Complete & Verified (
2026-11-19) (BUG-18,BUG-19,BUG-20,BUG-29):- Added bounded Redis flush lock wait/retry, periodic Lua
PEXPIRETTL renewal with ownership fencing beforeLTRIM, and explicitFLUSH_LOCK_TIMEOUT(503on/internal/flush) across bothcore/data/buffer.tsandserver/index.js. - Added full deleted-client buffer purge (
DataBufferEngine.purgeClientEventsAsync, Port 4000purgeClientFromLocalBuffer) plus flush-time and recent-read tombstone filtering so deleted client directories never resurrect on subsequent flushes; orderedSDKManager.deleteClientAsync()to purge buffers and sync Port 4000 before deleting disk files. - Converted
server/index.jsonBufferPushtoasyncand awaited it before firing Socket.IO ACK callbacks inserver/socket/handlers.js(sync_dataandbatch_sync), returningBUFFER_PERSIST_FAILEDon persistence error. - Unified canonical CSV storage root resolution via
DATASTACK_STORAGE_DIR(resolveDefaultStorageDir()) acrosscore/Storage/csvStorage.ts,server/workers/csvWriter.js,core/sdk/sdkManager.ts, and.env.example. - Verified:
51/51TS tests,6/6Python tests,npm run build, andeslint .clean.
- Added bounded Redis flush lock wait/retry, periodic Lua
-
Phase-04 Complete & Verified (
2026-11-19) (BUG-21,BUG-23,BUG-24):- Aligned HTTP
POST /api/eventssingle-event envelope stripping 1:1 with Port 4000 WSsync_dataand eliminated fabricated{ sample, rand }telemetry on empty{}. - Converted
GET /api/datadirectory/stat scans to non-blocking asyncnode:fs/promiseswith newest-first date-partition pruning, and scaledREDIS_RECENT_LIMITboundedly (500..5000) viaresolveRedisRecentLimit()/DATASTACK_RECENT_LIMITacross Port 3000 & Port 4000. - Replaced all
Math.random()client secrets (crypto.randomBytes(16)) and fallbackeventIds (crypto.randomBytes(6)) with CSPRNG acrosscore/sdk/sdkManager.ts,app/api/events/route.ts, andserver/socket/handlers.js. - Unified canonical PostgreSQL metadata DDL (
CORE_SCHEMA_TABLE_DDLS+CORE_SCHEMA_ALTER_DDLS+ensureCoreSchemaOnPool()) incore/Database/database.ts, delegatedcore/auth/adminUsers.tsto it, mirrored all 5 tables + 3 additiveALTER TABLEguards 1:1 in standaloneserver/index.js, and documented canonicalpgownership inprisma/schema.prisma. - Verified:
55/55TS tests (tests/phase04-ingestion-hygiene.test.tsadded),6/6Python tests,npm run build, andeslint .clean.
- Aligned HTTP
-
Phase-05 Complete & Verified (
2026-11-19) (BUG-07,BUG-08,BUG-09,BUG-14):- Fixed Simulator HTTP
Burst 10to send canonical{ clientId, batch }(witheventsbatch alias & explicitacceptedcount inPOST /api/events) and summarize exact server-confirmed counts viasummarizeHttpBatchResult()withoutres.ok ? count : 0falsification. - Fixed
parseEventTimestamp()& addedgetEventUtcDate()inlib/vconUiHelpers.tsfor integer and fractional epoch-seconds (< 1e11 * 1000) 1:1 with backend CSV parsers. - Added pending-queue-only
getPendingBufferedSummary()(?bufferedOnly=true/?includeBufferedSummary=true) and fair multi-client partition scanning (selectRecentPartitionFiles(), lifting the global 5-file cap) inGET /api/events. - Updated
ExportWorkspace.tsxto match buffered events strictly against active client/date filters viacountMatchingBufferedEvents()(preventing empty filtered header downloads) and refresh/api/storage,/api/metrics, andvconCacheimmediately post-export; updatedDashboardOverview.tsxto attribute strictly UTC-today buffered events toEvents todayand callawait refresh()after Quick Export. - Verified:
58/58TS tests,6/6Python tests,npm run build,eslint ., and Playwright E2E browser check (/vcon/simulator,/vcon/export,/vcon/dashboard,0console errors/warnings) clean.
- Fixed Simulator HTTP
-
Phase-06 Complete & Verified (
2026-11-19) (BUG-10,BUG-11,BUG-12,BUG-13,BUG-15,BUG-16):- Returned explicit
SESSION_EXPIRED/AUTH_UNAVAILABLEvsINVALID_PASSWORDcodes in/api/auth/profile, replaced dead401 && !currentPasswordredirect withshouldRedirectProfileUnauthorized(), and formattedSession Expireswith calendar date + 24h time (formatSessionExpiryLabel()). - Added opt-in
?purgeFiles=truecheckbox inClientsTable.tsxclient deletion modal and removed the misleading unusedSecret keyrow fromCredentialsDisplay. - Refactored
SettingsWorkspace.tsxnumeric inputs to string drafts (NumericSettingDrafts) validated inline viavalidateEditableSettingsDraft()(noValidatesubmit + tab jump to first invalid field, zero snap-to-0, zero silent clamping) and clarifiedSocket port (PORT_SOCKET)as read-only runtime config. - Unified public endpoint resolution (
resolvePublicEndpoints()) acrossSettingsWorkspace,/vcon/sdk, and/vcon/simulator(consistent on port-80localhostproxies) and updatedSdkSection.tsx+/vcon/sdkwith external path/wheelpip installguidance. - Added pre-hydration
datastack-themebootstrap script +suppressHydrationWarninginapp/layout.tsx(zero light-mode FOUC) and theme restoration/toggle inAdminSetup.tsx. - Upgraded
public/sw.js(datastack-cache-v3) with an uncached503HTML offline navigation fallback shell (buildOfflineShellResponse(),Cache-Control: no-store) for/vcon/dashboardand public navigations. - Verified:
64/64TS tests,6/6Python tests,npm run build,eslint ., and Playwright Chromium E2E check (/vcon/profile,/vcon/clients,/vcon/settings,/vcon/sdk, Light Mode reload,0console errors/warnings) clean.
- Returned explicit
-
Fresh Post-Phase-07 Forensic Audit & Baseline Freeze (
BASELINE-2026-11-19-R2):- Previous
Phase-07hardening (BUG-25–BUG-28) landed in Python SDK (client.py), legacysdk-python,scripts/start-all.js/dev-all.js,nginx/default.conf,server/index.js,docker/docker-compose.yml, and.env.example. - Conducted a brand-new 100% raw-source-code audit (
notes/FORENSIC_AUDIT_REPORT.md) identifying 18 verified remaining bugs (0Critical,6High,8Medium,4Low). - Frozen baseline (
BASELINE-2026-11-19-R2) and created the complete 7-document governance & Scope-Lock suite innotes/(README.md,BASELINE_FREEZE.md,PRODUCTION_READINESS_SCOPE_LOCK.md,PRODUCTION_ROADMAP.md,PHASE_EXECUTION_LOG.md,ERROR_HANDLING_MATRIX.md,ACTUAL_IMPLEMENTATION_STATUS.md). - R2 Phase-01 Complete & Verified (
2026-11-19) (BUG-01,BUG-02):- Moved same-origin CSRF enforcement in
middleware.tsahead of the public-route early return soPOST /api/auth/loginandPOST /api/auth/setupreject cross-origin/missing-origin requests with403while preserving same-origin browser submissions,sec-fetch-site: same-originfallback, safeGETroutes, and BearerPOST /api/events. - Extracted shared reverse-proxy-safe
extractClientIp()inlib/adminAuth.ts(trustingx-real-ipfirst, then rightmost non-emptyx-forwarded-forhop) and wired it into bothapp/api/auth/login/route.tsandapp/api/auth/setup/route.tsto block spoofed-XFF brute-force lockout bypasses. - Verified:
14/14tests/phase01-auth.test.tspass,npm run build,npm run lint, live HTTP403/401CSRF probes, and Chromium/vcon/login→/vcon/dashboardE2E (0errors).
- Moved same-origin CSRF enforcement in
- R2 Phase-02 Complete & Verified (
2026-11-19) (BUG-05,BUG-06,BUG-07,BUG-08,BUG-09,BUG-10):- Attached
.on('error', ...)listeners toDatabaseEngine.getPool()(max: 10) andserver/index.jsdbPool, and consolidatedcore/auth/adminUsers.tsonto the sharedDatabaseEngine.getPool(). - Replaced non-atomic Redis
LTRIM+LPUSHclient deletion purging incore/data/buffer.tswith atomic Lua scriptPURGE_CLIENT_LIST_LUA_SCRIPT(cjson.decode+ chunkedRPUSH), and replaced non-atomic RedisINCR+PEXPIRErate limiting with atomic Lua scriptREDIS_RATE_LIMIT_LUA_SCRIPTacross bothcore/Security/security.tsandserver/socket/handlers.js. - Enforced
throwOnLockTimeout: truelocally and remotely onPOST /internal/flushinGET /api/exportandPOST /api/storage, surfacing explicit503 FLUSH_LOCK_TIMEOUT. - Added
DataBufferEngine.registerShutdownHook()and registeredSDKManager.flushPendingClientActivityAsync()so Port3000flushes coalescedtotal_eventsdeltas onSIGTERM/SIGINT. - Wrapped
GET,PATCH(regenerate_token& update), andDELETEinapp/api/clients/[id]/route.tswith structured JSONtry/catcherror responses. - Verified:
14/14tests/phase02-sync.test.tspass,3/3R2 PHASE-02buffer/storage tests pass,npm run buildpass,npm run lintpass (0errors).
- Attached
- R2 Phase-03 Complete & Verified (
2026-11-19) (BUG-03,BUG-04,BUG-11,BUG-12,BUG-13,BUG-15):- Unified
SettingsBridgeSyncStatus&POST /api/settingsbridgeSyncresponse (attempted,ok,synced,status,error,checkedAt) and updateddescribeSettingsBridgeSync()+SettingsWorkspace.tsxso Port4000warnings/online state work reliably. - Counted regular non-symlink
.csvfiles inSDKManager.purgeClientStorageDir()(deleteClientWithStatsAsync()) and returned exactpurgedFilesfromDELETE /api/clients/[id]?purgeFiles=true. - Aligned flat batch item metadata stripping (
eventId,timestamp,clientId) 1:1 with single-event stripping across HTTPPOST /api/events, WSbatch_sync, and Python SDKsend_batch()/send_file(). - Exposed
DataBufferEngine.getLocalBufferSnapshot()for exact Port3000RAM pending summary ingetPendingBufferedSummary(). - Added non-blocking
pruneExpiredPartitionsAsync()off live event loops incore/Storage/csvStorage.ts&server/workers/csvWriter.js(wired intosettingsManager.ts&server/index.js) and deduplicatedGET /api/storageto a singlelistAllFilesAsync()scan with accurate client-scopedstats. - Cleared
selectedClientTokenon/vcon/sdkclient switch (with loading placeholder + disabledCopy Config), added400 INVALID_DATE_RANGEforfrom > toinGET /api/export, and normalizedclientIdfiltering inApiEngine.getExportFiles*& Port4000/internal/recent. - Verified:
21/21phase04-*tests pass,2/2R2 PHASE-03storage tests pass,9/9Python SDK tests pass, Chromium E2E across/vcon/settings,/vcon/clients,/vcon/sdk,/vcon/storage(0console errors),npm run build&npm run lint(0errors).
- Unified
- R2 Phase-04 Complete & Verified (
2026-11-19) (BUG-14,BUG-16,BUG-17,BUG-18):- Added
from __future__ import annotations, importedTuple, and replaced PEP 585tuple[int, Dict[str, Any]]with Python3.8-safeTuple[int, Dict[str, Any]]insdk/python/datastack/datastack/client.py, verified by a full-package Python3.8AST scan test intests/phase05_sdk_test.py. - Added unbuffered streaming
location = /api/export(proxy_buffering off;,proxy_read_timeout 300s;) targetinghttp://nextjs_upstreamin bothnginx/default.confanddocker/coolify-nginx.conf. - Removed
eslint: { ignoreDuringBuilds: true }fromnext.config.ts, setpackage.json"name": "datastack", pruned unused template packages (@google/genai,firebase-tools), and synchronizedpackage-lock.json. - Isolated
DATASTACK_DISABLE_LISTEN=1and live Port4000coupling acrosstests/phase01-auth.test.ts,tests/phase02-sync.test.ts,tests/phase03-storage-buffer.test.ts,tests/phase04-ingestion-hygiene.test.ts, andtests/phase07-deployment.test.ts. - Verified:
83/83full TS suite pass (with live Port4000running, without manual env flags),10/10Python SDK tests +py_compilepass,npm run build(with ESLint active) pass,npm run lint(0errors,0warnings).
- Added
- Progress:
4 / 4phases complete (18 / 18bugs resolved — 100% Production Ready).
- Previous
-
Pre-Release Docker & Coolify Deployment Hardening Complete & Verified (
2026-11-19) (DEPLOY-BUG-01–05, audited innotes/DEPLOYMENT_COOLIFY_AUDIT_REPORT.md):DEPLOY-BUG-01: Replaced staticupstreamblocks indocker/coolify-nginx.confwith Docker embedded DNSresolver 127.0.0.11 valid=10s ipv6=off;and variableproxy_passtargets ($nextjs_upstream=http://web:3000,$socketio_upstream=http://ingestion:4000), preventing502 Bad Gatewaystale container IP outages after container restarts/redeploys.DEPLOY-BUG-02: Refactored Port4000startup inserver/index.jsintostartIngestionServer()awaiting boundedsyncSettingsFromDatabase()beforeserver.listen(PORT)so rotated PGjwt_secretand rate-limit settings hydrate prior to/health: 200and/internal/*bridge traffic.DEPLOY-BUG-03: MemoizedensureDatabaseSchema()per activedbPoolinstance inserver/index.js(resetting only on pool swap or DDL failure) so 3-secondsyncSettingsFromDatabase()ticks no longer issue 8CREATE/ALTER TABLEDDLs repeatedly.DEPLOY-BUG-04: ReorderedgracefulShutdown()inserver/index.jsto close Socket.IO/HTTP ingress first (closeServerGracefully()), drainpendingRedisWritesandflushBuffer()/flushPendingClientActivity()with bounded timeouts (2000ms/6000ms), and close Redis (quit()with2000mstimeout +.disconnect()fallback) and PostgreSQL (dbPool.end()bounded2000ms); addedstop_grace_period: 20sto bothwebandingestionindocker/docker-compose.yml.DEPLOY-BUG-05: Clarified CoolifyBase Directory: /andDocker Compose Location: /docker/docker-compose.ymlindocs/users/installation.mdandREADME.md.- Verified:
85/85full TS suite pass (11/11intests/phase07-deployment.test.ts),10/10Python SDK tests pass,npm run buildclean,npm run lint(0errors,0warnings).
-
V2 Baseline Freeze & Governance Planning Complete (
BASELINE-2026-11-19-V2):- Froze the clean, Deployment-Hardened runtime state (
85/85TS tests,10/10Python tests, cleannpm run build&eslint ., zero runtime code changes) asBASELINE-2026-11-19-V2. - Created raw-source verified V2 planning & contract documents in
notes/:notes/V2_FEATURE_GAP_AUDIT.md(V2-FEAT-01–V2-FEAT-07gap analysis + full/vcon/*control mapping).notes/V2_DATABASE_AND_SETTINGS_CONTRACT.md(zero-loss additive PostgreSQL DDL rules preservingadmin_usersid = 1 CHECK (id = 1), AES-256-GCM DB secret encryption + maskedGET /api/settings, Port3000↔4000live sync).notes/BASELINE_FREEZE.md,notes/PRODUCTION_READINESS_SCOPE_LOCK.md(Doc001Scope Locks forV2-Phase-01–04),notes/PRODUCTION_ROADMAP.md(Doc002),notes/PHASE_EXECUTION_LOG.md(Doc003,0 / 4V2 phases done),notes/ERROR_HANDLING_MATRIX.md(Doc004, includingV2-E01–V2-E12),notes/ACTUAL_IMPLEMENTATION_STATUS.md, andnotes/README.md.
- Roadmap grouped into 4 Scope-Locked V2 Phases (
V2-Phase-01Archival/Gzip + Quota/Disk Guard;V2-Phase-02Webhooks/Alerts + IP Allowlist/Expiry + Prometheus/NDJSON;V2-Phase-03Multi-Admin RBAC + Audit Log;V2-Phase-04Node.js/TypeScript SDK +/vcon/sdkguide). V2-Phase-01Complete & Verified (2026-11-19) (V2-FEAT-01,V2-FEAT-03,V2-E01–V2-E05):- AES-256-GCM Secret Vault (
core/Security/secretVault.ts): Deterministic HKDF-SHA256 key derivation (encryptSecret,decryptSecret,getMaskedHintFromEncrypted,SecretVaultError) sos3_secret_access_key_encis encrypted at rest in PostgreSQL and never leaked in plaintext byGET /api/settings(hasS3SecretKey+s3SecretKeyMask: "••••xxxx"). - S3 / Cloudflare R2 / MinIO SigV4 Client (
core/Storage/s3Archive.ts): Pure Nodenode:crypto+fetchAWS Signature v4 client (testS3Connection,uploadFileToS3,buildS3ObjectKey,normalizeS3EndpointUrl) withPOST /api/storage/s3-testandPOST /api/storage/archive. - Transparent
.csv.gzAuto-Compression & Safe Archive-Before-Prune (core/Storage/csvStorage.ts,server/workers/csvWriter.js): Cooperatively locked atomic.csv→.csv.gzcompression (compressOldPartitionsAsync/compressOldPartitions) excluding today's active UTC partition; transparent.csv+.csv.gzreading/streaming acrossGET /api/data(surfacing500 Corrupt .csv.gz partitionon corrupt gzip —V2-E03),GET /api/events, andGET /api/export(application/gzipfor raw single.csv.gzdownload); fail-safebeforeDeleteFileS3 upload hook (V2-E02) that never deletes local partitions if S3 upload fails (502 S3_ARCHIVE_FAILED). - Per-Client Storage Quota (
storageQuotaMb) & VPS Disk Pause Guard (diskPauseThresholdPct): Additive PostgreSQL columns (client_sdks.storage_quota_mb+ 9admin_settingscolumns incore/Database/database.ts, synced inserver/index.jswhile preservingddlCount === 8),30sTTL cachedstatfs+ directory byte counters, and HTTP507 Insufficient Storage(DISK_SPACE_LOW/STORAGE_QUOTA_EXCEEDED—V2-E04,V2-E05) onPOST /api/eventsand Port4000sync_data/batch_syncwhile keeping all Admin read/export/settings routes online. - Admin UI Controls (
SettingsWorkspace.tsx,NewClientModal.tsx,ClientsTable.tsx,lib/vconUiHelpers.ts): Added live VPS disk usage bar,Auto-compress (.csv → .csv.gz),Disk pause threshold (0 or 50–99%), fullS3 / R2 off-site archiveconfiguration withTest S3 Connection&Archive Now, and per-clientStorage quota (MB, 0 = unlimited)in modal and table. - Verified:
89/89TS tests (4/4intests/v2-phase01-storage-quotas.test.ts),10/10Python SDK tests,npm run buildclean,npm run lint(0errors /0warnings), and Chromium E2E across/vcon/settings(Storage tab) and/vcon/clients(0console errors).
- AES-256-GCM Secret Vault (
V2-Phase-02Complete & Verified (2026-11-19) (V2-FEAT-02,V2-FEAT-04,V2-FEAT-06,V2-E06–V2-E09):- Per-Client IP / CIDR Allowlist & Token Expiration (
core/Security/ipAllowlist.ts,core/sdk/sdkManager.ts,app/api/clients/**,app/api/events/route.ts,server/index.js,server/socket/handlers.js): Validates and normalizes comma-separated IPv4, IPv4 CIDR (/0..32), IPv6, IPv6 CIDR (/0..128), IPv4-mapped IPv6 (::ffff:), loopback equivalence (127.0.0.1↔::1), and wildcard*; validates ISOexpiresAttimestamps. Enforced across both Port3000POST /api/eventsand Port4000Socket.IO handshakes/events (403 CLIENT_IP_FORBIDDEN—V2-E07;401 CLIENT_TOKEN_EXPIRED—V2-E08), synced live via/internal/client-syncand additive PostgreSQL columns (client_sdks.allowed_ips,client_sdks.expires_at). - Outbound Webhook Alerting (
core/alerts/webhookNotifier.ts,core/adminSettings/settingsManager.ts,app/api/settings/webhook-test/route.ts): Sends Slack (text), Discord (content), and structured JSON webhook payloads signed with optional HMAC-SHA256 (X-DataStack-Signature: sha256=<hex>) and deduplicated perevent:clientIdvia configurablewebhookCooldownSec(10..86400s, default300s). Triggers non-blocking alerts onDISK_SPACE_LOW,STORAGE_QUOTA_EXCEEDED,S3_ARCHIVE_FAILED,CLIENT_IP_FORBIDDEN, andCLIENT_TOKEN_EXPIRED, plus on-demandPOST /api/settings/webhook-test(200on delivery,502 WEBHOOK_DELIVERY_FAILEDon upstream error —V2-E06).webhook_secret_encis encrypted via AES-256-GCM Secret Vault and masked inGET /api/settings(hasWebhookSecret,webhookSecretMask). - Prometheus Scrape Endpoint (
app/api/prometheus/route.ts):GET /api/prometheusoutputs Prometheus 0.0.4 text exposition format (datastack_up,datastack_ingestion_bridge_up,datastack_active_sockets,datastack_events_total,datastack_buffer_events_total,datastack_storage_files_total,datastack_storage_bytes_total,datastack_storage_rows_total,datastack_disk_used_percent,datastack_disk_free_bytes,datastack_disk_total_bytes,datastack_clients_total,datastack_client_events_total,datastack_client_storage_bytes). Protected by constant-time SHA-256 Bearer token comparison (prometheus_metrics_token_encencrypted at rest) or active Admin Console session cookie (401 PROMETHEUS_UNAUTHORIZED—V2-E09). - Streaming
.ndjsonExport (app/api/export/route.ts,components/sections/ExportWorkspace.tsx): Addedformat=ndjson(Content-Type: application/x-ndjson; charset=utf-8,.ndjsonextension) streaming newline-delimited JSON objects across both.csvand.csv.gzpartitions, withNDJSONformat radio selector in/vcon/export. - Admin UI Controls (
SettingsWorkspace.tsx,NewClientModal.tsx,ClientsTable.tsx,lib/vconUiHelpers.ts): AddedOutbound webhook alertsandPrometheus metrics (/api/prometheus)groups under/vcon/settings→ Security tab, andAllowed IPs / CIDR+Token expirationcontrols in/vcon/clientsNew/Edit modals and table badges. - Verified:
93/93TS tests (4/4intests/v2-phase02-alerts-ip-prometheus.test.ts),10/10Python SDK tests,npm run buildclean,npm run lint(0errors /0warnings), and Chromium E2E across/vcon/settings(0console errors).
- Per-Client IP / CIDR Allowlist & Token Expiration (
V2-Phase-03Complete & Verified (2026-11-19) (V2-FEAT-05,V2-E10–V2-E11):- Multi-Admin Role-Based Access Control (
core/Database/database.ts,core/auth/adminSession.ts,core/auth/adminUsers.ts,lib/adminAuth.ts,app/api/team/route.ts,app/api/team/[id]/route.ts): Added canonicaladmin_team_memberstable in PostgreSQL (plus in-memory fallback) while preservingadmin_users(id = 1 CHECK (id = 1)) as the immutable Primary Owner (owner-1, protected by400 CANNOT_REVOKE_PRIMARY_OWNER—V2-E11). Supports rolesowner(rank 3 — full platform settings, secrets & team management),admin/operator(rank 2 — manage SDK clients, rotate tokens, manual buffer flushes, S3 test/archive), andviewer(rank 1 — read-only dashboards, events, clients, settings & exports). Enforced centrally viarequireAdminApiSession(req, { minRole })returning403 INSUFFICIENT_ROLE_PERMISSION(V2-E10). Role/status/password changes on any team member immediately bumptoken_versionto revoke existing sessions. - Immutable Admin Audit Log (
core/auth/auditLog.ts,app/api/audit/route.ts): Added canonicaladmin_audit_logstable in PostgreSQL (plus bounded 1,000-entry in-memory fallback) with automatic secret key redaction. Records governance events (ADMIN_LOGIN,ADMIN_PROFILE_UPDATED,ADMIN_PASSWORD_UPDATED,TEAM_MEMBER_CREATED,TEAM_MEMBER_UPDATED,TEAM_MEMBER_REVOKED,TEAM_MEMBER_DELETED,CLIENT_CREATED,CLIENT_UPDATED,CLIENT_TOKEN_ROTATED,CLIENT_DELETED,SETTINGS_UPDATED,S3_ARCHIVE_TRIGGERED,WEBHOOK_TEST_TRIGGERED) without ever inserting telemetry events into PostgreSQL. Exposed filterableGET /api/audit(?action=,?actor=,?q=,?limit=). - Admin Console UI (
components/sections/TeamWorkspace.tsx,components/sections/AuditLogViewer.tsx,app/vcon/team/page.tsx,app/vcon/audit/page.tsx,components/layout/Sidebar.tsx): Added/vcon/team(Team & RBAC directory, add member modal, role selector, session revocation & deletion) and/vcon/audit(filterable governance trail viewer) linked fromSidebar.tsx. - Verified:
97/97TS tests (4/4intests/v2-phase03-rbac-audit.test.ts),10/10Python SDK tests,npm run buildclean,npm run lint(0errors /0warnings), and Chromium E2E across/vcon/teamand/vcon/audit(0console errors).
- Multi-Admin Role-Based Access Control (
V2-Phase-04Complete & Verified (2026-11-20) (V2-FEAT-07,V2-E12):- Official Node.js / TypeScript Client SDK (
sdk/node/datastack/—@datastack/sdkv2.2.0): Createdsdk/node/datastack/package.json,types.ts,spool.ts,index.ts, andREADME.md. Supportsdatastack.config.json/DATASTACK_*env / constructor options with strictnormalizeClientIdvs JWTclientId/subverification (DataStackConfigError—V2-E12), Socket.IO primary streaming (connect(),send()/sync(),sendBatch()/batchSync()), automatic HTTPPOST /api/eventsfallback, recursive413 PAYLOAD_TOO_LARGEbatch splitting, partial-batch429 RATE_LIMIT_EXCEEDEDretry, typed terminal error mapping (401 CLIENT_TOKEN_EXPIRED&403 CLIENT_IP_FORBIDDEN->DataStackAuthError,507 STORAGE_QUOTA_EXCEEDED&DISK_SPACE_LOW->DataStackQuotaExceededError), durable.datastack_spool/spool.jsonllocal disk spooling (DiskSpooler) &flushSpool(), direct.csv/.json/.jsonl/.ndjsonfile ingestion (sendFile()with deterministic SHA-256eventIdviamakeDeterministicFileEventId), and gracefulawait client.shutdown(timeoutMs)drain. - Multi-Language SDK Guide (
app/vcon/sdk/page.tsx): Added interactive language tabs (Python SDK (v2.2.0),Node.js / TypeScript SDK (v2.2.0),HTTP / cURL) with live client JWT token population and copy-readyconfig.datastack.py/datastack.config.jsonsnippets. - Verified:
101/101TS tests (4/4intests/v2-phase04-node-sdk.test.ts),10/10Python SDK tests,npm run buildclean, andnpm run lint(0errors /0warnings).
- Official Node.js / TypeScript Client SDK (
- Post-V2 Full-Codebase Forensic Audit & Baseline Freeze (
BASELINE-2026-11-20-V2-R2):- Conducted a 100% raw-source-code forensic audit across all Port 3000 APIs/engines, Port 4000 Socket.IO workers,
/vcon/*UI pages, and Python/Node SDKs (notes/FORENSIC_AUDIT_REPORT.md), identifying 10 verified bugs (BUG-V2-AUDIT-01..BUG-V2-AUDIT-10:5High,4Medium,1Low). - Froze the current verified codebase (
102/102TS tests,10/10Python tests,0build/lint errors,0source code changes) asBASELINE-2026-11-20-V2-R2innotes/BASELINE_FREEZE.md. - Prepared the complete 7-document governance & Scope-Lock suite in
notes/:notes/FORENSIC_AUDIT_REPORT.md(10verified bugs with verbatim code blocks & line numbers)notes/BASELINE_FREEZE.md(1:1 baseline-to-audit mapping)notes/PRODUCTION_READINESS_SCOPE_LOCK.md(Doc001— exact fix contracts & strict per-phase Scope Locks)notes/PRODUCTION_ROADMAP.md(Doc002— A–Z 3-phase execution roadmap:V2-Audit-Phase-01Auth/Email/Audit,V2-Audit-Phase-02Port 4000 Webhooks/IPv6 CIDR/Health/Python SDK,V2-Audit-Phase-03UI Disk Usage/Client Expiry/CSV.GZ Badges)notes/PHASE_EXECUTION_LOG.md(Doc003—3 / 3phases completed,0 / 3remaining,10 / 10bugs resolved)notes/ERROR_HANDLING_MATRIX.md(Doc004— active error catalog +ERR-V2-AUDIT-01..05all5 / 5completed)notes/ACTUAL_IMPLEMENTATION_STATUS.md(ground-truth working vs per-phase delta —0remaining items)notes/README.md(governance index & 6-step execution protocol)
V2-Audit-Phase-01Complete & Verified (2026-11-20) (BUG-V2-AUDIT-02,BUG-V2-AUDIT-03,BUG-V2-AUDIT-04,ERR-V2-AUDIT-01):BUG-V2-AUDIT-04: UpdatedrevokeAdminSessions(target)incore/auth/adminUsers.tsto accept explicit{ memberId, email }and passed{ memberId: check.session.memberId, email: check.session.email }fromPOST /api/auth/logout(app/api/auth/logout/route.ts), eliminating the multi-admin logout global race condition (globals.__dataStackLastVerifiedMemberId).BUG-V2-AUDIT-03(ERR-V2-AUDIT-01): Enforced cross-table (admin_users↔admin_team_members) and in-memory peer duplicate email checks inupdateAdminAccount()(core/auth/adminUsers.ts) returningcode: 'ADMIN_EMAIL_CONFLICT', mapped to HTTP409 ConflictinPUT /api/auth/profile(app/api/auth/profile/route.ts).BUG-V2-AUDIT-02: Wired all 4 missing audit actions (ADMIN_LOGOUTinPOST /api/auth/logout,ADMIN_SETUPinPOST /api/auth/setup,STORAGE_FLUSH_TRIGGEREDinPOST /api/storage, andCLIENT_REVOKEDwhenupdates.status === 'revoked'inPATCH /api/clients/[id]) and synchronizedACTION_FILTER_OPTIONSincomponents/sections/AuditLogViewer.tsx1:1 withAuditActionTypeincore/auth/auditLog.ts(all 18/18 governance actions active).- Verified:
103/103TS tests (5/5intests/v2-phase03-rbac-audit.test.ts),10/10Python SDK tests,npm run buildclean, andnpm run lint(0errors /0warnings).
V2-Audit-Phase-02Complete & Verified (2026-11-20) (BUG-V2-AUDIT-05,BUG-V2-AUDIT-06,BUG-V2-AUDIT-08,BUG-V2-AUDIT-09,ERR-V2-AUDIT-02..05):BUG-V2-AUDIT-05(ERR-V2-AUDIT-02): Wired outbound HMAC-SHA256 webhook alert dispatching (triggerWebhookAlert+dispatchRejectionWebhook) on Port 4000 Socket.IO (server/index.js,server/socket/handlers.js) forCLIENT_TOKEN_EXPIRED,CLIENT_IP_FORBIDDEN,DISK_SPACE_LOW, andSTORAGE_QUOTA_EXCEEDEDacross handshake,sync_data,batch_sync, andping_verify, with AES-256-GCMdecryptVaultSecretinsyncSettingsFromDatabase()and livewebhookSecretpropagation incore/adminSettings/settingsManager.ts.BUG-V2-AUDIT-06(ERR-V2-AUDIT-03): UpgradedisClientIpAllowed()andstripIpv6MappedPrefix()in bothcore/Security/ipAllowlist.tsandserver/index.jsto use Node's built-innet.BlockList(addSubnet) for all IPv4 (/0..32) and IPv6 (/0..128) CIDR subnets and normalize[ipv6]:port,ipv4:port,%zone, and::ffff:.BUG-V2-AUDIT-08(ERR-V2-AUDIT-04): EnforcedisClientTokenExpired(registeredClient.expiresAt)(401 CLIENT_TOKEN_EXPIRED) andisClientIpAllowed(extractClientIp(req), registeredClient.allowedIps)(403 CLIENT_IP_FORBIDDEN) inGET /api/health(app/api/health/route.ts).BUG-V2-AUDIT-09(ERR-V2-AUDIT-05): Added"CLIENT_TOKEN_EXPIRED"and"CLIENT_IP_FORBIDDEN"to_is_non_retryable_auth_error()and_extract_auth_error_code()insdk/python/datastack/datastack/client.py, immediately raisingDataStackAuthError(msg, code=...)matching the Node.js SDK 1:1.- Verified:
104/104TS tests (5/5intests/v2-phase02-alerts-ip-prometheus.test.ts),11/11Python SDK tests (tests/phase05_sdk_test.py),npm run buildclean, andnpm run lint(0errors /0warnings).
V2-Audit-Phase-03Complete & Verified (2026-11-20) (BUG-V2-AUDIT-01,BUG-V2-AUDIT-07,BUG-V2-AUDIT-10):BUG-V2-AUDIT-01: AlignedDiskUsageSnapshot(core/Storage/csvStorage.ts),GET /api/storage(app/api/storage/route.ts), andSettingsWorkspace.tsx(components/sections/SettingsWorkspace.tsx) withavailable: boolean(totalBytes > 0) andfreeBytes: number(availableBytes) so/vcon/settings-> Storage -> "VPS disk usage" renders real disk utilization percentage, progress bar, and free/total GiB instead of"Statfs unavailable".BUG-V2-AUDIT-07: AddedisClientExpired(expiresAt, nowMs)inapp/vcon/clients/page.tsxand excluded expired tokens fromonlineCount,pendingCount, and"online"/"pending"filter tabs (classifying non-revoked expired clients under"offline"), matchingClientsTable.tsx"expired"badges 1:1.BUG-V2-AUDIT-10: Updatedcomponents/sections/ExportWorkspace.tsxandapp/vcon/storage/page.tsxto render aCSV.GZformat badge wheneverfile.compressed || file.filename.toLowerCase().endsWith('.csv.gz')is true (CSVotherwise).- Verified:
105/105TS tests (4/4intests/v2-phase01-storage-quotas.test.ts),11/11Python SDK tests (tests/phase05_sdk_test.py),npm run buildclean, andnpm run lint(0errors /0warnings). All3 / 3phases (10 / 10bugs) resolved.
- Conducted a 100% raw-source-code forensic audit across all Port 3000 APIs/engines, Port 4000 Socket.IO workers,
- R3 Full-Stack Production Remediation (
BASELINE-2026-11-20-R3—16Verified Bugs across4Scope-Locked Phases):R3-Phase-01Complete & Verified (2026-11-20) (BUG-01,BUG-03,BUG-04,BUG-06,BUG-10,BUG-13,BUG-15;ERR-R3-01..ERR-R3-04):BUG-01: Aligned Port 4000decryptVaultSecret()(server/index.js) 1:1 withcore/Security/secretVault.ts(v1:prefix,DATASTACK_ADMIN_SESSION_SECRET,base64urldecoding, fail-closed""+ warning) and injectedDATASTACK_ADMIN_SESSION_SECRET(SERVICE_REALBASE64_64_ADMINSESSION) intodocker/docker-compose.ymlservices.ingestion.environment.BUG-03: UpdatedAuthService.generateClientToken(payload, expiresIn?)(core/auth/jwt.ts) to omit JWTexpwhenexpiresInisnull/undefined/"", and passednullacross all 6 minting sites incore/sdk/sdkManager.tsso DBclient_sdks.expires_at(Never expires,>30d, live extension) is authoritative.BUG-04: EnforcedrequireAdminApiSession(req, { minRole: 'operator' })on admin-session calls toPOST /api/events(app/api/events/route.ts), returning403 INSUFFICIENT_ROLE_PERMISSIONfor read-onlyvieweraccounts.BUG-06: WrappeddecryptSecret(settings.webhookSecretEnc)inSettingsManager.notifyIngestionSettingsSync()(core/adminSettings/settingsManager.ts) intry/catch(""fallback + warning) and deferreddecryptSecretinPOST /api/settings/webhook-test(app/api/settings/webhook-test/route.ts) until neither draftwebhookSecretnorclearWebhookSecretis provided.BUG-10: Wired non-blockingADMIN_LOGIN_LOCKOUTwebhook alert dispatch inside the429 TOO_MANY_ATTEMPTSlockout branch ofPOST /api/auth/login(app/api/auth/login/route.ts).BUG-13: Unified Port 4000formatOutboundWebhookBody()(server/index.js) with Port 3000core/alerts/webhookNotifier.ts(instance,event,severity,title,message,clientId,details,timestamp,text,content).BUG-15: TightenedPOST /api/storage/s3-test(app/api/storage/s3-test/route.ts) fromoperatortoowner-only RBAC.- Verified:
6/6intests/r3-phase01-security-vault-webhooks.test.ts,15/15intests/v2-phase01..03,40/40intests/phase01/02/07,npm run buildclean,npm run lint(0errors).0UI/SDK/CSV files modified.
R3-Phase-02Complete & Verified (2026-11-20) (BUG-05,BUG-09):BUG-05: Added singleton unref'd periodic maintenance scheduler (ensureMaintenanceScheduler,runScheduledMaintenanceTick,stopMaintenanceScheduler,stopMaintenanceSchedulerForTests,isMaintenanceSchedulerActive) incore/adminSettings/settingsManager.tswith overlap locks (__dataStackMaintenanceRunning,__dataStackArchivePruneInFlight,__dataStackCompressInFlight), cooldown guards, and unhydrated-DB protection so S3 off-site archival + retention pruning and partition compression run unattended on schedule whenenableS3Backup === true.BUG-09: UpdatedextractDateFromFilename()in bothcore/Storage/csvStorage.tsandserver/workers/csvWriter.jsto strip.csv(.gz)and trailing-(?:part|gz)\d+suffixes so-gzN.csv.gzand-partN-gzM.csv.gzcollision files work identically across Port 3000/4000 pruning, S3 archiving,/api/data, and/api/export.- Verified:
3/3intests/r3-phase02-storage-scheduler-gzn.test.ts,40/40intests/phase03/v2-phase01/phase07/r3-phase01,npm run buildclean,npm run lint(0errors).0UI/SDK/Auth/Docker files modified.
R3-Phase-03Complete & Verified (2026-11-20) (BUG-02,BUG-07,BUG-08;ERR-R3-05,ERR-R3-06):BUG-02: Fixed Node.js SDK (sdk/node/datastack/index.ts) WSbatch_syncto emit a rawArraydirectly (1:1 with Port 4000server/socket/handlers.js:595), parseack.processed ?? ack.processedCount ?? ack.acceptedandack.duplicates ?? 0across WS & HTTP batch responses, and propagateduplicatesthroughsendBatch(),batchSync(), andsendFile().BUG-07(ERR-R3-05): Implemented realheartbeatIntervalMsmonitor in Node.js SDK (startHeartbeatMonitorwith.unref?.(), zero-CPUIDLE_WSpause, automaticping_verifyrecovery & spool drain, clean timer teardown,status().heartbeatActive); fixed single HTTP/WSDUPLICATE_DROPPEDreporting (accepted: 0, duplicates: 1); wiredTOKEN_ROTATED(401) andCLIENT_REVOKED(403) fast-failDataStackAuthErroracross WSconnect_error,ping_verify,sync_data,batch_sync, and HTTP fallback; compiled CommonJSsdk/node/datastack/dist/{index,types,spool}.{js,d.ts}viatsconfig.jsonand pointedpackage.jsonmain/types/exportsto./dist/index.js&./dist/index.d.ts.BUG-08(ERR-R3-06): Added non-retryableDataStackQuotaExceededError(DataStackError, retryable=False)insdk/python/datastack/datastack/exceptions.py, re-exported it in bothdatastackanddatastack_sdk, and wired immediate fail-fast handling for HTTP507and WSSTORAGE_QUOTA_EXCEEDED/DISK_SPACE_LOWinsdk/python/datastack/datastack/client.pywith0retries,0HTTP fallback after WS quota rejection, and0offline spooling.- Verified:
13/13Node SDK/packaging tests (5/5tests/r3-phase03-sdk-parity.test.tsagainst live Socket.IO handlers +4/4v2-phase04-node-sdk.test.ts+4/4phase05-packaging.test.ts),14/14Python SDK tests (3/3tests/r3_phase03_python_sdk_test.py+11/11phase05_sdk_test.py),py_compile,tsc,npm run buildclean,npm run lint(0errors,0warnings).0Port 3000/4000/UI/Docker files modified.
R3-Phase-04Complete & Verified (2026-11-20) (BUG-11,BUG-12,BUG-14,BUG-16;ERR-R3-07):BUG-11(ERR-R3-07): AddedshouldRedirectSimulatorUnauthorized(status, body)inlib/vconUiHelpers.tsand wired it post-JSON-parse across both single & batch HTTP sends inapp/vcon/simulator/page.tsxso expired/rotated/revoked client401responses display inline without logging out the authenticated admin.BUG-12: Added browser-localformatDateForLocalInput(iso)inlib/vconUiHelpers.tsand replaced UTC.toISOString().slice(0, 16)insideClientsTable.tsxopenEditModal(), eliminating local timezone drift on client edits.BUG-14: AddedisUiClientExpired(),classifyUiClientStatus(), andsummarizeUiClientCounts()inlib/vconUiHelpers.ts; excluded expired clients fromactiveClientsand exposedexpiredClientsincore/apiEngine/apiService.tsandDashboardOverview.tsx; emitteddatastack_clients_total{status="expired"}and excluded expired clients fromstatus="active"inapp/api/prometheus/route.ts; expanded/vcon/clientswith a 6-columnStatStrip+ 6-pill status filter (Total,Online,Offline,Pending,Expired,Revoked).BUG-16: Propagated verified session role fromapp/vcon/layout.tsxthroughcomponents/layout/VconShell.tsx(VconRoleContext/useVconRole()) +getVconRoleCapabilities()(lib/vconUiHelpers.ts) to gate all role-restricted UI controls acrossSettingsWorkspace,ClientsTable,/vcon/clients,DashboardOverview,/vcon/storage, and/vcon/simulator.- Verified:
29/29TS unit/integration tests (4/4r3-phase04-ui-metrics-rbac.test.ts+15/15phase04-ui-sync.test.ts+5/5v2-phase02-alerts-ip-prometheus.test.ts+5/5v2-phase03-rbac-audit.test.ts), live headless Chromium (playwright-cli,1440x900) verification across/vcon/dashboard,/vcon/clients,/vcon/simulator,/vcon/storage, and/vcon/settings(0JS/React pageerrors),npm run buildclean,npm run lint(0errors,0warnings).0Port 4000/SDK/Auth/Storage/Docker files modified. All4 / 4R3 phases (16 / 16bugs) complete.
- Full-Project Production Forensic Audit &
R4Baseline Freeze (BASELINE-2026-11-20-R4):-
Cleaned all 8 prior Files Manager files from
notes/and created the 100% raw-source-verifiednotes/FULL_PROJECT_FORENSIC_AUDIT.md(20 verified findings:3Critical,6High,7Medium,4Low —19actionable +1verified intentional previewAUDIT-17). -
Froze the untouched codebase as
BASELINE-2026-11-20-R4and created the complete 8-fileR4governance & Scope-Lock suite innotes/(README.md,FULL_PROJECT_FORENSIC_AUDIT.md,BASELINE_FREEZE.md,PRODUCTION_READINESS_SCOPE_LOCK.md,PRODUCTION_ROADMAP.md,PHASE_EXECUTION_LOG.md,ERROR_HANDLING_MATRIX.md,ACTUAL_IMPLEMENTATION_STATUS.md) across4Scope-Locked phases (R4-Phase-01..R4-Phase-04). -
R4-Phase-01Complete & Verified (2026-11-21) (AUDIT-01,AUDIT-03,AUDIT-04,AUDIT-09;ERR-R4-01–ERR-R4-05):AUDIT-01(ERR-R4-01): Created shared SSRF validatorcore/Security/ssrfGuard.tsblocking non-HTTP(S) schemes, embedded credentials, loopback, RFC1918/CGNAT/ULA,0.0.0.0/8, cloud metadata (169.254.0.0/16,fe80::/10,metadata.google.internal), bare internal hostnames, and DNS-rebinding targets, plusredirect: "manual"3xx rejection; enforced uniformly acrosscore/alerts/webhookNotifier.ts,core/Storage/s3Archive.ts,app/api/settings/route.ts,app/api/settings/webhook-test/route.ts,app/api/storage/s3-test/route.ts, andserver/index.jstriggerWebhookAlert, withDATASTACK_ALLOW_PRIVATE_URLS=truedocumented in.env.example(metadata still unconditionally blocked).AUDIT-03(ERR-R4-02): Mirroredadmin_team_membersandadmin_audit_logsDDL 1:1 intoserver/index.jsensureDatabaseSchema()for full 7-table parity across Port3000and Port4000on fresh PostgreSQL boots.AUDIT-04(ERR-R4-03): Added Next.jsinstrumentation.ts(register()onNEXT_RUNTIME === "nodejs") invokingSettingsManager.bootstrapMaintenanceSchedulerOnBoot()to hydrate settings and start the periodic storage maintenance scheduler automatically on Port3000startup.AUDIT-09(ERR-R4-04,ERR-R4-05): Addedconsole.warn+storageMode,degraded: true,code: "AUDIT_STORAGE_DEGRADED",warning, andX-DataStack-Audit-Degraded: 1header incore/auth/auditLog.ts&GET /api/audit; wiredPOST /internal/admin-session-syncinserver/index.js,syncInMemoryAdminSessionState/checkInMemorySessionRevocationinserver/filesManager.js, andnotifyIngestionAdminSessionSyncincore/auth/adminUsers.tsso!dbPoolFiles Manager sessions revoke immediately on logout/disable/delete/credential rotation.- Verified:
5/5tests/r4-phase01-security-boot.test.ts+85/85regression tests (phase01..04,phase07,files-manager,r3-phase01,r3-phase02,v2-phase03,v2-phase04=90/90PASS),npm run build,npm run lint(0errors).0forbidden UI/SDK/Docker/ingestion-batch files modified.
-
R4-Phase-02Complete & Verified (2026-11-21) (AUDIT-05,AUDIT-06,AUDIT-07,AUDIT-10,AUDIT-11,AUDIT-12;ERR-R4-06–ERR-R4-08):AUDIT-05(ERR-R4-06): EnforcedDATASTACK_MAX_BATCH_ITEMS(default1000, clamped1..10000, documented in.env.example) acrossPOST /api/events(400 BATCH_LIMIT_EXCEEDED, evaluated before storage admission scan) and Socket.IObatch_sync(server/socket/handlers.js).AUDIT-06: AddedFILES_MANAGER_MUTATIONandDATA_EXPORT_DOWNLOADEDtocore/auth/auditLog.tsandACTION_FILTER_OPTIONSincomponents/sections/AuditLogViewer.tsx; wired export download audit logs inapp/api/export/route.tsand CloudCmd HTTP/Socket mutation audit logs (admin_audit_logsPG + RAM fallback) inserver/filesManager.js.AUDIT-07(ERR-R4-07): Isolated corrupt.csv.gzdecompression failures insideGET /api/data(app/api/data/route.ts) so healthy client partitions return HTTP200alongsidecorruptPartitions(CORRUPT_GZIP_PARTITION) andwarnings.AUDIT-10: AddedserializeClientResponse()inapp/api/clients/[id]/route.tsstrippingsecretand non-rotationtokenfromPATCH /api/clients/[id]responses while exposingtokenPreview(and fulltokenonly onaction === "regenerate_token").AUDIT-11: MadePOST /internal/client-sync(action === "updated") inserver/index.jsdisconnect live sockets only on security-relevant updates (status === "revoked", expiredexpiresAt, token rotation,allowedIpschange, or live socket credential mismatch), keeping metadata-only updates (name,rateLimit,storageQuotaMb) connected without disruption.AUDIT-12(ERR-R4-08): ConverteduploadFileToS3(core/Storage/s3Archive.ts) to streaming SHA-256 pre-pass +Readable.toWeb(fs.createReadStream(...))(duplex: "half") with.destroy()teardown, and hardenedCSVStorageManager.openPartitionTextStream+countCsvDataRowsStreamAsync(public static) withfor await+.destroy()cleanup.- Verified:
5/5tests/r4-phase02-ingestion-audit-streaming.test.tsPASS, regression suites (r4-phase01,files-manager,phase02-04,v2-phase02-03,r3-phase02,r3-phase04) PASS, Playwright/vcon/audit(0console errors, new action filter options visible),npm run build, andnpm run lint(0errors).0forbidden SDK/Docker/Nginx/middleware/auth/unrelated UI files touched.
-
R4-Phase-03Complete & Verified (2026-11-21) (AUDIT-02,AUDIT-08,AUDIT-13,AUDIT-20;ERR-R4-09):AUDIT-02: Added"prepare": "npm run build"and"devDependencies": { "typescript": "^5.9.3" }tosdk/node/datastack/package.jsonand compiledsdk/node/datastack/dist/{index,spool,types}.{js,d.ts}so CommonJSrequire('./sdk/node/datastack')works cleanly out of the box.AUDIT-20: Added atomicDiskSpooler.ackAndRequeueFront(ackedCount, unprocessed)insdk/node/datastack/spool.tsand wireddeliverChunkWithSplitAndRetry()+flushSpool()insdk/node/datastack/index.ts(withdist/rebuilt) so partial-batch rate-limit recoveries commit acknowledged events without dropping or reordering unprocessed envelopes ahead of tail items.AUDIT-08(ERR-R4-09): AddedDATASTACK_ADMIN_EMAIL: "${DATASTACK_ADMIN_EMAIL:-admin@datastack.dev}"toservices.ingestion.environmentindocker/docker-compose.yml; addedproxyMode/wsSupported/wsHintdetection inGET /api/files/status(app/api/files/status/route.ts) and rendered a non-blocking[data-testid="files-manager-proxy-hint"]strip incomponents/sections/FilesManagerWorkspace.tsxwhen running standalone without Nginx WebSocket proxying.AUDIT-13: Removed dead/api/auth/setup/statuswhitelist line frommiddleware.tsand passed explicitrole: "owner",memberId: "owner-1"arguments tocreateAdminSessionToken()inapp/api/auth/setup/route.ts.- Verified:
4/4tests/r4-phase03-sdk-deploy-auth.test.ts+43/43regression checks (47/47total PASS acrossv2-phase04-node-sdk,r3-phase03-sdk-parity,phase05-packaging,phase07-deployment,phase01-auth,files-manager),node -e "require('./sdk/node/datastack')"PASS (2.2.0), Playwright/vcon/files(0console errors),npm run build, andnpm run lint(0errors).0forbidden Storage/Webhook/Client/Team/Profile/Landing files modified.
-
R4-Phase-04Complete & Verified (2026-11-21) (AUDIT-14,AUDIT-15,AUDIT-16,AUDIT-18,AUDIT-19;ERR-R4-10, plusAUDIT-07non-blockingcorruptPartitionsUI notice):AUDIT-14: Updatedcomponents/sections/TeamWorkspace.tsxto initialize role fromuseVconRole()(shellRole || 'viewer'), preserve distinctoperatorandadmin<Select>options and row values without coercingoperatortoadmin, label StatStrip'Operators / Admins', and require an explicit<Modal>("Confirm Team Member Removal",[data-testid="confirm-remove-member"]) beforeDELETE /api/team/${id}.AUDIT-15(ERR-R4-10): WireduseVconRole()+getVconRoleCapabilities(shellRole)(canOperate) inapp/vcon/sdk/page.tsxsoviewersessions skip?includeToken=truerequests cleanly (0HTTP403s) and display inline[data-testid="sdk-viewer-rbac-banner"]RBAC guidance while preserving existingloadingTokendisabled guards.AUDIT-16: Seededcomponents/layout/Sidebar.tsxrole badge fromuseVconRole()(formatShellRoleLabel()), renamed'Python SDK'to'Client SDKs', and addedcomponents/layout/Navbar.tsxmobileVCON_LABELSfor'/vcon/files'('Files Manager'),'/vcon/sdk'('Client SDKs'),'/vcon/team'('Team & RBAC'), and'/vcon/audit'('Audit Log').AUDIT-19: Replaced fake Demo fallback constants incomponents/sections/ProfileWorkspace.tsxwith neutralEMPTY_PROFILEplaceholders ('Loading...'/'—') until/api/auth/profileresolves.AUDIT-07UI Hook: Surfaced non-blocking[data-testid="data-corrupt-partitions-warning"]banners insidecomponents/sections/DataBrowser.tsxwhenGET /api/datareportscorruptPartitions.AUDIT-18: Alignedcomponents/sections/Hero.tsx,components/sections/SdkSection.tsx, andcomponents/sections/LandingAbout.tsxaround'Client SDKs'with side-by-side Python (datastack v2.2.0) and Node.js / TypeScript (@datastack/sdk v2.2.0) quick-start cards.- Verified:
5/5tests/r4-phase04-ui-rbac-copy.test.ts+46/46regression checks (51/51total PASS), live headless Chromium desktop (1440x900) & mobile (390x844) verification across/,/vcon/team,/vcon/profile,/vcon/data,/vcon/files,/vcon/audit,/vcon/sdk(0console/page errors,0viewer403s),npm run build, andnpm run lint(0errors).0forbidden backend/SDK/Docker/Nginx files modified. All4 / 4R4 phases (19 / 19actionable findings) complete.
-
PREPUSH-FIX-01Complete & Verified (2026-11-21) (DEPLOY-BLK-01,DEPLOY-BUG-02–05):DEPLOY-BLK-01: Resynchronizedpackage-lock.json(lockfileVersion: 3) withpackage.json(cloudcmd ^19.23.2+ full dependency tree) and verifiednpm ci --include=dev --dry-run(Dockerfile.web) andnpm ci --omit=dev --dry-run(Dockerfile.server) both exit0.DEPLOY-BUG-02: Hardeneddocker/coolify-nginx.confandnginx/default.confwithngx_http_realip_module(set_real_ip_fromloopback/RFC1918/Docker bridges + IPv6 ULA,real_ip_header X-Forwarded-For,real_ip_recursive on) soproxy_set_header X-Real-IP $remote_addrpreserves real external client IPs behind Coolify Traefik without permitting untrusted public IP spoofing.DEPLOY-BUG-03&DEPLOY-BUG-04: AddedARG INGESTION_INTERNAL_URL=http://ingestion:4000+ENV INGESTION_INTERNAL_URLindocker/Dockerfile.webbuilderstage for standalone Next.js rewrites, and forwarded optionalDATASTACK_ALLOW_PRIVATE_URLS,DATASTACK_MAX_BATCH_ITEMS,DATASTACK_RECENT_LIMIT,DATASTACK_MAINTENANCE_INTERVAL_MS, andDATASTACK_PROXY_MODEenv vars indocker/docker-compose.yml.DEPLOY-BUG-05: Updatedsdk/node/datastack/index.ts(emitSocketSingleandsendBatchChunkInternal) to callthis.disconnect()on terminalDataStackAuthErrorrejections (TOKEN_ROTATED,CLIENT_REVOKED, etc.) and recompiledsdk/node/datastack/dist/*so subsequentclient.connect()performs a fresh handshake (tests/r3-phase03-sdk-parity.test.ts5/5PASS).
-
- Froze the clean, Deployment-Hardened runtime state (
-
- Phase-01 Complete & Verified (
5. Known Constraints & Operational Notes for Future Contributors
- Never add per-row telemetry
INSERTqueries to PostgreSQL or addDROP/TRUNCATEauto-migrations. - Never move Socket.IO handlers into Next.js
app/api/*routes. - When deploying updates to an existing Coolify VPS, always preserve existing
SERVICE_*environment variables and avoiddocker compose down -vsopgdata,datastack-data, andredisdatavolumes remain intact. - After any code change, verify with
npm run build,npm run lint,npm ci --dry-run, and the test suites, then synchronize thisMEMORY.md.