Files
DataStack/docs/users/configuration.md
T
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

6.0 KiB
Raw Blame History

Platform Configuration Guide

This guide covers all environment variables, initial administrator bootstrap secrets, runtime tuning parameters, and synchronization behaviors across the DataStack architecture.


1. Environment Variables Overview

DataStack reads runtime configuration from system environment variables (or a root .env file). In development, safe fallback values are used. In production (NODE_ENV=production), all security keys must be explicitly configured with at least 32 cryptographically secure characters.

# Security & Session Secrets (REQUIRED in production, min 32 characters)
DATASTACK_ADMIN_SESSION_SECRET=your-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-32-plus-char-bootstrap-setup-key-here
DATASTACK_INTERNAL_SECRET=your-32-plus-char-internal-service-secret

# Administrator Account Initial Bootstrap
DATASTACK_ADMIN_EMAIL=admin@example.com

# PostgreSQL Persistence (recommended for production metadata)
DATABASE_URL=postgresql://datastack:password@postgres:5432/datastack_db

# Shared Redis Buffer (optional; enables cluster buffer queue & distributed locks)
REDIS_URL=redis://redis:6379

# Service Networking & Ports
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000

2. Environment Variables Specification

Security & Authentication

Variable Required in Prod Default (Dev) Description
DATASTACK_ADMIN_SESSION_SECRET Yes Built-in fallback Used by WebCrypto HMAC-SHA256 to sign and verify operator session cookies (datastack_admin_session). Must be ≥ 32 characters.
DATASTACK_JWT_SECRET Yes Built-in fallback Master signing secret used to generate and verify HMAC-SHA256 JWT tokens for registered SDK clients.
DATASTACK_ADMIN_SETUP_KEY Yes Built-in fallback Secret key required to complete the initial admin onboarding wizard at /vcon/setup.
DATASTACK_INTERNAL_SECRET Recommended Derived from JWT Secret used for authenticated HTTP communication between Port 3000 and the Port 4000 ingestion server via x-datastack-internal-secret. If omitted, automatically derived via SHA-256 of DATASTACK_JWT_SECRET.
DATASTACK_ALLOW_DEMO_ADMIN No false When set to true, enables quick demo credentials on the login screen (admin@example.com). Strictly prohibited in production.

PostgreSQL Database

Variable Required in Prod Default (Dev) Description
DATABASE_URL Recommended undefined PostgreSQL connection URI (postgresql://user:pass@host:port/dbname). If omitted, DataStack operates in an in-memory mock mode.

Redis Shared Buffer

Variable Required in Prod Default (Dev) Description
REDIS_URL Optional undefined Redis connection URI (redis://host:6379). When present, events are queued in Redis list datastack:buffer:events, surviving container restarts and enabling multi-worker flush locking.

Service Networking

Variable Required in Prod Default Description
PORT No 3000 Port for the Next.js 15 Web Application and REST API.
PORT_SOCKET No 4000 Port for the Express 5 + Socket.IO ingestion service.
INGESTION_INTERNAL_URL No http://127.0.0.1:4000 Internal URL used by Port 3000 to send bridge notifications to Port 4000 (/internal/*).

3. Platform Runtime Settings (/vcon/settings)

In addition to environment variables, operators can adjust platform operational thresholds at runtime through the Settings Workspace (/vcon/settings).

These settings are stored in PostgreSQL (admin_settings table) and broadcast in real time to the Port 4000 ingestion service via /internal/settings-sync.

Parameter Type Default Permitted Range Purpose
maxEventsPerSec Number 100 1 – 10,000 Ingestion rate ceiling. Events exceeding this limit receive RATE_LIMIT_EXCEEDED (429).
maxPayloadSizeKb Number 100 1 – 10,240 (10 MB) Maximum permitted size in kilobytes for any single event payload. Oversized payloads receive PAYLOAD_TOO_LARGE (413).
batchFlushThreshold Number 500 10 – 50,000 Queue depth threshold that triggers an immediate write of buffered events to CSV files on disk.
batchFlushIntervalMs Number 2000 200 – 60,000 (60s) Maximum duration pending events can remain buffered in memory/Redis before being flushed to disk.
fileRotationMb Number 100 5 – 1,024 (1 GB) File size threshold per daily CSV partition. Once reached, DataStack writes to {YYYY-MM-DD}-part2.csv.
enableIdempotency Boolean true true / false Enables 5-minute sliding-window deduplication based on client-provided eventId.
autoPruneDays Number 30 0 – 3,650 Automatically purges daily CSV partition files older than this value. Setting to 0 disables pruning and keeps all archives indefinitely.
jwtSecret String .env value ≥ 16 characters Master JWT signing secret. Rotating this value automatically re-signs all active SDK client credentials and notifies Port 4000.

4. Cross-Process Synchronization Rules

When settings are modified in /vcon/settings:

  1. Database Persistence: If PostgreSQL is configured, settings are written to admin_settings WHERE id = 'default'.
  2. Web Process Hydration: Port 3000 updates its local singleton instances (SecurityEngine, DataBufferEngine, CSVStorageManager).
  3. Bridge Notification: Port 3000 issues an authenticated HTTP POST to ${INGESTION_INTERNAL_URL}/internal/settings-sync carrying the x-datastack-internal-secret header.
  4. Ingestion Reload: Port 4000 receives the updated thresholds and immediately applies them to its socket handlers and background flusher worker without requiring a process restart.