- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
6.0 KiB
Platform Configuration Guide
This guide covers all environment variables, initial administrator bootstrap secrets, runtime tuning parameters, and synchronization behaviors across the DataStack architecture.
1. Environment Variables Overview
DataStack reads runtime configuration from system environment variables (or a root .env file). In development, safe fallback values are used. In production (NODE_ENV=production), all security keys must be explicitly configured with at least 32 cryptographically secure characters.
# Security & Session Secrets (REQUIRED in production, min 32 characters)
DATASTACK_ADMIN_SESSION_SECRET=your-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-32-plus-char-bootstrap-setup-key-here
DATASTACK_INTERNAL_SECRET=your-32-plus-char-internal-service-secret
# Administrator Account Initial Bootstrap
DATASTACK_ADMIN_EMAIL=admin@example.com
# PostgreSQL Persistence (recommended for production metadata)
DATABASE_URL=postgresql://datastack:password@postgres:5432/datastack_db
# Shared Redis Buffer (optional; enables cluster buffer queue & distributed locks)
REDIS_URL=redis://redis:6379
# Service Networking & Ports
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
2. Environment Variables Specification
Security & Authentication
| Variable | Required in Prod | Default (Dev) | Description |
|---|---|---|---|
DATASTACK_ADMIN_SESSION_SECRET |
Yes | Built-in fallback | Used by WebCrypto HMAC-SHA256 to sign and verify operator session cookies (datastack_admin_session). Must be ≥ 32 characters. |
DATASTACK_JWT_SECRET |
Yes | Built-in fallback | Master signing secret used to generate and verify HMAC-SHA256 JWT tokens for registered SDK clients. |
DATASTACK_ADMIN_SETUP_KEY |
Yes | Built-in fallback | Secret key required to complete the initial admin onboarding wizard at /vcon/setup. |
DATASTACK_INTERNAL_SECRET |
Recommended | Derived from JWT | Secret used for authenticated HTTP communication between Port 3000 and the Port 4000 ingestion server via x-datastack-internal-secret. If omitted, automatically derived via SHA-256 of DATASTACK_JWT_SECRET. |
DATASTACK_ALLOW_DEMO_ADMIN |
No | false |
When set to true, enables quick demo credentials on the login screen (admin@example.com). Strictly prohibited in production. |
PostgreSQL Database
| Variable | Required in Prod | Default (Dev) | Description |
|---|---|---|---|
DATABASE_URL |
Recommended | undefined |
PostgreSQL connection URI (postgresql://user:pass@host:port/dbname). If omitted, DataStack operates in an in-memory mock mode. |
Redis Shared Buffer
| Variable | Required in Prod | Default (Dev) | Description |
|---|---|---|---|
REDIS_URL |
Optional | undefined |
Redis connection URI (redis://host:6379). When present, events are queued in Redis list datastack:buffer:events, surviving container restarts and enabling multi-worker flush locking. |
Service Networking
| Variable | Required in Prod | Default | Description |
|---|---|---|---|
PORT |
No | 3000 |
Port for the Next.js 15 Web Application and REST API. |
PORT_SOCKET |
No | 4000 |
Port for the Express 5 + Socket.IO ingestion service. |
INGESTION_INTERNAL_URL |
No | http://127.0.0.1:4000 |
Internal URL used by Port 3000 to send bridge notifications to Port 4000 (/internal/*). |
3. Platform Runtime Settings (/vcon/settings)
In addition to environment variables, operators can adjust platform operational thresholds at runtime through the Settings Workspace (/vcon/settings).
These settings are stored in PostgreSQL (admin_settings table) and broadcast in real time to the Port 4000 ingestion service via /internal/settings-sync.
| Parameter | Type | Default | Permitted Range | Purpose |
|---|---|---|---|---|
maxEventsPerSec |
Number | 100 |
1 – 10,000 |
Ingestion rate ceiling. Events exceeding this limit receive RATE_LIMIT_EXCEEDED (429). |
maxPayloadSizeKb |
Number | 100 |
1 – 10,240 (10 MB) |
Maximum permitted size in kilobytes for any single event payload. Oversized payloads receive PAYLOAD_TOO_LARGE (413). |
batchFlushThreshold |
Number | 500 |
10 – 50,000 |
Queue depth threshold that triggers an immediate write of buffered events to CSV files on disk. |
batchFlushIntervalMs |
Number | 2000 |
200 – 60,000 (60s) |
Maximum duration pending events can remain buffered in memory/Redis before being flushed to disk. |
fileRotationMb |
Number | 100 |
5 – 1,024 (1 GB) |
File size threshold per daily CSV partition. Once reached, DataStack writes to {YYYY-MM-DD}-part2.csv. |
enableIdempotency |
Boolean | true |
true / false |
Enables 5-minute sliding-window deduplication based on client-provided eventId. |
autoPruneDays |
Number | 30 |
0 – 3,650 |
Automatically purges daily CSV partition files older than this value. Setting to 0 disables pruning and keeps all archives indefinitely. |
jwtSecret |
String | .env value |
≥ 16 characters | Master JWT signing secret. Rotating this value automatically re-signs all active SDK client credentials and notifies Port 4000. |
4. Cross-Process Synchronization Rules
When settings are modified in /vcon/settings:
- Database Persistence: If PostgreSQL is configured, settings are written to
admin_settings WHERE id = 'default'. - Web Process Hydration: Port 3000 updates its local singleton instances (
SecurityEngine,DataBufferEngine,CSVStorageManager). - Bridge Notification: Port 3000 issues an authenticated HTTP
POSTto${INGESTION_INTERNAL_URL}/internal/settings-synccarrying thex-datastack-internal-secretheader. - Ingestion Reload: Port 4000 receives the updated thresholds and immediately applies them to its socket handlers and background flusher worker without requiring a process restart.