Files
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

96 lines
6.0 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Platform Configuration Guide
This guide covers all environment variables, initial administrator bootstrap secrets, runtime tuning parameters, and synchronization behaviors across the DataStack architecture.
---
## 1. Environment Variables Overview
DataStack reads runtime configuration from system environment variables (or a root `.env` file). In development, safe fallback values are used. In production (`NODE_ENV=production`), all security keys must be explicitly configured with at least 32 cryptographically secure characters.
```dotenv
# Security & Session Secrets (REQUIRED in production, min 32 characters)
DATASTACK_ADMIN_SESSION_SECRET=your-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-32-plus-char-bootstrap-setup-key-here
DATASTACK_INTERNAL_SECRET=your-32-plus-char-internal-service-secret
# Administrator Account Initial Bootstrap
DATASTACK_ADMIN_EMAIL=admin@example.com
# PostgreSQL Persistence (recommended for production metadata)
DATABASE_URL=postgresql://datastack:password@postgres:5432/datastack_db
# Shared Redis Buffer (optional; enables cluster buffer queue & distributed locks)
REDIS_URL=redis://redis:6379
# Service Networking & Ports
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
```
---
## 2. Environment Variables Specification
### Security & Authentication
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `DATASTACK_ADMIN_SESSION_SECRET` | **Yes** | Built-in fallback | Used by WebCrypto HMAC-SHA256 to sign and verify operator session cookies (`datastack_admin_session`). Must be ≥ 32 characters. |
| `DATASTACK_JWT_SECRET` | **Yes** | Built-in fallback | Master signing secret used to generate and verify HMAC-SHA256 JWT tokens for registered SDK clients. |
| `DATASTACK_ADMIN_SETUP_KEY` | **Yes** | Built-in fallback | Secret key required to complete the initial admin onboarding wizard at `/vcon/setup`. |
| `DATASTACK_INTERNAL_SECRET` | Recommended | Derived from JWT | Secret used for authenticated HTTP communication between Port 3000 and the Port 4000 ingestion server via `x-datastack-internal-secret`. If omitted, automatically derived via SHA-256 of `DATASTACK_JWT_SECRET`. |
| `DATASTACK_ALLOW_DEMO_ADMIN` | No | `false` | When set to `true`, enables quick demo credentials on the login screen (`admin@example.com`). Strictly prohibited in production. |
### PostgreSQL Database
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `DATABASE_URL` | Recommended | `undefined` | PostgreSQL connection URI (`postgresql://user:pass@host:port/dbname`). If omitted, DataStack operates in an in-memory mock mode. |
### Redis Shared Buffer
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `REDIS_URL` | Optional | `undefined` | Redis connection URI (`redis://host:6379`). When present, events are queued in Redis list `datastack:buffer:events`, surviving container restarts and enabling multi-worker flush locking. |
### Service Networking
| Variable | Required in Prod | Default | Description |
| :--- | :--- | :--- | :--- |
| `PORT` | No | `3000` | Port for the Next.js 15 Web Application and REST API. |
| `PORT_SOCKET` | No | `4000` | Port for the Express 5 + Socket.IO ingestion service. |
| `INGESTION_INTERNAL_URL` | No | `http://127.0.0.1:4000` | Internal URL used by Port 3000 to send bridge notifications to Port 4000 (`/internal/*`). |
---
## 3. Platform Runtime Settings (`/vcon/settings`)
In addition to environment variables, operators can adjust platform operational thresholds at runtime through the **Settings Workspace** (`/vcon/settings`).
These settings are stored in PostgreSQL (`admin_settings` table) and broadcast in real time to the Port 4000 ingestion service via `/internal/settings-sync`.
| Parameter | Type | Default | Permitted Range | Purpose |
| :--- | :--- | :--- | :--- | :--- |
| `maxEventsPerSec` | Number | `100` | `1` – `10,000` | Ingestion rate ceiling. Events exceeding this limit receive `RATE_LIMIT_EXCEEDED (429)`. |
| `maxPayloadSizeKb` | Number | `100` | `1` – `10,240` (10 MB) | Maximum permitted size in kilobytes for any single event payload. Oversized payloads receive `PAYLOAD_TOO_LARGE (413)`. |
| `batchFlushThreshold` | Number | `500` | `10` – `50,000` | Queue depth threshold that triggers an immediate write of buffered events to CSV files on disk. |
| `batchFlushIntervalMs` | Number | `2000` | `200` – `60,000` (60s) | Maximum duration pending events can remain buffered in memory/Redis before being flushed to disk. |
| `fileRotationMb` | Number | `100` | `5` – `1,024` (1 GB) | File size threshold per daily CSV partition. Once reached, DataStack writes to `{YYYY-MM-DD}-part2.csv`. |
| `enableIdempotency` | Boolean | `true` | `true` / `false` | Enables 5-minute sliding-window deduplication based on client-provided `eventId`. |
| `autoPruneDays` | Number | `30` | `0` – `3,650` | Automatically purges daily CSV partition files older than this value. Setting to `0` disables pruning and keeps all archives indefinitely. |
| `jwtSecret` | String | `.env` value | ≥ 16 characters | Master JWT signing secret. Rotating this value automatically re-signs all active SDK client credentials and notifies Port 4000. |
---
## 4. Cross-Process Synchronization Rules
When settings are modified in `/vcon/settings`:
1. **Database Persistence**: If PostgreSQL is configured, settings are written to `admin_settings WHERE id = 'default'`.
2. **Web Process Hydration**: Port 3000 updates its local singleton instances (`SecurityEngine`, `DataBufferEngine`, `CSVStorageManager`).
3. **Bridge Notification**: Port 3000 issues an authenticated HTTP `POST` to `${INGESTION_INTERNAL_URL}/internal/settings-sync` carrying the `x-datastack-internal-secret` header.
4. **Ingestion Reload**: Port 4000 receives the updated thresholds and immediately applies them to its socket handlers and background flusher worker without requiring a process restart.