feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
@@ -0,0 +1,44 @@
|
||||
# Complete Configuration Reference
|
||||
|
||||
This document provides the exhaustive specification for every environment variable, `.env` parameter, and runtime database setting supported by DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Environment Variables (`.env`)
|
||||
|
||||
Environment variables are loaded on container or process boot.
|
||||
|
||||
| Variable Name | Type | Production Requirement | Default Value (Dev) | Description & Constraints |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| `NODE_ENV` | String | **Required** | `development` | Runtime mode. Must be set to `production` in live deployments to enforce fail-closed security and secure cookies. |
|
||||
| `DATASTACK_ADMIN_SESSION_SECRET` | String | **Required** | Fallback key | Cryptographic secret for signing WebCrypto HMAC-SHA256 operator session cookies. Min length: 32 chars. |
|
||||
| `DATASTACK_JWT_SECRET` | String | **Required** | Fallback key | Master HMAC-SHA256 secret used to sign and verify client SDK authentication tokens. Min length: 32 chars. |
|
||||
| `DATASTACK_ADMIN_SETUP_KEY` | String | **Required** | Fallback key | One-time secret key required to access `/vcon/setup` and create the initial administrator account. Min length: 32 chars. |
|
||||
| `DATASTACK_INTERNAL_SECRET` | String | Recommended | SHA-256 of JWT secret | Shared secret authorizing internal IPC between Port 3000 and Port 4000 via `x-datastack-internal-secret`. Min length: 32 chars. |
|
||||
| `DATASTACK_ADMIN_EMAIL` | String | Optional | `admin@example.com` | Default admin contact email used during development or bootstrap. |
|
||||
| `DATASTACK_ALLOW_DEMO_ADMIN` | Boolean | Strictly Forbidden in Prod | `false` | When `true`, displays demo credentials on login screen. Throws fatal error if enabled in production. |
|
||||
| `DATABASE_URL` | String | Recommended | `undefined` | PostgreSQL connection string (`postgresql://user:pass@host:5432/dbname`). If omitted, in-memory mock mode is activated. |
|
||||
| `REDIS_URL` | String | Optional | `undefined` | Redis connection URI (`redis://host:6379`). Enables multi-worker buffer queuing and distributed flush locks. |
|
||||
| `PORT` | Integer | Optional | `3000` | Port for the Next.js 15 web application and REST API. |
|
||||
| `PORT_SOCKET` | Integer | Optional | `4000` | Port for the Express 5 + Socket.IO ingestion service. |
|
||||
| `INGESTION_INTERNAL_URL` | String | Optional | `http://127.0.0.1:4000` | Base URL used by Port 3000 to send bridge notifications to Port 4000 (`/internal/*`). |
|
||||
|
||||
---
|
||||
|
||||
## 2. Runtime Platform Settings (`admin_settings` Table)
|
||||
|
||||
Runtime settings are persisted in PostgreSQL (`admin_settings` row with `id = 'default'`), configurable via `/vcon/settings`, and synchronized instantly across processes.
|
||||
|
||||
| Parameter Key | Storage Column | Type | Default | Range / Constraints | Functional Impact |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| `serverPort` | `server_port` | Integer | `4000` | `1024` – `65535` | Ingestion service port reported across the cluster. |
|
||||
| `webPort` | `web_port` | Integer | `3000` | `1024` – `65535` | Web dashboard port reported across the cluster. |
|
||||
| `maxEventsPerSec` | `max_events_per_sec` | Integer | `100` | `1` – `10,000` | Global ingestion rate limit ceiling. Exceeding events receive HTTP `429` or socket error. |
|
||||
| `maxPayloadSizeKb` | `max_payload_size_kb` | Integer | `100` | `1` – `10,240` (10 MB) | Maximum permitted size in KB for any single event. Larger payloads rejected with HTTP `413`. |
|
||||
| `batchFlushThreshold` | `batch_flush_threshold` | Integer | `500` | `10` – `50,000` | Buffer queue depth that triggers an immediate synchronous flush to disk. |
|
||||
| `batchFlushIntervalMs` | `batch_flush_interval_ms` | Integer | `2000` | `200` – `60,000` (60s) | Maximum age in milliseconds before pending events in queue are written to disk. |
|
||||
| `fileRotationMb` | `file_rotation_mb` | Integer | `100` | `5` – `1,024` (1 GB) | Partition size threshold on disk. When exceeded, output continues to `{YYYY-MM-DD}-part2.csv`. |
|
||||
| `enableIdempotency` | `enable_idempotency` | Boolean | `true` | `true` / `false` | Enables 5-minute sliding-window deduplication for events carrying an `eventId`. |
|
||||
| `autoPruneDays` | `auto_prune_days` | Integer | `30` | `0` – `3,650` | Purges partition files older than this day threshold. Setting to `0` retains archives indefinitely. |
|
||||
| `jwtSecret` | `jwt_secret` | String | `.env` value | Min 16 chars | Master JWT signing secret. Rotating triggers automated re-signing of all client tokens. |
|
||||
| `updatedAt` | `updated_at` | DateTime | `now()` | ISO 8601 string | Timestamp recording the last administrative modification of platform settings. |
|
||||
@@ -0,0 +1,51 @@
|
||||
# Dependencies Inventory & Rationales
|
||||
|
||||
This document audits all runtime dependencies, development tools, and client libraries used in DataStack, providing version numbers, licenses, and architectural rationales.
|
||||
|
||||
---
|
||||
|
||||
## 1. Node.js Production Dependencies
|
||||
|
||||
| Package | Version | License | Architectural Rationale |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `next` | `^15.4.9` | MIT | App Router frontend framework, SSR rendering, Edge middleware, and standalone container packaging. |
|
||||
| `react` | `^19.2.1` | MIT | Core UI component engine. |
|
||||
| `react-dom` | `^19.2.1` | MIT | DOM rendering for React 19. |
|
||||
| `express` | `^5.3.0` | MIT | High-performance, lightweight HTTP listener powering the dedicated Port 4000 ingestion server and `/internal/*` service bridge. |
|
||||
| `socket.io` | `^4.8.4` | MIT | Real-time WebSocket server engine with automatic reconnection, heartbeat pings, and room/client management. |
|
||||
| `socket.io-client`| `^4.8.4` | MIT | Client library for in-browser simulation (`/vcon/simulator`) and automated integration tests. |
|
||||
| `pg` | `^8.23.1` | MIT | Robust PostgreSQL connection pool (`pg.Pool`) used strictly for control-plane metadata persistence. |
|
||||
| `ioredis` | `^6.0.0` | MIT | High-speed Redis client supporting list buffering (`RPUSH`/`LPOP`), atomic trimming, and distributed flush locks (`SET NX PX`). |
|
||||
| `jsonwebtoken` | `^9.0.3` | MIT | Fast HMAC-SHA256 signing and verification for SDK client authentication tokens. |
|
||||
| `json2csv` | `^6.0.0-alpha.2` | MIT | High-performance transform streams for non-blocking CSV dataset extraction in `/api/export`. |
|
||||
| `csv-parse` | `^7.0.3` | MIT | Stream-based parser for structured file ingestion and test verification. |
|
||||
| `cors` | `^2.8.6` | MIT | Cross-origin resource sharing middleware for HTTP ingestion APIs. |
|
||||
| `lucide-react` | `^0.553.0` | ISC | Clean, consistent SVG icon set across all UI components. |
|
||||
| `motion` | `^12.23.24` | MIT | Smooth, declarative animations for dashboard counters, page transitions, and status badges. |
|
||||
| `clsx` & `tailwind-merge` | `^2.1.1` / `^3.3.1` | MIT | Utility functions for conditional class concatenation and Tailwind utility conflict resolution (`cn()`). |
|
||||
| `class-variance-authority`| `^0.7.1` | Apache-2.0 | Type-safe variant styling for UI primitives (`Button`, `Badge`, `Panel`). |
|
||||
| `@hookform/resolvers` | `^5.2.1` | MIT | Form validation resolver support. |
|
||||
| `@google/genai` | `^2.4.0` | Apache-2.0 | Google GenAI TypeScript SDK. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Development & Tooling Dependencies
|
||||
|
||||
| Package | Version | License | Architectural Rationale |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `typescript` | `^5` | Apache-2.0 | Static typing and compile-time contract enforcement across all source files. |
|
||||
| `tsx` | `^4.21.0` | MIT | Native Node.js TypeScript runner used to execute integration test suites directly (`npx tsx --test`). |
|
||||
| `@tailwindcss/postcss` | `4.1.11` | MIT | Next-generation Tailwind CSS v4 compiler. |
|
||||
| `@tailwindcss/typography` | `^0.5.19` | MIT | Typography utility classes for documentation and markdown rendering. |
|
||||
| `eslint` & `eslint-config-next` | `^9` / `^15.4.9` | MIT | Code quality analysis and style enforcement. |
|
||||
| `@types/*` | Various | MIT | High-fidelity TypeScript definitions for Node, React, Express, PG, and JWT. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Python SDK Dependencies (`sdk/python/datastack/`)
|
||||
|
||||
| Package | Version | License | Architectural Rationale |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `python-socketio[client]` | `>=5.11.0` | MIT | WebSocket transport enabling real-time streaming with automatic reconnection. |
|
||||
| `requests` | `>=2.31.0` | Apache-2.0 | Reliable HTTP transport for REST ingestion fallback and export downloads. |
|
||||
| `urllib3` | `>=2.0.0` | MIT | Connection pooling and retry logic for network resiliency. |
|
||||
@@ -0,0 +1,41 @@
|
||||
# Technical Glossary
|
||||
|
||||
This glossary defines technical terms, architectural concepts, and domain language used throughout the DataStack platform and documentation.
|
||||
|
||||
---
|
||||
|
||||
### Control Plane
|
||||
The administrative and configuration layer of DataStack. Handled on **Port 3000** (Next.js 15), the control plane manages administrator authentication, SDK client credential issuance, rate limits, and runtime limits. State is stored in PostgreSQL.
|
||||
|
||||
### Data Plane
|
||||
The high-throughput telemetry ingestion and storage subsystem. Handled on **Port 4000** (Express 5 + Socket.IO 4), the data plane handles persistent client connections, in-memory/Redis buffering, and writes to partitioned CSV files on disk.
|
||||
|
||||
### Cooperative Locking
|
||||
A lightweight filesystem locking protocol used to coordinate writes between independent processes (Port 3000 Web and Port 4000 Ingestion). Writers create a companion `{partition}.csv.lock` file before writing. If a writer crashes, locks older than 30 seconds are treated as stale and automatically unlinked.
|
||||
|
||||
### Sliding-Window Rate Limiting
|
||||
A rate limiting algorithm implemented in `core/Security/security.ts`. Tracks the number of events received per client within the current one-second sliding window. Exceeding events are rejected with HTTP 429 or Socket.IO error messages.
|
||||
|
||||
### Idempotency Envelope
|
||||
A deduplication safeguard that tracks client-provided `eventId` values within a 5-minute sliding window. If an event with the same ID arrives multiple times (due to client retries or network replays), duplicate writes to disk are prevented.
|
||||
|
||||
### Partitioned CSV Storage
|
||||
The core data storage model of DataStack. Telemetry streams are organized into directories by client identifier, with records partitioned into daily files named `YYYY-MM-DD.csv` conforming to RFC 4180 standard formatting.
|
||||
|
||||
### File Rotation
|
||||
The automatic division of daily CSV files when disk size exceeds `fileRotationMb` (default 100 MB). The active partition seamlessly transitions to `{YYYY-MM-DD}-part2.csv`, preventing unwieldy file sizes.
|
||||
|
||||
### Internal Service Bridge (`/internal/*`)
|
||||
The private inter-process communication protocol between Port 3000 and Port 4000. Guarded by `DATASTACK_INTERNAL_SECRET`, it allows administrative changes in `/vcon/` (such as client revocations or secret rotations) to take effect in Port 4000 without restarting services.
|
||||
|
||||
### Token Versioning (`tokenVersion`)
|
||||
A security mechanism in `admin_users` and session cookies. An integer counter is embedded in each signed session cookie. Updating an operator password or selecting "Log Out Other Sessions" increments this number in PostgreSQL, instantly invalidating all active sessions across all devices.
|
||||
|
||||
### Tombstone Registry (`deleted_clients`)
|
||||
A PostgreSQL audit table storing identifiers of permanently deleted clients. Ensures that previously signed JWT tokens or stale producer credentials can never be reused to send data.
|
||||
|
||||
### Standalone Output
|
||||
A Next.js build configuration mode (`output: "standalone"` in `next.config.ts`) that bundles only required production dependencies into `.next/standalone`, enabling lightweight, fast-starting Docker containers without bulky `node_modules` folders.
|
||||
|
||||
### Offline Spooling
|
||||
A reliability feature of the Python SDK. If the server is unreachable or network connectivity drops, telemetry events are buffered to a local disk directory (`.datastack_spool/`) and automatically drained to the server once connectivity is restored.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Technical Reference Portal
|
||||
|
||||
Welcome to the DataStack Technical Reference Portal. This section contains definitive specifications, configuration parameter reference tables, dependency audits, and technical terminology.
|
||||
|
||||
---
|
||||
|
||||
## Reference Documents
|
||||
|
||||
| Reference Document | Scope & Contents |
|
||||
| :--- | :--- |
|
||||
| [Configuration Reference (`configuration-reference.md`)](./configuration-reference.md) | Exhaustive parameter matrix of all environment variables, `.env` directives, runtime `/vcon/settings` keys, default values, and valid ranges. |
|
||||
| [Dependencies Inventory (`dependencies.md`)](./dependencies.md) | Complete audit of all runtime dependencies, dev tools, and Python client packages, including version numbers, licenses, and architectural rationales. |
|
||||
| [Technical Glossary (`glossary.md`)](./glossary.md) | Clear definitions of domain concepts, telemetry paradigms, cooperative locking, idempotency envelopes, and dual-runtime boundaries. |
|
||||
Reference in new issue
Block a user