feat(sdk): refactor Python SDK documentation

- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
Team Dark Devil committed 2026-11-18 10:03:40 +06:00
1 parent 4279da45bb
commit 40a6855724
38 files changed
+3315 -141

No files matched your search

+95
View File
@@ -0,0 +1,95 @@
# Platform Configuration Guide
This guide covers all environment variables, initial administrator bootstrap secrets, runtime tuning parameters, and synchronization behaviors across the DataStack architecture.
---
## 1. Environment Variables Overview
DataStack reads runtime configuration from system environment variables (or a root `.env` file). In development, safe fallback values are used. In production (`NODE_ENV=production`), all security keys must be explicitly configured with at least 32 cryptographically secure characters.
```dotenv
# Security & Session Secrets (REQUIRED in production, min 32 characters)
DATASTACK_ADMIN_SESSION_SECRET=your-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-32-plus-char-bootstrap-setup-key-here
DATASTACK_INTERNAL_SECRET=your-32-plus-char-internal-service-secret
# Administrator Account Initial Bootstrap
DATASTACK_ADMIN_EMAIL=admin@example.com
# PostgreSQL Persistence (recommended for production metadata)
DATABASE_URL=postgresql://datastack:password@postgres:5432/datastack_db
# Shared Redis Buffer (optional; enables cluster buffer queue & distributed locks)
REDIS_URL=redis://redis:6379
# Service Networking & Ports
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
```
---
## 2. Environment Variables Specification
### Security & Authentication
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `DATASTACK_ADMIN_SESSION_SECRET` | **Yes** | Built-in fallback | Used by WebCrypto HMAC-SHA256 to sign and verify operator session cookies (`datastack_admin_session`). Must be ≥ 32 characters. |
| `DATASTACK_JWT_SECRET` | **Yes** | Built-in fallback | Master signing secret used to generate and verify HMAC-SHA256 JWT tokens for registered SDK clients. |
| `DATASTACK_ADMIN_SETUP_KEY` | **Yes** | Built-in fallback | Secret key required to complete the initial admin onboarding wizard at `/vcon/setup`. |
| `DATASTACK_INTERNAL_SECRET` | Recommended | Derived from JWT | Secret used for authenticated HTTP communication between Port 3000 and the Port 4000 ingestion server via `x-datastack-internal-secret`. If omitted, automatically derived via SHA-256 of `DATASTACK_JWT_SECRET`. |
| `DATASTACK_ALLOW_DEMO_ADMIN` | No | `false` | When set to `true`, enables quick demo credentials on the login screen (`admin@example.com`). Strictly prohibited in production. |
### PostgreSQL Database
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `DATABASE_URL` | Recommended | `undefined` | PostgreSQL connection URI (`postgresql://user:pass@host:port/dbname`). If omitted, DataStack operates in an in-memory mock mode. |
### Redis Shared Buffer
| Variable | Required in Prod | Default (Dev) | Description |
| :--- | :--- | :--- | :--- |
| `REDIS_URL` | Optional | `undefined` | Redis connection URI (`redis://host:6379`). When present, events are queued in Redis list `datastack:buffer:events`, surviving container restarts and enabling multi-worker flush locking. |
### Service Networking
| Variable | Required in Prod | Default | Description |
| :--- | :--- | :--- | :--- |
| `PORT` | No | `3000` | Port for the Next.js 15 Web Application and REST API. |
| `PORT_SOCKET` | No | `4000` | Port for the Express 5 + Socket.IO ingestion service. |
| `INGESTION_INTERNAL_URL` | No | `http://127.0.0.1:4000` | Internal URL used by Port 3000 to send bridge notifications to Port 4000 (`/internal/*`). |
---
## 3. Platform Runtime Settings (`/vcon/settings`)
In addition to environment variables, operators can adjust platform operational thresholds at runtime through the **Settings Workspace** (`/vcon/settings`).
These settings are stored in PostgreSQL (`admin_settings` table) and broadcast in real time to the Port 4000 ingestion service via `/internal/settings-sync`.
| Parameter | Type | Default | Permitted Range | Purpose |
| :--- | :--- | :--- | :--- | :--- |
| `maxEventsPerSec` | Number | `100` | `1` – `10,000` | Ingestion rate ceiling. Events exceeding this limit receive `RATE_LIMIT_EXCEEDED (429)`. |
| `maxPayloadSizeKb` | Number | `100` | `1` – `10,240` (10 MB) | Maximum permitted size in kilobytes for any single event payload. Oversized payloads receive `PAYLOAD_TOO_LARGE (413)`. |
| `batchFlushThreshold` | Number | `500` | `10` – `50,000` | Queue depth threshold that triggers an immediate write of buffered events to CSV files on disk. |
| `batchFlushIntervalMs` | Number | `2000` | `200` – `60,000` (60s) | Maximum duration pending events can remain buffered in memory/Redis before being flushed to disk. |
| `fileRotationMb` | Number | `100` | `5` – `1,024` (1 GB) | File size threshold per daily CSV partition. Once reached, DataStack writes to `{YYYY-MM-DD}-part2.csv`. |
| `enableIdempotency` | Boolean | `true` | `true` / `false` | Enables 5-minute sliding-window deduplication based on client-provided `eventId`. |
| `autoPruneDays` | Number | `30` | `0` – `3,650` | Automatically purges daily CSV partition files older than this value. Setting to `0` disables pruning and keeps all archives indefinitely. |
| `jwtSecret` | String | `.env` value | ≥ 16 characters | Master JWT signing secret. Rotating this value automatically re-signs all active SDK client credentials and notifies Port 4000. |
---
## 4. Cross-Process Synchronization Rules
When settings are modified in `/vcon/settings`:
1. **Database Persistence**: If PostgreSQL is configured, settings are written to `admin_settings WHERE id = 'default'`.
2. **Web Process Hydration**: Port 3000 updates its local singleton instances (`SecurityEngine`, `DataBufferEngine`, `CSVStorageManager`).
3. **Bridge Notification**: Port 3000 issues an authenticated HTTP `POST` to `${INGESTION_INTERNAL_URL}/internal/settings-sync` carrying the `x-datastack-internal-secret` header.
4. **Ingestion Reload**: Port 4000 receives the updated thresholds and immediately applies them to its socket handlers and background flusher worker without requiring a process restart.
+52
View File
@@ -0,0 +1,52 @@
# Administration Console (`/vcon/`)
The DataStack Administration Console (`/vcon/`) provides operators and systems engineers with a central interface for observing telemetry ingestion, tracking disk utilization, managing administrator accounts, and tuning ingestion limits.
---
## 1. Authentication & Access
Access the console by visiting:
```text
https://your-domain.example/vcon/login
```
- **Authentication Method**: Signed HTTP-only session cookies (`datastack_admin_session`) using WebCrypto HMAC-SHA256.
- **Session Security**: Session tokens encode the administrator's ID and current `tokenVersion`. Changing a password or clicking **Sign Out** immediately invalidates all active sessions globally.
- **Brute-Force Protection**: IP + Email failed attempt tracking locks out callers for 15 minutes after 5 consecutive failures.
---
## 2. Dashboard Workspace (`/vcon/dashboard`)
The dashboard provides real-time operational visibility:
### Dual-Runtime Health Indicators
- **Web App (Port 3000)**: Displays status, uptime, and database connection state.
- **Ingestion Service (Port 4000)**: Verifies the real-time bridge via `GET /internal/metrics`. Displays active Socket.IO connection counts and server status.
### Metric Gauges & Telemetry Cards
- **Total Ingested Events**: Lifetime count of telemetry events recorded across all registered client partitions.
- **Active Clients**: Number of SDK client credentials currently active and authorized to send data.
- **Total Storage Used**: Total disk footprint of all CSV partitions stored under `/app/storage/data`.
- **Partition Files**: Count of daily partition files currently retained on disk.
- **Unflushed Buffer Depth**: Number of events currently awaiting disk write in local RAM or Redis.
- **Flush Threshold / Interval**: Configured batch write limits (e.g., 500 events or 2000 ms).
### Live Activity Ticker
- Displays a real-time list of recently ingested events across all connected clients.
- Shows timestamp (UTC), client identifier, event ID, and an expandable preview of the JSON payload.
- Auto-refreshes periodically or on-demand.
### Manual Buffer Flush
- Clicking **Flush Buffer** dispatches a request to `/api/events/flush` which synchronously notifies Port 4000 (`POST /internal/flush`) and flushes Port 3000's RAM queue to disk.
---
## 3. Profile & Security Workspace (`/vcon/profile`)
Manage administrator credentials and session tokens:
- **Current Operator Info**: Displays email address and creation timestamp.
- **Password Modification**: Update account passwords with mandatory current password verification. Passwords are encrypted using `crypto.scrypt` (64-byte derived key, 16-byte cryptographically secure random salt).
- **Session Revocation**: Clicking **Log Out Other Sessions** increments the user's `tokenVersion` in PostgreSQL, instantly invalidating any other browsers or cookies.
+58
View File
@@ -0,0 +1,58 @@
# Client SDK Registry & Management (`/vcon/clients`)
DataStack enforces strict authentication for all telemetry sources. The **Client Management** workspace (`/vcon/clients`) allows operators to register client applications, generate and rotate JWT credentials, adjust per-client rate limits, and revoke access instantly.
---
## 1. Registering an SDK Client
To onboard a new application or telemetry producer:
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
2. Complete the form:
- **Client Identifier (`clientId`)**: A unique lowercase slug (e.g., `pos_terminal_01`, `mobile_app_prod`, `fleet_vehicle_42`). Allowed characters: `a-z`, `0-9`, `_`, `-`.
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
- **Rate Limit (Events/sec)**: Maximum allowed throughput for this specific client (default: `100`).
3. Click **Register Client**.
4. **Copy the Generated Token**:
- DataStack generates a cryptographically random 32-character client secret.
- Using `DATASTACK_JWT_SECRET`, DataStack signs an HMAC-SHA256 JWT payload containing:
```json
{
"clientId": "pos_terminal_01",
"name": "Point of Sale Terminal #1",
"rateLimit": 100,
"iat": 1700000000
}
```
- Store this token in your client's environment or configuration.
---
## 2. Managing Client Credentials & Lifecycle
### Rotating Client Tokens
If a client secret or JWT token is compromised or needs routine rotation:
1. Locate the client in the table and click **Rotate Token**.
2. Confirm the prompt.
3. DataStack generates a fresh client secret, signs a new JWT, and updates PostgreSQL.
4. **Instant Disconnection**: Port 3000 calls `POST /internal/client-sync`. The Port 4000 ingestion server immediately severs all active WebSocket connections holding the old token. The client must reconnect using the newly generated token.
### Revoking Access
To temporarily pause ingestion from a specific client without deleting its history:
1. Click the status dropdown or action button and select **Revoke**.
2. The client status changes to `revoked`.
3. Port 4000 immediately rejects new handshakes and terminates existing connections with `CLIENT_REVOKED (403)`.
4. HTTP requests to `POST /api/events` with this token are rejected with `403 Forbidden`.
### Restoring Access
1. Select a revoked client and click **Activate**.
2. Handshakes and HTTP ingestion are instantly permitted again.
### Deleting a Client
When an application is permanently decommissioned:
1. Click **Delete Client**.
2. Choose whether to:
- **Keep Historical Data**: Retains `/app/storage/data/{clientId}/` on disk for future analytics and export.
- **Purge Historical Data**: Deletes the partition folder and all CSV files for this client permanently.
3. **Tombstone Protection**: DataStack records the client ID and secret in the `deleted_clients` PostgreSQL table. Even if someone attempts to reuse an old signed JWT, the ingestion server checks the tombstone list and rejects the request.
+74
View File
@@ -0,0 +1,74 @@
# Streaming Data Export (`/vcon/export`)
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
---
## 1. Streaming Architecture
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
---
## 2. Export Configuration & Options
The export workspace provides targeted filtering:
| Option | Values | Description |
| :--- | :--- | :--- |
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
---
## 3. Supported Formats & Tool Compatibility
### CSV Format (`text/csv`)
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
#### Importing CSV into Python (pandas)
```python
import pandas as pd
import json
# Load exported CSV
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
# Parse JSON payloads into a structured DataFrame
df["parsed_payload"] = df["payload"].apply(json.loads)
print(df.head())
```
#### Importing CSV into DuckDB
```sql
-- Query directly from the exported CSV without importing
SELECT
clientId,
timestamp,
json_extract_string(payload, '$.temperature') AS temperature
FROM read_csv_auto('datastack_export.csv')
WHERE temperature > 25.0;
```
### JSON Format (`application/json`)
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
---
## 4. Programmatic Export via API
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
```bash
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
--output export.csv
```
+83
View File
@@ -0,0 +1,83 @@
# Data Simulator (`/vcon/simulator`)
The DataStack Data Simulator (`/vcon/simulator`) enables operators and developers to validate ingestion pipelines, test rate limits, and inspect buffer flushes directly from the browser without setting up external producer scripts.
---
## 1. Overview & Capabilities
The Simulator allows you to:
- Test ingestion using either **HTTP REST** (`POST /api/events`) or **Socket.IO WebSocket** (`sync_data`).
- Select any active client from your registered client list, or simulate unauthenticated/custom payloads.
- Dispatch single events or high-volume bursts (from 10 to 500 events).
- Select from realistic schema presets or construct custom JSON objects.
- Observe response status, payload transmission size, and round-trip latency in real time.
---
## 2. Telemetry Presets
The simulator includes four industry-standard telemetry presets:
### 1. IoT Temperature Sensor
```json
{
"sensorId": "temp-north-01",
"temperature": 23.4,
"humidity": 45.2,
"battery": 98,
"unit": "celsius"
}
```
### 2. E-Commerce Order Checkout
```json
{
"orderId": "ord_9872134",
"customerId": "cust_11029",
"currency": "USD",
"totalAmount": 149.50,
"itemCount": 3,
"paymentMethod": "credit_card"
}
```
### 3. Server System Resource Metrics
```json
{
"host": "prod-api-worker-03",
"cpuUsagePercent": 34.2,
"memoryUsedMb": 1024,
"memoryTotalMb": 4096,
"loadAverage": [0.45, 0.62, 0.58]
}
```
### 4. User Audit Trail & Security Event
```json
{
"actorId": "usr_9912",
"action": "client_token_rotated",
"ipAddress": "192.168.1.100",
"userAgent": "Mozilla/5.0",
"success": true
}
```
---
## 3. Simulator Workflows
### Testing Single Event Ingestion
1. Select your target **Client** from the dropdown.
2. Select **Single Event** mode.
3. Choose a preset or customize the JSON editor.
4. Click **Send Event**.
5. The output console displays the assigned `eventId`, server response code (`200 OK`), and processing time.
### Testing High-Throughput Batch Flushing
1. Select **Batch Dispatch** mode.
2. Set the burst count (e.g., `500` events).
3. Click **Start Batch**.
4. Observe the batch progress bar.
5. Navigate to `/vcon/dashboard` to verify that the buffer threshold flush automatically triggered, writing the batch to the client's daily CSV partition file on disk.
+24
View File
@@ -0,0 +1,24 @@
# Features Guide
Welcome to the DataStack Feature Documentation. This section details the complete feature set of the DataStack high-throughput telemetry platform.
---
## Feature Index
| Feature Document | Purpose & Capabilities | Primary Navigation Path |
| :--- | :--- | :--- |
| [Admin Console (`admin-console.md`)](./admin-console.md) | System health monitoring, dual-runtime status, memory/buffer gauges, and administrator account management. | `/vcon/dashboard`, `/vcon/profile` |
| [Client Management (`client-management.md`)](./client-management.md) | Client SDK registration, HMAC-SHA256 JWT generation, credential rotation, socket revocation, and tombstone deletion. | `/vcon/clients` |
| [Data Simulator (`data-simulator.md`)](./data-simulator.md) | Browser-based event generation, single vs. burst testing, WebSocket/HTTP transport modes, and telemetry payload presets. | `/vcon/simulator` |
| [Partitioned CSV Storage (`partitioned-storage.md`)](./partitioned-storage.md) | Daily client directory partitioning, 100MB rotation (`-part2.csv`), cooperative lock files, and automated retention pruning. | `/vcon/storage` |
| [Data Export (`data-export.md`)](./data-export.md) | Streaming CSV/JSON extraction, date range filters, client filtering, Excel/pandas compatibility, and non-blocking reads. | `/vcon/export` |
---
## Architecture Summary
Every feature in DataStack follows a strict separation of concerns:
- **Control Plane**: Handled on Port 3000 via Next.js 15 App Router and PostgreSQL metadata tables (`admin_users`, `client_sdks`, `admin_settings`).
- **Data Plane**: Handled on Port 4000 via Express 5, Socket.IO 4, and persistent append-only CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
- **Zero SQL Insertions**: Telemetry events never create rows in relational tables, ensuring predictable query performance and infinite scale-to-zero storage efficiency.
@@ -0,0 +1,74 @@
# Partitioned CSV Storage (`/vcon/storage`)
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
---
## 1. Storage Architecture & Directory Layout
Telemetry files are partitioned by client identifier and UTC date:
```text
storage/
└── data/
├── pos_terminal_01/
│ ├── 2026-11-16.csv
│ ├── 2026-11-17.csv
│ └── 2026-11-17-part2.csv
└── mobile_app_prod/
├── 2026-11-17.csv
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
```
### Partition File Format
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
```csv
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
```
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
- **Newline**: Strict Unix standard `\n` line delimiters.
---
## 2. File Rotation Mechanism
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
- **Rotation Behavior**:
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
- Part 1: `2026-11-17.csv`
- Part 2: `2026-11-17-part2.csv`
- Part 3: `2026-11-17-part3.csv`
3. Subsequent writes immediately target the active part file.
---
## 3. Cooperative Locking Protocol
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
---
## 4. Automated Retention Pruning
DataStack manages disk usage through an automated pruning engine:
- **Setting**: `autoPruneDays` (default: `30` days).
- **Pruning Logic**:
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
- If a client's folder becomes empty, the folder is safely removed.
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.
+164
View File
@@ -0,0 +1,164 @@
# Getting Started with DataStack
This fast-track tutorial guides you from a fresh server to a fully running DataStack instance collecting live telemetry in under 5 minutes.
---
## Step 1: Deploy with Docker Compose
Clone the repository and copy the environment template:
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
cp .env.example .env
```
Generate 32+ character secrets for production:
```bash
# Generate random keys in Linux or macOS:
openssl rand -hex 32
```
In `.env`, set your secrets and admin email:
```dotenv
DATASTACK_ADMIN_SESSION_SECRET=your-32-char-session-secret-here
DATASTACK_JWT_SECRET=your-32-char-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-32-char-setup-key-here
DATASTACK_INTERNAL_SECRET=your-32-char-internal-secret-here
DATASTACK_ADMIN_EMAIL=admin@example.com
```
Launch the stack:
```bash
docker compose -f docker/docker-compose.yml up -d
```
Verify that all 5 services are healthy:
```bash
docker compose -f docker/docker-compose.yml ps
```
---
## Step 2: Complete Initial Administrator Setup
1. Open your browser and navigate to:
```text
http://localhost/vcon/setup
# Or https://your-domain.example/vcon/setup
```
2. Enter:
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` (or your chosen admin email).
- **Administrator Password**: Minimum 8 characters.
- **Setup Security Key**: Enter the exact string from `DATASTACK_ADMIN_SETUP_KEY`.
3. Click **Complete Initial Setup**. You are automatically logged in and redirected to `/vcon/dashboard`.
---
## Step 3: Register Your First SDK Client
1. In the sidebar, click **Clients** (`/vcon/clients`).
2. Click **+ Register Client**.
3. Fill in the fields:
- **Client Identifier**: `sensor_gateway_01` (lowercase alphanumeric and underscores).
- **Client Name**: `Primary Sensor Gateway`.
- **Rate Limit**: `100` (max events per second).
4. Click **Register Client**.
5. Copy the generated **Bearer JWT Token**. Store it securely.
---
## Step 4: Dispatch Your First Event via `curl`
Send a test telemetry event to the REST ingestion API:
```bash
curl -X POST "http://localhost/api/events" \
-H "Authorization: Bearer YOUR_GENERATED_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"eventId": "test_event_001",
"timestamp": "2026-11-17T12:00:00Z",
"data": {
"temperature": 23.4,
"humidity": 45,
"status": "ONLINE"
}
}'
```
**Expected Response**:
```json
{
"success": true,
"eventId": "test_event_001",
"event": {
"clientId": "sensor_gateway_01",
"eventId": "test_event_001",
"timestamp": "2026-11-17T12:00:00Z",
"payload": "{\"temperature\":23.4,\"humidity\":45,\"status\":\"ONLINE\"}"
}
}
```
---
## Step 5: Connect with the Python Client SDK
Install the official Python SDK:
```bash
pip install datastack
```
In your Python project directory, create `config.datastack.py`:
```python
DATASTACK_CONFIG = {
"SERVER_URL": "http://localhost",
"CLIENT_TOKEN": "YOUR_GENERATED_JWT_TOKEN",
"AUTO_CONNECT": True,
}
```
Create `main.py` and run it:
```python
from datastack import DataStack
import time
# Auto-discovers config.datastack.py in the current directory
client = DataStack()
print(f"Connected to DataStack as client: {client.client_id}")
# Stream 5 test events
for i in range(5):
res = client.send(
data={"reading_index": i, "voltage": 3.3 + (i * 0.05)},
event_id=f"reading_{i}_{int(time.time())}"
)
print(f"Dispatched event {i}: {res.get('eventId')} via {res.get('transport')}")
time.sleep(1)
client.disconnect()
```
---
## Step 6: Verify Data in the Admin Console
1. **Dashboard Overview (`/vcon/dashboard`)**:
- Check the **Active Buffer** gauge — incoming events are queued in Redis / memory.
- Observe the **Live Activity Stream** updating in real time.
2. **Data Browser (`/vcon/data`)**:
- Filter by client `sensor_gateway_01`.
- View your ingested events with exact timestamps and formatted JSON payloads.
3. **Data Export (`/vcon/export`)**:
- Click **Export Dataset** to download a clean, consolidated CSV file ready for Excel or Python analysis.
---
## Next Steps
- Consult the complete **[Python SDK Documentation](../../sdk/python/datastack/docs/README.md)** for API reference, offline spooling, and file ingestion.
- Learn how to stream files (`.csv`, `.json`, `.jsonl`) with **[Tutorials](./tutorials.md)**.
- Understand storage rotation and auto-pruning in **[Partitioned CSV Storage](./features/partitioned-storage.md)**.
- Explore production deployment and backups in **[Installation Guide](./installation.md)**.
+55
View File
@@ -0,0 +1,55 @@
# DataStack User & Operator Guide
Welcome to the **DataStack User & Operator Documentation**. This portal contains comprehensive guides for deploying, configuring, operating, and integrating applications with DataStack.
---
## Guide Directory
```text
docs/users/
├── index.md # This directory index
├── getting-started.md # 5-minute setup and quickstart tutorial
├── installation.md # Production Docker Compose & Coolify deployment
├── configuration.md # Secrets, environment variables & settings reference
├── features/ # Detailed functional walkthroughs
│ ├── index.md # Features directory index
│ ├── admin-console.md # /vcon/ dashboard, monitors, and profile workspace
│ ├── client-management.md # SDK client registration, JWT rotation, and revocation
│ ├── data-simulator.md # Interactive browser telemetry simulator & presets
│ ├── partitioned-storage.md # Daily CSV storage, file rotation, and auto-pruning
│ └── data-export.md # Streaming CSV and JSON dataset export workspace
├── workflows.md # Step-by-step user journeys & operations
├── troubleshooting.md # Forensic error recovery & diagnostics
└── faq.md # Frequently asked questions & storage architecture
```
---
## Core Reading Pathways
Choose the pathway that matches your immediate goal:
### Pathway 1: "I want to self-host DataStack on my VPS"
1. Start with **[Installation & Deployment](./installation.md)** for Docker Compose or Coolify setup.
2. Review **[Configuration Guide](./configuration.md)** to generate 32+ character secrets.
3. Follow **[Getting Started](./getting-started.md)** to run the `/vcon/setup` wizard and log in.
### Pathway 2: "I want to connect my Python application"
1. Review **[Client Token Management](./features/client-management.md)** to generate a client JWT token.
2. Read **[Tutorial: Python SDK Integration](./getting-started.md#step-4-connect-with-the-python-sdk)** to set up `config.datastack.py`.
3. Check **[Troubleshooting](./troubleshooting.md)** if you encounter rate limits or connection drops.
### Pathway 3: "I want to inspect or export collected data"
1. Read **[Partitioned CSV Storage](./features/partitioned-storage.md)** to understand directory and file layouts.
2. Read **[Streaming Data Export](./features/data-export.md)** to extract datasets via browser or `curl`.
3. Follow the **[Data Analysis Workflow](./workflows.md#workflow-3-exporting-and-analyzing-data)** to load records into pandas or Excel.
---
## Platform Philosophy
DataStack is built to solve the **"High-Velocity Telemetry Dilemma"**:
- **Why not PostgreSQL for everything?** Inserting thousands of events per second into relational database tables causes index thrashing, lock contention, and WAL exhaustion.
- **The DataStack Solution**: PostgreSQL is used strictly for administrative accounts and client credentials. Ingested events flow through in-memory and Redis buffers directly into organized, daily partitioned CSV files (`/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`).
- **The Benefit**: Virtually unlimited ingestion throughput, zero SQL lock contention, deterministic storage costs, and instant portability.
+114
View File
@@ -0,0 +1,114 @@
# Troubleshooting & Forensic Diagnostics
This guide provides systematic troubleshooting instructions for diagnosing and resolving common operational issues, connection failures, and error codes in DataStack.
---
## 1. Quick Diagnostic Checklist
When investigating an issue, verify the following baseline checks:
1. **Check Container Status**:
```bash
docker compose ps
```
All five services (`datastack-web`, `datastack-ingestion`, `datastack-proxy`, `datastack-postgres`, `datastack-redis`) should display `Up` and `(healthy)`.
2. **Check Nginx Proxy Health Endpoint**:
```bash
curl -i http://localhost/health
# Expected response: HTTP/1.1 200 OK
# Body: OK
```
3. **Check Web Application Status**:
```bash
curl -i http://localhost:3000/api/health
# Expected response: {"status":"ok","uptime":...}
```
4. **Inspect Live Logs**:
```bash
docker compose logs -f --tail=100 datastack-web datastack-ingestion
```
---
## 2. Ingestion & API Errors
### HTTP 401 Unauthorized (`INVALID_TOKEN`)
- **Cause**: The client request did not supply an `Authorization` header, or the provided JWT signature is invalid.
- **Resolution**:
- Ensure the client request header contains: `Authorization: Bearer <TOKEN>`.
- Check whether the platform JWT secret was recently rotated in `/vcon/settings`. If rotated, copy the newly signed token from `/vcon/clients`.
### HTTP 403 Forbidden (`CLIENT_REVOKED` or `CLIENT_DELETED`)
- **Cause**: The client was marked as `revoked` in `/vcon/clients`, or the client ID exists in the `deleted_clients` tombstone registry.
- **Resolution**:
- In `/vcon/clients`, verify the client's status. If revoked, click **Activate**.
- If the client was permanently deleted, register a new client slug.
### HTTP 403 Forbidden (`CSRF_FAILED`)
- **Cause**: State-changing POST/PUT/DELETE requests to administrative routes did not match the required `Origin` or `Referer` headers.
- **Resolution**:
- When accessing behind a reverse proxy (e.g., Cloudflare, Traefik, Coolify), ensure the proxy forwards the original host header:
```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
```
### HTTP 413 Payload Too Large (`PAYLOAD_TOO_LARGE`)
- **Cause**: An incoming event payload exceeds `maxPayloadSizeKb` (default: 100 KB).
- **Resolution**:
- Split large batches into smaller chunks, or increase the payload limit in `/vcon/settings` (up to 10 MB).
### HTTP 429 Too Many Requests (`RATE_LIMIT_EXCEEDED`)
- **Cause**: The client has exceeded its configured rate limit (events/second) or the platform-wide limit.
- **Resolution**:
- Navigate to `/vcon/clients`, click **Edit Client**, and increase the per-client rate limit.
- Or in `/vcon/settings`, raise `maxEventsPerSec`.
---
## 3. WebSocket & Socket.IO Connection Issues
### Handshake Fails or Drops Back to Long-Polling
- **Symptoms**: Browser or Python client reports continuous reconnect attempts; delays in data synchronization.
- **Cause**: The upstream reverse proxy is not passing the WebSocket upgrade headers.
- **Resolution**:
- Verify your reverse proxy configuration includes:
```nginx
location /socket.io/ {
proxy_pass http://datastack-ingestion:4000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
}
```
---
## 4. Storage & Filesystem Issues
### Write Delays or Stale Lock Warning in Logs
- **Symptoms**: Log message shows: `Acquiring lock for partition ... timeout`.
- **Cause**: A previous worker process was abruptly terminated (e.g., `kill -9` or OOM) while writing to disk, leaving a `.lock` file behind.
- **Resolution**:
- DataStack automatically cleans up locks older than 30 seconds.
- If an immediate manual clear is needed, check for `.lock` files in the partition directory:
```bash
find storage/data -name "*.lock" -type f
```
- Unlink locks older than 60 seconds:
```bash
find storage/data -name "*.lock" -type f -mmin +1 -delete
```
### Missing Historical Partitions
- **Cause**: Automated retention pruning (`autoPruneDays`) purges partition files older than the retention threshold.
- **Resolution**:
- Check the configured value in `/vcon/settings`. Set to `0` if you want all partition files retained indefinitely.
+125
View File
@@ -0,0 +1,125 @@
# Operational Workflows & User Journeys
This document provides step-by-step procedures for common operational workflows, from first-time administrator onboarding to credential rotation and disaster recovery.
---
## Workflow 1: Initial Administrator Onboarding
When setting up a newly deployed instance:
```text
Deploy Stack ──► Open /vcon/setup ──► Enter Setup Key & Admin Email ──► Create Password ──► Redirect to /vcon/dashboard
```
1. Visit your instance URL at `/vcon/setup` (e.g., `https://your-domain.example/vcon/setup`).
2. If an administrator already exists, DataStack redirects to `/vcon/login`.
3. Provide the required credentials:
- **Setup Key**: Enter the `DATASTACK_ADMIN_SETUP_KEY` configured in your `.env`.
- **Admin Email**: Enter the primary operational contact email.
- **Password**: Enter a strong password (minimum 8 characters, checked against common passwords).
4. Click **Complete Setup**.
5. DataStack hashes the password using `crypto.scrypt`, creates the first row in `admin_users`, establishes a signed session cookie, and redirects to `/vcon/dashboard`.
---
## Workflow 2: Onboarding an External Client Application
Connecting a new producer (Python app, IoT gateway, mobile app, microservice):
```text
/vcon/clients ──► Register Client ──► Copy JWT Token ──► Configure Client App ──► Stream Events
```
1. Navigate to **Clients** (`/vcon/clients`).
2. Click **+ Register Client**.
3. Specify:
- `Client Identifier`: e.g., `checkout_service_prod`
- `Client Name`: e.g., `Checkout Service Production`
- `Rate Limit`: e.g., `250` events/second
4. Click **Register Client**.
5. A modal appears displaying the newly signed HMAC-SHA256 JWT token.
6. **Copy the token** and configure your application environment:
```dotenv
DATASTACK_CLIENT_ID=checkout_service_prod
DATASTACK_AUTH_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
DATASTACK_SERVER_URL=https://your-domain.example
```
7. Start your producer. Events will immediately stream into the system and appear in the live dashboard activity feed.
---
## Workflow 3: Rotating a Client Token
If a client token has leaked or requires routine security rotation:
```text
/vcon/clients ──► Rotate Token ──► Copy New Token ──► Update Client Config ──► Reconnect
```
1. Open `/vcon/clients`.
2. Locate the client row and click **Rotate Token**.
3. Confirm the action in the confirmation modal.
4. DataStack generates a new client secret, signs a fresh JWT, updates PostgreSQL, and notifies Port 4000 via `/internal/client-sync`.
5. Active WebSocket connections using the prior token are disconnected immediately.
6. Copy the new token and update your producer application's environment. Once restarted, the client reconnects with the new token.
---
## Workflow 4: Rotating the Platform JWT Secret
In the event of an infrastructure-wide security review or platform secret compromise:
1. Open `/vcon/settings`.
2. In the **Security & Secrets** card, click **Rotate Platform Secret**.
3. Confirm the security alert.
4. DataStack:
- Generates a new cryptographically secure 48-character platform JWT secret.
- Saves the secret to `admin_settings`.
- Iterates through all registered clients in PostgreSQL and re-signs their tokens using the new master secret (`SDKManager.resignAllClientTokensAsync()`).
- Pushes the new secret and re-signed tokens to Port 4000 via `/internal/settings-sync`.
5. Navigate to `/vcon/clients` to export or view the refreshed client tokens to distribute to client application teams.
---
## Workflow 5: Backup & Disaster Recovery
DataStack stores state in two distinct layers: PostgreSQL (metadata) and filesystem volumes (telemetry data).
### Backing Up the PostgreSQL Control Plane
```bash
docker exec -t datastack-postgres pg_dump -U datastack datastack_db > datastack_meta_$(date +%F).sql
```
### Backing Up Telemetry Partitions
Telemetry files reside in the named Docker volume `datastack-data` (or mounted host path `./storage/data`).
```bash
# Archive all client partition directories
docker run --rm \
-v datastack-data:/data:ro \
-v $(pwd):/backup \
alpine tar czf /backup/datastack_partitions_$(date +%F).tar.gz -C /data .
```
### Restoring from Backup
1. Stop the ingestion and web containers:
```bash
docker compose down
```
2. Restore the database:
```bash
docker compose up -d postgres
cat datastack_meta_2026-11-17.sql | docker exec -i datastack-postgres psql -U datastack datastack_db
```
3. Restore the partition volume:
```bash
docker run --rm \
-v datastack-data:/data \
-v $(pwd):/backup \
alpine tar xzf /backup/datastack_partitions_2026-11-17.tar.gz -C /data
```
4. Restart the full stack:
```bash
docker compose up -d
```