feat(sdk): refactor Python SDK documentation

- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
Team Dark Devil committed 2026-11-18 10:03:40 +06:00
1 parent 4279da45bb
commit 40a6855724
38 files changed
+3315 -141

No files matched your search

+52
View File
@@ -0,0 +1,52 @@
# Administration Console (`/vcon/`)
The DataStack Administration Console (`/vcon/`) provides operators and systems engineers with a central interface for observing telemetry ingestion, tracking disk utilization, managing administrator accounts, and tuning ingestion limits.
---
## 1. Authentication & Access
Access the console by visiting:
```text
https://your-domain.example/vcon/login
```
- **Authentication Method**: Signed HTTP-only session cookies (`datastack_admin_session`) using WebCrypto HMAC-SHA256.
- **Session Security**: Session tokens encode the administrator's ID and current `tokenVersion`. Changing a password or clicking **Sign Out** immediately invalidates all active sessions globally.
- **Brute-Force Protection**: IP + Email failed attempt tracking locks out callers for 15 minutes after 5 consecutive failures.
---
## 2. Dashboard Workspace (`/vcon/dashboard`)
The dashboard provides real-time operational visibility:
### Dual-Runtime Health Indicators
- **Web App (Port 3000)**: Displays status, uptime, and database connection state.
- **Ingestion Service (Port 4000)**: Verifies the real-time bridge via `GET /internal/metrics`. Displays active Socket.IO connection counts and server status.
### Metric Gauges & Telemetry Cards
- **Total Ingested Events**: Lifetime count of telemetry events recorded across all registered client partitions.
- **Active Clients**: Number of SDK client credentials currently active and authorized to send data.
- **Total Storage Used**: Total disk footprint of all CSV partitions stored under `/app/storage/data`.
- **Partition Files**: Count of daily partition files currently retained on disk.
- **Unflushed Buffer Depth**: Number of events currently awaiting disk write in local RAM or Redis.
- **Flush Threshold / Interval**: Configured batch write limits (e.g., 500 events or 2000 ms).
### Live Activity Ticker
- Displays a real-time list of recently ingested events across all connected clients.
- Shows timestamp (UTC), client identifier, event ID, and an expandable preview of the JSON payload.
- Auto-refreshes periodically or on-demand.
### Manual Buffer Flush
- Clicking **Flush Buffer** dispatches a request to `/api/events/flush` which synchronously notifies Port 4000 (`POST /internal/flush`) and flushes Port 3000's RAM queue to disk.
---
## 3. Profile & Security Workspace (`/vcon/profile`)
Manage administrator credentials and session tokens:
- **Current Operator Info**: Displays email address and creation timestamp.
- **Password Modification**: Update account passwords with mandatory current password verification. Passwords are encrypted using `crypto.scrypt` (64-byte derived key, 16-byte cryptographically secure random salt).
- **Session Revocation**: Clicking **Log Out Other Sessions** increments the user's `tokenVersion` in PostgreSQL, instantly invalidating any other browsers or cookies.
+58
View File
@@ -0,0 +1,58 @@
# Client SDK Registry & Management (`/vcon/clients`)
DataStack enforces strict authentication for all telemetry sources. The **Client Management** workspace (`/vcon/clients`) allows operators to register client applications, generate and rotate JWT credentials, adjust per-client rate limits, and revoke access instantly.
---
## 1. Registering an SDK Client
To onboard a new application or telemetry producer:
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
2. Complete the form:
- **Client Identifier (`clientId`)**: A unique lowercase slug (e.g., `pos_terminal_01`, `mobile_app_prod`, `fleet_vehicle_42`). Allowed characters: `a-z`, `0-9`, `_`, `-`.
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
- **Rate Limit (Events/sec)**: Maximum allowed throughput for this specific client (default: `100`).
3. Click **Register Client**.
4. **Copy the Generated Token**:
- DataStack generates a cryptographically random 32-character client secret.
- Using `DATASTACK_JWT_SECRET`, DataStack signs an HMAC-SHA256 JWT payload containing:
```json
{
"clientId": "pos_terminal_01",
"name": "Point of Sale Terminal #1",
"rateLimit": 100,
"iat": 1700000000
}
```
- Store this token in your client's environment or configuration.
---
## 2. Managing Client Credentials & Lifecycle
### Rotating Client Tokens
If a client secret or JWT token is compromised or needs routine rotation:
1. Locate the client in the table and click **Rotate Token**.
2. Confirm the prompt.
3. DataStack generates a fresh client secret, signs a new JWT, and updates PostgreSQL.
4. **Instant Disconnection**: Port 3000 calls `POST /internal/client-sync`. The Port 4000 ingestion server immediately severs all active WebSocket connections holding the old token. The client must reconnect using the newly generated token.
### Revoking Access
To temporarily pause ingestion from a specific client without deleting its history:
1. Click the status dropdown or action button and select **Revoke**.
2. The client status changes to `revoked`.
3. Port 4000 immediately rejects new handshakes and terminates existing connections with `CLIENT_REVOKED (403)`.
4. HTTP requests to `POST /api/events` with this token are rejected with `403 Forbidden`.
### Restoring Access
1. Select a revoked client and click **Activate**.
2. Handshakes and HTTP ingestion are instantly permitted again.
### Deleting a Client
When an application is permanently decommissioned:
1. Click **Delete Client**.
2. Choose whether to:
- **Keep Historical Data**: Retains `/app/storage/data/{clientId}/` on disk for future analytics and export.
- **Purge Historical Data**: Deletes the partition folder and all CSV files for this client permanently.
3. **Tombstone Protection**: DataStack records the client ID and secret in the `deleted_clients` PostgreSQL table. Even if someone attempts to reuse an old signed JWT, the ingestion server checks the tombstone list and rejects the request.
+74
View File
@@ -0,0 +1,74 @@
# Streaming Data Export (`/vcon/export`)
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
---
## 1. Streaming Architecture
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
---
## 2. Export Configuration & Options
The export workspace provides targeted filtering:
| Option | Values | Description |
| :--- | :--- | :--- |
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
---
## 3. Supported Formats & Tool Compatibility
### CSV Format (`text/csv`)
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
#### Importing CSV into Python (pandas)
```python
import pandas as pd
import json
# Load exported CSV
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
# Parse JSON payloads into a structured DataFrame
df["parsed_payload"] = df["payload"].apply(json.loads)
print(df.head())
```
#### Importing CSV into DuckDB
```sql
-- Query directly from the exported CSV without importing
SELECT
clientId,
timestamp,
json_extract_string(payload, '$.temperature') AS temperature
FROM read_csv_auto('datastack_export.csv')
WHERE temperature > 25.0;
```
### JSON Format (`application/json`)
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
---
## 4. Programmatic Export via API
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
```bash
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
--output export.csv
```
+83
View File
@@ -0,0 +1,83 @@
# Data Simulator (`/vcon/simulator`)
The DataStack Data Simulator (`/vcon/simulator`) enables operators and developers to validate ingestion pipelines, test rate limits, and inspect buffer flushes directly from the browser without setting up external producer scripts.
---
## 1. Overview & Capabilities
The Simulator allows you to:
- Test ingestion using either **HTTP REST** (`POST /api/events`) or **Socket.IO WebSocket** (`sync_data`).
- Select any active client from your registered client list, or simulate unauthenticated/custom payloads.
- Dispatch single events or high-volume bursts (from 10 to 500 events).
- Select from realistic schema presets or construct custom JSON objects.
- Observe response status, payload transmission size, and round-trip latency in real time.
---
## 2. Telemetry Presets
The simulator includes four industry-standard telemetry presets:
### 1. IoT Temperature Sensor
```json
{
"sensorId": "temp-north-01",
"temperature": 23.4,
"humidity": 45.2,
"battery": 98,
"unit": "celsius"
}
```
### 2. E-Commerce Order Checkout
```json
{
"orderId": "ord_9872134",
"customerId": "cust_11029",
"currency": "USD",
"totalAmount": 149.50,
"itemCount": 3,
"paymentMethod": "credit_card"
}
```
### 3. Server System Resource Metrics
```json
{
"host": "prod-api-worker-03",
"cpuUsagePercent": 34.2,
"memoryUsedMb": 1024,
"memoryTotalMb": 4096,
"loadAverage": [0.45, 0.62, 0.58]
}
```
### 4. User Audit Trail & Security Event
```json
{
"actorId": "usr_9912",
"action": "client_token_rotated",
"ipAddress": "192.168.1.100",
"userAgent": "Mozilla/5.0",
"success": true
}
```
---
## 3. Simulator Workflows
### Testing Single Event Ingestion
1. Select your target **Client** from the dropdown.
2. Select **Single Event** mode.
3. Choose a preset or customize the JSON editor.
4. Click **Send Event**.
5. The output console displays the assigned `eventId`, server response code (`200 OK`), and processing time.
### Testing High-Throughput Batch Flushing
1. Select **Batch Dispatch** mode.
2. Set the burst count (e.g., `500` events).
3. Click **Start Batch**.
4. Observe the batch progress bar.
5. Navigate to `/vcon/dashboard` to verify that the buffer threshold flush automatically triggered, writing the batch to the client's daily CSV partition file on disk.
+24
View File
@@ -0,0 +1,24 @@
# Features Guide
Welcome to the DataStack Feature Documentation. This section details the complete feature set of the DataStack high-throughput telemetry platform.
---
## Feature Index
| Feature Document | Purpose & Capabilities | Primary Navigation Path |
| :--- | :--- | :--- |
| [Admin Console (`admin-console.md`)](./admin-console.md) | System health monitoring, dual-runtime status, memory/buffer gauges, and administrator account management. | `/vcon/dashboard`, `/vcon/profile` |
| [Client Management (`client-management.md`)](./client-management.md) | Client SDK registration, HMAC-SHA256 JWT generation, credential rotation, socket revocation, and tombstone deletion. | `/vcon/clients` |
| [Data Simulator (`data-simulator.md`)](./data-simulator.md) | Browser-based event generation, single vs. burst testing, WebSocket/HTTP transport modes, and telemetry payload presets. | `/vcon/simulator` |
| [Partitioned CSV Storage (`partitioned-storage.md`)](./partitioned-storage.md) | Daily client directory partitioning, 100MB rotation (`-part2.csv`), cooperative lock files, and automated retention pruning. | `/vcon/storage` |
| [Data Export (`data-export.md`)](./data-export.md) | Streaming CSV/JSON extraction, date range filters, client filtering, Excel/pandas compatibility, and non-blocking reads. | `/vcon/export` |
---
## Architecture Summary
Every feature in DataStack follows a strict separation of concerns:
- **Control Plane**: Handled on Port 3000 via Next.js 15 App Router and PostgreSQL metadata tables (`admin_users`, `client_sdks`, `admin_settings`).
- **Data Plane**: Handled on Port 4000 via Express 5, Socket.IO 4, and persistent append-only CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
- **Zero SQL Insertions**: Telemetry events never create rows in relational tables, ensuring predictable query performance and infinite scale-to-zero storage efficiency.
@@ -0,0 +1,74 @@
# Partitioned CSV Storage (`/vcon/storage`)
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
---
## 1. Storage Architecture & Directory Layout
Telemetry files are partitioned by client identifier and UTC date:
```text
storage/
└── data/
├── pos_terminal_01/
│ ├── 2026-11-16.csv
│ ├── 2026-11-17.csv
│ └── 2026-11-17-part2.csv
└── mobile_app_prod/
├── 2026-11-17.csv
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
```
### Partition File Format
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
```csv
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
```
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
- **Newline**: Strict Unix standard `\n` line delimiters.
---
## 2. File Rotation Mechanism
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
- **Rotation Behavior**:
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
- Part 1: `2026-11-17.csv`
- Part 2: `2026-11-17-part2.csv`
- Part 3: `2026-11-17-part3.csv`
3. Subsequent writes immediately target the active part file.
---
## 3. Cooperative Locking Protocol
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
---
## 4. Automated Retention Pruning
DataStack manages disk usage through an automated pruning engine:
- **Setting**: `autoPruneDays` (default: `30` days).
- **Pruning Logic**:
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
- If a client's folder becomes empty, the folder is safely removed.
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.