feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
@@ -0,0 +1,52 @@
|
||||
# Administration Console (`/vcon/`)
|
||||
|
||||
The DataStack Administration Console (`/vcon/`) provides operators and systems engineers with a central interface for observing telemetry ingestion, tracking disk utilization, managing administrator accounts, and tuning ingestion limits.
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication & Access
|
||||
|
||||
Access the console by visiting:
|
||||
```text
|
||||
https://your-domain.example/vcon/login
|
||||
```
|
||||
|
||||
- **Authentication Method**: Signed HTTP-only session cookies (`datastack_admin_session`) using WebCrypto HMAC-SHA256.
|
||||
- **Session Security**: Session tokens encode the administrator's ID and current `tokenVersion`. Changing a password or clicking **Sign Out** immediately invalidates all active sessions globally.
|
||||
- **Brute-Force Protection**: IP + Email failed attempt tracking locks out callers for 15 minutes after 5 consecutive failures.
|
||||
|
||||
---
|
||||
|
||||
## 2. Dashboard Workspace (`/vcon/dashboard`)
|
||||
|
||||
The dashboard provides real-time operational visibility:
|
||||
|
||||
### Dual-Runtime Health Indicators
|
||||
- **Web App (Port 3000)**: Displays status, uptime, and database connection state.
|
||||
- **Ingestion Service (Port 4000)**: Verifies the real-time bridge via `GET /internal/metrics`. Displays active Socket.IO connection counts and server status.
|
||||
|
||||
### Metric Gauges & Telemetry Cards
|
||||
- **Total Ingested Events**: Lifetime count of telemetry events recorded across all registered client partitions.
|
||||
- **Active Clients**: Number of SDK client credentials currently active and authorized to send data.
|
||||
- **Total Storage Used**: Total disk footprint of all CSV partitions stored under `/app/storage/data`.
|
||||
- **Partition Files**: Count of daily partition files currently retained on disk.
|
||||
- **Unflushed Buffer Depth**: Number of events currently awaiting disk write in local RAM or Redis.
|
||||
- **Flush Threshold / Interval**: Configured batch write limits (e.g., 500 events or 2000 ms).
|
||||
|
||||
### Live Activity Ticker
|
||||
- Displays a real-time list of recently ingested events across all connected clients.
|
||||
- Shows timestamp (UTC), client identifier, event ID, and an expandable preview of the JSON payload.
|
||||
- Auto-refreshes periodically or on-demand.
|
||||
|
||||
### Manual Buffer Flush
|
||||
- Clicking **Flush Buffer** dispatches a request to `/api/events/flush` which synchronously notifies Port 4000 (`POST /internal/flush`) and flushes Port 3000's RAM queue to disk.
|
||||
|
||||
---
|
||||
|
||||
## 3. Profile & Security Workspace (`/vcon/profile`)
|
||||
|
||||
Manage administrator credentials and session tokens:
|
||||
|
||||
- **Current Operator Info**: Displays email address and creation timestamp.
|
||||
- **Password Modification**: Update account passwords with mandatory current password verification. Passwords are encrypted using `crypto.scrypt` (64-byte derived key, 16-byte cryptographically secure random salt).
|
||||
- **Session Revocation**: Clicking **Log Out Other Sessions** increments the user's `tokenVersion` in PostgreSQL, instantly invalidating any other browsers or cookies.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Client SDK Registry & Management (`/vcon/clients`)
|
||||
|
||||
DataStack enforces strict authentication for all telemetry sources. The **Client Management** workspace (`/vcon/clients`) allows operators to register client applications, generate and rotate JWT credentials, adjust per-client rate limits, and revoke access instantly.
|
||||
|
||||
---
|
||||
|
||||
## 1. Registering an SDK Client
|
||||
|
||||
To onboard a new application or telemetry producer:
|
||||
|
||||
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
|
||||
2. Complete the form:
|
||||
- **Client Identifier (`clientId`)**: A unique lowercase slug (e.g., `pos_terminal_01`, `mobile_app_prod`, `fleet_vehicle_42`). Allowed characters: `a-z`, `0-9`, `_`, `-`.
|
||||
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
|
||||
- **Rate Limit (Events/sec)**: Maximum allowed throughput for this specific client (default: `100`).
|
||||
3. Click **Register Client**.
|
||||
4. **Copy the Generated Token**:
|
||||
- DataStack generates a cryptographically random 32-character client secret.
|
||||
- Using `DATASTACK_JWT_SECRET`, DataStack signs an HMAC-SHA256 JWT payload containing:
|
||||
```json
|
||||
{
|
||||
"clientId": "pos_terminal_01",
|
||||
"name": "Point of Sale Terminal #1",
|
||||
"rateLimit": 100,
|
||||
"iat": 1700000000
|
||||
}
|
||||
```
|
||||
- Store this token in your client's environment or configuration.
|
||||
|
||||
---
|
||||
|
||||
## 2. Managing Client Credentials & Lifecycle
|
||||
|
||||
### Rotating Client Tokens
|
||||
If a client secret or JWT token is compromised or needs routine rotation:
|
||||
1. Locate the client in the table and click **Rotate Token**.
|
||||
2. Confirm the prompt.
|
||||
3. DataStack generates a fresh client secret, signs a new JWT, and updates PostgreSQL.
|
||||
4. **Instant Disconnection**: Port 3000 calls `POST /internal/client-sync`. The Port 4000 ingestion server immediately severs all active WebSocket connections holding the old token. The client must reconnect using the newly generated token.
|
||||
|
||||
### Revoking Access
|
||||
To temporarily pause ingestion from a specific client without deleting its history:
|
||||
1. Click the status dropdown or action button and select **Revoke**.
|
||||
2. The client status changes to `revoked`.
|
||||
3. Port 4000 immediately rejects new handshakes and terminates existing connections with `CLIENT_REVOKED (403)`.
|
||||
4. HTTP requests to `POST /api/events` with this token are rejected with `403 Forbidden`.
|
||||
|
||||
### Restoring Access
|
||||
1. Select a revoked client and click **Activate**.
|
||||
2. Handshakes and HTTP ingestion are instantly permitted again.
|
||||
|
||||
### Deleting a Client
|
||||
When an application is permanently decommissioned:
|
||||
1. Click **Delete Client**.
|
||||
2. Choose whether to:
|
||||
- **Keep Historical Data**: Retains `/app/storage/data/{clientId}/` on disk for future analytics and export.
|
||||
- **Purge Historical Data**: Deletes the partition folder and all CSV files for this client permanently.
|
||||
3. **Tombstone Protection**: DataStack records the client ID and secret in the `deleted_clients` PostgreSQL table. Even if someone attempts to reuse an old signed JWT, the ingestion server checks the tombstone list and rejects the request.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Streaming Data Export (`/vcon/export`)
|
||||
|
||||
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Streaming Architecture
|
||||
|
||||
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
|
||||
|
||||
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
|
||||
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
|
||||
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
|
||||
|
||||
---
|
||||
|
||||
## 2. Export Configuration & Options
|
||||
|
||||
The export workspace provides targeted filtering:
|
||||
|
||||
| Option | Values | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
|
||||
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
|
||||
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
|
||||
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported Formats & Tool Compatibility
|
||||
|
||||
### CSV Format (`text/csv`)
|
||||
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
|
||||
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
|
||||
|
||||
#### Importing CSV into Python (pandas)
|
||||
```python
|
||||
import pandas as pd
|
||||
import json
|
||||
|
||||
# Load exported CSV
|
||||
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
|
||||
|
||||
# Parse JSON payloads into a structured DataFrame
|
||||
df["parsed_payload"] = df["payload"].apply(json.loads)
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
#### Importing CSV into DuckDB
|
||||
```sql
|
||||
-- Query directly from the exported CSV without importing
|
||||
SELECT
|
||||
clientId,
|
||||
timestamp,
|
||||
json_extract_string(payload, '$.temperature') AS temperature
|
||||
FROM read_csv_auto('datastack_export.csv')
|
||||
WHERE temperature > 25.0;
|
||||
```
|
||||
|
||||
### JSON Format (`application/json`)
|
||||
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
|
||||
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
|
||||
|
||||
---
|
||||
|
||||
## 4. Programmatic Export via API
|
||||
|
||||
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
|
||||
|
||||
```bash
|
||||
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
|
||||
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
|
||||
--output export.csv
|
||||
```
|
||||
@@ -0,0 +1,83 @@
|
||||
# Data Simulator (`/vcon/simulator`)
|
||||
|
||||
The DataStack Data Simulator (`/vcon/simulator`) enables operators and developers to validate ingestion pipelines, test rate limits, and inspect buffer flushes directly from the browser without setting up external producer scripts.
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview & Capabilities
|
||||
|
||||
The Simulator allows you to:
|
||||
- Test ingestion using either **HTTP REST** (`POST /api/events`) or **Socket.IO WebSocket** (`sync_data`).
|
||||
- Select any active client from your registered client list, or simulate unauthenticated/custom payloads.
|
||||
- Dispatch single events or high-volume bursts (from 10 to 500 events).
|
||||
- Select from realistic schema presets or construct custom JSON objects.
|
||||
- Observe response status, payload transmission size, and round-trip latency in real time.
|
||||
|
||||
---
|
||||
|
||||
## 2. Telemetry Presets
|
||||
|
||||
The simulator includes four industry-standard telemetry presets:
|
||||
|
||||
### 1. IoT Temperature Sensor
|
||||
```json
|
||||
{
|
||||
"sensorId": "temp-north-01",
|
||||
"temperature": 23.4,
|
||||
"humidity": 45.2,
|
||||
"battery": 98,
|
||||
"unit": "celsius"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. E-Commerce Order Checkout
|
||||
```json
|
||||
{
|
||||
"orderId": "ord_9872134",
|
||||
"customerId": "cust_11029",
|
||||
"currency": "USD",
|
||||
"totalAmount": 149.50,
|
||||
"itemCount": 3,
|
||||
"paymentMethod": "credit_card"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Server System Resource Metrics
|
||||
```json
|
||||
{
|
||||
"host": "prod-api-worker-03",
|
||||
"cpuUsagePercent": 34.2,
|
||||
"memoryUsedMb": 1024,
|
||||
"memoryTotalMb": 4096,
|
||||
"loadAverage": [0.45, 0.62, 0.58]
|
||||
}
|
||||
```
|
||||
|
||||
### 4. User Audit Trail & Security Event
|
||||
```json
|
||||
{
|
||||
"actorId": "usr_9912",
|
||||
"action": "client_token_rotated",
|
||||
"ipAddress": "192.168.1.100",
|
||||
"userAgent": "Mozilla/5.0",
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Simulator Workflows
|
||||
|
||||
### Testing Single Event Ingestion
|
||||
1. Select your target **Client** from the dropdown.
|
||||
2. Select **Single Event** mode.
|
||||
3. Choose a preset or customize the JSON editor.
|
||||
4. Click **Send Event**.
|
||||
5. The output console displays the assigned `eventId`, server response code (`200 OK`), and processing time.
|
||||
|
||||
### Testing High-Throughput Batch Flushing
|
||||
1. Select **Batch Dispatch** mode.
|
||||
2. Set the burst count (e.g., `500` events).
|
||||
3. Click **Start Batch**.
|
||||
4. Observe the batch progress bar.
|
||||
5. Navigate to `/vcon/dashboard` to verify that the buffer threshold flush automatically triggered, writing the batch to the client's daily CSV partition file on disk.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Features Guide
|
||||
|
||||
Welcome to the DataStack Feature Documentation. This section details the complete feature set of the DataStack high-throughput telemetry platform.
|
||||
|
||||
---
|
||||
|
||||
## Feature Index
|
||||
|
||||
| Feature Document | Purpose & Capabilities | Primary Navigation Path |
|
||||
| :--- | :--- | :--- |
|
||||
| [Admin Console (`admin-console.md`)](./admin-console.md) | System health monitoring, dual-runtime status, memory/buffer gauges, and administrator account management. | `/vcon/dashboard`, `/vcon/profile` |
|
||||
| [Client Management (`client-management.md`)](./client-management.md) | Client SDK registration, HMAC-SHA256 JWT generation, credential rotation, socket revocation, and tombstone deletion. | `/vcon/clients` |
|
||||
| [Data Simulator (`data-simulator.md`)](./data-simulator.md) | Browser-based event generation, single vs. burst testing, WebSocket/HTTP transport modes, and telemetry payload presets. | `/vcon/simulator` |
|
||||
| [Partitioned CSV Storage (`partitioned-storage.md`)](./partitioned-storage.md) | Daily client directory partitioning, 100MB rotation (`-part2.csv`), cooperative lock files, and automated retention pruning. | `/vcon/storage` |
|
||||
| [Data Export (`data-export.md`)](./data-export.md) | Streaming CSV/JSON extraction, date range filters, client filtering, Excel/pandas compatibility, and non-blocking reads. | `/vcon/export` |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
Every feature in DataStack follows a strict separation of concerns:
|
||||
- **Control Plane**: Handled on Port 3000 via Next.js 15 App Router and PostgreSQL metadata tables (`admin_users`, `client_sdks`, `admin_settings`).
|
||||
- **Data Plane**: Handled on Port 4000 via Express 5, Socket.IO 4, and persistent append-only CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
- **Zero SQL Insertions**: Telemetry events never create rows in relational tables, ensuring predictable query performance and infinite scale-to-zero storage efficiency.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Partitioned CSV Storage (`/vcon/storage`)
|
||||
|
||||
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
|
||||
|
||||
---
|
||||
|
||||
## 1. Storage Architecture & Directory Layout
|
||||
|
||||
Telemetry files are partitioned by client identifier and UTC date:
|
||||
|
||||
```text
|
||||
storage/
|
||||
└── data/
|
||||
├── pos_terminal_01/
|
||||
│ ├── 2026-11-16.csv
|
||||
│ ├── 2026-11-17.csv
|
||||
│ └── 2026-11-17-part2.csv
|
||||
└── mobile_app_prod/
|
||||
├── 2026-11-17.csv
|
||||
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
|
||||
```
|
||||
|
||||
### Partition File Format
|
||||
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
|
||||
|
||||
```csv
|
||||
clientId,eventId,timestamp,payload
|
||||
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
|
||||
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
|
||||
```
|
||||
|
||||
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
|
||||
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
|
||||
- **Newline**: Strict Unix standard `\n` line delimiters.
|
||||
|
||||
---
|
||||
|
||||
## 2. File Rotation Mechanism
|
||||
|
||||
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
|
||||
|
||||
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
|
||||
- **Rotation Behavior**:
|
||||
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
|
||||
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
|
||||
- Part 1: `2026-11-17.csv`
|
||||
- Part 2: `2026-11-17-part2.csv`
|
||||
- Part 3: `2026-11-17-part3.csv`
|
||||
3. Subsequent writes immediately target the active part file.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cooperative Locking Protocol
|
||||
|
||||
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
|
||||
|
||||
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
|
||||
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
|
||||
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
|
||||
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
|
||||
|
||||
---
|
||||
|
||||
## 4. Automated Retention Pruning
|
||||
|
||||
DataStack manages disk usage through an automated pruning engine:
|
||||
|
||||
- **Setting**: `autoPruneDays` (default: `30` days).
|
||||
- **Pruning Logic**:
|
||||
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
|
||||
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
|
||||
- If a client's folder becomes empty, the folder is safely removed.
|
||||
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
|
||||
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.
|
||||
Reference in new issue
Block a user