Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
90 lines
5.2 KiB
Markdown
90 lines
5.2 KiB
Markdown
# Feature Guide
|
|
|
|
This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (`/vcon/`).
|
|
|
|
---
|
|
|
|
## 1. The Administration Console (`/vcon/`)
|
|
|
|
The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.
|
|
|
|
Access the console by signing in at:
|
|
```text
|
|
https://your-domain.example/vcon/login
|
|
```
|
|
|
|
### Dashboard Overview (`/vcon/dashboard`)
|
|
The primary operational cockpit displays:
|
|
- **System Health**: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
|
|
- **Storage Metrics**: Total CSV partition file count, active client directories, and total disk usage.
|
|
- **Buffer Gauges**: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
|
|
- **Live Activity Stream**: Real-time ticker showing recently ingested events with timestamps and payload previews.
|
|
|
|
---
|
|
|
|
## 2. Client SDK Registry & Security (`/vcon/clients`)
|
|
|
|
DataStack enforces strict authentication for all incoming telemetry streams.
|
|
|
|
### Registering a New Client
|
|
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
|
|
2. Provide:
|
|
- **Client Identifier**: Unique lowercase alphanumeric slug (e.g., `pos_terminal_01`, `mobile_app_prod`).
|
|
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
|
|
- **Rate Limit**: Maximum allowed events per second for this client (default: `100 events/sec`).
|
|
3. Click **Register Client**. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
|
|
4. **Copy the Token**: Store the generated token securely in your client application's environment or configuration file.
|
|
|
|
### Credential Rotation & Access Control
|
|
- **Rotate Token**: Click **Rotate Token** to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
|
|
- **Revoke Access**: Set the client status to `Revoked`. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with `CLIENT_REVOKED (403)`.
|
|
- **Permanent Deletion**: Deleting a client removes it from PostgreSQL and adds it to `deleted_clients` to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.
|
|
|
|
---
|
|
|
|
## 3. Data Simulator (`/vcon/simulator`)
|
|
|
|
The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:
|
|
- **Single Event**: Send ad-hoc JSON payloads with custom event IDs.
|
|
- **Batch Generator**: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
|
|
- **Preset Schemas**: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.
|
|
|
|
---
|
|
|
|
## 4. Partitioned CSV Storage & Retention (`/vcon/storage`)
|
|
|
|
DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:
|
|
- **Directory Layout**: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
|
- **CSV Schema**: Each file contains 4 clean columns:
|
|
```csv
|
|
clientId,eventId,timestamp,payload
|
|
pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
|
|
```
|
|
- **Size-Based Rotation**: When a partition reaches the configured threshold (default: `100 MB`), DataStack automatically closes the file and continues writing to `{YYYY-MM-DD}-part2.csv`.
|
|
- **Automated Retention Pruning**: Configure `autoPruneDays` (e.g., `30` days). DataStack automatically removes expired partition files, or set to `0` ("Keep all") to retain historical archives indefinitely.
|
|
- **Manual Buffer Flush**: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.
|
|
|
|
---
|
|
|
|
## 5. Streaming Data Export (`/vcon/export`)
|
|
|
|
The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:
|
|
- **Client Filter**: Export all clients or filter by a specific client ID.
|
|
- **Date Range**: Filter records within precise UTC start and end dates.
|
|
- **Formats**:
|
|
- **CSV**: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
|
|
- **JSON**: Formatted JSON array stream.
|
|
- **Non-Blocking Streaming**: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.
|
|
|
|
---
|
|
|
|
## 6. Platform Runtime Settings (`/vcon/settings`)
|
|
|
|
Fine-tune ingestion rules with instant cross-process propagation:
|
|
- **Max Events Per Second**: Global ingestion rate ceiling.
|
|
- **Max Payload Size (KB)**: Maximum permitted size per event payload (default: `100 KB`).
|
|
- **Batch Flush Threshold**: Number of buffered events that triggers an immediate write to disk (default: `500 events`).
|
|
- **Batch Flush Interval**: Maximum elapsed time before flushing pending events to disk (default: `2000 ms`).
|
|
- **Idempotency Protection**: Deduplicates events with identical `eventId` values within a 5-minute sliding window.
|
|
- **Connection Probing**: Test the live internal bridge between the Web console and the Port 4000 ingestion service.
|