Files
DataStack/docs/users/features.md
Team Dark Devil 4279da45bb chore: establish cross-IDE synchronization protocol
Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
2026-11-18 07:11:11 +06:00

5.2 KiB

Feature Guide

This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (/vcon/).


1. The Administration Console (/vcon/)

The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.

Access the console by signing in at:

https://your-domain.example/vcon/login

Dashboard Overview (/vcon/dashboard)

The primary operational cockpit displays:

  • System Health: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
  • Storage Metrics: Total CSV partition file count, active client directories, and total disk usage.
  • Buffer Gauges: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
  • Live Activity Stream: Real-time ticker showing recently ingested events with timestamps and payload previews.

2. Client SDK Registry & Security (/vcon/clients)

DataStack enforces strict authentication for all incoming telemetry streams.

Registering a New Client

  1. Navigate to Clients (/vcon/clients) and click + Register Client.
  2. Provide:
    • Client Identifier: Unique lowercase alphanumeric slug (e.g., pos_terminal_01, mobile_app_prod).
    • Client Name: Human-readable label (e.g., Point of Sale Terminal #1).
    • Rate Limit: Maximum allowed events per second for this client (default: 100 events/sec).
  3. Click Register Client. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
  4. Copy the Token: Store the generated token securely in your client application's environment or configuration file.

Credential Rotation & Access Control

  • Rotate Token: Click Rotate Token to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
  • Revoke Access: Set the client status to Revoked. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with CLIENT_REVOKED (403).
  • Permanent Deletion: Deleting a client removes it from PostgreSQL and adds it to deleted_clients to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.

3. Data Simulator (/vcon/simulator)

The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:

  • Single Event: Send ad-hoc JSON payloads with custom event IDs.
  • Batch Generator: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
  • Preset Schemas: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.

4. Partitioned CSV Storage & Retention (/vcon/storage)

DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:

  • Directory Layout: /app/storage/data/{clientId}/{YYYY-MM-DD}.csv
  • CSV Schema: Each file contains 4 clean columns:
    clientId,eventId,timestamp,payload
    pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
    
  • Size-Based Rotation: When a partition reaches the configured threshold (default: 100 MB), DataStack automatically closes the file and continues writing to {YYYY-MM-DD}-part2.csv.
  • Automated Retention Pruning: Configure autoPruneDays (e.g., 30 days). DataStack automatically removes expired partition files, or set to 0 ("Keep all") to retain historical archives indefinitely.
  • Manual Buffer Flush: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.

5. Streaming Data Export (/vcon/export)

The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:

  • Client Filter: Export all clients or filter by a specific client ID.
  • Date Range: Filter records within precise UTC start and end dates.
  • Formats:
    • CSV: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
    • JSON: Formatted JSON array stream.
  • Non-Blocking Streaming: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.

6. Platform Runtime Settings (/vcon/settings)

Fine-tune ingestion rules with instant cross-process propagation:

  • Max Events Per Second: Global ingestion rate ceiling.
  • Max Payload Size (KB): Maximum permitted size per event payload (default: 100 KB).
  • Batch Flush Threshold: Number of buffered events that triggers an immediate write to disk (default: 500 events).
  • Batch Flush Interval: Maximum elapsed time before flushing pending events to disk (default: 2000 ms).
  • Idempotency Protection: Deduplicates events with identical eventId values within a 5-minute sliding window.
  • Connection Probing: Test the live internal bridge between the Web console and the Port 4000 ingestion service.