- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
8.0 KiB
8.0 KiB
DataStack Documentation Hub
Welcome to the official, authoritative documentation for DataStack — a self-hosted, universal real-time telemetry ingestion engine, administrative governance platform (/vcon/), and partitioned CSV persistence system developed by Vib Tools.
Documentation Portals
DataStack documentation is organized into three specialized portals designed for different audiences:
┌────────────────────────┐
│ DataStack Docs Hub │
│ docs/index.md │
└───────────┬────────────┘
│
┌────────────────────────────┼────────────────────────────┐
▼ ▼ ▼
┌───────────────────────────┐┌───────────────────────────┐┌───────────────────────────┐
│ User Portal ││ Developer Portal ││ Reference Portal │
│ docs/users/ ││ docs/dev/ ││ docs/reference/ │
├───────────────────────────┤├───────────────────────────┤├───────────────────────────┤
│ - Getting Started ││ - Architecture Blueprint ││ - Config Variable Bounds │
│ - Docker & Coolify Install││ - Codebase Structure Map ││ - Dependency Inventory │
│ - Feature Deep Dives ││ - REST & Socket.IO APIs ││ - Technical Glossary │
│ - End-to-End Workflows ││ - Database & DDL Schema ││ - Error Code Index │
│ - Troubleshooting & FAQ ││ - Security Architecture ││ │
└───────────────────────────┘└───────────────────────────┘└───────────────────────────┘
1. End-User & Operator Portal (docs/users/)
Designed for system administrators, DevOps engineers, and non-technical platform operators:
- Getting Started: Zero-to-one walkthrough: deploying DataStack, setting up the administrator account, and dispatching your first telemetry event.
- Installation Guide: Production deployment via Docker Compose and Coolify, persistent volume layouts, and reverse proxy networking.
- Configuration Guide: Environment variable reference, security secrets setup, and runtime limit tuning.
- Feature Guides: Detailed guides on the Admin Console, Client Token Management, Data Simulator, Partitioned Storage, and Streaming Data Export.
- Operational Workflows: Step-by-step operator workflows for token rotation, client onboarding, data exports, and manual buffer flushes.
- Troubleshooting & Diagnostics: Forensic troubleshooting procedures for connection drops, rate-limiting (
429), payload rejections (413), and lock recoveries. - Frequently Asked Questions: In-depth answers regarding CSV storage philosophy, Redis AOF durability, and container updates.
2. Developer & Contributor Portal (docs/dev/)
Designed for software engineers, backend architects, and open-source contributors:
- Architecture Blueprint: Deep dive into the decoupled dual-runtime model (Port 3000 Web vs Port 4000 Ingestion), the authenticated
/internal/*service bridge, Redis durability, and cooperative partition file locking. - Codebase Structure: Complete file-by-file inventory of directories, source modules, and UI primitives.
- Local Development Setup: Step-by-step local workstation configuration with dual-terminal development (
npm run dev+npm run server). - API Reference: Exhaustive REST API specification, headers, request/response JSON schemas, and Socket.IO protocol specification (
sync_data,batch_sync,ping_verify). - Database Schema & Models: PostgreSQL metadata schema definitions (
prisma/schema.prisma), rawpg.Poolexecution, and non-destructiveADD COLUMN IF NOT EXISTSmigrations. - Security Architecture: Forensic breakdown of password hashing (
scrypt), WebCrypto HMAC-SHA256 session cookies, brute-force IP lockout, CSRF Origin validation, and client JWT issuance. - Building & Packaging: Next.js standalone compilation (
.next/standalone), multi-stage Docker builds, and Nginx proxy containerization. - Testing Suites: Running the automated TypeScript integration suites (
node:testviatsx) and Python SDK unit tests. - Contributing Guidelines: Architectural guardrails, single-source-of-truth UI standards, and the universal multi-IDE persistent memory synchronization rule (
MEMORY.md&.ai-rules).
3. Technical Reference Portal (docs/reference/)
Authoritative specification tables and indexes:
- Configuration Reference: Complete parameter reference for
.envvariables and runtime/vcon/settingsbounds. - Dependency Inventory: Comprehensive audit of all runtime, dev, and optional dependencies.
- Technical Glossary: Definitions for platform concepts (partitioned CSV, distributed flush lock, cooperative lock, idempotency key, etc.).
- Audit & Coverage Report: Forensic codebase audit verification matrix and testing coverage summary.
System Overview at a Glance
| Component | Port | Technology Stack | Primary Purpose |
|---|---|---|---|
proxy |
80 / 443 |
Nginx Alpine | Public ingress routing /socket.io/ and /health to ingestion server, and all other traffic to Next.js. |
web |
3000 |
Next.js 15, React 19, TypeScript | Public overview, /vcon/* admin console, authenticated REST ingestion (POST /api/events), and streaming exports (GET /api/export). |
ingestion |
4000 |
Express 5, Socket.IO 4, Node 22 | High-throughput WebSocket ingestion, JWT client verification, per-client sliding-window rate limiting, and Redis queue buffering. |
redis |
6379 |
Redis 7 Alpine (--appendonly yes) |
Shared queue (datastack:buffer:events), recent event cache, and distributed flush coordinator (datastack:flush:lock). |
postgres |
5432 |
PostgreSQL 16 Alpine | Control-plane storage for admin accounts (admin_users), brute-force login attempts (admin_login_attempts), and client credentials (client_sdks). |
storage |
Local Mount | Partitioned RFC 4180 CSV | Telemetry persistence root: /app/storage/data/{clientId}/{YYYY-MM-DD}.csv. Zero telemetry is inserted into PostgreSQL. |
Governance & Master Manifest
For documentation authoring rules, directory maintenance guidelines, and cross-IDE sync instructions, see the Documentation Manifest (docs/MANIFEST.md).