Files
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

78 lines
8.0 KiB
Markdown

# DataStack Documentation Hub
Welcome to the official, authoritative documentation for **DataStack** — a self-hosted, universal real-time telemetry ingestion engine, administrative governance platform (`/vcon/`), and partitioned CSV persistence system developed by **[Vib Tools](https://vib.tools/)**.
---
## Documentation Portals
DataStack documentation is organized into three specialized portals designed for different audiences:
```text
┌────────────────────────┐
│ DataStack Docs Hub │
│ docs/index.md │
└───────────┬────────────┘
│
┌────────────────────────────┼────────────────────────────┐
▼ ▼ ▼
┌───────────────────────────┐┌───────────────────────────┐┌───────────────────────────┐
│ User Portal ││ Developer Portal ││ Reference Portal │
│ docs/users/ ││ docs/dev/ ││ docs/reference/ │
├───────────────────────────┤├───────────────────────────┤├───────────────────────────┤
│ - Getting Started ││ - Architecture Blueprint ││ - Config Variable Bounds │
│ - Docker & Coolify Install││ - Codebase Structure Map ││ - Dependency Inventory │
│ - Feature Deep Dives ││ - REST & Socket.IO APIs ││ - Technical Glossary │
│ - End-to-End Workflows ││ - Database & DDL Schema ││ - Error Code Index │
│ - Troubleshooting & FAQ ││ - Security Architecture ││ │
└───────────────────────────┘└───────────────────────────┘└───────────────────────────┘
```
### 1. [End-User & Operator Portal (`docs/users/`)](./users/index.md)
Designed for system administrators, DevOps engineers, and non-technical platform operators:
- **[Getting Started](./users/getting-started.md)**: Zero-to-one walkthrough: deploying DataStack, setting up the administrator account, and dispatching your first telemetry event.
- **[Installation Guide](./users/installation.md)**: Production deployment via Docker Compose and Coolify, persistent volume layouts, and reverse proxy networking.
- **[Configuration Guide](./users/configuration.md)**: Environment variable reference, security secrets setup, and runtime limit tuning.
- **[Feature Guides](./users/features/index.md)**: Detailed guides on the [Admin Console](./users/features/admin-console.md), [Client Token Management](./users/features/client-management.md), [Data Simulator](./users/features/data-simulator.md), [Partitioned Storage](./users/features/partitioned-storage.md), and [Streaming Data Export](./users/features/data-export.md).
- **[Operational Workflows](./users/workflows.md)**: Step-by-step operator workflows for token rotation, client onboarding, data exports, and manual buffer flushes.
- **[Troubleshooting & Diagnostics](./users/troubleshooting.md)**: Forensic troubleshooting procedures for connection drops, rate-limiting (`429`), payload rejections (`413`), and lock recoveries.
- **[Frequently Asked Questions](./users/faq.md)**: In-depth answers regarding CSV storage philosophy, Redis AOF durability, and container updates.
### 2. [Developer & Contributor Portal (`docs/dev/`)](./dev/index.md)
Designed for software engineers, backend architects, and open-source contributors:
- **[Architecture Blueprint](./dev/architecture.md)**: Deep dive into the decoupled dual-runtime model (Port 3000 Web vs Port 4000 Ingestion), the authenticated `/internal/*` service bridge, Redis durability, and cooperative partition file locking.
- **[Codebase Structure](./dev/codebase-structure.md)**: Complete file-by-file inventory of directories, source modules, and UI primitives.
- **[Local Development Setup](./dev/local-setup.md)**: Step-by-step local workstation configuration with dual-terminal development (`npm run dev` + `npm run server`).
- **[API Reference](./dev/api-reference.md)**: Exhaustive REST API specification, headers, request/response JSON schemas, and Socket.IO protocol specification (`sync_data`, `batch_sync`, `ping_verify`).
- **[Database Schema & Models](./dev/database-schema.md)**: PostgreSQL metadata schema definitions (`prisma/schema.prisma`), raw `pg.Pool` execution, and non-destructive `ADD COLUMN IF NOT EXISTS` migrations.
- **[Security Architecture](./dev/security.md)**: Forensic breakdown of password hashing (`scrypt`), WebCrypto HMAC-SHA256 session cookies, brute-force IP lockout, CSRF Origin validation, and client JWT issuance.
- **[Building & Packaging](./dev/building-packaging.md)**: Next.js standalone compilation (`.next/standalone`), multi-stage Docker builds, and Nginx proxy containerization.
- **[Testing Suites](./dev/testing.md)**: Running the automated TypeScript integration suites (`node:test` via `tsx`) and Python SDK unit tests.
- **[Contributing Guidelines](./dev/contributing.md)**: Architectural guardrails, single-source-of-truth UI standards, and the universal multi-IDE persistent memory synchronization rule (`MEMORY.md` & `.ai-rules`).
### 3. [Technical Reference Portal (`docs/reference/`)](./reference/index.md)
Authoritative specification tables and indexes:
- **[Configuration Reference](./reference/configuration-reference.md)**: Complete parameter reference for `.env` variables and runtime `/vcon/settings` bounds.
- **[Dependency Inventory](./reference/dependencies.md)**: Comprehensive audit of all runtime, dev, and optional dependencies.
- **[Technical Glossary](./reference/glossary.md)**: Definitions for platform concepts (partitioned CSV, distributed flush lock, cooperative lock, idempotency key, etc.).
- **[Audit & Coverage Report](./audit-coverage-report.md)**: Forensic codebase audit verification matrix and testing coverage summary.
---
## System Overview at a Glance
| Component | Port | Technology Stack | Primary Purpose |
| :--- | :--- | :--- | :--- |
| **`proxy`** | `80` / `443` | Nginx Alpine | Public ingress routing `/socket.io/` and `/health` to ingestion server, and all other traffic to Next.js. |
| **`web`** | `3000` | Next.js 15, React 19, TypeScript | Public overview, `/vcon/*` admin console, authenticated REST ingestion (`POST /api/events`), and streaming exports (`GET /api/export`). |
| **`ingestion`** | `4000` | Express 5, Socket.IO 4, Node 22 | High-throughput WebSocket ingestion, JWT client verification, per-client sliding-window rate limiting, and Redis queue buffering. |
| **`redis`** | `6379` | Redis 7 Alpine (`--appendonly yes`) | Shared queue (`datastack:buffer:events`), recent event cache, and distributed flush coordinator (`datastack:flush:lock`). |
| **`postgres`**| `5432` | PostgreSQL 16 Alpine | Control-plane storage for admin accounts (`admin_users`), brute-force login attempts (`admin_login_attempts`), and client credentials (`client_sdks`). |
| **`storage`** | Local Mount | Partitioned RFC 4180 CSV | Telemetry persistence root: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. Zero telemetry is inserted into PostgreSQL. |
---
## Governance & Master Manifest
For documentation authoring rules, directory maintenance guidelines, and cross-IDE sync instructions, see the **[Documentation Manifest (`docs/MANIFEST.md`)](./MANIFEST.md)**.