chore: establish cross-IDE synchronization protocol

Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
This commit is contained in:
Team Dark Devil committed 2026-11-18 07:11:11 +06:00
1 parent 05407fd338
commit 4279da45bb
14 files changed
+1162

No files matched your search

+76
View File
@@ -0,0 +1,76 @@
# Documentation Manifest (`docs/MANIFEST.md`)
> **Location**: `/docs/MANIFEST.md`
> **Audience**: Contributors, Maintainers, and Documentation Authors
> **Last Updated**: 2026-11-17
---
## 1. Directory Purpose & Structure
The `docs/` directory contains the complete official documentation for **DataStack**, structured into distinct audiences to provide clarity, depth, and ease of navigation:
```text
docs/
├── MANIFEST.md # This document (directory index & governance)
├── users/ # End-User & Operator Documentation
│ ├── README.md # User documentation overview & quick links
│ ├── installation.md # Self-hosting, Docker, and Coolify setup guides
│ ├── features.md # Guide to /vcon/ console, client tokens, and telemetry
│ ├── tutorials.md # Step-by-step user tutorials and integration walkthroughs
│ └── faq.md # Frequently asked questions and troubleshooting
└── dev/ # Developer & Contributor Documentation
├── README.md # Developer guide overview & roadmap
├── architecture.md # Deep dive into dual-runtime architecture and storage
├── setup.md # Local development environment setup & configuration
├── build.md # Build, compilation, Docker containerization & test suites
└── contributing.md # Contribution rules, conventions, and workflow
```
---
## 2. Directory Breakdown
### `docs/users/` (End-Users, System Operators & Administrators)
Designed for non-technical users, platform administrators, and data engineers operating or integrating with a live DataStack deployment:
| File | Purpose | Key Topics Covered |
| :--- | :--- | :--- |
| `README.md` | Entry point for users | Documentation map, navigation guide, and core concepts overview. |
| `installation.md` | Deployment guide | Production Docker Compose, Coolify 1-click deployment, environment variables, reverse proxy setup, and persistent volume management. |
| `features.md` | Feature breakdown | Web console (`/vcon/`), client SDK tokens, real-time event simulation, daily partitioned CSV storage, streaming export, and platform settings. |
| `tutorials.md` | Practical guides | Step-by-step tutorials: first telemetry event, connecting Python applications, viewing live events in the console, and exporting datasets. |
| `faq.md` | Troubleshooting & FAQ | Common questions on data retention, token revocation, port configurations, CSV rotation, and connection diagnostics. |
### `docs/dev/` (Developers, Contributors & System Architects)
Designed for engineers extending the codebase, fixing bugs, writing client SDKs, or contributing to the core platform:
| File | Purpose | Key Topics Covered |
| :--- | :--- | :--- |
| `README.md` | Entry point for devs | Contributor handbook, repository layout, and code navigation. |
| `architecture.md` | Architectural blueprint | Dual-runtime model (Port 3000 Web vs Port 4000 Ingestion), Redis durability queue, partitioned CSV engine with cooperative locks, and PostgreSQL metadata store. |
| `setup.md` | Local development | Prerequisites (Node.js 22, PostgreSQL, Redis), dual-terminal development, environment secrets, and sample configurations. |
| `build.md` | Build & compilation | Next.js standalone compilation (`npm run build`), ESLint validation, Docker multi-stage builds, and running TypeScript (`npx tsx --test`) and Python unit test suites. |
| `contributing.md` | Contribution standards | Multi-IDE memory synchronization (`MEMORY.md` & `.ai-rules`), UI component governance (`components/ui/`), code conventions, and Pull Request checklist. |
---
## 3. Maintenance & Documentation Update Guidelines
Documentation accuracy is a core quality requirement in DataStack. Whenever modifying code or configurations, follow these update rules:
### When to Update `docs/users/`
- **New User-Facing Features**: Whenever adding or altering pages in `/vcon/*` (e.g., changes to export options, client token regeneration, or UI settings), update `docs/users/features.md`.
- **Installation / Deployment Changes**: Whenever modifying `docker/docker-compose.yml`, Dockerfiles, Nginx configurations, or `.env.example`, update `docs/users/installation.md`.
- **API Endpoints & Integration**: If new parameters are added to `POST /api/events` or `GET /api/export`, update `docs/users/tutorials.md` and `docs/users/faq.md`.
### When to Update `docs/dev/`
- **Architectural Changes**: Whenever altering the communication bridge between Port 3000 and Port 4000, changing Redis queues, or altering CSV storage mechanisms, update `docs/dev/architecture.md`.
- **New Dependencies or Scripts**: Whenever adding npm dependencies, changing build targets, or adding test files, update `docs/dev/setup.md` and `docs/dev/build.md`.
- **Contribution Standards**: If code standards, linting rules, or multi-IDE memory rules are revised, update `docs/dev/contributing.md` and sync with `AGENTS.md` and `.ai-rules`.
### Quality & Style Standards
1. **Markdown Formatting**: Use clean GitHub-Flavored Markdown (GFM) with clear headings, bulleted lists, and language-tagged code blocks (e.g. ` ```bash `, ` ```python `).
2. **Absolute vs Relative Links**: Inside markdown files, use relative links to reference other documentation files (e.g., `[Architecture Overview](../dev/architecture.md)`).
3. **No Stale Placeholders**: Avoid unverified configuration keys or generic placeholder URLs; align variables with `.env.example`.
4. **Cross-IDE Sync**: After any documentation update, update `MEMORY.md` in the root directory to maintain synchronization across all IDEs.
+58
View File
@@ -0,0 +1,58 @@
# Developer & Contributor Guide
Welcome to the **DataStack Developer Documentation**. This section contains in-depth architectural specifications, development workflows, compilation instructions, and contribution conventions for engineers working on the DataStack codebase.
---
## Developer Documentation Index
Explore the core technical guides:
### 1. [Architecture Blueprint](./architecture.md)
Detailed breakdown of DataStack's decoupled dual-runtime model:
- **Port 3000 (`web`)**: Next.js 15 App Router, React 19, serverless REST API, streaming CSV/JSON export.
- **Port 4000 (`ingestion`)**: Dedicated Express 5 + Socket.IO real-time ingestion server.
- **Internal Service Bridge**: Authenticated `/internal/*` communication via `DATASTACK_INTERNAL_SECRET`.
- **Storage Subsystem**: In-memory and Redis-backed buffering, cooperative partition file locking, and streaming row counters.
- **PostgreSQL Control-Plane**: Schema provisioning and metadata storage rules (`pg.Pool` + `prisma/schema.prisma`).
### 2. [Local Development Setup](./setup.md)
Instructions for configuring your local machine:
- Prerequisites (Node.js 22, npm, PostgreSQL 16, Redis 7).
- Environment variable configuration (`.env.example`).
- Dual-terminal local workflow (`npm run dev` + `npm run server`).
- Database schema initialization.
### 3. [Build, Compilation & Testing](./build.md)
Standard build and testing commands:
- Compiling Next.js standalone output (`npm run build`).
- Code quality and linting (`npm run lint`).
- Running the comprehensive TypeScript test suite with `node:test` and `tsx`.
- Executing Python SDK unit tests.
- Docker multi-stage container builds.
### 4. [Contribution Rules & Conventions](./contributing.md)
Guidelines for submitting pull requests and modifying code:
- **Universal Multi-IDE Memory Rule** (`MEMORY.md` & `.ai-rules`).
- Single-source-of-truth UI primitives in `components/ui/` (`0rem` radius, zero shadows).
- Architectural guardrails (zero telemetry rows in PostgreSQL, no Socket.IO in Next.js).
- Pull Request and code review checklist.
---
## High-Level Repository Layout
```text
datastack/
├── app/ # Next.js 15 App Router (Frontend, /vcon/*, and /api/*)
├── components/ # UI primitives (ui/), layout shells (layout/), and sections (sections/)
├── core/ # Backend TypeScript modules (auth, Database, data, Storage, Security, sdk)
├── server/ # Dedicated Port 4000 Express + Socket.IO service & CSV batch worker
├── sdk/python/datastack/ # Official pip-installable Python SDK
├── sdk-python/ # Backward-compatible Python SDK wrapper
├── storage/data/ # Partitioned CSV telemetry storage root
├── docker/ # Production Dockerfiles, Nginx proxy configs, and docker-compose.yml
├── nginx/ # Local Nginx development configuration
├── tests/ # Automated test suites (phase01 - phase07)
└── docs/ # User and developer documentation
```
+105
View File
@@ -0,0 +1,105 @@
# Architecture Blueprint
This document details the architectural principles, runtime boundaries, and subsystems powering DataStack.
---
## 1. Core Architectural Principles
DataStack is designed around three foundational engineering constraints:
1. **Decoupled Runtimes**: Long-lived WebSocket connections require stateful, persistent event loops, while modern administrative consoles benefit from serverless, SSR App Router paradigms. DataStack strictly decouples these into **Port 3000** (Web & REST API) and **Port 4000** (Real-Time Ingestion).
2. **Zero Per-Row Database Writes**: Telemetry streams are never inserted per-row into relational tables. Events are buffered in memory and Redis, then flushed in sequential batches directly to daily partitioned CSV files on disk.
3. **Control-Plane vs Data-Plane Isolation**: PostgreSQL is strictly reserved for the control plane (user authentication, client token registry, rate limits, and platform configuration). The data plane resides entirely on disk in partitioned CSV files.
---
## 2. Dual-Runtime Architecture
```text
┌────────────────────────────────────────────────────────┐
│ Nginx Reverse Proxy (Port 80) │
└───────────────┬────────────────────────┬───────────────┘
│ │
/socket.io/ │ │ / & /api/*
▼ ▼
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
│ Ingestion Service (Port 4000) │ │ Web & Admin Console (Port 3000) │
│ Express 5 + Socket.IO 4 │ │ Next.js 15 App Router + React 19 │
├───────────────────────────────────┤ ├───────────────────────────────────┤
│ - WebSocket Handshake Auth (JWT) │ │ - Public Landing & PWA Lifecycle │
│ - Real-time Rate Limiting │ │ - /vcon/* Admin Console │
│ - 5-min Sliding Window Idempotency│ │ - Authenticated HTTP Ingestion │
│ - Memory & Redis Batch Buffer │ │ - Streaming CSV / JSON Export │
│ - CsvWriterWorker Flusher │ │ - Session Auth & Brute-Force Guard│
└─────────────────┬─────────────────┘ └─────────────────┬─────────────────┘
│ │
│ Authenticated /internal/* Bridge │
└────────────────◄═══════════════════════┘
(DATASTACK_INTERNAL_SECRET)
```
### Port 3000 (`web`): Next.js 15 App Router
- **Role**: Serves the user interface and HTTP REST endpoints.
- **Key Modules**:
- `middleware.ts`: Authenticates `/vcon/*` routes using signed WebCrypto session cookies, and enforces CSRF protection by validating `Origin`/`Referer` headers against `Host` and `X-Forwarded-Host`.
- `app/api/events/route.ts`: Accepts HTTP ingestion for single events, arrays (`{ batch: [...] }`), or structured file imports (`{ fileName, records: [...] }`).
- `app/api/export/route.ts`: Streams filtered CSV or JSON files directly from disk partitions to the client browser without buffering entire datasets in server RAM.
### Port 4000 (`ingestion`): Dedicated Ingestion Server
- **Role**: High-throughput Socket.IO WebSocket server (`server/index.js`).
- **Key Modules**:
- `server/socket/handlers.js`: Enforces client JWT authentication, sliding-window rate limits (per-client and global), payload size validation, and idempotency deduplication.
- `server/workers/csvWriter.js`: Flushes buffered batches to daily partitioned CSV files with file rotation and cooperative locking.
---
## 3. Internal Service Bridge (`/internal/*`)
Because Port 3000 and Port 4000 operate as separate processes, administrative actions in `/vcon/` must synchronize with the real-time ingestion service instantly:
- **Security**: All `/internal/*` routes require a shared, timing-safe authentication header: `x-datastack-internal-secret`.
- **Secret Derivation**: `DATASTACK_INTERNAL_SECRET` can be explicitly defined in `.env` or is automatically derived from the active JWT secret via SHA-256.
### Bridge Endpoints:
1. `GET /internal/metrics`: Aggregates active Socket.IO connections, buffer depths, and security stats for the `/vcon/dashboard`.
2. `POST /internal/settings-sync`: Pushes updated rate limits, payload sizes, flush thresholds, or rotated JWT secrets instantly to Port 4000.
3. `POST /internal/client-sync`: Pushes registered, rotated, revoked, or deleted client updates. If a client is revoked, Port 4000 immediately kicks all active sockets for that client.
4. `POST /internal/flush`: Triggers an immediate write of Port 4000's buffer to disk when an admin initiates a manual flush or export.
5. `GET /internal/recent`: Fetches recently buffered events from Port 4000's queue so `/vcon/data` displays zero-latency activity.
---
## 4. Storage Subsystem & Durability
### Data Ingestion Flow:
1. **Push**: Events enter via WebSocket (`sync_data`) or HTTP (`POST /api/events`).
2. **Buffer Queue**: Events are written to the Redis queue (`datastack:buffer:events`) with local RAM fallback.
3. **Batch Flush Condition**: A flush triggers when:
- Buffer size reaches `batchFlushThreshold` (default: 500 events), OR
- Time elapsed reaches `batchFlushIntervalMs` (default: 2000 ms), OR
- A manual flush or export is requested.
4. **Distributed Flush Lock**: In multi-process deployments, workers acquire `datastack:flush:lock` in Redis (`SET lockToken PX 15000 NX`) to prevent concurrent writes.
5. **Safe Trim**: Redis queue items are trimmed (`LTRIM`) **strictly after** the batch append operation confirms success on disk.
### Cooperative Partition File Locking
When writing to `/storage/data/{clientId}/{YYYY-MM-DD}.csv`:
- The writer creates a cooperative `.lock` file alongside the partition (`{file}.lock`).
- Stale locks (older than 10 seconds) created by crashed processes are automatically cleared.
- This prevents race conditions when both Port 3000 and Port 4000 flush to the same client partition simultaneously.
---
## 5. PostgreSQL Metadata Subsystem
PostgreSQL persistence is managed via raw `pg.Pool` connections (`core/Database/database.ts`), mirrored by the reference schema in `prisma/schema.prisma`.
### Managed Tables:
1. `admin_users`: Stores admin credentials with `crypto.scrypt` password hashing and a `token_version` column for instant session invalidation.
2. `admin_login_attempts`: Tracks failed login attempts per composite IP + Email hash with 15-minute brute-force lockout.
3. `client_sdks`: Stores registered client metadata, hashed secrets, signed tokens, rate limits, status (`active`/`revoked`), and total event counters.
4. `deleted_clients`: Tombstone table ensuring deleted client IDs and tokens can never be revived or accepted.
5. `admin_settings`: Single-row configuration storing platform limits, ports, thresholds, and secrets.
### Non-Destructive Migrations
On startup, `DatabaseEngine.ensureSchema()` runs non-destructive `CREATE TABLE IF NOT EXISTS` and additive `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` DDL statements, guaranteeing seamless upgrades on existing production databases without data loss.
+94
View File
@@ -0,0 +1,94 @@
# Build, Compilation & Testing Guide
This guide outlines compilation commands, linter checks, test suite execution, and Docker container packaging.
---
## 1. Production Web Compilation
DataStack uses Next.js 15 App Router configured for **Standalone Output** (`output: "standalone"` in `next.config.ts`), which bundles only the required Node.js runtime files into `.next/standalone`.
### Compiling the Application:
```bash
npm run build
```
### Build Verification Checklist:
- Compiles TypeScript without type errors (`tsc --noEmit`).
- Generates serverless routes and static assets under `.next/`.
- Prepares standalone bundle under `.next/standalone/server.js`.
---
## 2. Code Quality & Linting
DataStack uses ESLint 9 with `eslint-config-next`:
```bash
npm run lint
```
Ensure this command exits cleanly (`code: 0`) with zero warnings or errors prior to committing code or opening a pull request.
---
## 3. Automated Test Suites
DataStack includes comprehensive, fast-executing automated test suites verifying security, synchronization, storage durability, PWA compliance, and deployment health.
### Running TypeScript Integration & Unit Tests
Run the test runner using `npx --yes tsx --test`:
```bash
# Run all core test suites
npx --yes tsx --test tests/phase01-auth.test.ts \
tests/phase02-sync.test.ts \
tests/phase03-storage-buffer.test.ts \
tests/phase06-pwa-landing.test.ts \
tests/phase07-deployment.test.ts
```
#### What These Tests Verify:
- **`phase01-auth.test.ts`**: Secret fail-closed checks, `tokenVersion` session invalidation, brute-force IP throttling, client token redaction, and CSRF Origin validation.
- **`phase02-sync.test.ts`**: Port 3000 ↔ Port 4000 internal secret authentication, settings propagation, instant client revocation, and JWT rotation disconnection.
- **`phase03-storage-buffer.test.ts`**: Redis AOF durability, distributed flush locking, cooperative partition `.lock` files, streaming row counting, and filtered CSV export.
- **`phase06-pwa-landing.test.ts`**: Web App Manifest validity, Service Worker registration, and navigation integrity.
- **`phase07-deployment.test.ts`**: Docker Compose configuration, Redis AOF persistence, Nginx `/health` proxying, deterministic lockfile parity, startup ownership checks, and unhydrated-DB auto-prune guards.
### Running Python SDK Unit Tests
```bash
python3 -m unittest tests/phase05_sdk_test.py
```
Verifies Python SDK WebSocket streaming, Heartbeat HTTP fallback, disk spooling (`.datastack_spool/`), and `.csv`/`.json` file streaming.
---
## 4. Docker Container Builds
To test production containerization locally:
### 1. Build the Web Container (Port 3000)
```bash
docker build -f docker/Dockerfile.web -t datastack-web:latest .
```
### 2. Build the Ingestion Container (Port 4000)
```bash
docker build -f docker/Dockerfile.server -t datastack-ingestion:latest .
```
### 3. Build the Nginx Proxy Container (Port 80)
```bash
docker build -f docker/Dockerfile.proxy -t datastack-proxy:latest .
```
### 4. Build & Run the Entire Stack with Compose
```bash
docker compose -f docker/docker-compose.yml build
docker compose -f docker/docker-compose.yml up -d
```
Verify container health with:
```bash
docker compose -f docker/docker-compose.yml ps
```
All containers (`web`, `ingestion`, `proxy`, `postgres`, `redis`) should display `(healthy)`.
+64
View File
@@ -0,0 +1,64 @@
# Contribution Rules & Guidelines
Thank you for contributing to **DataStack**! To maintain architectural purity, production reliability, and seamless collaboration across multiple development environments, all contributors must adhere to these guidelines.
---
## 1. Universal Multi-IDE Persistent Memory (`MEMORY.md`)
> **CRITICAL RULE**: DataStack is developed across multiple IDEs and AI toolchains (such as VS Code, Google Antigravity, Cursor, and terminal environments). Whenever any modification, update, or bug fix is completed in your session, you must automatically update the `MEMORY.md` file (or provide the exact markdown blocks) so the project context remains perfectly synchronized across all IDEs (see `.ai-rules` and `AGENTS.md`).
When updating `MEMORY.md`:
- Note any new files, directories, routes, or modules added to the repository.
- Record any updated environment variables, runtime parameters, or architectural adjustments.
- Update the status of active features or resolved bugs.
---
## 2. Architectural Guardrails (Strict Enforcement)
1. **Dual-Runtime Separation**:
- Next.js 15 App Router (`app/`) runs on **Port 3000**.
- Express 5 + Socket.IO (`server/index.js`) runs on **Port 4000**.
- **Never** initialize Socket.IO server instances inside Next.js serverless route handlers.
2. **Zero Telemetry Writes to PostgreSQL**:
- Incoming telemetry events **must never** be inserted as rows into PostgreSQL.
- Events are buffered in memory and Redis, then appended to daily CSV partitions in `/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
- PostgreSQL is strictly reserved for `admin_users`, `admin_login_attempts`, `client_sdks`, `deleted_clients`, and `admin_settings`.
3. **Non-Destructive Database Migrations**:
- Never write code that executes `DROP TABLE`, `DROP COLUMN`, or `TRUNCATE` against production databases.
- All schema upgrades must use defensive `CREATE TABLE IF NOT EXISTS` and `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` statements.
4. **Cooperative CSV Locking**:
- All disk write operations must acquire cooperative file locks (`.lock`) with automated stale-lock recovery to prevent dual-writer corruption.
---
## 3. UI Component & Design System Standards
- **Single Source of Truth**: All UI primitives must reside in `components/ui/` (`Button`, `Input`, `Modal`, `Table`, `Panel`, `StatCard`, etc.) and layout shells in `components/layout/`. Do not introduce ad-hoc isolated CSS or duplicate button components.
- **Design Tokens**:
- Border radius: `--radius: 0rem` across all components.
- Shadows: None (`--shadow: none`). Crisp, clean 1px borders with dark and light mode support.
- Font stack: `"Segoe UI", "Segoe UI Web", system-ui, -apple-system, "Helvetica Neue", sans-serif`.
- **Icons**: Import all icons exclusively from `lucide-react`.
---
## 4. Pull Request & Pre-Submission Checklist
Before submitting a Pull Request:
1. **Lint Check**: Run `npm run lint` and verify zero errors and warnings.
2. **Build Check**: Run `npm run build` and ensure the Next.js standalone application compiles cleanly.
3. **Automated Tests**:
- Run the TypeScript integration tests:
```bash
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts
```
- Run the Python SDK tests if modifying SDK code:
```bash
python3 -m unittest tests/phase05_sdk_test.py
```
4. **Documentation**:
- If adding or changing features, update the relevant files in `docs/users/` or `docs/dev/`.
- Update `docs/MANIFEST.md` if new documentation files were introduced.
5. **Memory Synchronization**: Ensure `MEMORY.md` reflects all changes made during your session.
+96
View File
@@ -0,0 +1,96 @@
# Local Development Setup
This guide walks contributors through configuring and running DataStack locally on your development workstation.
---
## 1. Prerequisites
Ensure the following tools are installed:
- **Node.js**: Version `22.x` (LTS recommended)
- **Package Manager**: `npm` (v10+)
- **Python**: Version `3.10+` (optional, for developing and testing the Python client SDK)
- **Docker & Docker Compose**: Optional, useful for running local PostgreSQL and Redis instances.
---
## 2. Clone & Install Dependencies
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
# Install root dependencies deterministically using the committed lockfile
npm ci
```
---
## 3. Environment Configuration
Copy the example environment configuration:
```bash
cp .env.example .env
```
### Local Development Defaults:
In local development (`NODE_ENV=development`), DataStack includes built-in fallbacks:
- If `DATABASE_URL` is omitted, DataStack operates seamlessly in **In-Memory Mock Mode**, allowing you to test the full UI, client registration, and REST endpoints without running PostgreSQL.
- If `REDIS_URL` is omitted, DataStack uses an **in-memory RAM buffer queue**, persisting directly to `./storage/data`.
For full production parity during development, you can spin up local PostgreSQL and Redis containers:
```bash
docker run --name datastack-postgres -e POSTGRES_USER=datastack -e POSTGRES_PASSWORD=password -e POSTGRES_DB=datastack_db -p 5432:5432 -d postgres:16-alpine
docker run --name datastack-redis -p 6379:6379 -d redis:7-alpine redis-server --appendonly yes
```
Then update your `.env`:
```dotenv
DATABASE_URL=postgresql://datastack:password@127.0.0.1:5432/datastack_db
REDIS_URL=redis://127.0.0.1:6379
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
DATASTACK_ADMIN_EMAIL=admin@example.com
```
---
## 4. Running the Development Servers
DataStack requires two concurrent runtime processes. Open two terminal tabs in the project root:
### Terminal 1: Web Console & REST API (Port 3000)
```bash
npm run dev
```
Starts the Next.js 15 development server with Fast Refresh at `http://localhost:3000`.
### Terminal 2: Ingestion Server (Port 4000)
```bash
npm run server
```
Starts the dedicated Express 5 + Socket.IO ingestion service at `http://localhost:4000`.
---
## 5. First-Time Setup in Development
1. Open `http://localhost:3000/vcon/setup` in your browser.
2. Complete the setup wizard using your `DATASTACK_ADMIN_SETUP_KEY` from `.env`.
3. In `/vcon/clients`, register a test client application to generate your first JWT token.
4. In `/vcon/simulator`, test sending live events and observe the buffer counter in `/vcon/dashboard`.
---
## 6. Python SDK Development
If you are modifying the Python SDK in `sdk/python/datastack/`:
```bash
cd sdk/python/datastack
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r ../../../tests/requirements.txt # If running SDK test suites
```
+52
View File
@@ -0,0 +1,52 @@
# DataStack User & Operator Guide
Welcome to the **DataStack User Documentation**. This guide is designed for operators, administrators, product teams, and engineers who wish to deploy, operate, and collect real-time telemetry using DataStack.
---
## What is DataStack?
**DataStack** is a self-hosted, lightweight telemetry and data collection platform. It enables your applications to stream events in real time, securely buffers them with Redis and in-memory batching, and persists them into clean, partitioned daily CSV files without overloading a relational database.
---
## Documentation Navigation
Explore the following sections to get up and running:
### 1. [Installation & Deployment](./installation.md)
Step-by-step instructions for deploying DataStack in production:
- Running with **Docker Compose**
- 1-Click deployment on **Coolify**
- Configuring environment variables & persistent volumes
- Reverse proxy and SSL configuration
### 2. [Feature Walkthrough](./features.md)
Comprehensive overview of all platform capabilities:
- **Admin Console (`/vcon/`)**: Dashboard overview, live activity preview, and storage monitors.
- **Client SDK Management**: Registering client applications, issuing JWT tokens, and instant revocation.
- **Data Simulator**: Interactive in-browser telemetry generation for testing without writing code.
- **Partitioned Storage**: Size-based file rotation and automatic daily retention pruning.
- **Streaming Export**: High-performance streaming of filtered CSV or JSON datasets.
### 3. [Usage Tutorials](./tutorials.md)
Hands-on walkthroughs for common tasks:
- Sending your first telemetry event via HTTP REST
- Streaming real-time events using the official Python SDK
- Streaming local `.csv`, `.json`, and `.jsonl` data files
- Exporting datasets for analysis in Excel, pandas, or DuckDB
### 4. [Frequently Asked Questions & Troubleshooting](./faq.md)
Answers to common operational questions:
- How does DataStack prevent data loss during container restarts?
- How to rotate client credentials safely without downtime
- What to do if WebSocket handshakes fail or time out
- How to adjust event rate limits and payload size thresholds
---
## Quick Reference Links
- [Developer & Contributor Guide](../dev/README.md)
- [Architecture Blueprint](../dev/architecture.md)
- [Documentation Index](../MANIFEST.md)
- [Official Website](https://vib.tools/)
+83
View File
@@ -0,0 +1,83 @@
# Frequently Asked Questions (FAQ) & Troubleshooting
This document answers common operational, configuration, and troubleshooting questions.
---
## General Architecture & Data Storage
### Why does DataStack store events in CSV files instead of PostgreSQL?
Relational databases degrade significantly when handling thousands of concurrent writes per second due to WAL bottlenecks, indexing contention, and connection limits. DataStack separates concerns:
- **PostgreSQL** is used exclusively for administrative accounts, client credentials, and platform configuration.
- **Partitioned CSV Files** (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`) receive high-throughput batched writes sequentially on disk. This enables lightning-fast ingestion, predictable disk space usage, zero SQL table locking, and trivial backup and compression.
### What happens if the server crashes while events are buffered?
DataStack implements multi-layer crash protection:
1. **Redis AOF Durability**: In production with Redis enabled (`--appendonly yes`), incoming events are pushed immediately to a durable Redis queue (`datastack:buffer:events`).
2. **Post-Write Trimming**: The Redis list is trimmed (`LTRIM`) **strictly after** the batch append to the CSV file succeeds on disk. If a crash occurs mid-write, Redis retains the items and flushes them upon reboot.
3. **Graceful Signal Handling**: On `SIGTERM` or `SIGINT` (such as during container updates), DataStack halts new input, awaits the in-flight disk flush, and releases file locks before shutting down.
### How does file rotation and retention pruning work?
- **Rotation (`fileRotationMb`)**: By default, when a daily CSV file reaches `100 MB`, DataStack safely rotates to `{YYYY-MM-DD}-part2.csv`.
- **Retention Pruning (`autoPruneDays`)**: By default, partitions older than 30 days are automatically pruned. To retain all historical records indefinitely, set **Auto Prune Days** to `0` ("Keep all") in `/vcon/settings`.
---
## Networking & Ports
### What ports does DataStack use?
In a standard Docker Compose deployment, the services communicate over an internal network:
- **Port `80`**: Exposed by the Nginx reverse proxy (`proxy` container) to the public or host.
- **Port `3000`**: Internal Next.js web application, Admin Console, and REST API (`web` container).
- **Port `4000`**: Internal Express + Socket.IO real-time ingestion service (`ingestion` container).
- **Port `5432`**: Internal PostgreSQL database (`postgres` container).
- **Port `6379`**: Internal Redis buffer (`redis` container).
You only need to expose Port `80` (or `443` with SSL) through your firewall.
---
## Authentication & Client Security
### How does token rotation work? Does it disconnect active connections?
When you click **Rotate Token** in `/vcon/clients`:
1. A new HMAC-SHA256 token and secret are generated and stored in PostgreSQL.
2. The change is instantly communicated to the Port 4000 ingestion server over the internal authenticated bridge.
3. Any active WebSocket connections using the old token are immediately disconnected with `TOKEN_ROTATED`.
4. Subsequent requests using the old token are rejected with `401 Unauthorized`.
### What does `CLIENT_REVOKED` mean?
If a client registration is marked as `Revoked` or permanently deleted by an administrator, incoming requests are rejected with HTTP status `403 Forbidden` (`CLIENT_REVOKED`). If a WebSocket connection was open, it is kicked immediately.
---
## Troubleshooting Guide
### Issue: "WebSocket ACK timeout" or Handshake Fails
1. **Check Nginx Proxy Headers**: If using an external reverse proxy (like Cloudflare, Traefik, or Caddy) in front of DataStack, ensure `Upgrade` and `Connection: Upgrade` headers are passed through for WebSocket support:
```nginx
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
2. **Verify Port 4000**: In `/vcon/settings`, click **Connections -> Probe**. If the status reports offline, inspect the logs of the `ingestion` container:
```bash
docker compose -f docker/docker-compose.yml logs ingestion
```
### Issue: "RATE_LIMIT_EXCEEDED" Errors
Each client has a configurable rate limit (default: 100 events/sec) and a global platform limit. If your client bursts above this rate, the server returns status `429`.
- **Solution**: In `/vcon/clients`, increase the client's rate limit, or in `/vcon/settings`, adjust the global maximum events per second.
- **Python SDK**: The official Python SDK automatically handles rate limits by backing off and retrying remaining events without dropping data.
### Issue: "PAYLOAD_TOO_LARGE" Errors
By default, individual events cannot exceed `100 KB` (`maxPayloadSizeKb`).
- **Solution**: In `/vcon/settings`, increase the **Max Payload Size (KB)** setting to accommodate larger JSON objects.
### How do I back up my collected telemetry data?
All partition files are located in the `datastack-data` volume (`/app/storage/data`). You can create a compressed backup at any time while the containers are running:
```bash
docker run --rm \
--volumes-from datastack-web-1 \
-v $(pwd):/backup \
alpine tar czf /backup/datastack_backup_$(date +%F).tar.gz /app/storage/data
```
+89
View File
@@ -0,0 +1,89 @@
# Feature Guide
This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (`/vcon/`).
---
## 1. The Administration Console (`/vcon/`)
The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.
Access the console by signing in at:
```text
https://your-domain.example/vcon/login
```
### Dashboard Overview (`/vcon/dashboard`)
The primary operational cockpit displays:
- **System Health**: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
- **Storage Metrics**: Total CSV partition file count, active client directories, and total disk usage.
- **Buffer Gauges**: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
- **Live Activity Stream**: Real-time ticker showing recently ingested events with timestamps and payload previews.
---
## 2. Client SDK Registry & Security (`/vcon/clients`)
DataStack enforces strict authentication for all incoming telemetry streams.
### Registering a New Client
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
2. Provide:
- **Client Identifier**: Unique lowercase alphanumeric slug (e.g., `pos_terminal_01`, `mobile_app_prod`).
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
- **Rate Limit**: Maximum allowed events per second for this client (default: `100 events/sec`).
3. Click **Register Client**. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
4. **Copy the Token**: Store the generated token securely in your client application's environment or configuration file.
### Credential Rotation & Access Control
- **Rotate Token**: Click **Rotate Token** to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
- **Revoke Access**: Set the client status to `Revoked`. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with `CLIENT_REVOKED (403)`.
- **Permanent Deletion**: Deleting a client removes it from PostgreSQL and adds it to `deleted_clients` to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.
---
## 3. Data Simulator (`/vcon/simulator`)
The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:
- **Single Event**: Send ad-hoc JSON payloads with custom event IDs.
- **Batch Generator**: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
- **Preset Schemas**: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.
---
## 4. Partitioned CSV Storage & Retention (`/vcon/storage`)
DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:
- **Directory Layout**: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`
- **CSV Schema**: Each file contains 4 clean columns:
```csv
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
```
- **Size-Based Rotation**: When a partition reaches the configured threshold (default: `100 MB`), DataStack automatically closes the file and continues writing to `{YYYY-MM-DD}-part2.csv`.
- **Automated Retention Pruning**: Configure `autoPruneDays` (e.g., `30` days). DataStack automatically removes expired partition files, or set to `0` ("Keep all") to retain historical archives indefinitely.
- **Manual Buffer Flush**: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.
---
## 5. Streaming Data Export (`/vcon/export`)
The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:
- **Client Filter**: Export all clients or filter by a specific client ID.
- **Date Range**: Filter records within precise UTC start and end dates.
- **Formats**:
- **CSV**: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
- **JSON**: Formatted JSON array stream.
- **Non-Blocking Streaming**: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.
---
## 6. Platform Runtime Settings (`/vcon/settings`)
Fine-tune ingestion rules with instant cross-process propagation:
- **Max Events Per Second**: Global ingestion rate ceiling.
- **Max Payload Size (KB)**: Maximum permitted size per event payload (default: `100 KB`).
- **Batch Flush Threshold**: Number of buffered events that triggers an immediate write to disk (default: `500 events`).
- **Batch Flush Interval**: Maximum elapsed time before flushing pending events to disk (default: `2000 ms`).
- **Idempotency Protection**: Deduplicates events with identical `eventId` values within a 5-minute sliding window.
- **Connection Probing**: Test the live internal bridge between the Web console and the Port 4000 ingestion service.
+111
View File
@@ -0,0 +1,111 @@
# Installation & Deployment Guide
This guide explains how to install and self-host DataStack on your own server or cloud VPS.
---
## Prerequisites
Before starting, ensure your system meets the following minimum requirements:
- **Operating System**: Linux (Ubuntu 22.04+, Debian 12+, Alpine), macOS, or Windows WSL2.
- **Docker Engine**: Docker `v24.0+` and Docker Compose `v2.20+`.
- **System Resources**:
- Minimum: 1 vCPU, 1 GB RAM, 10 GB disk space.
- Recommended: 2 vCPU, 2–4 GB RAM, SSD storage scaled to your expected data volume.
- **Networking**: Ports `80` (HTTP) and `443` (HTTPS) accessible via your reverse proxy or cloud firewall.
---
## Method 1: Docker Compose (Recommended)
DataStack includes a complete, production-ready Docker Compose stack located in the [`docker/`](../../docker/) directory.
### 1. Clone the Repository
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
```
### 2. Configure Environment Variables
Copy the example environment configuration:
```bash
cp .env.example .env
```
Generate secure, random 32+ character secrets for production:
```bash
# In Linux / macOS, generate 48-char random hex keys:
openssl rand -hex 32
```
Edit `.env` and fill in the required variables:
```dotenv
# Production Security Secrets (Minimum 32 characters each)
DATASTACK_ADMIN_SESSION_SECRET=your-random-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-random-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-random-32-plus-char-admin-setup-key-here
DATASTACK_INTERNAL_SECRET=your-random-32-plus-char-internal-service-secret-here
# Administrator Initial Email
DATASTACK_ADMIN_EMAIL=admin@yourcompany.com
```
### 3. Launch the Stack
Run Docker Compose from the project root:
```bash
docker compose -f docker/docker-compose.yml up -d
```
The stack launches 5 containers:
1. **`proxy`**: Nginx reverse proxy exposed on port `80`, routing `/socket.io/` to port `4000` and all other routes to port `3000`.
2. **`web`**: Next.js 15 standalone server running on port `3000`.
3. **`ingestion`**: Dedicated Express + Socket.IO ingestion server running on port `4000`.
4. **`postgres`**: PostgreSQL 16 database storing administrative credentials, SDK client records, and settings.
5. **`redis`**: Redis 7 container running with `--appendonly yes` for crash-resilient queue buffering.
---
## Method 2: Coolify Cloud / Self-Hosted PaaS
DataStack is optimized for deployment via [Coolify](https://coolify.io/):
1. **Create Resource**: In Coolify, click **+ Add Resource** → **Docker Compose**.
2. **Repository Settings**:
- Point to your DataStack Git repository.
- Set **Base Directory** to `/docker`.
- Set **Docker Compose Location** to `/docker/docker-compose.yml`.
3. **Domain Assignment**:
- Attach your custom domain (e.g. `https://datastack.yourdomain.com`) to the **`proxy`** service on internal port `80`.
- Enable automatic SSL/TLS via Coolify's built-in Let's Encrypt integration.
4. **Environment Variables**:
- Coolify automatically injects generated `SERVICE_*` variables for database and JWT secrets.
5. **Deploy**: Click **Deploy** and monitor the build logs until all 5 services are healthy.
---
## First-Run Administrator Setup (`/vcon/setup`)
Once the stack is running:
1. Open your browser and navigate to:
```text
https://your-domain.example/vcon/setup
```
2. You will be prompted to enter:
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` or your chosen admin email.
- **Administrator Password**: Choose a strong password (minimum 8 characters).
- **Setup Security Key**: Enter the exact value configured in `DATASTACK_ADMIN_SETUP_KEY`.
3. Click **Complete Initial Setup**. You will be securely logged in and redirected to the `/vcon/dashboard` console.
---
## Persistent Storage & Volumes
DataStack uses three named volumes to safeguard your data across container updates:
| Volume Name | Container Path | Purpose |
| :--- | :--- | :--- |
| **`datastack-data`** | `/app/storage` | Contains `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. Shared by `web` and `ingestion`. |
| **`pgdata`** | `/var/lib/postgresql/data` | Persists administrative users, client credentials, and settings. |
| **`redisdata`** | `/data` | Persists Redis Append-Only File (`appendonly.aof`) for unflushed event buffers. |
> **IMPORTANT**: When upgrading or redeploying containers, never execute `docker compose down -v`, as the `-v` flag deletes named volumes. Use standard `docker compose down` followed by `docker compose up -d`.
+166
View File
@@ -0,0 +1,166 @@
# Usage Tutorials
This document provides hands-on tutorials for sending data to DataStack, integrating client SDKs, and exporting datasets.
---
## Tutorial 1: Sending Events via HTTP REST (`curl`)
The fastest way to send data to DataStack without writing application code is via the HTTP REST API.
### 1. Retrieve Your Client Token
In `/vcon/clients`, register a client (e.g., `bash_script_01`) and copy the generated Bearer JWT token.
### 2. Send a Single Event
```bash
curl -X POST "https://your-domain.example/api/events" \
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"eventId": "order_1001",
"timestamp": "2026-11-17T12:00:00Z",
"data": {
"item": "Mechanical Keyboard",
"price": 129.99,
"status": "completed"
}
}'
```
**Expected Response**:
```json
{
"success": true,
"eventId": "order_1001",
"event": {
"clientId": "bash_script_01",
"eventId": "order_1001",
"timestamp": "2026-11-17T12:00:00Z",
"payload": "{\"item\":\"Mechanical Keyboard\",\"price\":129.99,\"status\":\"completed\"}"
}
}
```
### 3. Send a Batch of Events
```bash
curl -X POST "https://your-domain.example/api/events" \
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"batch": [
{ "eventId": "evt_01", "data": { "metric": "cpu", "value": 45.2 } },
{ "eventId": "evt_02", "data": { "metric": "ram", "value": 72.8 } }
]
}'
```
---
## Tutorial 2: Integrating the Python Client SDK
DataStack provides an official, pip-installable Python SDK featuring real-time Socket.IO streaming, automatic HTTP fallback, and offline disk spooling.
### 1. Install the SDK
```bash
pip install datastack
# Or install backward-compatible package:
pip install datastack-sdk
```
### 2. Create Configuration (`config.datastack.py`)
In the root of your Python project, create a `config.datastack.py` file:
```python
DATASTACK_CONFIG = {
"SERVER_URL": "https://your-domain.example", # WebSocket & proxy endpoint
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Token from /vcon/clients
"AUTO_CONNECT": True,
"ENABLE_HTTP_FALLBACK": True,
"OFFLINE_SPOOL_ENABLED": True,
}
```
### 3. Stream Telemetry from Python
```python
from datastack import DataStack
import time
# Client automatically discovers and loads config.datastack.py
client = DataStack()
print(f"Connected to DataStack as client: {client.client_id}")
for i in range(5):
response = client.send(
data={
"sensor_id": "temp_probe_04",
"temperature_celsius": 24.5 + i * 0.2,
"humidity_percent": 55,
},
event_id=f"reading_{i}_{int(time.time())}"
)
print(f"Event sent: {response}")
time.sleep(1)
client.disconnect()
```
---
## Tutorial 3: Streaming Local Files (`send_file`)
If you have offline log files or CSV datasets that need to be ingested into DataStack, use the SDK's built-in file streaming:
```python
from datastack import DataStack
client = DataStack()
# Supports .csv, .json, .jsonl, and .ndjson
result = client.send_file(
file_path="./daily_orders.csv",
chunk_size=250, # Stream in batches of 250 records
extra_metadata={"imported_by": "nightly_cron", "env": "production"}
)
print(f"File Ingestion Complete:")
print(f" Processed rows: {result['processed']}")
print(f" Duplicates skipped: {result['duplicates']}")
print(f" Chunks sent: {result['chunksSent']}")
```
- **Lossless Handling**: If your server rate limits the file, the SDK automatically pauses and retries.
- **Deduplication**: Rows without an explicit `eventId` receive deterministic SHA-256 IDs, preventing duplicate rows if the script runs twice.
---
## Tutorial 4: Exporting Data for Analysis
You can stream historical partitions for analysis in pandas, Python, or command-line scripts.
### 1. Stream CSV Export via `curl`
```bash
curl -X GET "https://your-domain.example/api/export?format=csv&clientId=bash_script_01" \
-H "Cookie: datastack_admin_session=YOUR_ADMIN_SESSION_TOKEN" \
-o export_bash_script.csv
```
### 2. Load Directly into pandas
```python
import pandas as pd
import requests
import io
session_cookie = "YOUR_ADMIN_SESSION_COOKIE"
export_url = "https://your-domain.example/api/export?format=csv&clientId=all"
response = requests.get(
export_url,
cookies={"datastack_admin_session": session_cookie},
stream=True
)
df = pd.read_csv(io.StringIO(response.text))
print("Exported DataFrame:")
print(df.head())
```