chore: establish cross-IDE synchronization protocol

Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
This commit is contained in:
Team Dark Devil committed 2026-11-18 07:11:11 +06:00
1 parent 05407fd338
commit 4279da45bb
14 files changed
+1162

No files matched your search

+52
View File
@@ -0,0 +1,52 @@
# DataStack User & Operator Guide
Welcome to the **DataStack User Documentation**. This guide is designed for operators, administrators, product teams, and engineers who wish to deploy, operate, and collect real-time telemetry using DataStack.
---
## What is DataStack?
**DataStack** is a self-hosted, lightweight telemetry and data collection platform. It enables your applications to stream events in real time, securely buffers them with Redis and in-memory batching, and persists them into clean, partitioned daily CSV files without overloading a relational database.
---
## Documentation Navigation
Explore the following sections to get up and running:
### 1. [Installation & Deployment](./installation.md)
Step-by-step instructions for deploying DataStack in production:
- Running with **Docker Compose**
- 1-Click deployment on **Coolify**
- Configuring environment variables & persistent volumes
- Reverse proxy and SSL configuration
### 2. [Feature Walkthrough](./features.md)
Comprehensive overview of all platform capabilities:
- **Admin Console (`/vcon/`)**: Dashboard overview, live activity preview, and storage monitors.
- **Client SDK Management**: Registering client applications, issuing JWT tokens, and instant revocation.
- **Data Simulator**: Interactive in-browser telemetry generation for testing without writing code.
- **Partitioned Storage**: Size-based file rotation and automatic daily retention pruning.
- **Streaming Export**: High-performance streaming of filtered CSV or JSON datasets.
### 3. [Usage Tutorials](./tutorials.md)
Hands-on walkthroughs for common tasks:
- Sending your first telemetry event via HTTP REST
- Streaming real-time events using the official Python SDK
- Streaming local `.csv`, `.json`, and `.jsonl` data files
- Exporting datasets for analysis in Excel, pandas, or DuckDB
### 4. [Frequently Asked Questions & Troubleshooting](./faq.md)
Answers to common operational questions:
- How does DataStack prevent data loss during container restarts?
- How to rotate client credentials safely without downtime
- What to do if WebSocket handshakes fail or time out
- How to adjust event rate limits and payload size thresholds
---
## Quick Reference Links
- [Developer & Contributor Guide](../dev/README.md)
- [Architecture Blueprint](../dev/architecture.md)
- [Documentation Index](../MANIFEST.md)
- [Official Website](https://vib.tools/)
+83
View File
@@ -0,0 +1,83 @@
# Frequently Asked Questions (FAQ) & Troubleshooting
This document answers common operational, configuration, and troubleshooting questions.
---
## General Architecture & Data Storage
### Why does DataStack store events in CSV files instead of PostgreSQL?
Relational databases degrade significantly when handling thousands of concurrent writes per second due to WAL bottlenecks, indexing contention, and connection limits. DataStack separates concerns:
- **PostgreSQL** is used exclusively for administrative accounts, client credentials, and platform configuration.
- **Partitioned CSV Files** (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`) receive high-throughput batched writes sequentially on disk. This enables lightning-fast ingestion, predictable disk space usage, zero SQL table locking, and trivial backup and compression.
### What happens if the server crashes while events are buffered?
DataStack implements multi-layer crash protection:
1. **Redis AOF Durability**: In production with Redis enabled (`--appendonly yes`), incoming events are pushed immediately to a durable Redis queue (`datastack:buffer:events`).
2. **Post-Write Trimming**: The Redis list is trimmed (`LTRIM`) **strictly after** the batch append to the CSV file succeeds on disk. If a crash occurs mid-write, Redis retains the items and flushes them upon reboot.
3. **Graceful Signal Handling**: On `SIGTERM` or `SIGINT` (such as during container updates), DataStack halts new input, awaits the in-flight disk flush, and releases file locks before shutting down.
### How does file rotation and retention pruning work?
- **Rotation (`fileRotationMb`)**: By default, when a daily CSV file reaches `100 MB`, DataStack safely rotates to `{YYYY-MM-DD}-part2.csv`.
- **Retention Pruning (`autoPruneDays`)**: By default, partitions older than 30 days are automatically pruned. To retain all historical records indefinitely, set **Auto Prune Days** to `0` ("Keep all") in `/vcon/settings`.
---
## Networking & Ports
### What ports does DataStack use?
In a standard Docker Compose deployment, the services communicate over an internal network:
- **Port `80`**: Exposed by the Nginx reverse proxy (`proxy` container) to the public or host.
- **Port `3000`**: Internal Next.js web application, Admin Console, and REST API (`web` container).
- **Port `4000`**: Internal Express + Socket.IO real-time ingestion service (`ingestion` container).
- **Port `5432`**: Internal PostgreSQL database (`postgres` container).
- **Port `6379`**: Internal Redis buffer (`redis` container).
You only need to expose Port `80` (or `443` with SSL) through your firewall.
---
## Authentication & Client Security
### How does token rotation work? Does it disconnect active connections?
When you click **Rotate Token** in `/vcon/clients`:
1. A new HMAC-SHA256 token and secret are generated and stored in PostgreSQL.
2. The change is instantly communicated to the Port 4000 ingestion server over the internal authenticated bridge.
3. Any active WebSocket connections using the old token are immediately disconnected with `TOKEN_ROTATED`.
4. Subsequent requests using the old token are rejected with `401 Unauthorized`.
### What does `CLIENT_REVOKED` mean?
If a client registration is marked as `Revoked` or permanently deleted by an administrator, incoming requests are rejected with HTTP status `403 Forbidden` (`CLIENT_REVOKED`). If a WebSocket connection was open, it is kicked immediately.
---
## Troubleshooting Guide
### Issue: "WebSocket ACK timeout" or Handshake Fails
1. **Check Nginx Proxy Headers**: If using an external reverse proxy (like Cloudflare, Traefik, or Caddy) in front of DataStack, ensure `Upgrade` and `Connection: Upgrade` headers are passed through for WebSocket support:
```nginx
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
2. **Verify Port 4000**: In `/vcon/settings`, click **Connections -> Probe**. If the status reports offline, inspect the logs of the `ingestion` container:
```bash
docker compose -f docker/docker-compose.yml logs ingestion
```
### Issue: "RATE_LIMIT_EXCEEDED" Errors
Each client has a configurable rate limit (default: 100 events/sec) and a global platform limit. If your client bursts above this rate, the server returns status `429`.
- **Solution**: In `/vcon/clients`, increase the client's rate limit, or in `/vcon/settings`, adjust the global maximum events per second.
- **Python SDK**: The official Python SDK automatically handles rate limits by backing off and retrying remaining events without dropping data.
### Issue: "PAYLOAD_TOO_LARGE" Errors
By default, individual events cannot exceed `100 KB` (`maxPayloadSizeKb`).
- **Solution**: In `/vcon/settings`, increase the **Max Payload Size (KB)** setting to accommodate larger JSON objects.
### How do I back up my collected telemetry data?
All partition files are located in the `datastack-data` volume (`/app/storage/data`). You can create a compressed backup at any time while the containers are running:
```bash
docker run --rm \
--volumes-from datastack-web-1 \
-v $(pwd):/backup \
alpine tar czf /backup/datastack_backup_$(date +%F).tar.gz /app/storage/data
```
+89
View File
@@ -0,0 +1,89 @@
# Feature Guide
This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (`/vcon/`).
---
## 1. The Administration Console (`/vcon/`)
The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.
Access the console by signing in at:
```text
https://your-domain.example/vcon/login
```
### Dashboard Overview (`/vcon/dashboard`)
The primary operational cockpit displays:
- **System Health**: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
- **Storage Metrics**: Total CSV partition file count, active client directories, and total disk usage.
- **Buffer Gauges**: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
- **Live Activity Stream**: Real-time ticker showing recently ingested events with timestamps and payload previews.
---
## 2. Client SDK Registry & Security (`/vcon/clients`)
DataStack enforces strict authentication for all incoming telemetry streams.
### Registering a New Client
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
2. Provide:
- **Client Identifier**: Unique lowercase alphanumeric slug (e.g., `pos_terminal_01`, `mobile_app_prod`).
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
- **Rate Limit**: Maximum allowed events per second for this client (default: `100 events/sec`).
3. Click **Register Client**. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
4. **Copy the Token**: Store the generated token securely in your client application's environment or configuration file.
### Credential Rotation & Access Control
- **Rotate Token**: Click **Rotate Token** to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
- **Revoke Access**: Set the client status to `Revoked`. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with `CLIENT_REVOKED (403)`.
- **Permanent Deletion**: Deleting a client removes it from PostgreSQL and adds it to `deleted_clients` to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.
---
## 3. Data Simulator (`/vcon/simulator`)
The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:
- **Single Event**: Send ad-hoc JSON payloads with custom event IDs.
- **Batch Generator**: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
- **Preset Schemas**: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.
---
## 4. Partitioned CSV Storage & Retention (`/vcon/storage`)
DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:
- **Directory Layout**: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`
- **CSV Schema**: Each file contains 4 clean columns:
```csv
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
```
- **Size-Based Rotation**: When a partition reaches the configured threshold (default: `100 MB`), DataStack automatically closes the file and continues writing to `{YYYY-MM-DD}-part2.csv`.
- **Automated Retention Pruning**: Configure `autoPruneDays` (e.g., `30` days). DataStack automatically removes expired partition files, or set to `0` ("Keep all") to retain historical archives indefinitely.
- **Manual Buffer Flush**: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.
---
## 5. Streaming Data Export (`/vcon/export`)
The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:
- **Client Filter**: Export all clients or filter by a specific client ID.
- **Date Range**: Filter records within precise UTC start and end dates.
- **Formats**:
- **CSV**: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
- **JSON**: Formatted JSON array stream.
- **Non-Blocking Streaming**: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.
---
## 6. Platform Runtime Settings (`/vcon/settings`)
Fine-tune ingestion rules with instant cross-process propagation:
- **Max Events Per Second**: Global ingestion rate ceiling.
- **Max Payload Size (KB)**: Maximum permitted size per event payload (default: `100 KB`).
- **Batch Flush Threshold**: Number of buffered events that triggers an immediate write to disk (default: `500 events`).
- **Batch Flush Interval**: Maximum elapsed time before flushing pending events to disk (default: `2000 ms`).
- **Idempotency Protection**: Deduplicates events with identical `eventId` values within a 5-minute sliding window.
- **Connection Probing**: Test the live internal bridge between the Web console and the Port 4000 ingestion service.
+111
View File
@@ -0,0 +1,111 @@
# Installation & Deployment Guide
This guide explains how to install and self-host DataStack on your own server or cloud VPS.
---
## Prerequisites
Before starting, ensure your system meets the following minimum requirements:
- **Operating System**: Linux (Ubuntu 22.04+, Debian 12+, Alpine), macOS, or Windows WSL2.
- **Docker Engine**: Docker `v24.0+` and Docker Compose `v2.20+`.
- **System Resources**:
- Minimum: 1 vCPU, 1 GB RAM, 10 GB disk space.
- Recommended: 2 vCPU, 2–4 GB RAM, SSD storage scaled to your expected data volume.
- **Networking**: Ports `80` (HTTP) and `443` (HTTPS) accessible via your reverse proxy or cloud firewall.
---
## Method 1: Docker Compose (Recommended)
DataStack includes a complete, production-ready Docker Compose stack located in the [`docker/`](../../docker/) directory.
### 1. Clone the Repository
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
```
### 2. Configure Environment Variables
Copy the example environment configuration:
```bash
cp .env.example .env
```
Generate secure, random 32+ character secrets for production:
```bash
# In Linux / macOS, generate 48-char random hex keys:
openssl rand -hex 32
```
Edit `.env` and fill in the required variables:
```dotenv
# Production Security Secrets (Minimum 32 characters each)
DATASTACK_ADMIN_SESSION_SECRET=your-random-32-plus-char-session-secret-here
DATASTACK_JWT_SECRET=your-random-32-plus-char-client-jwt-secret-here
DATASTACK_ADMIN_SETUP_KEY=your-random-32-plus-char-admin-setup-key-here
DATASTACK_INTERNAL_SECRET=your-random-32-plus-char-internal-service-secret-here
# Administrator Initial Email
DATASTACK_ADMIN_EMAIL=admin@yourcompany.com
```
### 3. Launch the Stack
Run Docker Compose from the project root:
```bash
docker compose -f docker/docker-compose.yml up -d
```
The stack launches 5 containers:
1. **`proxy`**: Nginx reverse proxy exposed on port `80`, routing `/socket.io/` to port `4000` and all other routes to port `3000`.
2. **`web`**: Next.js 15 standalone server running on port `3000`.
3. **`ingestion`**: Dedicated Express + Socket.IO ingestion server running on port `4000`.
4. **`postgres`**: PostgreSQL 16 database storing administrative credentials, SDK client records, and settings.
5. **`redis`**: Redis 7 container running with `--appendonly yes` for crash-resilient queue buffering.
---
## Method 2: Coolify Cloud / Self-Hosted PaaS
DataStack is optimized for deployment via [Coolify](https://coolify.io/):
1. **Create Resource**: In Coolify, click **+ Add Resource** → **Docker Compose**.
2. **Repository Settings**:
- Point to your DataStack Git repository.
- Set **Base Directory** to `/docker`.
- Set **Docker Compose Location** to `/docker/docker-compose.yml`.
3. **Domain Assignment**:
- Attach your custom domain (e.g. `https://datastack.yourdomain.com`) to the **`proxy`** service on internal port `80`.
- Enable automatic SSL/TLS via Coolify's built-in Let's Encrypt integration.
4. **Environment Variables**:
- Coolify automatically injects generated `SERVICE_*` variables for database and JWT secrets.
5. **Deploy**: Click **Deploy** and monitor the build logs until all 5 services are healthy.
---
## First-Run Administrator Setup (`/vcon/setup`)
Once the stack is running:
1. Open your browser and navigate to:
```text
https://your-domain.example/vcon/setup
```
2. You will be prompted to enter:
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` or your chosen admin email.
- **Administrator Password**: Choose a strong password (minimum 8 characters).
- **Setup Security Key**: Enter the exact value configured in `DATASTACK_ADMIN_SETUP_KEY`.
3. Click **Complete Initial Setup**. You will be securely logged in and redirected to the `/vcon/dashboard` console.
---
## Persistent Storage & Volumes
DataStack uses three named volumes to safeguard your data across container updates:
| Volume Name | Container Path | Purpose |
| :--- | :--- | :--- |
| **`datastack-data`** | `/app/storage` | Contains `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. Shared by `web` and `ingestion`. |
| **`pgdata`** | `/var/lib/postgresql/data` | Persists administrative users, client credentials, and settings. |
| **`redisdata`** | `/data` | Persists Redis Append-Only File (`appendonly.aof`) for unflushed event buffers. |
> **IMPORTANT**: When upgrading or redeploying containers, never execute `docker compose down -v`, as the `-v` flag deletes named volumes. Use standard `docker compose down` followed by `docker compose up -d`.
+166
View File
@@ -0,0 +1,166 @@
# Usage Tutorials
This document provides hands-on tutorials for sending data to DataStack, integrating client SDKs, and exporting datasets.
---
## Tutorial 1: Sending Events via HTTP REST (`curl`)
The fastest way to send data to DataStack without writing application code is via the HTTP REST API.
### 1. Retrieve Your Client Token
In `/vcon/clients`, register a client (e.g., `bash_script_01`) and copy the generated Bearer JWT token.
### 2. Send a Single Event
```bash
curl -X POST "https://your-domain.example/api/events" \
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"eventId": "order_1001",
"timestamp": "2026-11-17T12:00:00Z",
"data": {
"item": "Mechanical Keyboard",
"price": 129.99,
"status": "completed"
}
}'
```
**Expected Response**:
```json
{
"success": true,
"eventId": "order_1001",
"event": {
"clientId": "bash_script_01",
"eventId": "order_1001",
"timestamp": "2026-11-17T12:00:00Z",
"payload": "{\"item\":\"Mechanical Keyboard\",\"price\":129.99,\"status\":\"completed\"}"
}
}
```
### 3. Send a Batch of Events
```bash
curl -X POST "https://your-domain.example/api/events" \
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"batch": [
{ "eventId": "evt_01", "data": { "metric": "cpu", "value": 45.2 } },
{ "eventId": "evt_02", "data": { "metric": "ram", "value": 72.8 } }
]
}'
```
---
## Tutorial 2: Integrating the Python Client SDK
DataStack provides an official, pip-installable Python SDK featuring real-time Socket.IO streaming, automatic HTTP fallback, and offline disk spooling.
### 1. Install the SDK
```bash
pip install datastack
# Or install backward-compatible package:
pip install datastack-sdk
```
### 2. Create Configuration (`config.datastack.py`)
In the root of your Python project, create a `config.datastack.py` file:
```python
DATASTACK_CONFIG = {
"SERVER_URL": "https://your-domain.example", # WebSocket & proxy endpoint
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Token from /vcon/clients
"AUTO_CONNECT": True,
"ENABLE_HTTP_FALLBACK": True,
"OFFLINE_SPOOL_ENABLED": True,
}
```
### 3. Stream Telemetry from Python
```python
from datastack import DataStack
import time
# Client automatically discovers and loads config.datastack.py
client = DataStack()
print(f"Connected to DataStack as client: {client.client_id}")
for i in range(5):
response = client.send(
data={
"sensor_id": "temp_probe_04",
"temperature_celsius": 24.5 + i * 0.2,
"humidity_percent": 55,
},
event_id=f"reading_{i}_{int(time.time())}"
)
print(f"Event sent: {response}")
time.sleep(1)
client.disconnect()
```
---
## Tutorial 3: Streaming Local Files (`send_file`)
If you have offline log files or CSV datasets that need to be ingested into DataStack, use the SDK's built-in file streaming:
```python
from datastack import DataStack
client = DataStack()
# Supports .csv, .json, .jsonl, and .ndjson
result = client.send_file(
file_path="./daily_orders.csv",
chunk_size=250, # Stream in batches of 250 records
extra_metadata={"imported_by": "nightly_cron", "env": "production"}
)
print(f"File Ingestion Complete:")
print(f" Processed rows: {result['processed']}")
print(f" Duplicates skipped: {result['duplicates']}")
print(f" Chunks sent: {result['chunksSent']}")
```
- **Lossless Handling**: If your server rate limits the file, the SDK automatically pauses and retries.
- **Deduplication**: Rows without an explicit `eventId` receive deterministic SHA-256 IDs, preventing duplicate rows if the script runs twice.
---
## Tutorial 4: Exporting Data for Analysis
You can stream historical partitions for analysis in pandas, Python, or command-line scripts.
### 1. Stream CSV Export via `curl`
```bash
curl -X GET "https://your-domain.example/api/export?format=csv&clientId=bash_script_01" \
-H "Cookie: datastack_admin_session=YOUR_ADMIN_SESSION_TOKEN" \
-o export_bash_script.csv
```
### 2. Load Directly into pandas
```python
import pandas as pd
import requests
import io
session_cookie = "YOUR_ADMIN_SESSION_COOKIE"
export_url = "https://your-domain.example/api/export?format=csv&clientId=all"
response = requests.get(
export_url,
cookies={"datastack_admin_session": session_cookie},
stream=True
)
df = pd.read_csv(io.StringIO(response.text))
print("Exported DataFrame:")
print(df.head())
```