> **Practical software for real workflows.** Stream structured events over WebSockets and REST, buffer batches in memory, and persist clean, portable CSV datasets with full operational control.
**DataStack** is a free, open-source, self-hosted real-time data collection and telemetry logging platform developed by **[Vib Tools](https://vib.tools/)**. It is designed for developers, technical teams, operators, and researchers who need a fast, inspectable, and maintainable system to collect structured JSON events over **Socket.IO WebSockets** and **HTTP REST APIs**, buffer them in memory, and persist them into daily and size-rotated **CSV partitions** without the complexity of heavyweight log clusters or per-row database bottlenecks.
Many teams need to collect high-frequency telemetry, service heartbeats, IoT sensor readings, or structured operational events, but face two extremes:
- Inserting thousands of events per second row-by-row into a relational database creates lock contention and storage bloat.
- Deploying distributed log clusters introduces high infrastructure costs and operational overhead.
**DataStack solves this with a focused, transparent architecture:**
- **Zero Per-Row Database Inserts**: Incoming events are validated in memory, queued in a batch buffer, and appended in batches to partitioned CSV files (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`).
- **PostgreSQL for Control-Plane Metadata Only**: PostgreSQL (`pg`) is dedicated strictly to administrator authentication (`admin_users`), brute-force login protection (`admin_login_attempts`), SDK client credentials (`client_sdks`), and platform configuration (`admin_settings`).
- **Portable, Inspectable Datasets**: Collected data lives in standard CSV partitions that can be searched in the browser, exported as streaming CSV or JSON archives, or loaded directly into Python, Pandas, DuckDB, R, or Excel.
---
## Complete A–Z Feature Overview
### 1. Real-Time Ingestion Engine (Port 4000 & Port 3000)
- **Dedicated Socket.IO Server (`Port 4000`)**: Handles persistent WebSocket connections (`sync_data` for single events and `batch_sync` for batch arrays) completely decoupled from Next.js serverless routes.
- **Authenticated HTTP Ingestion (`POST /api/events`)**: Supports standard HTTP POST ingestion authenticated via either a Client Bearer JWT token (`Authorization: Bearer <token>`) or an active Administrator Session Cookie.
- **Sub-Millisecond In-Memory Batching**: Queues incoming events in memory and flushes to disk automatically when reaching `batchFlushThreshold` (default `500` events) or on a periodic timer (`batchFlushIntervalMs`, default `2000ms`).
- **On-Demand Cross-Process Buffer Flush**: Clicking **Flush Buffer** in `/vcon/storage` (`POST /api/storage`) or triggering an export (`GET /api/export`) flushes both the Next.js buffer and the Port 4000 Socket.IO buffer immediately.
- **Client & Date Partitioning**: Stores events under `/storage/data/{clientId}/{YYYY-MM-DD}.csv` with a consistent 4-column schema: `clientId,eventId,timestamp,payload`.
- **Automatic Size-Based File Rotation**: Automatically rotates active daily partitions to `{YYYY-MM-DD}-part2.csv`, `-part3.csv`, etc. when a file reaches `fileRotationMb` (default `100 MB`).
- **Automated Retention Pruning (`autoPruneDays`)**: Automatically removes expired daily CSV partitions older than the configured retention window (default `30` days).
- **Streaming CSV & JSON Exports**: Streams filtered datasets across any client and date range (`Today`, `7d`, `30d`, `All`, or custom dates) in either `.csv` or parsed `.json` format (`GET /api/export`).
### 3. Client SDK Management & Access Governance
- **Per-Client JWT Credentials**: Issue unique `clientId`, secret key, and signed HMAC-SHA256 JWT tokens from `/vcon/clients`.
- **Configurable Per-Client Rate Limiting**: Enforce per-second sliding-window rate limits customized for each client (`client.rateLimit` capped by global `maxEventsPerSec`).
- **5-Minute TTL Idempotency Deduplication**: Prevents duplicate event ingestion for identical `clientId:eventId` pairs within a 5-minute sliding window with automatic memory cleanup.
- **Live Key Rotation, Revocation & Deletion**: Rotate client JWT tokens, temporarily revoke/reactivate client access (`403 CLIENT_REVOKED`), or permanently delete client registrations with immediate enforcement across both HTTP and Socket.IO engines.
- **Automatic Token Re-Signing**: Updating the platform `jwtSecret` in `/vcon/settings` automatically re-signs active client tokens in PostgreSQL so client credentials stay synchronized.
### 4. Security & Administrator Authentication
- **First-Run Installation Setup (`/vcon/setup`)**: Protected by a one-time installation key (`DATASTACK_ADMIN_SETUP_KEY`); permanently locks once the initial administrator account is created in PostgreSQL.
- **Scrypt Password Hashing & Timing-Safe Verification**: Administrator passwords are hashed with Node.js `crypto.scrypt` (`N=16384, r=8, p=1`) and verified using `timingSafeEqual`.
- **Brute-Force Login Lockout**: Tracks failed login attempts in PostgreSQL (`admin_login_attempts`), locking sign-in for 15 minutes after 5 failed attempts within a 15-minute window and returning a `Retry-After` header.
- **WebCrypto HMAC-SHA256 Sessions**: Issues `HttpOnly`, `SameSite=Lax` session cookies (`8 hours` standard or `30 days` with **Remember me**), preserving remaining session lifetime across profile updates.
| `/vcon/dashboard` | **Dashboard** | View total stored rows, active Socket.IO connections, disk usage, buffered event counts, registered clients, and live recent events. |
| `/vcon/clients` | **Clients** | Register new SDK clients, copy signed JWT tokens, inspect per-client event counters and last-seen timestamps, rotate keys, revoke access, or delete clients. |
| `/vcon/data` | **Data Browser** | Search and filter persisted CSV rows by client and keyword, paginate through records, and inspect formatted JSON payloads. |
| `/vcon/export` | **Export** | Build custom exports filtered by client and date range (`Today`, `7d`, `30d`, `All`), preview row/byte counts, and stream `.csv` or `.json` downloads. |
| `/vcon/storage` | **Storage** | Monitor disk usage, total CSV partition files, active rotation rules (`100MB / Daily`), and trigger an immediate manual buffer flush. |
| `/vcon/sdk` | **Python SDK** | Select any registered client from the dropdown to generate a ready-to-run Python snippet pre-populated with that client's signed JWT token. |
| `/vcon/simulator` | **Simulator** | Emit single events or configurable batch test payloads directly from the browser to verify ingestion, rate limits, and CSV flushing. |
| `/vcon/profile` | **Admin Profile** | Update the administrator email address or password with current-password verification. |
---
## Quick Start (Local Development)
### Prerequisites
- **Node.js** `20+` and `npm`
- **PostgreSQL** `15+` (optional for local development; required for persistent multi-process production state)
- **Python** `3.8+` (optional, for using `datastack-sdk`)
The official production Python SDK is located in [`./sdk/python/datastack`](./sdk/python/datastack) (with full backward compatibility in [`./sdk-python`](./sdk-python)). It provides zero-boilerplate `config.datastack.py` auto-discovery, low-latency Socket.IO streaming, automatic Heartbeat HTTP fallback (`POST /api/events`), offline disk spooling (`.datastack_spool/spool.jsonl`), and direct `.csv`, `.json`, and `.jsonl` file ingestion.
| `GET /api/storage` | Admin Session Cookie | Get CSV partition file list and disk usage statistics |
| `POST /api/storage` | Admin Session Cookie | Flush both Next.js and Port 4000 buffers to disk immediately |
| `GET / PUT /api/settings` | Admin Session Cookie | Read or update platform runtime settings |
| `GET /api/metrics` | Admin Session Cookie | Consolidated system health, buffer, storage, and Socket.IO metrics |
---
## Docker Compose & Coolify Deployment
DataStack includes a production-ready Docker Compose stack in [`docker/docker-compose.yml`](./docker/docker-compose.yml):
- **`proxy`**: Nginx reverse proxy on internal port `80` routing `/socket.io/` to `ingestion:4000` and all other traffic to `web:3000`.
- **`web`**: Standalone Next.js 15 container on port `3000` (runs as non-root user).
- **`ingestion`**: Express + Socket.IO container on port `4000` (runs as non-root user).
- **`postgres`**: PostgreSQL 16 with persistent `pgdata` volume.
- **`redis`**: Redis 7 with persistent `redisdata` volume.
- **`datastack-data` Volume**: Mounted at `/app/storage/data` in both `web` and `ingestion` so CSV partitions are shared seamlessly across containers.
### Coolify Deployment Steps
1. Create a **Docker Compose** resource in Coolify pointing to your repository.
2. Set **Base Directory** to `/docker` and **Docker Compose Location** to `/docker/docker-compose.yml`.
3. Attach your HTTPS domain to the `proxy` service on port `80`.
4. Deploy the stack, open `https://your-domain.example/vcon/setup`, and enter your administrator email, password (8+ characters), and the generated `SERVICE_REALBASE64_64_ADMININSTALL` key from Coolify's environment variables.
**DataStack** is developed and maintained by **[Vib Tools](https://vib.tools/)** — a software organization building practical desktop applications, self-hosted software, automation tooling, developer utilities, reusable frameworks, and open-source projects for real workflows.