i vib fcc4f4db2d chore: publish current DataStack updates
Publish the current application, design, and Phase 1 deployment updates. Excludes ignored personal notes, local MCP config, and generated TypeScript build info.
2026-10-23 19:56:38 -07:00

DataStack — Self-Hosted Universal Realtime Data Collection Platform

DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.

Architecture

Process Port Role
Next.js App Router 3000 Frontend UI, /vcon/* admin console, streaming CSV export /api/export
Express + Socket.IO 4000 WebSocket handshake, JWT client auth, event intake
CSV Writer Worker — In-memory buffer flush to CSV (500 rows / 2s)

Data Persistence

  • CSV Storage Path: /storage/data/{clientId}/{YYYY-MM-DD}.csv
  • Schema: clientId, eventId, timestamp(ISO), payload(JSON)
  • Rotation Rule: Per-day OR 100MB threshold (whichever triggers first).
  • PostgreSQL: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.

Python SDK

Install the client:

pip install ./sdk-python

Usage:

from datastack_sdk import DataStackClient

client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
    client.sync({"metric": "load", "value": 42})
    client.disconnect()

Running the Platform

# Terminal 1: Web Interface & Admin Console
npm run dev

# Terminal 2: Express Socket.IO Ingestion Engine
npm run server

Production Containers

Docker Compose builds the standalone Next.js web image and the separate Socket.IO ingestion image. Docker Engine with the Compose plugin is required.

  1. Copy .env.example to .env and set unique values for DATASTACK_JWT_SECRET, DATASTACK_ADMIN_SESSION_SECRET, DATASTACK_ADMIN_EMAIL, DATASTACK_ADMIN_PASSWORD, DATABASE_URL, and POSTGRES_PASSWORD. Keep .env private and out of source control. DATABASE_URL must use the postgres service host and a password matching POSTGRES_PASSWORD.
  2. From the repository root, validate and start the stack:
docker compose --env-file .env -f docker/docker-compose.yml config
docker compose --env-file .env -f docker/docker-compose.yml up --build -d

The web console is published on port 3000 and the ingestion service on port 4000. Redis and PostgreSQL are private to the Compose network. CSV files written by the ingestion container persist in the datastack-data named volume; PostgreSQL metadata persists in pgdata.

The current web process still has a separate in-memory buffer and container-local storage view. Redis and Prisma/PostgreSQL are not yet used by application code, and the web process does not yet read the ingestion volume. Cross-process event visibility and shared storage are Phase P4 work; metadata persistence is Phase P5 work. Do not treat this Compose stack as production-ready until those phases and runtime checks are complete.

S
Description
DataStack is a free, open-source, self-hosted real-time data collection and telemetry logging platform developed by Vib Tools.
Readme MIT
4.3 MiB
0 Stars 1 Watchers 0 Forks
Languages
TypeScript 74.1%
JavaScript 11.6%
HTML 7.9%
Python 6.2%
CSS 0.2%