Files
DataStack/README.md
T
2026-10-25 06:12:47 -07:00

4.1 KiB

DataStack — Self-Hosted Universal Realtime Data Collection Platform

DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.

Architecture

Process Port Role
Next.js App Router 3000 Frontend UI, /vcon/* admin console, streaming CSV export /api/export
Express + Socket.IO 4000 WebSocket handshake, JWT client auth, event intake
CSV Writer Worker — In-memory buffer flush to CSV (500 rows / 2s)

Data Persistence

  • CSV Storage Path: /storage/data/{clientId}/{YYYY-MM-DD}.csv
  • Schema: clientId, eventId, timestamp(ISO), payload(JSON)
  • Rotation Rule: Per-day OR 100MB threshold (whichever triggers first).
  • PostgreSQL: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.

Python SDK

Install the client:

pip install ./sdk-python

Usage:

from datastack_sdk import DataStackClient

client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
    client.sync({"metric": "load", "value": 42})
    client.disconnect()

Running the Platform

# Terminal 1: Web Interface & Admin Console
npm run dev

# Terminal 2: Express Socket.IO Ingestion Engine
npm run server

Production Containers

Coolify Deployment

The repository includes a Docker Compose stack for a Coolify Git-based Docker Compose application. Coolify must be installed on a VPS with a domain and HTTPS configured.

  1. In Coolify, add a Docker Compose application from https://gitlab.com/ivibtools/datastack.git, branch main.
  2. Set Base Directory to / and Docker Compose Location to docker/docker-compose.yml.
  3. Set DATASTACK_ADMIN_EMAIL in the application's Environment Variables. Coolify auto-generates the database name/password, JWT/session secrets, and one-time setup key from the SERVICE_* variables in Compose. DATABASE_URL is assembled internally from the generated database name/password. Do not override generated database identity values after first deployment.
  4. In Domains, add the public HTTPS domain to service proxy on internal port 80. Do not expose web, ingestion, Redis, or PostgreSQL directly. Coolify supplies proxy labels for the selected service.
  5. Deploy and check service health and runtime logs in Coolify.

The proxy sends / to Next.js and /socket.io/ to the Socket.IO service over the private Compose network. The Settings page uses the current origin for a deployed WebSocket URL; local development on port 3000 still displays port 4000.

After deployment, open the first-run setup page. Copy the generated SERVICE_REALBASE64_64_ADMININSTALL key from Coolify into the setup form, confirm the DATASTACK_ADMIN_EMAIL configured for this deployment, and choose a strong password. The server accepts only that configured email, and initial admin setup closes permanently after the first account is stored.

Compose mounts PostgreSQL's pgdata named volume, which is intended to survive normal rebuilds/redeploys; never remove the volume or change the generated database name to “reset” the app. Back up pgdata before database upgrades or maintenance. Changing the PostgreSQL password also requires updating the existing database role, not just its environment variable. The setup/login code uses PostgreSQL for the bootstrap admin credential only; Prisma-backed client/settings metadata remains P5 work. The datastack-data volume persists ingestion CSVs, but the web process does not yet read that volume; cross-process event visibility/shared storage remains P4. This stack has not yet been built or deployed on a Coolify VPS from this environment, so volume persistence still needs target-runtime verification.