DataStack — Self-Hosted Universal Realtime Data Collection Platform
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
Architecture
| Process | Port | Role |
|---|---|---|
| Next.js App Router | 3000 | Frontend UI, /vcon/* admin console, streaming CSV export /api/export |
| Express + Socket.IO | 4000 | WebSocket handshake, JWT client auth, event intake |
| CSV Writer Worker | — | In-memory buffer flush to CSV (500 rows / 2s) |
Data Persistence
- CSV Storage Path:
/storage/data/{clientId}/{YYYY-MM-DD}.csv - Schema:
clientId, eventId, timestamp(ISO), payload(JSON) - Rotation Rule: Per-day OR 100MB threshold (whichever triggers first).
- PostgreSQL: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.
Python SDK
Install the client:
pip install ./sdk-python
Usage:
from datastack_sdk import DataStackClient
client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
client.sync({"metric": "load", "value": 42})
client.disconnect()
Running the Platform
# Terminal 1: Web Interface & Admin Console
npm run dev
# Terminal 2: Express Socket.IO Ingestion Engine
npm run server
Production Containers
Coolify Deployment
The repository includes a Docker Compose stack for a Coolify Git-based Docker Compose application. Coolify must be installed on a VPS with a domain and HTTPS configured.
- In Coolify, add a Docker Compose application from
https://gitlab.com/ivibtools/datastack.git, branchmain. - Set Base Directory to
/dockerand Docker Compose Location to/docker/docker-compose.yml. Confirm Coolify resolves the Compose file to the repository'sdocker/docker-compose.yml; do not change the Compose file path when changing the Base Directory. - Set
DATASTACK_ADMIN_EMAILin the application's Environment Variables. Coolify auto-generates the database name/password, JWT/session secrets, and one-time setup key from theSERVICE_*variables in Compose.DATABASE_URLis assembled internally from the generated database name/password. Do not override generated database identity values after first deployment. - In Domains, add the public HTTPS domain to service
proxyon internal port80. Do not exposeweb,ingestion, Redis, or PostgreSQL directly. Coolify supplies proxy labels for the selected service. - Deploy and check service health and runtime logs in Coolify.
The proxy sends / to Next.js and /socket.io/ to the Socket.IO service over the private Compose network. The Settings page uses the current origin for a deployed WebSocket URL; local development on port 3000 still displays port 4000.
After deployment, open the first-run setup page. Copy the generated SERVICE_REALBASE64_64_ADMININSTALL key from Coolify into the setup form, confirm the DATASTACK_ADMIN_EMAIL configured for this deployment, and choose a strong password. The server accepts only that configured email, and initial admin setup closes permanently after the first account is stored.
Compose mounts PostgreSQL's pgdata named volume, which is intended to survive normal rebuilds/redeploys; never remove the volume or change the generated database name to “reset” the app. Back up pgdata before database upgrades or maintenance. Changing the PostgreSQL password also requires updating the existing database role, not just its environment variable. The setup/login code uses PostgreSQL for the bootstrap admin credential only; Prisma-backed client/settings metadata remains P5 work. The datastack-data volume persists ingestion CSVs, but the web process does not yet read that volume; cross-process event visibility/shared storage remains P4. This stack has not yet been built or deployed on a Coolify VPS from this environment, so volume persistence still needs target-runtime verification.