Publish the current application, design, and Phase 1 deployment updates. Excludes ignored personal notes, local MCP config, and generated TypeScript build info.
3.1 KiB
DataStack — Self-Hosted Universal Realtime Data Collection Platform
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
Architecture
| Process | Port | Role |
|---|---|---|
| Next.js App Router | 3000 | Frontend UI, /vcon/* admin console, streaming CSV export /api/export |
| Express + Socket.IO | 4000 | WebSocket handshake, JWT client auth, event intake |
| CSV Writer Worker | — | In-memory buffer flush to CSV (500 rows / 2s) |
Data Persistence
- CSV Storage Path:
/storage/data/{clientId}/{YYYY-MM-DD}.csv - Schema:
clientId, eventId, timestamp(ISO), payload(JSON) - Rotation Rule: Per-day OR 100MB threshold (whichever triggers first).
- PostgreSQL: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.
Python SDK
Install the client:
pip install ./sdk-python
Usage:
from datastack_sdk import DataStackClient
client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
client.sync({"metric": "load", "value": 42})
client.disconnect()
Running the Platform
# Terminal 1: Web Interface & Admin Console
npm run dev
# Terminal 2: Express Socket.IO Ingestion Engine
npm run server
Production Containers
Docker Compose builds the standalone Next.js web image and the separate Socket.IO ingestion image. Docker Engine with the Compose plugin is required.
- Copy
.env.exampleto.envand set unique values forDATASTACK_JWT_SECRET,DATASTACK_ADMIN_SESSION_SECRET,DATASTACK_ADMIN_EMAIL,DATASTACK_ADMIN_PASSWORD,DATABASE_URL, andPOSTGRES_PASSWORD. Keep.envprivate and out of source control.DATABASE_URLmust use thepostgresservice host and a password matchingPOSTGRES_PASSWORD. - From the repository root, validate and start the stack:
docker compose --env-file .env -f docker/docker-compose.yml config
docker compose --env-file .env -f docker/docker-compose.yml up --build -d
The web console is published on port 3000 and the ingestion service on port 4000. Redis and PostgreSQL are private to the Compose network. CSV files written by the ingestion container persist in the datastack-data named volume; PostgreSQL metadata persists in pgdata.
The current web process still has a separate in-memory buffer and container-local storage view. Redis and Prisma/PostgreSQL are not yet used by application code, and the web process does not yet read the ingestion volume. Cross-process event visibility and shared storage are Phase P4 work; metadata persistence is Phase P5 work. Do not treat this Compose stack as production-ready until those phases and runtime checks are complete.