# DataStack — Self-Hosted Universal Realtime Data Collection Platform DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV. ## Architecture | Process | Port | Role | | :---------------------- | :--- | :----------------------------------------------------------------------- | | **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` | | **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake | | **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) | ## Data Persistence - **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv` - **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)` - **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first). - **PostgreSQL**: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL. ## Python SDK Install the client: ```bash pip install ./sdk-python ``` Usage: ```python from datastack_sdk import DataStackClient client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN") if client.connect(): client.sync({"metric": "load", "value": 42}) client.disconnect() ``` ## Running the Platform ```bash # Terminal 1: Web Interface & Admin Console npm run dev # Terminal 2: Express Socket.IO Ingestion Engine npm run server ``` ## Production Containers ## Coolify Deployment The repository includes a Docker Compose stack for a Coolify Git-based Docker Compose application. Coolify must be installed on a VPS with a domain and HTTPS configured. 1. In Coolify, add a Docker Compose application from `https://gitlab.com/ivibtools/datastack.git`, branch `main`. 2. Set Base Directory to `/` and Docker Compose Location to `docker/docker-compose.yml`. 3. Set `DATASTACK_ADMIN_EMAIL` in the application's Environment Variables. Coolify auto-generates the database name/password, JWT/session secrets, and one-time setup key from the `SERVICE_*` variables in Compose. `DATABASE_URL` is assembled internally from the generated database name/password. Do not override generated database identity values after first deployment. 4. In Domains, add the public HTTPS domain to service `proxy` on internal port `80`. Do not expose `web`, `ingestion`, Redis, or PostgreSQL directly. Coolify supplies proxy labels for the selected service. 5. Deploy and check service health and runtime logs in Coolify. The proxy sends `/` to Next.js and `/socket.io/` to the Socket.IO service over the private Compose network. The Settings page uses the current origin for a deployed WebSocket URL; local development on port 3000 still displays port 4000. After deployment, open the first-run setup page. Copy the generated `SERVICE_REALBASE64_64_ADMININSTALL` key from Coolify into the setup form, confirm the `DATASTACK_ADMIN_EMAIL` configured for this deployment, and choose a strong password. The server accepts only that configured email, and initial admin setup closes permanently after the first account is stored. Compose mounts PostgreSQL's `pgdata` named volume, which is intended to survive normal rebuilds/redeploys; never remove the volume or change the generated database name to “reset” the app. Back up `pgdata` before database upgrades or maintenance. Changing the PostgreSQL password also requires updating the existing database role, not just its environment variable. The setup/login code uses PostgreSQL for the bootstrap admin credential only; Prisma-backed client/settings metadata remains P5 work. The `datastack-data` volume persists ingestion CSVs, but the web process does not yet read that volume; cross-process event visibility/shared storage remains P4. This stack has not yet been built or deployed on a Coolify VPS from this environment, so volume persistence still needs target-runtime verification.