Files
DataStack/README.md
T

63 lines
3.4 KiB
Markdown
Raw Normal View History

2026-10-23 11:43:49 -05:00
# DataStack — Self-Hosted Universal Realtime Data Collection Platform
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Architecture
2026-10-23 11:43:09 -05:00
2026-10-23 19:56:38 -07:00
| Process | Port | Role |
| :---------------------- | :--- | :----------------------------------------------------------------------- |
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) |
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Data Persistence
2026-10-23 19:56:38 -07:00
2026-10-23 11:43:49 -05:00
- **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
- **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)`
- **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first).
2026-10-23 19:56:38 -07:00
- **PostgreSQL**: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Python SDK
2026-10-23 19:56:38 -07:00
2026-10-23 11:43:49 -05:00
Install the client:
2026-10-23 19:56:38 -07:00
2026-10-23 11:43:49 -05:00
```bash
pip install ./sdk-python
```
Usage:
2026-10-23 19:56:38 -07:00
2026-10-23 11:43:49 -05:00
```python
from datastack_sdk import DataStackClient
client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
client.sync({"metric": "load", "value": 42})
client.disconnect()
```
## Running the Platform
2026-10-23 19:56:38 -07:00
2026-10-23 11:43:49 -05:00
```bash
# Terminal 1: Web Interface & Admin Console
npm run dev
# Terminal 2: Express Socket.IO Ingestion Engine
npm run server
```
2026-10-23 19:56:38 -07:00
## Production Containers
2026-10-23 20:53:34 -07:00
## Coolify Deployment
2026-10-23 19:56:38 -07:00
2026-10-23 20:53:34 -07:00
The repository includes a Docker Compose stack for a Coolify Git-based Docker Compose application. Coolify must be installed on a VPS with a domain and HTTPS configured.
2026-10-23 19:56:38 -07:00
2026-10-23 20:53:34 -07:00
1. In Coolify, add a Docker Compose application from `https://gitlab.com/ivibtools/datastack.git`, branch `main`.
2. Set Base Directory to `/` and Docker Compose Location to `docker/docker-compose.yml`.
3. Add the required variables in the Coolify application's Environment Variables: `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_ADMIN_EMAIL`, `DATASTACK_ADMIN_PASSWORD`, `DATABASE_URL`, and `POSTGRES_PASSWORD`. Use unique secrets; `DATABASE_URL` must use host `postgres`, database `datastack_db`, user `datastack`, and a password matching `POSTGRES_PASSWORD`.
4. In Domains, add the public HTTPS domain to service `proxy` on internal port `80`. Do not expose `web`, `ingestion`, Redis, or PostgreSQL directly. Coolify supplies proxy labels for the selected service.
5. Deploy and check service health and runtime logs in Coolify.
2026-10-23 19:56:38 -07:00
2026-10-23 20:53:34 -07:00
The proxy sends `/` to Next.js and `/socket.io/` to the Socket.IO service over the private Compose network. The Settings page uses the current origin for a deployed WebSocket URL; local development on port 3000 still displays port 4000.
2026-10-23 19:56:38 -07:00
2026-10-23 20:53:34 -07:00
The `datastack-data` named volume persists CSVs written by ingestion. The web process does not yet read that volume; Redis and Prisma/PostgreSQL are not yet used by application code. Cross-process event visibility/shared storage remains Phase P4 work and metadata persistence remains Phase P5 work. Coolify deployment does not resolve those production-readiness gaps, and this stack has not yet been built or deployed on a Coolify VPS from this environment.