Files
DataStack/README.md
T
i vib fcc4f4db2d chore: publish current DataStack updates
Publish the current application, design, and Phase 1 deployment updates. Excludes ignored personal notes, local MCP config, and generated TypeScript build info.
2026-10-23 19:56:38 -07:00

65 lines
3.1 KiB
Markdown

# DataStack — Self-Hosted Universal Realtime Data Collection Platform
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
## Architecture
| Process | Port | Role |
| :---------------------- | :--- | :----------------------------------------------------------------------- |
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) |
## Data Persistence
- **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
- **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)`
- **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first).
- **PostgreSQL**: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.
## Python SDK
Install the client:
```bash
pip install ./sdk-python
```
Usage:
```python
from datastack_sdk import DataStackClient
client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
if client.connect():
client.sync({"metric": "load", "value": 42})
client.disconnect()
```
## Running the Platform
```bash
# Terminal 1: Web Interface & Admin Console
npm run dev
# Terminal 2: Express Socket.IO Ingestion Engine
npm run server
```
## Production Containers
Docker Compose builds the standalone Next.js web image and the separate Socket.IO ingestion image. Docker Engine with the Compose plugin is required.
1. Copy `.env.example` to `.env` and set unique values for `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_ADMIN_EMAIL`, `DATASTACK_ADMIN_PASSWORD`, `DATABASE_URL`, and `POSTGRES_PASSWORD`. Keep `.env` private and out of source control.
`DATABASE_URL` must use the `postgres` service host and a password matching `POSTGRES_PASSWORD`.
2. From the repository root, validate and start the stack:
```powershell
docker compose --env-file .env -f docker/docker-compose.yml config
docker compose --env-file .env -f docker/docker-compose.yml up --build -d
```
The web console is published on port 3000 and the ingestion service on port 4000. Redis and PostgreSQL are private to the Compose network. CSV files written by the ingestion container persist in the `datastack-data` named volume; PostgreSQL metadata persists in `pgdata`.
The current web process still has a separate in-memory buffer and container-local storage view. Redis and Prisma/PostgreSQL are not yet used by application code, and the web process does not yet read the ingestion volume. Cross-process event visibility and shared storage are Phase P4 work; metadata persistence is Phase P5 work. Do not treat this Compose stack as production-ready until those phases and runtime checks are complete.