2026-10-23 11:43:49 -05:00
# DataStack — Self-Hosted Universal Realtime Data Collection Platform
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Architecture
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
| Process | Port | Role |
| :--- | :--- | :--- |
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s` ) |
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Data Persistence
- **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
- **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)`
- **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first).
- **PostgreSQL**: Used strictly for SDK client registration and admin settings. Zero collected data is inserted into the database.
2026-10-23 11:43:09 -05:00
2026-10-23 11:43:49 -05:00
## Python SDK
Install the client:
```bash
pip install ./sdk-python
```
Usage:
```python
from datastack_sdk import DataStackClient
client = DataStackClient ( server_url = "http://localhost:4000" , token = "JWT_TOKEN" )
if client . connect ():
client . sync ({ "metric" : "load" , "value" : 42 })
client . disconnect ()
```
## Running the Platform
```bash
# Terminal 1: Web Interface & Admin Console
npm run dev
# Terminal 2: Express Socket.IO Ingestion Engine
npm run server
```