Refresh design token inventory files to reflect the latest UI sizing and layout configurations from the sandbox prototype.
43 lines
1.4 KiB
Markdown
43 lines
1.4 KiB
Markdown
# DataStack — Self-Hosted Universal Realtime Data Collection Platform
|
|
|
|
DataStack is a high-throughput, self-hosted data collection platform designed for high-frequency event ingestion over WebSockets with buffered batch persistence to CSV.
|
|
|
|
## Architecture
|
|
|
|
| Process | Port | Role |
|
|
| :--- | :--- | :--- |
|
|
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
|
|
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
|
|
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) |
|
|
|
|
## Data Persistence
|
|
- **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
|
- **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)`
|
|
- **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first).
|
|
- **PostgreSQL**: Used strictly for SDK client registration and admin settings. Zero collected data is inserted into the database.
|
|
|
|
## Python SDK
|
|
Install the client:
|
|
```bash
|
|
pip install ./sdk-python
|
|
```
|
|
|
|
Usage:
|
|
```python
|
|
from datastack_sdk import DataStackClient
|
|
|
|
client = DataStackClient(server_url="http://localhost:4000", token="JWT_TOKEN")
|
|
if client.connect():
|
|
client.sync({"metric": "load", "value": 42})
|
|
client.disconnect()
|
|
```
|
|
|
|
## Running the Platform
|
|
```bash
|
|
# Terminal 1: Web Interface & Admin Console
|
|
npm run dev
|
|
|
|
# Terminal 2: Express Socket.IO Ingestion Engine
|
|
npm run server
|
|
```
|