chore: publish current DataStack updates
Publish the current application, design, and Phase 1 deployment updates. Excludes ignored personal notes, local MCP config, and generated TypeScript build info.
This commit is contained in:
1 parent
d39d9f92f9
commit
fcc4f4db2d
88 files changed
+21248
-2213
No files matched your search
@@ -4,25 +4,29 @@ DataStack is a high-throughput, self-hosted data collection platform designed fo
|
||||
|
||||
## Architecture
|
||||
|
||||
| Process | Port | Role |
|
||||
| :--- | :--- | :--- |
|
||||
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
|
||||
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
|
||||
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) |
|
||||
| Process | Port | Role |
|
||||
| :---------------------- | :--- | :----------------------------------------------------------------------- |
|
||||
| **Next.js App Router** | 3000 | Frontend UI, `/vcon/*` admin console, streaming CSV export `/api/export` |
|
||||
| **Express + Socket.IO** | 4000 | WebSocket handshake, JWT client auth, event intake |
|
||||
| **CSV Writer Worker** | — | In-memory buffer flush to CSV (`500 rows / 2s`) |
|
||||
|
||||
## Data Persistence
|
||||
|
||||
- **CSV Storage Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
||||
- **Schema**: `clientId, eventId, timestamp(ISO), payload(JSON)`
|
||||
- **Rotation Rule**: Per-day OR 100MB threshold (whichever triggers first).
|
||||
- **PostgreSQL**: Used strictly for SDK client registration and admin settings. Zero collected data is inserted into the database.
|
||||
- **PostgreSQL**: Reserved for SDK client registration and admin settings. Runtime persistence is not wired yet; collected data is never inserted into PostgreSQL.
|
||||
|
||||
## Python SDK
|
||||
|
||||
Install the client:
|
||||
|
||||
```bash
|
||||
pip install ./sdk-python
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```python
|
||||
from datastack_sdk import DataStackClient
|
||||
|
||||
@@ -33,6 +37,7 @@ if client.connect():
|
||||
```
|
||||
|
||||
## Running the Platform
|
||||
|
||||
```bash
|
||||
# Terminal 1: Web Interface & Admin Console
|
||||
npm run dev
|
||||
@@ -40,3 +45,20 @@ npm run dev
|
||||
# Terminal 2: Express Socket.IO Ingestion Engine
|
||||
npm run server
|
||||
```
|
||||
|
||||
## Production Containers
|
||||
|
||||
Docker Compose builds the standalone Next.js web image and the separate Socket.IO ingestion image. Docker Engine with the Compose plugin is required.
|
||||
|
||||
1. Copy `.env.example` to `.env` and set unique values for `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_ADMIN_EMAIL`, `DATASTACK_ADMIN_PASSWORD`, `DATABASE_URL`, and `POSTGRES_PASSWORD`. Keep `.env` private and out of source control.
|
||||
`DATABASE_URL` must use the `postgres` service host and a password matching `POSTGRES_PASSWORD`.
|
||||
2. From the repository root, validate and start the stack:
|
||||
|
||||
```powershell
|
||||
docker compose --env-file .env -f docker/docker-compose.yml config
|
||||
docker compose --env-file .env -f docker/docker-compose.yml up --build -d
|
||||
```
|
||||
|
||||
The web console is published on port 3000 and the ingestion service on port 4000. Redis and PostgreSQL are private to the Compose network. CSV files written by the ingestion container persist in the `datastack-data` named volume; PostgreSQL metadata persists in `pgdata`.
|
||||
|
||||
The current web process still has a separate in-memory buffer and container-local storage view. Redis and Prisma/PostgreSQL are not yet used by application code, and the web process does not yet read the ingestion volume. Cross-process event visibility and shared storage are Phase P4 work; metadata persistence is Phase P5 work. Do not treat this Compose stack as production-ready until those phases and runtime checks are complete.
|
||||
Reference in new issue
Block a user