feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
@@ -0,0 +1,289 @@
|
||||
# API Reference Specification
|
||||
|
||||
This document provides the authoritative API specification for all DataStack REST HTTP endpoints (`app/api/*`) and real-time Socket.IO WebSocket protocol events (`server/socket/handlers.js`).
|
||||
|
||||
---
|
||||
|
||||
## 1. REST API Specification (Port 3000)
|
||||
|
||||
### 1.1 Ingestion Endpoints
|
||||
|
||||
#### `POST /api/events`
|
||||
Ingests telemetry events over authenticated HTTP.
|
||||
|
||||
- **Authentication**: `Authorization: Bearer <CLIENT_JWT_TOKEN>`
|
||||
- **Content-Type**: `application/json`
|
||||
- **Supported Payload Shapes**:
|
||||
|
||||
##### Single Event Mode
|
||||
```json
|
||||
{
|
||||
"eventId": "evt_custom_12345",
|
||||
"payload": {
|
||||
"temperature": 23.4,
|
||||
"status": "normal"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
##### Batch Array Mode
|
||||
```json
|
||||
{
|
||||
"batch": [
|
||||
{ "eventId": "evt_1", "payload": { "val": 10 } },
|
||||
{ "eventId": "evt_2", "payload": { "val": 20 } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
##### File / Structured Record Mode
|
||||
```json
|
||||
{
|
||||
"fileName": "sensor_logs.csv",
|
||||
"records": [
|
||||
{ "val": 10 },
|
||||
{ "val": 20 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **Responses**:
|
||||
- `200 OK`:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"accepted": 1,
|
||||
"buffered": true
|
||||
}
|
||||
```
|
||||
- `401 Unauthorized`: Missing or invalid client JWT.
|
||||
- `403 Forbidden`: Client status is `revoked` or client ID is recorded in `deleted_clients`.
|
||||
- `413 Payload Too Large`: Total payload exceeds `maxPayloadSizeKb`.
|
||||
- `429 Too Many Requests`: Client or global rate limit exceeded.
|
||||
|
||||
---
|
||||
|
||||
#### `POST /api/events/flush`
|
||||
Forces an immediate write of all buffered RAM and Redis events to disk.
|
||||
|
||||
- **Authentication**: Operator Session Cookie (`datastack_admin_session`)
|
||||
- **Responses**:
|
||||
- `200 OK`:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"flushed": 150
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Export Endpoints
|
||||
|
||||
#### `GET /api/export`
|
||||
Streams filtered CSV or JSON records directly from disk partitions to the caller.
|
||||
|
||||
- **Authentication**: Operator Session Cookie (`datastack_admin_session`)
|
||||
- **Query Parameters**:
|
||||
- `clientId` (string, optional): Specific client identifier or `all` (default: `all`).
|
||||
- `from` (string, optional): Inclusive start date filter (`YYYY-MM-DD`).
|
||||
- `to` (string, optional): Inclusive end date filter (`YYYY-MM-DD`).
|
||||
- `format` (string, optional): `csv` (default) or `json`.
|
||||
- `includeHeader` (boolean, optional): Whether to output CSV headers (default: `true`).
|
||||
- `flush` (boolean, optional): Whether to flush pending memory/Redis buffers before streaming (default: `true`).
|
||||
- `filename` (string, optional): Specific single partition filename (e.g., `2026-11-17.csv`).
|
||||
- **Response Headers**:
|
||||
- `Content-Type`: `text/csv; charset=utf-8` or `application/json; charset=utf-8`
|
||||
- `Content-Disposition`: `attachment; filename="datastack_export_YYYY-MM-DD.csv"`
|
||||
- `Cache-Control`: `no-store`
|
||||
- `Transfer-Encoding`: `chunked`
|
||||
|
||||
---
|
||||
|
||||
### 1.3 System Health & Diagnostics
|
||||
|
||||
#### `GET /api/health`
|
||||
Returns process uptime and database connection status.
|
||||
|
||||
- **Authentication**: None (Public)
|
||||
- **Responses**:
|
||||
- `200 OK`:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"uptime": 12450.2,
|
||||
"timestamp": "2026-11-17T12:00:00.000Z",
|
||||
"database": "connected"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Operator Authentication Endpoints
|
||||
|
||||
#### `POST /api/auth/login`
|
||||
Authenticates administrator credentials and establishes session cookie.
|
||||
|
||||
- **Request Body**:
|
||||
```json
|
||||
{
|
||||
"email": "admin@example.com",
|
||||
"password": "strongPassword123!"
|
||||
}
|
||||
```
|
||||
- **Responses**:
|
||||
- `200 OK`: Sets `datastack_admin_session` cookie (HttpOnly, SameSite=Lax, Secure in production).
|
||||
- `401 Unauthorized`: Invalid credentials.
|
||||
- `429 Too Many Requests`: IP temporarily blocked due to 5 failed attempts in 15 minutes.
|
||||
|
||||
---
|
||||
|
||||
#### `POST /api/auth/logout`
|
||||
Terminates the operator session and clears the cookie.
|
||||
|
||||
---
|
||||
|
||||
#### `GET /api/auth/me`
|
||||
Returns current authenticated administrator profile.
|
||||
|
||||
- **Responses**:
|
||||
- `200 OK`: `{"authenticated": true, "user": {"id": "...", "email": "..."}}`
|
||||
- `401 Unauthorized`: `{"authenticated": false}`
|
||||
|
||||
---
|
||||
|
||||
#### `POST /api/auth/setup`
|
||||
Completes first-time installation wizard.
|
||||
|
||||
- **Request Body**:
|
||||
```json
|
||||
{
|
||||
"setupKey": "CONFIGURED_SETUP_KEY",
|
||||
"email": "admin@example.com",
|
||||
"password": "strongPassword123!"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `POST /api/auth/change-password`
|
||||
Updates operator password and increments `tokenVersion`.
|
||||
|
||||
- **Request Body**:
|
||||
```json
|
||||
{
|
||||
"currentPassword": "oldPassword",
|
||||
"newPassword": "newPassword123!"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.5 Client Registry Endpoints
|
||||
|
||||
#### `GET /api/clients`
|
||||
Lists all registered SDK clients and rate limits.
|
||||
|
||||
#### `POST /api/clients`
|
||||
Registers a new SDK client application.
|
||||
- **Request Body**:
|
||||
```json
|
||||
{
|
||||
"clientId": "pos_terminal_01",
|
||||
"name": "Point of Sale 1",
|
||||
"rateLimit": 100
|
||||
}
|
||||
```
|
||||
|
||||
#### `PUT /api/clients/[id]`
|
||||
Updates client rate limits, rotates secret tokens, or revokes access.
|
||||
|
||||
#### `DELETE /api/clients/[id]`
|
||||
Deletes client from registry, moves ID to `deleted_clients` tombstone, and optionally purges storage files.
|
||||
|
||||
---
|
||||
|
||||
### 1.6 Settings Endpoints
|
||||
|
||||
#### `GET /api/settings`
|
||||
Returns public platform runtime limits and buffer configurations.
|
||||
|
||||
#### `PUT /api/settings`
|
||||
Updates platform limits and notifies Port 4000 ingestion service.
|
||||
|
||||
---
|
||||
|
||||
## 2. Socket.IO Real-Time Protocol Specification (Port 4000)
|
||||
|
||||
### Handshake Authentication
|
||||
Clients connect via Socket.IO v4 client with a valid JWT token passed in the `auth` object or handshake `query`:
|
||||
|
||||
```javascript
|
||||
const socket = io("https://your-domain.example", {
|
||||
path: "/socket.io/",
|
||||
auth: {
|
||||
token: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
If the token is invalid, expired, revoked, or belongs to a tombstone client, the handshake is rejected immediately with an authentication error.
|
||||
|
||||
---
|
||||
|
||||
### Socket Events & Messages
|
||||
|
||||
#### Event: `sync_data`
|
||||
Transmits a single telemetry event.
|
||||
|
||||
- **Message Body**:
|
||||
```json
|
||||
{
|
||||
"eventId": "evt_client_9812",
|
||||
"payload": {
|
||||
"metric": "cpu",
|
||||
"value": 42.1
|
||||
}
|
||||
}
|
||||
```
|
||||
- **Acknowledgment Callback**:
|
||||
- Success: `{"status": "ok", "eventId": "evt_client_9812"}`
|
||||
- Error: `{"error": "Rate limit exceeded", "code": 429}`
|
||||
|
||||
---
|
||||
|
||||
#### Event: `sync_batch`
|
||||
Transmits an array of events in a single payload.
|
||||
|
||||
- **Message Body**:
|
||||
```json
|
||||
{
|
||||
"events": [
|
||||
{ "eventId": "evt_1", "payload": { "val": 1 } },
|
||||
{ "eventId": "evt_2", "payload": { "val": 2 } }
|
||||
]
|
||||
}
|
||||
```
|
||||
- **Acknowledgment Callback**:
|
||||
- Success: `{"status": "ok", "acceptedCount": 2}`
|
||||
|
||||
---
|
||||
|
||||
#### Event: `heartbeat`
|
||||
Keep-alive ping to measure latency and maintain connection health.
|
||||
|
||||
- **Message Body**: `{"timestamp": 1700000000000}`
|
||||
- **Acknowledgment Callback**: `{"status": "ok", "pong": true, "serverTime": 1700000000010}`
|
||||
|
||||
---
|
||||
|
||||
## 3. Internal Service Bridge Specification (`/internal/*`)
|
||||
|
||||
All internal endpoints on Port 4000 require the header:
|
||||
`x-datastack-internal-secret: <DATASTACK_INTERNAL_SECRET>`
|
||||
|
||||
- `GET /internal/metrics`: Returns connected socket count, active buffer queue depth, and memory stats.
|
||||
- `POST /internal/settings-sync`: Pushes updated limits and secrets to Port 4000.
|
||||
- `POST /internal/client-sync`: Pushes rotated, revoked, or deleted client updates.
|
||||
- `POST /internal/flush`: Dispatches synchronous buffer flush to disk.
|
||||
- `GET /internal/recent`: Fetches latest unwritten queue items for dashboard live activity streams.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Building, Packaging & Production Containers
|
||||
|
||||
This guide outlines compilation commands, linter checks, test suite execution, and Docker container packaging for DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Production Web Compilation
|
||||
|
||||
DataStack uses Next.js 15 App Router configured for **Standalone Output** (`output: "standalone"` in `next.config.ts`), which bundles only the required Node.js runtime files into `.next/standalone`.
|
||||
|
||||
### Compiling the Web App
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Build Verification Checklist
|
||||
- Compiles TypeScript without type errors (`tsc --noEmit`).
|
||||
- Generates serverless routes and static assets under `.next/`.
|
||||
- Prepares standalone bundle under `.next/standalone/server.js`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Code Quality & Linting
|
||||
|
||||
DataStack uses ESLint 9 with `eslint-config-next`:
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
```
|
||||
|
||||
Ensure this command exits cleanly (`code: 0`) with zero warnings or errors prior to committing code or opening a pull request.
|
||||
|
||||
---
|
||||
|
||||
## 3. Docker Container Builds
|
||||
|
||||
DataStack provides optimized, multi-stage Dockerfiles under `docker/`:
|
||||
|
||||
### 1. Build the Web Container (Port 3000)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.web -t datastack-web:latest .
|
||||
```
|
||||
- **Base**: `node:22-alpine`.
|
||||
- **Stage 1 (Builder)**: Installs dependencies deterministically (`npm ci`) and runs `npm run build`.
|
||||
- **Stage 2 (Runner)**: Copies `.next/standalone`, public assets, and sets non-root user `nextjs` (UID 1001).
|
||||
|
||||
### 2. Build the Ingestion Container (Port 4000)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.server -t datastack-ingestion:latest .
|
||||
```
|
||||
- **Base**: `node:22-alpine`.
|
||||
- Installs production dependencies and runs `node server/index.js` under a dedicated non-root user.
|
||||
|
||||
### 3. Build the Nginx Proxy Container (Port 80)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.proxy -t datastack-proxy:latest .
|
||||
```
|
||||
- **Base**: `nginx:1.27-alpine`.
|
||||
- Mounts custom reverse proxy configuration (`nginx/nginx.conf`) routing `/` to Port 3000 and `/socket.io/` to Port 4000.
|
||||
|
||||
### 4. Build & Run the Entire Stack with Compose
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml build
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
Verify container health:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml ps
|
||||
```
|
||||
All containers (`web`, `ingestion`, `proxy`, `postgres`, `redis`) should display `(healthy)`.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Codebase Structure & Directory Inventory
|
||||
|
||||
This document provides a comprehensive, file-by-file architectural inventory of the DataStack codebase, detailing module responsibilities, runtime boundaries, and inter-module dependencies.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Directory Overview
|
||||
|
||||
```text
|
||||
/
|
||||
├── app/ # Next.js 15 App Router: UI pages, layouts, and REST API routes (Port 3000)
|
||||
├── components/ # Reusable UI primitives and layout shells
|
||||
├── core/ # Shared business logic, database connectors, and security engines
|
||||
├── server/ # Dedicated Express 5 + Socket.IO 4 Ingestion Server (Port 4000)
|
||||
├── sdk/ # Official client SDKs (Python, with TypeScript SDK planned)
|
||||
├── docker/ # Production multi-stage Dockerfiles and Compose configurations
|
||||
├── nginx/ # Nginx reverse proxy routing and health check configuration
|
||||
├── prisma/ # Schema mirror documenting PostgreSQL control-plane tables
|
||||
├── public/ # Static assets, PWA icons, and manifests
|
||||
├── storage/ # Mounted partition volume for daily CSV files (/storage/data/)
|
||||
├── tests/ # Automated TypeScript integration and Python SDK test suites
|
||||
└── docs/ # Authoritative documentation suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Web Application & REST API (`app/`)
|
||||
|
||||
Operating on **Port 3000**, this Next.js 15 App Router subsystem handles operator authentication, dashboard rendering, authenticated HTTP ingestion, and streaming dataset exports.
|
||||
|
||||
### Core Entry Points
|
||||
- `app/layout.tsx`: Root HTML layout importing centralized CSS tokens, fonts, and PWA registration.
|
||||
- `app/page.tsx`: Public marketing and documentation overview page.
|
||||
- `app/middleware.ts`: Next.js Edge middleware. Enforces signed HMAC-SHA256 session cookies on `/vcon/*` routes (except `/vcon/login` and `/vcon/setup`) and validates CSRF `Origin`/`Referer` headers against `Host` and `X-Forwarded-Host`.
|
||||
- `app/globals.css`: Global design tokens, CSS variables, and zero-pill border radius resets.
|
||||
- `app/manifest.ts`: Web App Manifest generator for PWA installation.
|
||||
- `app/sw.ts`: Progressive Web App Service Worker script.
|
||||
|
||||
### Administration Console Pages (`app/vcon/`)
|
||||
- `app/vcon/login/page.tsx`: Operator sign-in page with brute-force rate-limiting feedback.
|
||||
- `app/vcon/setup/page.tsx`: First-time setup wizard accepting `DATASTACK_ADMIN_SETUP_KEY`.
|
||||
- `app/vcon/dashboard/page.tsx`: Primary operational dashboard displaying dual-runtime health, disk utilization, buffer depth, and recent live telemetry ticker.
|
||||
- `app/vcon/clients/page.tsx`: Client SDK registry for generating tokens, rotating secrets, and revoking clients.
|
||||
- `app/vcon/simulator/page.tsx`: Interactive in-browser telemetry generator with single and burst dispatch.
|
||||
- `app/vcon/storage/page.tsx`: CSV partition inspector and manual buffer flush controls.
|
||||
- `app/vcon/export/page.tsx`: Filtered streaming CSV and JSON export workspace.
|
||||
- `app/vcon/settings/page.tsx`: Platform runtime settings manager (rate limits, flush intervals, rotation size).
|
||||
- `app/vcon/profile/page.tsx`: Administrator credentials, password updates, and session revocation.
|
||||
|
||||
### REST API Route Handlers (`app/api/`)
|
||||
- `app/api/events/route.ts`: Authenticated HTTP ingestion endpoint. Accepts single events, `{ batch: [...] }`, or structured file imports (`{ fileName, records: [...] }`).
|
||||
- `app/api/events/flush/route.ts`: Forces immediate buffer flushes across both Port 3000 and Port 4000.
|
||||
- `app/api/export/route.ts`: Streams filtered CSV or JSON datasets directly from disk to the client response.
|
||||
- `app/api/health/route.ts`: Health check endpoint reporting process uptime and database status.
|
||||
- `app/api/auth/login/route.ts`: Authenticates admin email and password with `crypto.scrypt` verification and composite IP brute-force tracking.
|
||||
- `app/api/auth/logout/route.ts`: Clears session cookie and invalidates session token.
|
||||
- `app/api/auth/me/route.ts`: Returns current operator identity and session validity.
|
||||
- `app/api/auth/setup/route.ts`: Executes initial admin setup with setup key verification.
|
||||
- `app/api/auth/change-password/route.ts`: Updates operator password and increments `tokenVersion`.
|
||||
- `app/api/clients/route.ts`: Registers new SDK clients and lists existing client credentials.
|
||||
- `app/api/clients/[id]/route.ts`: Updates client rate limits, rotates secrets, revokes, or permanently deletes clients.
|
||||
- `app/api/settings/route.ts`: Reads and updates platform runtime settings with database persistence.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core Business Logic Singletons (`core/`)
|
||||
|
||||
The `core/` directory contains framework-agnostic TypeScript singletons shared between the Web application and ingestion workers.
|
||||
|
||||
| Module | Primary Class / Functions | Responsibility |
|
||||
| :--- | :--- | :--- |
|
||||
| `core/Database/database.ts` | `DatabaseEngine` | Manages raw `pg.Pool` PostgreSQL connections, executes non-destructive DDL migrations, and provides an in-memory mock store when `DATABASE_URL` is omitted. |
|
||||
| `core/auth/adminAuth.ts` | `AdminAuthService` | Enforces `crypto.scrypt` password hashing (64-byte key, 16-byte salt), constant-time string comparison, and composite IP + email failed attempt lockout. |
|
||||
| `core/auth/jwt.ts` | `AuthService` | Generates and verifies HMAC-SHA256 JWT tokens for SDK clients and WebCrypto HMAC-SHA256 session tokens for administrators. |
|
||||
| `core/Security/security.ts` | `SecurityEngine` | Implements sliding-window rate limiters, 5-minute idempotency caches, payload size guards, and timing-safe internal secret authentication. |
|
||||
| `core/data/buffer.ts` | `DataBufferEngine` | Manages in-memory RAM event queues and optional Redis list queues (`datastack:buffer:events`), with threshold and interval auto-flushing. |
|
||||
| `core/Storage/csvStorage.ts` | `CSVStorageManager` | Handles RFC 4180 CSV serialization, client/date directory partitioning, 100MB file rotation (`-part2.csv`), cooperative `.lock` file handling, and automated retention pruning. |
|
||||
| `core/sdk/sdkManager.ts` | `SDKManager` | Coordinates client CRUD operations, credential rotation, master secret re-signing, and tombstone checks. |
|
||||
| `core/adminSettings/settingsManager.ts` | `SettingsManager` | Manages system settings state, synchronizes with PostgreSQL, and notifies Port 4000 via `/internal/settings-sync`. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Ingestion Server Subsystem (`server/`)
|
||||
|
||||
Operating on **Port 4000**, this dedicated Express 5 + Socket.IO 4 service handles high-throughput, persistent WebSocket connections.
|
||||
|
||||
- `server/index.js`: Express application entry point. Listens on `PORT_SOCKET` (4000), mounts internal service bridge routes (`/internal/*`), and binds the Socket.IO server.
|
||||
- `server/socket/handlers.js`: Enforces JWT authentication during the Socket.IO handshake (`io.use()`), handles incoming telemetry events (`sync_data`, `sync_batch`), applies per-client rate limits, deduplicates idempotency envelopes, and supports administrative socket termination.
|
||||
- `server/workers/csvWriter.js`: Background flusher worker (`CsvWriterWorker`). Periodically polls the buffer queue and flushes batches to partitioned CSV files using cooperative file locks.
|
||||
|
||||
---
|
||||
|
||||
## 5. Client SDK Subsystem (`sdk/`)
|
||||
|
||||
- `sdk/python/datastack/`: Official Python Client Library.
|
||||
- `client.py`: Provides `DataStackClient` supporting WebSocket streaming (`connect()`), automatic fallback to HTTP (`send_event()`), offline disk spooling (`.datastack_spool/`), and batch file streaming (`stream_csv()`, `stream_json()`).
|
||||
- `__init__.py`: Package entry point exposing `DataStackClient`.
|
||||
- `pyproject.toml` & `setup.py`: Packaging specifications.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Database Schema & Migrations
|
||||
|
||||
This document specifies the PostgreSQL control-plane database schema, table definitions, constraints, and non-destructive additive migration architecture.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Boundary
|
||||
|
||||
> **IMPORTANT ARCHITECTURAL RULE**: PostgreSQL is used **strictly for control-plane metadata** (admin accounts, login throttling, client tokens, and runtime settings).
|
||||
>
|
||||
> **Zero Telemetry Writes**: Telemetry events are never written to PostgreSQL. All event data is buffered and persisted directly to daily CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Table Specifications
|
||||
|
||||
### 2.1 Table: `admin_users`
|
||||
Stores single-tenant administrator credentials and session token versions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS admin_users (
|
||||
id SMALLINT PRIMARY KEY DEFAULT 1 CHECK (id = 1),
|
||||
email TEXT NOT NULL UNIQUE,
|
||||
password_salt BYTEA NOT NULL,
|
||||
password_hash BYTEA NOT NULL,
|
||||
token_version INTEGER NOT NULL DEFAULT 1,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
- **`id`**: Enforces a single administrator row constraint (`CHECK (id = 1)`).
|
||||
- **`email`**: Unique administrator contact and login email.
|
||||
- **`password_salt`**: Cryptographically secure 16-byte random salt (`crypto.randomBytes(16)`).
|
||||
- **`password_hash`**: 64-byte scrypt derived hash (`crypto.scryptSync(password, salt, 64)`).
|
||||
- **`token_version`**: Integer counter embedded into session cookies. Incrementing this immediately invalidates all active sessions globally.
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Table: `admin_login_attempts`
|
||||
Tracks failed login attempts for brute-force protection.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS admin_login_attempts (
|
||||
key_hash TEXT PRIMARY KEY,
|
||||
window_started_at TIMESTAMPTZ NOT NULL,
|
||||
failed_attempts INTEGER NOT NULL,
|
||||
blocked_until TIMESTAMPTZ,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
- **`key_hash`**: SHA-256 hash of `normalized_ip + ":" + normalized_email`.
|
||||
- **`window_started_at`**: Start timestamp of the current 15-minute sliding window.
|
||||
- **`failed_attempts`**: Count of failed attempts within the window (locks at 5).
|
||||
- **`blocked_until`**: Timestamp until which further login attempts from this key are rejected.
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Table: `client_sdks`
|
||||
Maintains the registry of authorized SDK client credentials and per-client throughput quotas.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS client_sdks (
|
||||
client_id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
secret TEXT NOT NULL,
|
||||
token TEXT NOT NULL,
|
||||
rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
total_events BIGINT NOT NULL DEFAULT 0,
|
||||
last_active TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
- **`client_id`**: Lowercase slug identifying the producer.
|
||||
- **`secret`**: 32-character random client secret used for verification.
|
||||
- **`token`**: Current signed HMAC-SHA256 JWT token.
|
||||
- **`rate_limit`**: Quota of maximum allowed events per second.
|
||||
- **`status`**: Current access state (`active` or `revoked`).
|
||||
- **`total_events`**: Lifetime count of successfully ingested events.
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Table: `deleted_clients`
|
||||
Tombstone table preventing reused client IDs or compromised stale tokens from regaining access.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS deleted_clients (
|
||||
client_id TEXT PRIMARY KEY,
|
||||
deleted_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
- **`client_id`**: Identifier of permanently removed client. Checked during both HTTP ingestion and Socket.IO handshakes.
|
||||
|
||||
---
|
||||
|
||||
### 2.5 Table: `admin_settings`
|
||||
Single-row platform configuration table storing runtime operational parameters.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS admin_settings (
|
||||
id TEXT PRIMARY KEY DEFAULT 'default',
|
||||
server_port INTEGER NOT NULL DEFAULT 4000,
|
||||
web_port INTEGER NOT NULL DEFAULT 3000,
|
||||
max_events_per_sec INTEGER NOT NULL DEFAULT 100,
|
||||
max_payload_size_kb INTEGER NOT NULL DEFAULT 100,
|
||||
batch_flush_threshold INTEGER NOT NULL DEFAULT 500,
|
||||
batch_flush_interval_ms INTEGER NOT NULL DEFAULT 2000,
|
||||
file_rotation_mb INTEGER NOT NULL DEFAULT 100,
|
||||
jwt_secret TEXT NOT NULL,
|
||||
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
||||
auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Migration Architecture
|
||||
|
||||
DataStack executes automated, non-destructive schema migrations on application startup via `DatabaseEngine.ensureSchema()`:
|
||||
|
||||
1. **`CREATE TABLE IF NOT EXISTS`**: Ensures all tables exist without altering pre-existing data.
|
||||
2. **`ALTER TABLE ... ADD COLUMN IF NOT EXISTS`**: Adds new columns with backwards-compatible defaults if upgrading from an older version.
|
||||
3. **No Destructive Operations**: `DROP TABLE`, `DROP COLUMN`, and type truncations are strictly forbidden in production migrations. Existing databases retain all rows and constraints across container updates.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Developer & Contributor Documentation Hub
|
||||
|
||||
Welcome to the DataStack Developer Hub. This documentation suite is designed for software engineers, systems programmers, and open-source contributors working on or extending the DataStack platform.
|
||||
|
||||
---
|
||||
|
||||
## Developer Portals & Architecture Specifications
|
||||
|
||||
| Document | Purpose & Scope | Target Audience |
|
||||
| :--- | :--- | :--- |
|
||||
| [Architecture Blueprint (`architecture.md`)](./architecture.md) | Deep dive into the dual-runtime architecture (Port 3000 Web vs Port 4000 Ingestion), internal bridge protocol, Redis AOF durability, and storage layout. | Core Contributors, Architects |
|
||||
| [Codebase Structure (`codebase-structure.md`)](./codebase-structure.md) | File-by-file inventory of directories, source modules (`core/`, `app/`, `server/`, `components/`), and their relationships. | All Developers |
|
||||
| [Local Development Setup (`local-setup.md`)](./local-setup.md) | Workstation setup, Node.js 22 requirements, dual-terminal local workflow, and in-memory vs Docker container modes. | New Contributors |
|
||||
| [API Reference Specification (`api-reference.md`)](./api-reference.md) | OpenAPI-style contract reference for all REST endpoints (`app/api/*`) and Socket.IO real-time events (`server/socket/handlers.js`). | Integration Engineers |
|
||||
| [Database Schema & Migrations (`database-schema.md`)](./database-schema.md) | PostgreSQL control-plane tables, DDL migrations, column definitions, and `prisma/schema.prisma` mirror. | Backend Engineers |
|
||||
| [Security Architecture (`security.md`)](./security.md) | Cryptographic design: `crypto.scrypt`, WebCrypto HMAC sessions, brute-force IP lockout, CSRF Origin validation, and JWT verification. | Security Engineers |
|
||||
| [Building & Packaging (`building-packaging.md`)](./building-packaging.md) | Next.js standalone compilation (`.next/standalone`), multi-stage Dockerfiles (`web`, `server`, `proxy`), and production container assembly. | DevOps, Release Engineers |
|
||||
| [Testing Guide (`testing.md`)](./testing.md) | Guide to automated TypeScript integration tests (`npx tsx --test`) and Python SDK unit tests (`python3 -m unittest`). | QA, Test Engineers |
|
||||
| [Contributing Guidelines (`contributing.md`)](./contributing.md) | Development rules, zero-SQL-telemetry enforcement, single-source UI primitives, multi-IDE memory synchronization (`MEMORY.md`), and PR checklist. | All Contributors |
|
||||
|
||||
---
|
||||
|
||||
## Core Engineering Principles
|
||||
|
||||
1. **Decoupled Runtime Separation**: Next.js 15 App Router on Port 3000 handles user interaction and REST exports. Express 5 + Socket.IO 4 on Port 4000 handles persistent WebSocket handshakes. Socket.IO is never instantiated inside Next.js serverless handlers.
|
||||
2. **Zero Per-Row Relational Telemetry**: Telemetry streams are buffered in memory/Redis and flushed to daily CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. PostgreSQL is reserved exclusively for the control plane.
|
||||
3. **Fail-Closed Security**: In production (`NODE_ENV=production`), missing or placeholder secrets halt startup immediately.
|
||||
4. **Cooperative File Locking**: Concurrent flushes between Port 3000 and Port 4000 coordinate safely using `.lock` files with automatic stale lock expiration.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Local Development Setup
|
||||
|
||||
This guide walks contributors through configuring and running DataStack locally on your development workstation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Ensure the following tools are installed:
|
||||
- **Node.js**: Version `22.x` (LTS recommended)
|
||||
- **Package Manager**: `npm` (v10+)
|
||||
- **Python**: Version `3.10+` (optional, for developing and testing the Python client SDK)
|
||||
- **Docker & Docker Compose**: Optional, useful for running local PostgreSQL and Redis instances.
|
||||
|
||||
---
|
||||
|
||||
## 2. Clone & Install Dependencies
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vibtoolshq/datastack.git
|
||||
cd datastack
|
||||
|
||||
# Install root dependencies deterministically using the committed lockfile
|
||||
npm ci
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment Configuration
|
||||
|
||||
Copy the example environment configuration:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
### Local Development Defaults
|
||||
In local development (`NODE_ENV=development`), DataStack includes built-in fallbacks:
|
||||
- If `DATABASE_URL` is omitted, DataStack operates seamlessly in **In-Memory Mock Mode**, allowing you to test the full UI, client registration, and REST endpoints without running PostgreSQL.
|
||||
- If `REDIS_URL` is omitted, DataStack uses an **in-memory RAM buffer queue**, persisting directly to `./storage/data`.
|
||||
|
||||
For full production parity during development, you can spin up local PostgreSQL and Redis containers:
|
||||
```bash
|
||||
docker run --name datastack-postgres -e POSTGRES_USER=datastack -e POSTGRES_PASSWORD=password -e POSTGRES_DB=datastack_db -p 5432:5432 -d postgres:16-alpine
|
||||
docker run --name datastack-redis -p 6379:6379 -d redis:7-alpine redis-server --appendonly yes
|
||||
```
|
||||
|
||||
Then update your `.env`:
|
||||
```dotenv
|
||||
DATABASE_URL=postgresql://datastack:password@127.0.0.1:5432/datastack_db
|
||||
REDIS_URL=redis://127.0.0.1:6379
|
||||
PORT=3000
|
||||
PORT_SOCKET=4000
|
||||
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
|
||||
DATASTACK_ADMIN_EMAIL=admin@example.com
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Running the Development Servers
|
||||
|
||||
DataStack requires two concurrent runtime processes. Open two terminal tabs in the project root:
|
||||
|
||||
### Terminal 1: Web Console & REST API (Port 3000)
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
Starts the Next.js 15 development server with Fast Refresh at `http://localhost:3000`.
|
||||
|
||||
### Terminal 2: Ingestion Server (Port 4000)
|
||||
```bash
|
||||
npm run server
|
||||
```
|
||||
Starts the dedicated Express 5 + Socket.IO ingestion service at `http://localhost:4000`.
|
||||
|
||||
---
|
||||
|
||||
## 5. First-Time Setup in Development
|
||||
|
||||
1. Open `http://localhost:3000/vcon/setup` in your browser.
|
||||
2. Complete the setup wizard using your `DATASTACK_ADMIN_SETUP_KEY` from `.env`.
|
||||
3. In `/vcon/clients`, register a test client application to generate your first JWT token.
|
||||
4. In `/vcon/simulator`, test sending live events and observe the buffer counter in `/vcon/dashboard`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Python SDK Development
|
||||
|
||||
If you are modifying the Python SDK in `sdk/python/datastack/`:
|
||||
|
||||
```bash
|
||||
cd sdk/python/datastack
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
pip install -r ../../../tests/requirements.txt # If running SDK test suites
|
||||
```
|
||||
@@ -0,0 +1,96 @@
|
||||
# Security Architecture & Cryptographic Design
|
||||
|
||||
This document details the security principles, cryptographic implementations, authentication mechanisms, and attack mitigations enforced across DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Security Architecture Overview
|
||||
|
||||
DataStack follows a defense-in-depth model with strict boundary separation:
|
||||
|
||||
```text
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ Internet / External Clients │
|
||||
└──────────────────┬─────────────────────────────────┬───────────────────┘
|
||||
│ HTTPS Ingestion (JWT) │ HTTPS Web Console
|
||||
▼ ▼
|
||||
┌───────────────────┐ ┌───────────────────┐
|
||||
│ Port 4000 (Ing) │ │ Port 3000 (Web) │
|
||||
│ Socket.IO Hand │ │ Next.js Edge MW │
|
||||
│ JWT Verify │ │ HMAC Session Ver │
|
||||
│ Rate Limiting │ │ CSRF Origin Ver │
|
||||
└─────────┬─────────┘ └─────────┬─────────┘
|
||||
│ │
|
||||
└──────── Authenticated ──────────┘
|
||||
/internal/* Bridge
|
||||
(DATASTACK_INTERNAL_SECRET)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Administrator Authentication & Sessions
|
||||
|
||||
### Password Storage (`crypto.scrypt`)
|
||||
Administrator passwords are never stored in plaintext or reversible formats:
|
||||
- **Salt Generation**: 16 cryptographically secure random bytes generated via `crypto.randomBytes(16)`.
|
||||
- **Key Derivation Function**: Node.js native `crypto.scryptSync(password, salt, 64)`.
|
||||
- **Constant-Time Verification**: Hashes are compared using `crypto.timingSafeEqual` to prevent timing side-channel attacks.
|
||||
|
||||
### Session Cookies (`datastack_admin_session`)
|
||||
- **Signing Mechanism**: Signed with WebCrypto HMAC-SHA256 using `DATASTACK_ADMIN_SESSION_SECRET`.
|
||||
- **Session Payload**:
|
||||
```json
|
||||
{
|
||||
"userId": 1,
|
||||
"email": "admin@example.com",
|
||||
"tokenVersion": 1,
|
||||
"exp": 1700086400000
|
||||
}
|
||||
```
|
||||
- **Instant Global Invalidation**: Every administrative request validates `tokenVersion` against PostgreSQL. When an operator updates their password or selects **Log Out Other Sessions**, `tokenVersion` increments, instantly invalidating all issued cookies across all devices.
|
||||
- **Cookie Security Flags**: `HttpOnly; SameSite=Lax; Path=/; Priority=High;` (with `Secure` enforced when `NODE_ENV=production`).
|
||||
|
||||
### Brute-Force IP Lockout
|
||||
To prevent credential stuffing:
|
||||
- Tracks failed login attempts per composite SHA-256 hash of `normalized_ip + ":" + normalized_email`.
|
||||
- **Threshold**: 5 consecutive failed attempts locks the composite key for 15 minutes.
|
||||
- Attempts are recorded in PostgreSQL (`admin_login_attempts`) and survive container restarts.
|
||||
|
||||
---
|
||||
|
||||
## 3. Client SDK Authentication & Access Control
|
||||
|
||||
### HMAC-SHA256 Client JWTs
|
||||
- Each registered client receives a JWT signed by `DATASTACK_JWT_SECRET`.
|
||||
- The token payload embeds `{ clientId, name, rateLimit, iat }`.
|
||||
- Verified on both HTTP REST ingestion (`Authorization: Bearer <TOKEN>`) and Socket.IO handshakes (`auth.token`).
|
||||
|
||||
### Instant Credential Revocation & Tombstones
|
||||
- **Rotation**: Rotating a client token generates a new secret and pushes the update to Port 4000 via `/internal/client-sync`. Active WebSocket sockets using the old token are severed immediately.
|
||||
- **Revocation**: Setting status to `revoked` immediately halts ingestion with `CLIENT_REVOKED (403)`.
|
||||
- **Tombstones (`deleted_clients`)**: Deleting a client registers its identifier in `deleted_clients`. Stale, leaked, or intercepted tokens can never be reused to write telemetry.
|
||||
|
||||
---
|
||||
|
||||
## 4. Cross-Site Request Forgery (CSRF) Mitigation
|
||||
|
||||
State-changing HTTP methods (`POST`, `PUT`, `DELETE`, `PATCH`) handled by Next.js App Router are protected in `middleware.ts`:
|
||||
- Validates the request's `Origin` or `Referer` header against the incoming `Host` or `X-Forwarded-Host`.
|
||||
- Cross-origin requests attempting to trigger administrative endpoints or manual flushes are rejected with `403 Forbidden` (`CSRF_FAILED`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Internal Service Bridge Security
|
||||
|
||||
Communication between Port 3000 (Web) and Port 4000 (Ingestion Server) is isolated behind a shared secret:
|
||||
- Header: `x-datastack-internal-secret: <DATASTACK_INTERNAL_SECRET>`.
|
||||
- The ingestion server verifies incoming headers using `crypto.timingSafeEqual`.
|
||||
- Unauthorized internal requests are rejected immediately with `403 Forbidden`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Secret Fail-Closed Enforcement
|
||||
|
||||
When running in production (`NODE_ENV=production`):
|
||||
- All secret keys (`DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SETUP_KEY`, `DATASTACK_INTERNAL_SECRET`) must be explicitly defined and have a minimum length of 32 characters.
|
||||
- If any secret contains default placeholder strings (e.g., `replace-with-`, `your-secret`), the server halts during bootstrap with a fatal configuration error, preventing accidental deployment with insecure credentials.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Automated Testing Guide
|
||||
|
||||
This guide details the automated test suites, execution commands, coverage domains, and verification standards for DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Test Architecture
|
||||
|
||||
DataStack utilizes a lightweight, native test architecture built upon the Node.js native test runner (`node:test`) and Python's `unittest` framework:
|
||||
|
||||
- **Zero Heavy Test Framework Bloat**: Tests run using `npx tsx --test`, executing directly against TypeScript source modules without requiring Jest or Mocha.
|
||||
- **Fast Execution**: The complete TypeScript suite executes in under 2 seconds.
|
||||
- **Isolation**: Each test suite initializes mock stores or clean temporary directories, guaranteeing zero side effects on production data.
|
||||
|
||||
---
|
||||
|
||||
## 2. Running TypeScript Test Suites
|
||||
|
||||
To run all TypeScript test suites in sequence:
|
||||
|
||||
```bash
|
||||
npx --yes tsx --test tests/phase01-auth.test.ts \
|
||||
tests/phase02-sync.test.ts \
|
||||
tests/phase03-storage-buffer.test.ts \
|
||||
tests/phase06-pwa-landing.test.ts \
|
||||
tests/phase07-deployment.test.ts
|
||||
```
|
||||
|
||||
### Test Suite Coverage Breakdown
|
||||
|
||||
#### 1. `tests/phase01-auth.test.ts` (Security & Authentication)
|
||||
- **Production Secret Fail-Closed**: Verifies that empty, short (<32 chars), or placeholder secrets halt execution.
|
||||
- **Session Cookie Invalidation**: Asserts that incrementing `tokenVersion` immediately causes session rejection.
|
||||
- **Brute-Force Rate Limiting**: Validates that 5 failed login attempts lock out the composite IP + Email key for 15 minutes.
|
||||
- **Client Token Redaction**: Ensures client secrets and raw tokens are omitted from public audit responses.
|
||||
- **CSRF Origin Verification**: Tests that mismatched Origin/Referer headers return `403 Forbidden`.
|
||||
|
||||
#### 2. `tests/phase02-sync.test.ts` (Dual-Runtime Synchronization)
|
||||
- **Internal Secret Security**: Tests timing-safe header validation on `/internal/*` routes.
|
||||
- **Settings Synchronization**: Validates that updating settings in Port 3000 propagates instantly to Port 4000.
|
||||
- **Client Revocation**: Confirms that revoking a client immediately severs active WebSocket connections.
|
||||
- **JWT Secret Rotation**: Asserts that all client tokens are re-signed and old tokens are rejected.
|
||||
|
||||
#### 3. `tests/phase03-storage-buffer.test.ts` (Storage & Durability)
|
||||
- **Redis Queue Durability**: Tests RAM queue fallback and Redis list buffering.
|
||||
- **Batch Threshold & Interval**: Verifies that threshold count (500) and interval time (2000ms) trigger flushes.
|
||||
- **Cooperative Locking**: Proves that `.lock` files prevent concurrent write collisions and that stale locks (>30s) are safely reclaimed.
|
||||
- **Streaming Export**: Verifies non-blocking streaming CSV and JSON exports with client and date filtering.
|
||||
|
||||
#### 4. `tests/phase06-pwa-landing.test.ts` (PWA & Client Shell)
|
||||
- **Web App Manifest**: Validates manifest schema, icons, and display modes.
|
||||
- **Service Worker Lifecycle**: Tests cache strategies and offline shell handling.
|
||||
|
||||
#### 5. `tests/phase07-deployment.test.ts` (Containerization & Deployment Health)
|
||||
- **Docker Compose Configuration**: Audits volume mounts (`datastack-data`, `pgdata`, `redisdata`).
|
||||
- **Nginx Reverse Proxy**: Verifies routing rules for `/`, `/socket.io/`, and `/health`.
|
||||
- **Unhydrated DB Auto-Pruning Guard**: Confirms that partition pruning is deferred until database settings are fully hydrated.
|
||||
|
||||
---
|
||||
|
||||
## 3. Running Python SDK Test Suites
|
||||
|
||||
The Python SDK includes a dedicated integration test suite verifying WebSocket streaming, heartbeat pings, HTTP fallbacks, and offline disk spooling:
|
||||
|
||||
```bash
|
||||
python3 -m unittest tests/phase05_sdk_test.py
|
||||
```
|
||||
|
||||
### Verified Behaviors:
|
||||
- WebSocket connection and JWT handshake authentication.
|
||||
- Offline spooling to local disk directory (`.datastack_spool/`) during network disruption.
|
||||
- Automatic drain and delivery of spooled events when connectivity is restored.
|
||||
- Streaming large `.csv` and `.json` files in chunks.
|
||||
|
||||
---
|
||||
|
||||
## 4. Continuous Integration Checklist
|
||||
|
||||
Before opening a pull request:
|
||||
1. `npm run lint` must exit with `0` errors and `0` warnings.
|
||||
2. `npm run build` must compile the Next.js standalone bundle successfully.
|
||||
3. All 5 TypeScript test suites must pass (`23/23 tests passing`).
|
||||
4. Python SDK test suite must pass (`3/3 tests passing`).
|
||||
Reference in new issue
Block a user