feat(sdk): refactor Python SDK documentation

- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
Team Dark Devil committed 2026-11-18 10:03:40 +06:00
1 parent 4279da45bb
commit 40a6855724
38 files changed
+3315 -141

No files matched your search

+289
View File
@@ -0,0 +1,289 @@
# API Reference Specification
This document provides the authoritative API specification for all DataStack REST HTTP endpoints (`app/api/*`) and real-time Socket.IO WebSocket protocol events (`server/socket/handlers.js`).
---
## 1. REST API Specification (Port 3000)
### 1.1 Ingestion Endpoints
#### `POST /api/events`
Ingests telemetry events over authenticated HTTP.
- **Authentication**: `Authorization: Bearer <CLIENT_JWT_TOKEN>`
- **Content-Type**: `application/json`
- **Supported Payload Shapes**:
##### Single Event Mode
```json
{
"eventId": "evt_custom_12345",
"payload": {
"temperature": 23.4,
"status": "normal"
}
}
```
##### Batch Array Mode
```json
{
"batch": [
{ "eventId": "evt_1", "payload": { "val": 10 } },
{ "eventId": "evt_2", "payload": { "val": 20 } }
]
}
```
##### File / Structured Record Mode
```json
{
"fileName": "sensor_logs.csv",
"records": [
{ "val": 10 },
{ "val": 20 }
]
}
```
- **Responses**:
- `200 OK`:
```json
{
"success": true,
"accepted": 1,
"buffered": true
}
```
- `401 Unauthorized`: Missing or invalid client JWT.
- `403 Forbidden`: Client status is `revoked` or client ID is recorded in `deleted_clients`.
- `413 Payload Too Large`: Total payload exceeds `maxPayloadSizeKb`.
- `429 Too Many Requests`: Client or global rate limit exceeded.
---
#### `POST /api/events/flush`
Forces an immediate write of all buffered RAM and Redis events to disk.
- **Authentication**: Operator Session Cookie (`datastack_admin_session`)
- **Responses**:
- `200 OK`:
```json
{
"success": true,
"flushed": 150
}
```
---
### 1.2 Export Endpoints
#### `GET /api/export`
Streams filtered CSV or JSON records directly from disk partitions to the caller.
- **Authentication**: Operator Session Cookie (`datastack_admin_session`)
- **Query Parameters**:
- `clientId` (string, optional): Specific client identifier or `all` (default: `all`).
- `from` (string, optional): Inclusive start date filter (`YYYY-MM-DD`).
- `to` (string, optional): Inclusive end date filter (`YYYY-MM-DD`).
- `format` (string, optional): `csv` (default) or `json`.
- `includeHeader` (boolean, optional): Whether to output CSV headers (default: `true`).
- `flush` (boolean, optional): Whether to flush pending memory/Redis buffers before streaming (default: `true`).
- `filename` (string, optional): Specific single partition filename (e.g., `2026-11-17.csv`).
- **Response Headers**:
- `Content-Type`: `text/csv; charset=utf-8` or `application/json; charset=utf-8`
- `Content-Disposition`: `attachment; filename="datastack_export_YYYY-MM-DD.csv"`
- `Cache-Control`: `no-store`
- `Transfer-Encoding`: `chunked`
---
### 1.3 System Health & Diagnostics
#### `GET /api/health`
Returns process uptime and database connection status.
- **Authentication**: None (Public)
- **Responses**:
- `200 OK`:
```json
{
"status": "ok",
"uptime": 12450.2,
"timestamp": "2026-11-17T12:00:00.000Z",
"database": "connected"
}
```
---
### 1.4 Operator Authentication Endpoints
#### `POST /api/auth/login`
Authenticates administrator credentials and establishes session cookie.
- **Request Body**:
```json
{
"email": "admin@example.com",
"password": "strongPassword123!"
}
```
- **Responses**:
- `200 OK`: Sets `datastack_admin_session` cookie (HttpOnly, SameSite=Lax, Secure in production).
- `401 Unauthorized`: Invalid credentials.
- `429 Too Many Requests`: IP temporarily blocked due to 5 failed attempts in 15 minutes.
---
#### `POST /api/auth/logout`
Terminates the operator session and clears the cookie.
---
#### `GET /api/auth/me`
Returns current authenticated administrator profile.
- **Responses**:
- `200 OK`: `{"authenticated": true, "user": {"id": "...", "email": "..."}}`
- `401 Unauthorized`: `{"authenticated": false}`
---
#### `POST /api/auth/setup`
Completes first-time installation wizard.
- **Request Body**:
```json
{
"setupKey": "CONFIGURED_SETUP_KEY",
"email": "admin@example.com",
"password": "strongPassword123!"
}
```
---
#### `POST /api/auth/change-password`
Updates operator password and increments `tokenVersion`.
- **Request Body**:
```json
{
"currentPassword": "oldPassword",
"newPassword": "newPassword123!"
}
```
---
### 1.5 Client Registry Endpoints
#### `GET /api/clients`
Lists all registered SDK clients and rate limits.
#### `POST /api/clients`
Registers a new SDK client application.
- **Request Body**:
```json
{
"clientId": "pos_terminal_01",
"name": "Point of Sale 1",
"rateLimit": 100
}
```
#### `PUT /api/clients/[id]`
Updates client rate limits, rotates secret tokens, or revokes access.
#### `DELETE /api/clients/[id]`
Deletes client from registry, moves ID to `deleted_clients` tombstone, and optionally purges storage files.
---
### 1.6 Settings Endpoints
#### `GET /api/settings`
Returns public platform runtime limits and buffer configurations.
#### `PUT /api/settings`
Updates platform limits and notifies Port 4000 ingestion service.
---
## 2. Socket.IO Real-Time Protocol Specification (Port 4000)
### Handshake Authentication
Clients connect via Socket.IO v4 client with a valid JWT token passed in the `auth` object or handshake `query`:
```javascript
const socket = io("https://your-domain.example", {
path: "/socket.io/",
auth: {
token: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
});
```
If the token is invalid, expired, revoked, or belongs to a tombstone client, the handshake is rejected immediately with an authentication error.
---
### Socket Events & Messages
#### Event: `sync_data`
Transmits a single telemetry event.
- **Message Body**:
```json
{
"eventId": "evt_client_9812",
"payload": {
"metric": "cpu",
"value": 42.1
}
}
```
- **Acknowledgment Callback**:
- Success: `{"status": "ok", "eventId": "evt_client_9812"}`
- Error: `{"error": "Rate limit exceeded", "code": 429}`
---
#### Event: `sync_batch`
Transmits an array of events in a single payload.
- **Message Body**:
```json
{
"events": [
{ "eventId": "evt_1", "payload": { "val": 1 } },
{ "eventId": "evt_2", "payload": { "val": 2 } }
]
}
```
- **Acknowledgment Callback**:
- Success: `{"status": "ok", "acceptedCount": 2}`
---
#### Event: `heartbeat`
Keep-alive ping to measure latency and maintain connection health.
- **Message Body**: `{"timestamp": 1700000000000}`
- **Acknowledgment Callback**: `{"status": "ok", "pong": true, "serverTime": 1700000000010}`
---
## 3. Internal Service Bridge Specification (`/internal/*`)
All internal endpoints on Port 4000 require the header:
`x-datastack-internal-secret: <DATASTACK_INTERNAL_SECRET>`
- `GET /internal/metrics`: Returns connected socket count, active buffer queue depth, and memory stats.
- `POST /internal/settings-sync`: Pushes updated limits and secrets to Port 4000.
- `POST /internal/client-sync`: Pushes rotated, revoked, or deleted client updates.
- `POST /internal/flush`: Dispatches synchronous buffer flush to disk.
- `GET /internal/recent`: Fetches latest unwritten queue items for dashboard live activity streams.
+71
View File
@@ -0,0 +1,71 @@
# Building, Packaging & Production Containers
This guide outlines compilation commands, linter checks, test suite execution, and Docker container packaging for DataStack.
---
## 1. Production Web Compilation
DataStack uses Next.js 15 App Router configured for **Standalone Output** (`output: "standalone"` in `next.config.ts`), which bundles only the required Node.js runtime files into `.next/standalone`.
### Compiling the Web App
```bash
npm run build
```
### Build Verification Checklist
- Compiles TypeScript without type errors (`tsc --noEmit`).
- Generates serverless routes and static assets under `.next/`.
- Prepares standalone bundle under `.next/standalone/server.js`.
---
## 2. Code Quality & Linting
DataStack uses ESLint 9 with `eslint-config-next`:
```bash
npm run lint
```
Ensure this command exits cleanly (`code: 0`) with zero warnings or errors prior to committing code or opening a pull request.
---
## 3. Docker Container Builds
DataStack provides optimized, multi-stage Dockerfiles under `docker/`:
### 1. Build the Web Container (Port 3000)
```bash
docker build -f docker/Dockerfile.web -t datastack-web:latest .
```
- **Base**: `node:22-alpine`.
- **Stage 1 (Builder)**: Installs dependencies deterministically (`npm ci`) and runs `npm run build`.
- **Stage 2 (Runner)**: Copies `.next/standalone`, public assets, and sets non-root user `nextjs` (UID 1001).
### 2. Build the Ingestion Container (Port 4000)
```bash
docker build -f docker/Dockerfile.server -t datastack-ingestion:latest .
```
- **Base**: `node:22-alpine`.
- Installs production dependencies and runs `node server/index.js` under a dedicated non-root user.
### 3. Build the Nginx Proxy Container (Port 80)
```bash
docker build -f docker/Dockerfile.proxy -t datastack-proxy:latest .
```
- **Base**: `nginx:1.27-alpine`.
- Mounts custom reverse proxy configuration (`nginx/nginx.conf`) routing `/` to Port 3000 and `/socket.io/` to Port 4000.
### 4. Build & Run the Entire Stack with Compose
```bash
docker compose -f docker/docker-compose.yml build
docker compose -f docker/docker-compose.yml up -d
```
Verify container health:
```bash
docker compose -f docker/docker-compose.yml ps
```
All containers (`web`, `ingestion`, `proxy`, `postgres`, `redis`) should display `(healthy)`.
+98
View File
@@ -0,0 +1,98 @@
# Codebase Structure & Directory Inventory
This document provides a comprehensive, file-by-file architectural inventory of the DataStack codebase, detailing module responsibilities, runtime boundaries, and inter-module dependencies.
---
## 1. High-Level Directory Overview
```text
/
├── app/ # Next.js 15 App Router: UI pages, layouts, and REST API routes (Port 3000)
├── components/ # Reusable UI primitives and layout shells
├── core/ # Shared business logic, database connectors, and security engines
├── server/ # Dedicated Express 5 + Socket.IO 4 Ingestion Server (Port 4000)
├── sdk/ # Official client SDKs (Python, with TypeScript SDK planned)
├── docker/ # Production multi-stage Dockerfiles and Compose configurations
├── nginx/ # Nginx reverse proxy routing and health check configuration
├── prisma/ # Schema mirror documenting PostgreSQL control-plane tables
├── public/ # Static assets, PWA icons, and manifests
├── storage/ # Mounted partition volume for daily CSV files (/storage/data/)
├── tests/ # Automated TypeScript integration and Python SDK test suites
└── docs/ # Authoritative documentation suite
```
---
## 2. Web Application & REST API (`app/`)
Operating on **Port 3000**, this Next.js 15 App Router subsystem handles operator authentication, dashboard rendering, authenticated HTTP ingestion, and streaming dataset exports.
### Core Entry Points
- `app/layout.tsx`: Root HTML layout importing centralized CSS tokens, fonts, and PWA registration.
- `app/page.tsx`: Public marketing and documentation overview page.
- `app/middleware.ts`: Next.js Edge middleware. Enforces signed HMAC-SHA256 session cookies on `/vcon/*` routes (except `/vcon/login` and `/vcon/setup`) and validates CSRF `Origin`/`Referer` headers against `Host` and `X-Forwarded-Host`.
- `app/globals.css`: Global design tokens, CSS variables, and zero-pill border radius resets.
- `app/manifest.ts`: Web App Manifest generator for PWA installation.
- `app/sw.ts`: Progressive Web App Service Worker script.
### Administration Console Pages (`app/vcon/`)
- `app/vcon/login/page.tsx`: Operator sign-in page with brute-force rate-limiting feedback.
- `app/vcon/setup/page.tsx`: First-time setup wizard accepting `DATASTACK_ADMIN_SETUP_KEY`.
- `app/vcon/dashboard/page.tsx`: Primary operational dashboard displaying dual-runtime health, disk utilization, buffer depth, and recent live telemetry ticker.
- `app/vcon/clients/page.tsx`: Client SDK registry for generating tokens, rotating secrets, and revoking clients.
- `app/vcon/simulator/page.tsx`: Interactive in-browser telemetry generator with single and burst dispatch.
- `app/vcon/storage/page.tsx`: CSV partition inspector and manual buffer flush controls.
- `app/vcon/export/page.tsx`: Filtered streaming CSV and JSON export workspace.
- `app/vcon/settings/page.tsx`: Platform runtime settings manager (rate limits, flush intervals, rotation size).
- `app/vcon/profile/page.tsx`: Administrator credentials, password updates, and session revocation.
### REST API Route Handlers (`app/api/`)
- `app/api/events/route.ts`: Authenticated HTTP ingestion endpoint. Accepts single events, `{ batch: [...] }`, or structured file imports (`{ fileName, records: [...] }`).
- `app/api/events/flush/route.ts`: Forces immediate buffer flushes across both Port 3000 and Port 4000.
- `app/api/export/route.ts`: Streams filtered CSV or JSON datasets directly from disk to the client response.
- `app/api/health/route.ts`: Health check endpoint reporting process uptime and database status.
- `app/api/auth/login/route.ts`: Authenticates admin email and password with `crypto.scrypt` verification and composite IP brute-force tracking.
- `app/api/auth/logout/route.ts`: Clears session cookie and invalidates session token.
- `app/api/auth/me/route.ts`: Returns current operator identity and session validity.
- `app/api/auth/setup/route.ts`: Executes initial admin setup with setup key verification.
- `app/api/auth/change-password/route.ts`: Updates operator password and increments `tokenVersion`.
- `app/api/clients/route.ts`: Registers new SDK clients and lists existing client credentials.
- `app/api/clients/[id]/route.ts`: Updates client rate limits, rotates secrets, revokes, or permanently deletes clients.
- `app/api/settings/route.ts`: Reads and updates platform runtime settings with database persistence.
---
## 3. Core Business Logic Singletons (`core/`)
The `core/` directory contains framework-agnostic TypeScript singletons shared between the Web application and ingestion workers.
| Module | Primary Class / Functions | Responsibility |
| :--- | :--- | :--- |
| `core/Database/database.ts` | `DatabaseEngine` | Manages raw `pg.Pool` PostgreSQL connections, executes non-destructive DDL migrations, and provides an in-memory mock store when `DATABASE_URL` is omitted. |
| `core/auth/adminAuth.ts` | `AdminAuthService` | Enforces `crypto.scrypt` password hashing (64-byte key, 16-byte salt), constant-time string comparison, and composite IP + email failed attempt lockout. |
| `core/auth/jwt.ts` | `AuthService` | Generates and verifies HMAC-SHA256 JWT tokens for SDK clients and WebCrypto HMAC-SHA256 session tokens for administrators. |
| `core/Security/security.ts` | `SecurityEngine` | Implements sliding-window rate limiters, 5-minute idempotency caches, payload size guards, and timing-safe internal secret authentication. |
| `core/data/buffer.ts` | `DataBufferEngine` | Manages in-memory RAM event queues and optional Redis list queues (`datastack:buffer:events`), with threshold and interval auto-flushing. |
| `core/Storage/csvStorage.ts` | `CSVStorageManager` | Handles RFC 4180 CSV serialization, client/date directory partitioning, 100MB file rotation (`-part2.csv`), cooperative `.lock` file handling, and automated retention pruning. |
| `core/sdk/sdkManager.ts` | `SDKManager` | Coordinates client CRUD operations, credential rotation, master secret re-signing, and tombstone checks. |
| `core/adminSettings/settingsManager.ts` | `SettingsManager` | Manages system settings state, synchronizes with PostgreSQL, and notifies Port 4000 via `/internal/settings-sync`. |
---
## 4. Ingestion Server Subsystem (`server/`)
Operating on **Port 4000**, this dedicated Express 5 + Socket.IO 4 service handles high-throughput, persistent WebSocket connections.
- `server/index.js`: Express application entry point. Listens on `PORT_SOCKET` (4000), mounts internal service bridge routes (`/internal/*`), and binds the Socket.IO server.
- `server/socket/handlers.js`: Enforces JWT authentication during the Socket.IO handshake (`io.use()`), handles incoming telemetry events (`sync_data`, `sync_batch`), applies per-client rate limits, deduplicates idempotency envelopes, and supports administrative socket termination.
- `server/workers/csvWriter.js`: Background flusher worker (`CsvWriterWorker`). Periodically polls the buffer queue and flushes batches to partitioned CSV files using cooperative file locks.
---
## 5. Client SDK Subsystem (`sdk/`)
- `sdk/python/datastack/`: Official Python Client Library.
- `client.py`: Provides `DataStackClient` supporting WebSocket streaming (`connect()`), automatic fallback to HTTP (`send_event()`), offline disk spooling (`.datastack_spool/`), and batch file streaming (`stream_csv()`, `stream_json()`).
- `__init__.py`: Package entry point exposing `DataStackClient`.
- `pyproject.toml` & `setup.py`: Packaging specifications.
+128
View File
@@ -0,0 +1,128 @@
# Database Schema & Migrations
This document specifies the PostgreSQL control-plane database schema, table definitions, constraints, and non-destructive additive migration architecture.
---
## 1. Architectural Boundary
> **IMPORTANT ARCHITECTURAL RULE**: PostgreSQL is used **strictly for control-plane metadata** (admin accounts, login throttling, client tokens, and runtime settings).
>
> **Zero Telemetry Writes**: Telemetry events are never written to PostgreSQL. All event data is buffered and persisted directly to daily CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
---
## 2. Table Specifications
### 2.1 Table: `admin_users`
Stores single-tenant administrator credentials and session token versions.
```sql
CREATE TABLE IF NOT EXISTS admin_users (
id SMALLINT PRIMARY KEY DEFAULT 1 CHECK (id = 1),
email TEXT NOT NULL UNIQUE,
password_salt BYTEA NOT NULL,
password_hash BYTEA NOT NULL,
token_version INTEGER NOT NULL DEFAULT 1,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
- **`id`**: Enforces a single administrator row constraint (`CHECK (id = 1)`).
- **`email`**: Unique administrator contact and login email.
- **`password_salt`**: Cryptographically secure 16-byte random salt (`crypto.randomBytes(16)`).
- **`password_hash`**: 64-byte scrypt derived hash (`crypto.scryptSync(password, salt, 64)`).
- **`token_version`**: Integer counter embedded into session cookies. Incrementing this immediately invalidates all active sessions globally.
---
### 2.2 Table: `admin_login_attempts`
Tracks failed login attempts for brute-force protection.
```sql
CREATE TABLE IF NOT EXISTS admin_login_attempts (
key_hash TEXT PRIMARY KEY,
window_started_at TIMESTAMPTZ NOT NULL,
failed_attempts INTEGER NOT NULL,
blocked_until TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
- **`key_hash`**: SHA-256 hash of `normalized_ip + ":" + normalized_email`.
- **`window_started_at`**: Start timestamp of the current 15-minute sliding window.
- **`failed_attempts`**: Count of failed attempts within the window (locks at 5).
- **`blocked_until`**: Timestamp until which further login attempts from this key are rejected.
---
### 2.3 Table: `client_sdks`
Maintains the registry of authorized SDK client credentials and per-client throughput quotas.
```sql
CREATE TABLE IF NOT EXISTS client_sdks (
client_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
secret TEXT NOT NULL,
token TEXT NOT NULL,
rate_limit INTEGER NOT NULL DEFAULT 100,
status TEXT NOT NULL DEFAULT 'active',
total_events BIGINT NOT NULL DEFAULT 0,
last_active TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
- **`client_id`**: Lowercase slug identifying the producer.
- **`secret`**: 32-character random client secret used for verification.
- **`token`**: Current signed HMAC-SHA256 JWT token.
- **`rate_limit`**: Quota of maximum allowed events per second.
- **`status`**: Current access state (`active` or `revoked`).
- **`total_events`**: Lifetime count of successfully ingested events.
---
### 2.4 Table: `deleted_clients`
Tombstone table preventing reused client IDs or compromised stale tokens from regaining access.
```sql
CREATE TABLE IF NOT EXISTS deleted_clients (
client_id TEXT PRIMARY KEY,
deleted_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
- **`client_id`**: Identifier of permanently removed client. Checked during both HTTP ingestion and Socket.IO handshakes.
---
### 2.5 Table: `admin_settings`
Single-row platform configuration table storing runtime operational parameters.
```sql
CREATE TABLE IF NOT EXISTS admin_settings (
id TEXT PRIMARY KEY DEFAULT 'default',
server_port INTEGER NOT NULL DEFAULT 4000,
web_port INTEGER NOT NULL DEFAULT 3000,
max_events_per_sec INTEGER NOT NULL DEFAULT 100,
max_payload_size_kb INTEGER NOT NULL DEFAULT 100,
batch_flush_threshold INTEGER NOT NULL DEFAULT 500,
batch_flush_interval_ms INTEGER NOT NULL DEFAULT 2000,
file_rotation_mb INTEGER NOT NULL DEFAULT 100,
jwt_secret TEXT NOT NULL,
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
auto_prune_days INTEGER NOT NULL DEFAULT 30,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
---
## 3. Migration Architecture
DataStack executes automated, non-destructive schema migrations on application startup via `DatabaseEngine.ensureSchema()`:
1. **`CREATE TABLE IF NOT EXISTS`**: Ensures all tables exist without altering pre-existing data.
2. **`ALTER TABLE ... ADD COLUMN IF NOT EXISTS`**: Adds new columns with backwards-compatible defaults if upgrading from an older version.
3. **No Destructive Operations**: `DROP TABLE`, `DROP COLUMN`, and type truncations are strictly forbidden in production migrations. Existing databases retain all rows and constraints across container updates.
+28
View File
@@ -0,0 +1,28 @@
# Developer & Contributor Documentation Hub
Welcome to the DataStack Developer Hub. This documentation suite is designed for software engineers, systems programmers, and open-source contributors working on or extending the DataStack platform.
---
## Developer Portals & Architecture Specifications
| Document | Purpose & Scope | Target Audience |
| :--- | :--- | :--- |
| [Architecture Blueprint (`architecture.md`)](./architecture.md) | Deep dive into the dual-runtime architecture (Port 3000 Web vs Port 4000 Ingestion), internal bridge protocol, Redis AOF durability, and storage layout. | Core Contributors, Architects |
| [Codebase Structure (`codebase-structure.md`)](./codebase-structure.md) | File-by-file inventory of directories, source modules (`core/`, `app/`, `server/`, `components/`), and their relationships. | All Developers |
| [Local Development Setup (`local-setup.md`)](./local-setup.md) | Workstation setup, Node.js 22 requirements, dual-terminal local workflow, and in-memory vs Docker container modes. | New Contributors |
| [API Reference Specification (`api-reference.md`)](./api-reference.md) | OpenAPI-style contract reference for all REST endpoints (`app/api/*`) and Socket.IO real-time events (`server/socket/handlers.js`). | Integration Engineers |
| [Database Schema & Migrations (`database-schema.md`)](./database-schema.md) | PostgreSQL control-plane tables, DDL migrations, column definitions, and `prisma/schema.prisma` mirror. | Backend Engineers |
| [Security Architecture (`security.md`)](./security.md) | Cryptographic design: `crypto.scrypt`, WebCrypto HMAC sessions, brute-force IP lockout, CSRF Origin validation, and JWT verification. | Security Engineers |
| [Building & Packaging (`building-packaging.md`)](./building-packaging.md) | Next.js standalone compilation (`.next/standalone`), multi-stage Dockerfiles (`web`, `server`, `proxy`), and production container assembly. | DevOps, Release Engineers |
| [Testing Guide (`testing.md`)](./testing.md) | Guide to automated TypeScript integration tests (`npx tsx --test`) and Python SDK unit tests (`python3 -m unittest`). | QA, Test Engineers |
| [Contributing Guidelines (`contributing.md`)](./contributing.md) | Development rules, zero-SQL-telemetry enforcement, single-source UI primitives, multi-IDE memory synchronization (`MEMORY.md`), and PR checklist. | All Contributors |
---
## Core Engineering Principles
1. **Decoupled Runtime Separation**: Next.js 15 App Router on Port 3000 handles user interaction and REST exports. Express 5 + Socket.IO 4 on Port 4000 handles persistent WebSocket handshakes. Socket.IO is never instantiated inside Next.js serverless handlers.
2. **Zero Per-Row Relational Telemetry**: Telemetry streams are buffered in memory/Redis and flushed to daily CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. PostgreSQL is reserved exclusively for the control plane.
3. **Fail-Closed Security**: In production (`NODE_ENV=production`), missing or placeholder secrets halt startup immediately.
4. **Cooperative File Locking**: Concurrent flushes between Port 3000 and Port 4000 coordinate safely using `.lock` files with automatic stale lock expiration.
+96
View File
@@ -0,0 +1,96 @@
# Local Development Setup
This guide walks contributors through configuring and running DataStack locally on your development workstation.
---
## 1. Prerequisites
Ensure the following tools are installed:
- **Node.js**: Version `22.x` (LTS recommended)
- **Package Manager**: `npm` (v10+)
- **Python**: Version `3.10+` (optional, for developing and testing the Python client SDK)
- **Docker & Docker Compose**: Optional, useful for running local PostgreSQL and Redis instances.
---
## 2. Clone & Install Dependencies
```bash
git clone https://github.com/vibtoolshq/datastack.git
cd datastack
# Install root dependencies deterministically using the committed lockfile
npm ci
```
---
## 3. Environment Configuration
Copy the example environment configuration:
```bash
cp .env.example .env
```
### Local Development Defaults
In local development (`NODE_ENV=development`), DataStack includes built-in fallbacks:
- If `DATABASE_URL` is omitted, DataStack operates seamlessly in **In-Memory Mock Mode**, allowing you to test the full UI, client registration, and REST endpoints without running PostgreSQL.
- If `REDIS_URL` is omitted, DataStack uses an **in-memory RAM buffer queue**, persisting directly to `./storage/data`.
For full production parity during development, you can spin up local PostgreSQL and Redis containers:
```bash
docker run --name datastack-postgres -e POSTGRES_USER=datastack -e POSTGRES_PASSWORD=password -e POSTGRES_DB=datastack_db -p 5432:5432 -d postgres:16-alpine
docker run --name datastack-redis -p 6379:6379 -d redis:7-alpine redis-server --appendonly yes
```
Then update your `.env`:
```dotenv
DATABASE_URL=postgresql://datastack:password@127.0.0.1:5432/datastack_db
REDIS_URL=redis://127.0.0.1:6379
PORT=3000
PORT_SOCKET=4000
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
DATASTACK_ADMIN_EMAIL=admin@example.com
```
---
## 4. Running the Development Servers
DataStack requires two concurrent runtime processes. Open two terminal tabs in the project root:
### Terminal 1: Web Console & REST API (Port 3000)
```bash
npm run dev
```
Starts the Next.js 15 development server with Fast Refresh at `http://localhost:3000`.
### Terminal 2: Ingestion Server (Port 4000)
```bash
npm run server
```
Starts the dedicated Express 5 + Socket.IO ingestion service at `http://localhost:4000`.
---
## 5. First-Time Setup in Development
1. Open `http://localhost:3000/vcon/setup` in your browser.
2. Complete the setup wizard using your `DATASTACK_ADMIN_SETUP_KEY` from `.env`.
3. In `/vcon/clients`, register a test client application to generate your first JWT token.
4. In `/vcon/simulator`, test sending live events and observe the buffer counter in `/vcon/dashboard`.
---
## 6. Python SDK Development
If you are modifying the Python SDK in `sdk/python/datastack/`:
```bash
cd sdk/python/datastack
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r ../../../tests/requirements.txt # If running SDK test suites
```
+96
View File
@@ -0,0 +1,96 @@
# Security Architecture & Cryptographic Design
This document details the security principles, cryptographic implementations, authentication mechanisms, and attack mitigations enforced across DataStack.
---
## 1. Security Architecture Overview
DataStack follows a defense-in-depth model with strict boundary separation:
```text
┌────────────────────────────────────────────────────────────────────────┐
│ Internet / External Clients │
└──────────────────┬─────────────────────────────────┬───────────────────┘
│ HTTPS Ingestion (JWT) │ HTTPS Web Console
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Port 4000 (Ing) │ │ Port 3000 (Web) │
│ Socket.IO Hand │ │ Next.js Edge MW │
│ JWT Verify │ │ HMAC Session Ver │
│ Rate Limiting │ │ CSRF Origin Ver │
└─────────┬─────────┘ └─────────┬─────────┘
│ │
└──────── Authenticated ──────────┘
/internal/* Bridge
(DATASTACK_INTERNAL_SECRET)
```
---
## 2. Administrator Authentication & Sessions
### Password Storage (`crypto.scrypt`)
Administrator passwords are never stored in plaintext or reversible formats:
- **Salt Generation**: 16 cryptographically secure random bytes generated via `crypto.randomBytes(16)`.
- **Key Derivation Function**: Node.js native `crypto.scryptSync(password, salt, 64)`.
- **Constant-Time Verification**: Hashes are compared using `crypto.timingSafeEqual` to prevent timing side-channel attacks.
### Session Cookies (`datastack_admin_session`)
- **Signing Mechanism**: Signed with WebCrypto HMAC-SHA256 using `DATASTACK_ADMIN_SESSION_SECRET`.
- **Session Payload**:
```json
{
"userId": 1,
"email": "admin@example.com",
"tokenVersion": 1,
"exp": 1700086400000
}
```
- **Instant Global Invalidation**: Every administrative request validates `tokenVersion` against PostgreSQL. When an operator updates their password or selects **Log Out Other Sessions**, `tokenVersion` increments, instantly invalidating all issued cookies across all devices.
- **Cookie Security Flags**: `HttpOnly; SameSite=Lax; Path=/; Priority=High;` (with `Secure` enforced when `NODE_ENV=production`).
### Brute-Force IP Lockout
To prevent credential stuffing:
- Tracks failed login attempts per composite SHA-256 hash of `normalized_ip + ":" + normalized_email`.
- **Threshold**: 5 consecutive failed attempts locks the composite key for 15 minutes.
- Attempts are recorded in PostgreSQL (`admin_login_attempts`) and survive container restarts.
---
## 3. Client SDK Authentication & Access Control
### HMAC-SHA256 Client JWTs
- Each registered client receives a JWT signed by `DATASTACK_JWT_SECRET`.
- The token payload embeds `{ clientId, name, rateLimit, iat }`.
- Verified on both HTTP REST ingestion (`Authorization: Bearer <TOKEN>`) and Socket.IO handshakes (`auth.token`).
### Instant Credential Revocation & Tombstones
- **Rotation**: Rotating a client token generates a new secret and pushes the update to Port 4000 via `/internal/client-sync`. Active WebSocket sockets using the old token are severed immediately.
- **Revocation**: Setting status to `revoked` immediately halts ingestion with `CLIENT_REVOKED (403)`.
- **Tombstones (`deleted_clients`)**: Deleting a client registers its identifier in `deleted_clients`. Stale, leaked, or intercepted tokens can never be reused to write telemetry.
---
## 4. Cross-Site Request Forgery (CSRF) Mitigation
State-changing HTTP methods (`POST`, `PUT`, `DELETE`, `PATCH`) handled by Next.js App Router are protected in `middleware.ts`:
- Validates the request's `Origin` or `Referer` header against the incoming `Host` or `X-Forwarded-Host`.
- Cross-origin requests attempting to trigger administrative endpoints or manual flushes are rejected with `403 Forbidden` (`CSRF_FAILED`).
---
## 5. Internal Service Bridge Security
Communication between Port 3000 (Web) and Port 4000 (Ingestion Server) is isolated behind a shared secret:
- Header: `x-datastack-internal-secret: <DATASTACK_INTERNAL_SECRET>`.
- The ingestion server verifies incoming headers using `crypto.timingSafeEqual`.
- Unauthorized internal requests are rejected immediately with `403 Forbidden`.
---
## 6. Secret Fail-Closed Enforcement
When running in production (`NODE_ENV=production`):
- All secret keys (`DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SETUP_KEY`, `DATASTACK_INTERNAL_SECRET`) must be explicitly defined and have a minimum length of 32 characters.
- If any secret contains default placeholder strings (e.g., `replace-with-`, `your-secret`), the server halts during bootstrap with a fatal configuration error, preventing accidental deployment with insecure credentials.
+83
View File
@@ -0,0 +1,83 @@
# Automated Testing Guide
This guide details the automated test suites, execution commands, coverage domains, and verification standards for DataStack.
---
## 1. Test Architecture
DataStack utilizes a lightweight, native test architecture built upon the Node.js native test runner (`node:test`) and Python's `unittest` framework:
- **Zero Heavy Test Framework Bloat**: Tests run using `npx tsx --test`, executing directly against TypeScript source modules without requiring Jest or Mocha.
- **Fast Execution**: The complete TypeScript suite executes in under 2 seconds.
- **Isolation**: Each test suite initializes mock stores or clean temporary directories, guaranteeing zero side effects on production data.
---
## 2. Running TypeScript Test Suites
To run all TypeScript test suites in sequence:
```bash
npx --yes tsx --test tests/phase01-auth.test.ts \
tests/phase02-sync.test.ts \
tests/phase03-storage-buffer.test.ts \
tests/phase06-pwa-landing.test.ts \
tests/phase07-deployment.test.ts
```
### Test Suite Coverage Breakdown
#### 1. `tests/phase01-auth.test.ts` (Security & Authentication)
- **Production Secret Fail-Closed**: Verifies that empty, short (<32 chars), or placeholder secrets halt execution.
- **Session Cookie Invalidation**: Asserts that incrementing `tokenVersion` immediately causes session rejection.
- **Brute-Force Rate Limiting**: Validates that 5 failed login attempts lock out the composite IP + Email key for 15 minutes.
- **Client Token Redaction**: Ensures client secrets and raw tokens are omitted from public audit responses.
- **CSRF Origin Verification**: Tests that mismatched Origin/Referer headers return `403 Forbidden`.
#### 2. `tests/phase02-sync.test.ts` (Dual-Runtime Synchronization)
- **Internal Secret Security**: Tests timing-safe header validation on `/internal/*` routes.
- **Settings Synchronization**: Validates that updating settings in Port 3000 propagates instantly to Port 4000.
- **Client Revocation**: Confirms that revoking a client immediately severs active WebSocket connections.
- **JWT Secret Rotation**: Asserts that all client tokens are re-signed and old tokens are rejected.
#### 3. `tests/phase03-storage-buffer.test.ts` (Storage & Durability)
- **Redis Queue Durability**: Tests RAM queue fallback and Redis list buffering.
- **Batch Threshold & Interval**: Verifies that threshold count (500) and interval time (2000ms) trigger flushes.
- **Cooperative Locking**: Proves that `.lock` files prevent concurrent write collisions and that stale locks (>30s) are safely reclaimed.
- **Streaming Export**: Verifies non-blocking streaming CSV and JSON exports with client and date filtering.
#### 4. `tests/phase06-pwa-landing.test.ts` (PWA & Client Shell)
- **Web App Manifest**: Validates manifest schema, icons, and display modes.
- **Service Worker Lifecycle**: Tests cache strategies and offline shell handling.
#### 5. `tests/phase07-deployment.test.ts` (Containerization & Deployment Health)
- **Docker Compose Configuration**: Audits volume mounts (`datastack-data`, `pgdata`, `redisdata`).
- **Nginx Reverse Proxy**: Verifies routing rules for `/`, `/socket.io/`, and `/health`.
- **Unhydrated DB Auto-Pruning Guard**: Confirms that partition pruning is deferred until database settings are fully hydrated.
---
## 3. Running Python SDK Test Suites
The Python SDK includes a dedicated integration test suite verifying WebSocket streaming, heartbeat pings, HTTP fallbacks, and offline disk spooling:
```bash
python3 -m unittest tests/phase05_sdk_test.py
```
### Verified Behaviors:
- WebSocket connection and JWT handshake authentication.
- Offline spooling to local disk directory (`.datastack_spool/`) during network disruption.
- Automatic drain and delivery of spooled events when connectivity is restored.
- Streaming large `.csv` and `.json` files in chunks.
---
## 4. Continuous Integration Checklist
Before opening a pull request:
1. `npm run lint` must exit with `0` errors and `0` warnings.
2. `npm run build` must compile the Next.js standalone bundle successfully.
3. All 5 TypeScript test suites must pass (`23/23 tests passing`).
4. Python SDK test suite must pass (`3/3 tests passing`).