docs: update architecture documentation and manifests
Refine technical documentation to reflect current architecture, component structure, and routing conventions for the vcon console.
This commit is contained in:
1 parent
bb938c33c0
commit
367dd819f8
64 files changed
+910
-2661
No files matched your search
@@ -1,12 +1,14 @@
|
||||
# Database Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Database layer managing PostgreSQL schema via Prisma ORM.
|
||||
Manages the shared PostgreSQL connection pool (`pg.Pool`), automatic metadata table initialization, and live database health probes (`core/Database/database.ts`).
|
||||
|
||||
## Absolute Constraint
|
||||
PostgreSQL is utilized ONLY for:
|
||||
1. Client SDK registrations and API keys
|
||||
2. Admin settings and feature flags
|
||||
3. System audit logs
|
||||
## Managed PostgreSQL Tables
|
||||
PostgreSQL is dedicated strictly to platform metadata and security state:
|
||||
1. `admin_users`: Administrator account credentials (scrypt password hashes).
|
||||
2. `admin_login_attempts`: Brute-force login attempt counters and lockout timestamps.
|
||||
3. `client_sdks`: Registered SDK clients, signed JWT tokens, rate limits, status (`active` / `revoked`), and event counters.
|
||||
4. `admin_settings`: Global runtime settings (`max_events_per_sec`, `max_payload_size_kb`, `batch_flush_threshold`, `batch_flush_interval_ms`, `file_rotation_mb`, `jwt_secret`, `enable_idempotency`, `auto_prune_days`).
|
||||
|
||||
**Collected realtime data is NEVER stored per-row in PostgreSQL.** It is stored exclusively in rotated CSV files at `/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
## Storage Separation Rule
|
||||
Collected event streams are **never** inserted per-row into PostgreSQL; they are written exclusively in batches to partitioned CSV files under `/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
@@ -1,12 +1,9 @@
|
||||
# Security Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Enforces client rate-limiting, event idempotency verification, and payload size bounds.
|
||||
Enforces per-client sliding-window rate limits, payload size bounds, and event idempotency deduplication (`core/Security/security.ts`).
|
||||
|
||||
## Key Logic
|
||||
- **Idempotency**: Drops duplicate `clientId + eventId` within a 5-minute TTL window.
|
||||
- **Rate Limit**: Sliding per-second window configurable per client or globally.
|
||||
- **Payload Guard**: Rejects payloads exceeding 100KB per event with explicit codes.
|
||||
|
||||
## Dependencies
|
||||
- Native Node.js `Buffer`, `Map`
|
||||
## Key Responsibilities
|
||||
- **Idempotency Deduplication**: Rejects duplicate `clientId:eventId` pairs within a 5-minute TTL window.
|
||||
- **Sliding-Window Rate Limiter**: Enforces the lower of the client's configured `rateLimit` and the global `maxEventsPerSec` limit.
|
||||
- **Payload Size Guard**: Rejects individual event payloads exceeding `maxPayloadSizeBytes` (default `100 KB`).
|
||||
@@ -1,15 +0,0 @@
|
||||
# Server Runtime Manifest
|
||||
|
||||
## Purpose
|
||||
Express.js and Socket.IO server engine hosting WebSocket connections, JWT authentication handshakes, and event intake on port 4000.
|
||||
|
||||
## Key Logic
|
||||
- Initializes Express server on Port 4000 (decoupled from Next.js on Port 3000)
|
||||
- Mounts Socket.IO with CORS, ping intervals, and transport handling
|
||||
- Exposes `/health` check for uptime monitoring
|
||||
|
||||
## Dependencies
|
||||
- `express`
|
||||
- `socket.io`
|
||||
- `cors`
|
||||
- `core/connector/socketHandler`
|
||||
@@ -1,14 +1,10 @@
|
||||
# Storage Engine Manifest
|
||||
# CSV Storage Engine Manifest
|
||||
|
||||
## Purpose
|
||||
File-based CSV storage engine. All collected data is persisted strictly to CSV files (zero per-row database inserts).
|
||||
Partitioned CSV file storage engine (`core/Storage/csvStorage.ts`) responsible for writing, rotating, inspecting, and pruning collected event datasets on disk.
|
||||
|
||||
## Conventions
|
||||
- **Path**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
||||
- **Schema**: `clientId, eventId, timestamp, payload`
|
||||
- **Rotation**: Rotates per-day or when single file reaches 100MB limit (`-part2.csv`).
|
||||
- **Batching**: Never writes per-event. Flushes buffered arrays using `json2csv` parser.
|
||||
|
||||
## Dependencies
|
||||
- `json2csv`
|
||||
- Node.js `fs`, `path`
|
||||
## Storage Conventions
|
||||
- **Path Pattern**: `/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
||||
- **Columns**: `clientId,eventId,timestamp,payload`
|
||||
- **Size & Daily Rotation**: Creates a new daily file per client and rotates to `{YYYY-MM-DD}-part2.csv` when a file reaches `maxFileSizeBytes` (configurable via `fileRotationMb`, default `100MB`).
|
||||
- **Retention Pruning**: `pruneExpiredPartitions(autoPruneDays)` automatically removes daily CSV partitions older than the configured retention period.
|
||||
@@ -1,17 +1,15 @@
|
||||
# Admin Settings Manifest
|
||||
# Admin Settings Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Global administrative configuration store and live subsystem parameter tuning.
|
||||
Manages global runtime configuration (`core/adminSettings/settingsManager.ts`), persisting settings to the PostgreSQL `admin_settings` table and applying live updates across backend engines.
|
||||
|
||||
## Managed Parameters
|
||||
- Ports: Ingestion (:4000) and Web (:3000)
|
||||
- Rate limiting and maximum payload size bounds
|
||||
- Buffer flush thresholds (count & timer)
|
||||
- CSV rotation limits (100MB / per-day)
|
||||
- JWT signature secret keys
|
||||
|
||||
## Dependencies
|
||||
- `core/Security/security`
|
||||
- `core/data/buffer`
|
||||
- `core/auth/jwt`
|
||||
- `core/Storage/csvStorage`
|
||||
- `serverPort` (`4000`) and `webPort` (`3000`)
|
||||
- `maxEventsPerSec`: Default per-client throughput ceiling (events/sec)
|
||||
- `maxPayloadSizeKb`: Maximum event payload size in kilobytes
|
||||
- `batchFlushThreshold`: Number of queued events that triggers an immediate CSV flush
|
||||
- `batchFlushIntervalMs`: Background timer interval for periodic CSV flushes
|
||||
- `fileRotationMb`: Maximum CSV partition file size before rotating to `-part2.csv`
|
||||
- `autoPruneDays`: Retention window in days for automatic CSV partition pruning
|
||||
- `enableIdempotency`: 5-minute TTL duplicate event filtering toggle
|
||||
- `jwtSecret`: HMAC secret used to sign and verify Client SDK JWT tokens (updating this automatically re-signs existing client tokens)
|
||||
@@ -1,9 +1,8 @@
|
||||
# API Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Core administrative API services supplying telemetry, aggregation metrics, and export data feeds to Next.js route handlers.
|
||||
Aggregates system health, buffer status, client statistics, and CSV storage metrics for the `/vcon/dashboard` administration console (`core/apiEngine/apiService.ts`).
|
||||
|
||||
## Responsibilities
|
||||
- Aggregate telemetry across Buffer, CSV Storage, Sockets, and Security
|
||||
- Filter and list CSV export datasets
|
||||
- Provide consolidated administrative health payloads
|
||||
- Queries the Port 4000 ingestion server (`${INGESTION_INTERNAL_URL}/health`) for live `activeSockets` and remote `bufferSize`.
|
||||
- Combines remote Socket.IO metrics with local `DataBufferEngine`, `CSVStorageManager`, `SDKManager`, `SecurityEngine`, and `DatabaseEngine` health states.
|
||||
@@ -1,12 +1,9 @@
|
||||
# Core Auth Manifest
|
||||
# Authentication Engine Manifest
|
||||
|
||||
## Purpose
|
||||
JWT-based authentication engine for Client SDK connections during Socket.IO handshakes.
|
||||
Provides authentication and session security for both Administrator Console users and Client SDK connections (`core/auth/*`).
|
||||
|
||||
## Key Logic
|
||||
- Signs JWT with `clientId`, `rateLimit`, and timestamp.
|
||||
- Verifies bearer tokens during Socket.IO connection handshake.
|
||||
- Disconnects invalid or expired tokens immediately.
|
||||
|
||||
## Dependencies
|
||||
- `jsonwebtoken`
|
||||
## Modules
|
||||
- `jwt.ts`: Signs and verifies Client SDK JWT tokens (`clientId`, `name`, `rateLimit`) using `DATASTACK_JWT_SECRET` or the active `admin_settings.jwt_secret`.
|
||||
- `adminSession.ts`: WebCrypto HMAC-SHA256 administrator session token signer/verifier (`datastack_admin_session` HTTP-only cookie) compatible with both Next.js Edge Middleware and Node.js route handlers.
|
||||
- `adminUsers.ts`: PostgreSQL-backed administrator account store (`admin_users`) using `crypto.scrypt` password hashing, timing-safe verification, and brute-force login lockout tracking (`admin_login_attempts`).
|
||||
@@ -1,17 +0,0 @@
|
||||
# Connector Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Express + Socket.IO connection intake, WebSocket handshake validation, and real-time event router.
|
||||
|
||||
## Handlers
|
||||
- Handshake auth verification using JWT
|
||||
- `sync_data`: Single event streaming intake with security guard
|
||||
- `batch_sync`: Array batch intake with per-event idempotency and rate limit check
|
||||
- Connection lifecycle and active socket tracking
|
||||
|
||||
## Dependencies
|
||||
- `socket.io`
|
||||
- `core/auth/jwt`
|
||||
- `core/Security/security`
|
||||
- `core/data/buffer`
|
||||
- `core/sdk/sdkManager`
|
||||
@@ -18,6 +18,7 @@ interface BufferGlobalState {
|
||||
|
||||
type BufferGlobals = typeof globalThis & {
|
||||
__dataStackBufferState?: BufferGlobalState;
|
||||
__dataStackBufferShutdownRegistered?: boolean;
|
||||
};
|
||||
|
||||
const globals = globalThis as BufferGlobals;
|
||||
@@ -66,6 +67,19 @@ export class DataBufferEngine {
|
||||
void this.flush();
|
||||
}, state.batchFlushIntervalMs);
|
||||
}
|
||||
|
||||
if (
|
||||
typeof process !== 'undefined' &&
|
||||
typeof process.on === 'function' &&
|
||||
!globals.__dataStackBufferShutdownRegistered
|
||||
) {
|
||||
globals.__dataStackBufferShutdownRegistered = true;
|
||||
const flushOnExit = () => {
|
||||
void this.flush().catch(() => undefined);
|
||||
};
|
||||
process.on('SIGTERM', flushOnExit);
|
||||
process.on('SIGINT', flushOnExit);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,12 +1,9 @@
|
||||
# Data Management Manifest
|
||||
# Data Buffer Engine Manifest
|
||||
|
||||
## Purpose
|
||||
In-memory and Redis buffer engine for high-throughput batching of collected events.
|
||||
In-memory event batching engine (`core/data/buffer.ts`) that queues incoming events and flushes them in batches to partitioned CSV files on disk.
|
||||
|
||||
## Key Logic
|
||||
- Buffers events in memory to prevent per-event disk I/O.
|
||||
- Flushes batch to CSV Storage on threshold (500 items) or interval (2000ms).
|
||||
- Delivers live events to UI subscribers for real-time monitoring.
|
||||
|
||||
## Dependencies
|
||||
- `core/Storage/csvStorage`
|
||||
## Key Responsibilities
|
||||
- Queues validated events in memory to avoid per-event disk I/O overhead.
|
||||
- Automatically flushes queued batches to `CSVStorageManager.appendBatch()` when the queue reaches `batchFlushThreshold` (`500` events) or when the periodic flush timer (`batchFlushIntervalMs`, default `2000ms`) fires.
|
||||
- Supports dynamic timer rescheduling via `setFlushInterval(intervalMs)` when updated in `/vcon/settings`.
|
||||
+12
-13
@@ -1,15 +1,14 @@
|
||||
# Core Backend Engine Manifest
|
||||
# Core Backend Modules Manifest
|
||||
|
||||
## Purpose
|
||||
Modular backend subsystems for DataStack:
|
||||
- `auth`: JWT verification and generation
|
||||
- `connector`: Socket.IO intake and handshake handling
|
||||
- `data`: In-memory and Redis buffer management
|
||||
- `Storage`: CSV file partition writing and rotation
|
||||
- `Security`: Rate limiter, payload validator, idempotency checker
|
||||
- `sdk`: Client SDK registry and credentials
|
||||
- `adminSettings`: Configuration state
|
||||
- `Database`: PostgreSQL / Prisma schema metadata
|
||||
- `notification`: System alerts and audit events
|
||||
- `Server`: Express + Socket.IO process (Port 4000)
|
||||
- `apiEngine`: Aggregated data providers for REST API
|
||||
Core TypeScript backend engines powering authentication, client SDK management, event buffering, CSV partition storage, and PostgreSQL metadata persistence.
|
||||
|
||||
## Submodules
|
||||
- `auth/`: Client JWT signing/verification (`jwt.ts`), WebCrypto HMAC-SHA256 admin session tokens (`adminSession.ts`), and PostgreSQL scrypt admin user/lockout management (`adminUsers.ts`).
|
||||
- `Database/`: Shared PostgreSQL connection pool (`pg.Pool`), automatic schema initialization (`client_sdks`, `admin_settings`), and live `SELECT 1` health checks (`database.ts`).
|
||||
- `sdk/`: Client SDK registration, token rotation, status revocation, deletion, and event counters backed by PostgreSQL (`sdkManager.ts`).
|
||||
- `adminSettings/`: Platform runtime configuration store backed by PostgreSQL with live propagation to security, buffer, and storage engines (`settingsManager.ts`).
|
||||
- `data/`: In-memory event batching queue with configurable flush threshold and interval timer (`buffer.ts`).
|
||||
- `Storage/`: Partitioned CSV writer (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`), size-based file rotation (`-part2.csv`), and retention pruning (`csvStorage.ts`).
|
||||
- `Security/`: Per-client sliding-window rate limiter, 5-minute TTL idempotency deduplication, and payload byte-size validation (`security.ts`).
|
||||
- `apiEngine/`: Consolidated dashboard metrics aggregator combining local storage stats with Port 4000 `/health` telemetry (`apiService.ts`).
|
||||
@@ -1,9 +0,0 @@
|
||||
# Notification Engine Manifest
|
||||
|
||||
## Purpose
|
||||
System-level alerts and event notification service for admin awareness.
|
||||
|
||||
## Key Logic
|
||||
- Records security violations (unauthorized JWT attempts, rate limits)
|
||||
- Logs storage events (batch flushes, rotation to part files, CSV write failures)
|
||||
- Real-time notification feed for `/vcon/` admin UI.
|
||||
@@ -1,12 +1,9 @@
|
||||
# SDK Management Manifest
|
||||
# SDK Management Engine Manifest
|
||||
|
||||
## Purpose
|
||||
Manages client SDK credentials, client registration, tokens, and client-level throughput rate limits.
|
||||
Manages Client SDK registrations, JWT credentials, throughput rate limits, and activity counters (`core/sdk/sdkManager.ts`) backed by PostgreSQL (`client_sdks`).
|
||||
|
||||
## Key Logic
|
||||
- Generates API keys and signed JWT client tokens.
|
||||
- Tracks client registration status (active/revoked), event counters, and last-active timestamp.
|
||||
- Allows immediate token revocation or rate limit reconfiguration.
|
||||
|
||||
## Dependencies
|
||||
- `core/auth/jwt`
|
||||
## Key Responsibilities
|
||||
- Registers new clients with unique secret keys and signed JWT tokens.
|
||||
- Persists client status (`active` / `revoked`), per-client `rateLimit`, `totalEvents`, and `lastActive` timestamps in PostgreSQL `client_sdks`.
|
||||
- Supports one-click credential rotation (`regenerateTokenAsync`), status toggling (`updateClientAsync`), permanent removal (`deleteClientAsync`), and bulk token re-signing (`resignAllClientTokensAsync`) when the platform JWT secret is updated.
|
||||
Reference in new issue
Block a user