feat(auth): implement RBAC audit logging and session security
Integrate granular audit logging for admin actions, transition to role-based API access via `requireAdminApiSession`, and update session tokens to persist user roles and IDs for consistent multi-admin management.
This commit is contained in:
1 parent
2873b9904a
commit
5c977d6977
37 files changed
+5939
-349
No files matched your search
@@ -1,7 +1,7 @@
|
||||
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory
|
||||
|
||||
> **Last Synchronized**: 2026-11-19
|
||||
> **Status**: **V2-Phase-01 Complete & Verified (`BASELINE-2026-11-19-V2`) — `1 / 4` V2 Phases Done (Storage `.csv.gz` Auto-Compression, S3/R2 Off-Site Archive with AES-256-GCM Secret Vault, Per-Client `storageQuotaMb` & VPS Disk Pause Guard `HTTP 507`) — Verified (`89/89` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
|
||||
> **Last Synchronized**: 2026-11-20
|
||||
> **Status**: **V2-Phase-04 & `V2-Deploy-Fix-01` Complete & Verified (`BASELINE-2026-11-19-V2`) — `4 / 4` V2 Phases + Final Coolify Redeploy Hardening Done (`V2-Phase-01` Storage `.csv.gz` Compression, S3/R2 Off-Site Archive & Per-Client/Disk Quotas; `V2-Phase-02` Outbound Webhook Alerting, Per-Client IP/CIDR Allowlist & Token Expiry, Prometheus `/api/prometheus` Metrics Endpoint & Streaming `.ndjson` Export; `V2-Phase-03` Multi-Admin RBAC `owner`/`admin`/`operator`/`viewer` via `admin_team_members` & Immutable Admin Audit Log `admin_audit_logs`; `V2-Phase-04` Official Node.js / TypeScript Client SDK `sdk/node/datastack` & `/vcon/sdk` Multi-Language Guide; `V2-Deploy-Fix-01` `DEPLOY-V2-BUG-01..03`) — Verified (`102/102` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
|
||||
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
|
||||
|
||||
---
|
||||
@@ -314,6 +314,22 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **
|
||||
- **Per-Client Storage Quota (`storageQuotaMb`) & VPS Disk Pause Guard (`diskPauseThresholdPct`)**: Additive PostgreSQL columns (`client_sdks.storage_quota_mb` + 9 `admin_settings` columns in `core/Database/database.ts`, synced in `server/index.js` while preserving `ddlCount === 8`), `30s` TTL cached `statfs` + directory byte counters, and HTTP `507 Insufficient Storage` (`DISK_SPACE_LOW` / `STORAGE_QUOTA_EXCEEDED` — `V2-E04`, `V2-E05`) on `POST /api/events` and Port `4000` `sync_data`/`batch_sync` while keeping all Admin read/export/settings routes online.
|
||||
- **Admin UI Controls (`SettingsWorkspace.tsx`, `NewClientModal.tsx`, `ClientsTable.tsx`, `lib/vconUiHelpers.ts`)**: Added live VPS disk usage bar, `Auto-compress (.csv → .csv.gz)`, `Disk pause threshold (0 or 50–99%)`, full `S3 / R2 off-site archive` configuration with `Test S3 Connection` & `Archive Now`, and per-client `Storage quota (MB, 0 = unlimited)` in modal and table.
|
||||
- **Verified**: `89/89` TS tests (`4/4` in `tests/v2-phase01-storage-quotas.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/settings` (Storage tab) and `/vcon/clients` (`0` console errors).
|
||||
- **`V2-Phase-02` Complete & Verified (`2026-11-19`)** (`V2-FEAT-02`, `V2-FEAT-04`, `V2-FEAT-06`, `V2-E06`–`V2-E09`):
|
||||
- **Per-Client IP / CIDR Allowlist & Token Expiration (`core/Security/ipAllowlist.ts`, `core/sdk/sdkManager.ts`, `app/api/clients/**`, `app/api/events/route.ts`, `server/index.js`, `server/socket/handlers.js`)**: Validates and normalizes comma-separated IPv4, IPv4 CIDR (`/0..32`), IPv6, IPv6 CIDR (`/0..128`), IPv4-mapped IPv6 (`::ffff:`), loopback equivalence (`127.0.0.1` ↔ `::1`), and wildcard `*`; validates ISO `expiresAt` timestamps. Enforced across both Port `3000` `POST /api/events` and Port `4000` Socket.IO handshakes/events (`403 CLIENT_IP_FORBIDDEN` — `V2-E07`; `401 CLIENT_TOKEN_EXPIRED` — `V2-E08`), synced live via `/internal/client-sync` and additive PostgreSQL columns (`client_sdks.allowed_ips`, `client_sdks.expires_at`).
|
||||
- **Outbound Webhook Alerting (`core/alerts/webhookNotifier.ts`, `core/adminSettings/settingsManager.ts`, `app/api/settings/webhook-test/route.ts`)**: Sends Slack (`text`), Discord (`content`), and structured JSON webhook payloads signed with optional HMAC-SHA256 (`X-DataStack-Signature: sha256=<hex>`) and deduplicated per `event:clientId` via configurable `webhookCooldownSec` (`10..86400s`, default `300s`). Triggers non-blocking alerts on `DISK_SPACE_LOW`, `STORAGE_QUOTA_EXCEEDED`, `S3_ARCHIVE_FAILED`, `CLIENT_IP_FORBIDDEN`, and `CLIENT_TOKEN_EXPIRED`, plus on-demand `POST /api/settings/webhook-test` (`200` on delivery, `502 WEBHOOK_DELIVERY_FAILED` on upstream error — `V2-E06`). `webhook_secret_enc` is encrypted via AES-256-GCM Secret Vault and masked in `GET /api/settings` (`hasWebhookSecret`, `webhookSecretMask`).
|
||||
- **Prometheus Scrape Endpoint (`app/api/prometheus/route.ts`)**: `GET /api/prometheus` outputs Prometheus 0.0.4 text exposition format (`datastack_up`, `datastack_ingestion_bridge_up`, `datastack_active_sockets`, `datastack_events_total`, `datastack_buffer_events_total`, `datastack_storage_files_total`, `datastack_storage_bytes_total`, `datastack_storage_rows_total`, `datastack_disk_used_percent`, `datastack_disk_free_bytes`, `datastack_disk_total_bytes`, `datastack_clients_total`, `datastack_client_events_total`, `datastack_client_storage_bytes`). Protected by constant-time SHA-256 Bearer token comparison (`prometheus_metrics_token_enc` encrypted at rest) or active Admin Console session cookie (`401 PROMETHEUS_UNAUTHORIZED` — `V2-E09`).
|
||||
- **Streaming `.ndjson` Export (`app/api/export/route.ts`, `components/sections/ExportWorkspace.tsx`)**: Added `format=ndjson` (`Content-Type: application/x-ndjson; charset=utf-8`, `.ndjson` extension) streaming newline-delimited JSON objects across both `.csv` and `.csv.gz` partitions, with `NDJSON` format radio selector in `/vcon/export`.
|
||||
- **Admin UI Controls (`SettingsWorkspace.tsx`, `NewClientModal.tsx`, `ClientsTable.tsx`, `lib/vconUiHelpers.ts`)**: Added `Outbound webhook alerts` and `Prometheus metrics (/api/prometheus)` groups under `/vcon/settings` → **Security** tab, and `Allowed IPs / CIDR` + `Token expiration` controls in `/vcon/clients` New/Edit modals and table badges.
|
||||
- **Verified**: `93/93` TS tests (`4/4` in `tests/v2-phase02-alerts-ip-prometheus.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/settings` (`0` console errors).
|
||||
- **`V2-Phase-03` Complete & Verified (`2026-11-19`)** (`V2-FEAT-05`, `V2-E10`–`V2-E11`):
|
||||
- **Multi-Admin Role-Based Access Control (`core/Database/database.ts`, `core/auth/adminSession.ts`, `core/auth/adminUsers.ts`, `lib/adminAuth.ts`, `app/api/team/route.ts`, `app/api/team/[id]/route.ts`)**: Added canonical `admin_team_members` table in PostgreSQL (plus in-memory fallback) while preserving `admin_users` (`id = 1 CHECK (id = 1)`) as the immutable Primary Owner (`owner-1`, protected by `400 CANNOT_REVOKE_PRIMARY_OWNER` — `V2-E11`). Supports roles `owner` (rank 3 — full platform settings, secrets & team management), `admin` / `operator` (rank 2 — manage SDK clients, rotate tokens, manual buffer flushes, S3 test/archive), and `viewer` (rank 1 — read-only dashboards, events, clients, settings & exports). Enforced centrally via `requireAdminApiSession(req, { minRole })` returning `403 INSUFFICIENT_ROLE_PERMISSION` (`V2-E10`). Role/status/password changes on any team member immediately bump `token_version` to revoke existing sessions.
|
||||
- **Immutable Admin Audit Log (`core/auth/auditLog.ts`, `app/api/audit/route.ts`)**: Added canonical `admin_audit_logs` table in PostgreSQL (plus bounded 1,000-entry in-memory fallback) with automatic secret key redaction. Records governance events (`ADMIN_LOGIN`, `ADMIN_PROFILE_UPDATED`, `ADMIN_PASSWORD_UPDATED`, `TEAM_MEMBER_CREATED`, `TEAM_MEMBER_UPDATED`, `TEAM_MEMBER_REVOKED`, `TEAM_MEMBER_DELETED`, `CLIENT_CREATED`, `CLIENT_UPDATED`, `CLIENT_TOKEN_ROTATED`, `CLIENT_DELETED`, `SETTINGS_UPDATED`, `S3_ARCHIVE_TRIGGERED`, `WEBHOOK_TEST_TRIGGERED`) without ever inserting telemetry events into PostgreSQL. Exposed filterable `GET /api/audit` (`?action=`, `?actor=`, `?q=`, `?limit=`).
|
||||
- **Admin Console UI (`components/sections/TeamWorkspace.tsx`, `components/sections/AuditLogViewer.tsx`, `app/vcon/team/page.tsx`, `app/vcon/audit/page.tsx`, `components/layout/Sidebar.tsx`)**: Added `/vcon/team` (Team & RBAC directory, add member modal, role selector, session revocation & deletion) and `/vcon/audit` (filterable governance trail viewer) linked from `Sidebar.tsx`.
|
||||
- **Verified**: `97/97` TS tests (`4/4` in `tests/v2-phase03-rbac-audit.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/team` and `/vcon/audit` (`0` console errors).
|
||||
- **`V2-Phase-04` Complete & Verified (`2026-11-20`)** (`V2-FEAT-07`, `V2-E12`):
|
||||
- **Official Node.js / TypeScript Client SDK (`sdk/node/datastack/` — `@datastack/sdk` v2.2.0)**: Created `sdk/node/datastack/package.json`, `types.ts`, `spool.ts`, `index.ts`, and `README.md`. Supports `datastack.config.json` / `DATASTACK_*` env / constructor options with strict `normalizeClientId` vs JWT `clientId`/`sub` verification (`DataStackConfigError` — `V2-E12`), Socket.IO primary streaming (`connect()`, `send()`/`sync()`, `sendBatch()`/`batchSync()`), automatic HTTP `POST /api/events` fallback, recursive `413 PAYLOAD_TOO_LARGE` batch splitting, partial-batch `429 RATE_LIMIT_EXCEEDED` retry, typed terminal error mapping (`401 CLIENT_TOKEN_EXPIRED` & `403 CLIENT_IP_FORBIDDEN` -> `DataStackAuthError`, `507 STORAGE_QUOTA_EXCEEDED` & `DISK_SPACE_LOW` -> `DataStackQuotaExceededError`), durable `.datastack_spool/spool.jsonl` local disk spooling (`DiskSpooler`) & `flushSpool()`, direct `.csv`/`.json`/`.jsonl`/`.ndjson` file ingestion (`sendFile()` with deterministic SHA-256 `eventId` via `makeDeterministicFileEventId`), and graceful `await client.shutdown(timeoutMs)` drain.
|
||||
- **Multi-Language SDK Guide (`app/vcon/sdk/page.tsx`)**: Added interactive language tabs (`Python SDK (v2.2.0)`, `Node.js / TypeScript SDK (v2.2.0)`, `HTTP / cURL`) with live client JWT token population and copy-ready `config.datastack.py` / `datastack.config.json` snippets.
|
||||
- **Verified**: `101/101` TS tests (`4/4` in `tests/v2-phase04-node-sdk.test.ts`), `10/10` Python SDK tests, `npm run build` clean, and `npm run lint` (`0` errors / `0` warnings).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user