feat(auth): enhance security and audit logging

- Integrate webhook alerts for admin login lockouts.
- Implement detailed audit logging for login, logout, and setup actions.
- Improve client revocation event tracking.
- Standardize error handling in profile updates.
- Enforce 'operator' role for event API access.
This commit is contained in:
Team Dark Devil committed 2026-11-21 00:06:55 +06:00
1 parent 5c977d6977
commit 8132087085
52 files changed
+4434 -407

No files matched your search

+57 -10
View File
@@ -1,7 +1,7 @@
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory
> **Last Synchronized**: 2026-11-20
> **Status**: **V2-Phase-04 & `V2-Deploy-Fix-01` Complete & Verified (`BASELINE-2026-11-19-V2`) — `4 / 4` V2 Phases + Final Coolify Redeploy Hardening Done (`V2-Phase-01` Storage `.csv.gz` Compression, S3/R2 Off-Site Archive & Per-Client/Disk Quotas; `V2-Phase-02` Outbound Webhook Alerting, Per-Client IP/CIDR Allowlist & Token Expiry, Prometheus `/api/prometheus` Metrics Endpoint & Streaming `.ndjson` Export; `V2-Phase-03` Multi-Admin RBAC `owner`/`admin`/`operator`/`viewer` via `admin_team_members` & Immutable Admin Audit Log `admin_audit_logs`; `V2-Phase-04` Official Node.js / TypeScript Client SDK `sdk/node/datastack` & `/vcon/sdk` Multi-Language Guide; `V2-Deploy-Fix-01` `DEPLOY-V2-BUG-01..03`) — Verified (`102/102` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
> **Status**: **R3 Production Remediation (`BASELINE-2026-11-20-R3`): `3 / 4` Phases Complete (`12 / 16` Bugs Resolved — `R3-Phase-01`, `R3-Phase-02`, & `R3-Phase-03` Verified Complete; `R3-Phase-04` ready for `Proceed to R3-Phase-04`).**
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
---
@@ -146,15 +146,15 @@ datastack/
│ ├── configuration-reference.md # Complete environment variable & setting bounds
│ ├── dependencies.md # Runtime & dev dependencies audit with rationales
│ └── glossary.md # Technical domain terminology & concept glossary
├── notes/ # Production Hardening Governance & Scope-Lock Suite (`BASELINE-2026-11-19-R2`):
│ ├── README.md # Master governance index & 6-step scope-locked execution protocol
│ ├── FORENSIC_AUDIT_REPORT.md # 100% raw-code verified audit (18 bugs: 0C/6H/8M/4L), Demo vs Prod & UI-vs-Reality matrices
│ ├── BASELINE_FREEZE.md # Frozen baseline snapshot (`BASELINE-2026-11-19-R2`) & 1:1 bug alignment
│ ├── PRODUCTION_READINESS_SCOPE_LOCK.md # (Doc 001) Exact bug fix contracts & per-phase Scope Locks (`Phase-01`..`Phase-04`)
│ ├── PRODUCTION_ROADMAP.md # (Doc 002) Dependency-ordered A-Z 4-phase roadmap & burn-down tracker
│ ├── PHASE_EXECUTION_LOG.md # (Doc 003) Live phase completion log (`4/4` complete) & zero-regression guardrails
│ ├── ERROR_HANDLING_MATRIX.md # (Doc 004) Active error-handling catalog & 9 planned hardening additions
│ └── ACTUAL_IMPLEMENTATION_STATUS.md # Ground-truth working vs completed-per-phase vs remaining ledger
├── notes/ # Production Remediation Governance & Scope-Lock Suite (`BASELINE-2026-11-20-R3`):
│ ├── README.md # Master governance index, 6-step execution workflow & phase approval commands
│ ├── FORENSIC_AUDIT_REPORT.md # 100% raw-code verified audit (16 bugs: BUG-01..BUG-16) & Demo vs Prod matrix
│ ├── BASELINE_FREEZE.md # Frozen baseline snapshot (`BASELINE-2026-11-20-R3`) & 1:1 bug alignment
│ ├── PRODUCTION_READINESS_SCOPE_LOCK.md # (Doc 001) Exact bug fix contracts & per-phase Scope Locks (`R3-Phase-01`..`04`)
│ ├── PRODUCTION_ROADMAP.md # (Doc 002) Dependency-ordered A–Z 4-phase roadmap & burn-down tracker
│ ├── PHASE_EXECUTION_LOG.md # (Doc 003) Live phase completion log (`0/4` complete in R3 cycle) & zero-drift guardrails
│ ├── ERROR_HANDLING_MATRIX.md # (Doc 004) Active error-handling catalog & 7 planned fixes (`ERR-R3-01`..`07`)
│ └── ACTUAL_IMPLEMENTATION_STATUS.md # Ground-truth working vs completed-per-phase vs remaining ledger (16 items remaining)
└── tests/ # Automated verification suites:
├── phase01-auth.test.ts # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
├── phase02-sync.test.ts # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
@@ -330,6 +330,53 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **
- **Official Node.js / TypeScript Client SDK (`sdk/node/datastack/` — `@datastack/sdk` v2.2.0)**: Created `sdk/node/datastack/package.json`, `types.ts`, `spool.ts`, `index.ts`, and `README.md`. Supports `datastack.config.json` / `DATASTACK_*` env / constructor options with strict `normalizeClientId` vs JWT `clientId`/`sub` verification (`DataStackConfigError` — `V2-E12`), Socket.IO primary streaming (`connect()`, `send()`/`sync()`, `sendBatch()`/`batchSync()`), automatic HTTP `POST /api/events` fallback, recursive `413 PAYLOAD_TOO_LARGE` batch splitting, partial-batch `429 RATE_LIMIT_EXCEEDED` retry, typed terminal error mapping (`401 CLIENT_TOKEN_EXPIRED` & `403 CLIENT_IP_FORBIDDEN` -> `DataStackAuthError`, `507 STORAGE_QUOTA_EXCEEDED` & `DISK_SPACE_LOW` -> `DataStackQuotaExceededError`), durable `.datastack_spool/spool.jsonl` local disk spooling (`DiskSpooler`) & `flushSpool()`, direct `.csv`/`.json`/`.jsonl`/`.ndjson` file ingestion (`sendFile()` with deterministic SHA-256 `eventId` via `makeDeterministicFileEventId`), and graceful `await client.shutdown(timeoutMs)` drain.
- **Multi-Language SDK Guide (`app/vcon/sdk/page.tsx`)**: Added interactive language tabs (`Python SDK (v2.2.0)`, `Node.js / TypeScript SDK (v2.2.0)`, `HTTP / cURL`) with live client JWT token population and copy-ready `config.datastack.py` / `datastack.config.json` snippets.
- **Verified**: `101/101` TS tests (`4/4` in `tests/v2-phase04-node-sdk.test.ts`), `10/10` Python SDK tests, `npm run build` clean, and `npm run lint` (`0` errors / `0` warnings).
- **Post-V2 Full-Codebase Forensic Audit & Baseline Freeze (`BASELINE-2026-11-20-V2-R2`)**:
- Conducted a 100% raw-source-code forensic audit across all Port 3000 APIs/engines, Port 4000 Socket.IO workers, `/vcon/*` UI pages, and Python/Node SDKs (`notes/FORENSIC_AUDIT_REPORT.md`), identifying **10 verified bugs** (`BUG-V2-AUDIT-01` .. `BUG-V2-AUDIT-10`: `5` High, `4` Medium, `1` Low).
- Froze the current verified codebase (`102/102` TS tests, `10/10` Python tests, `0` build/lint errors, `0` source code changes) as **`BASELINE-2026-11-20-V2-R2`** in `notes/BASELINE_FREEZE.md`.
- Prepared the complete 7-document governance & Scope-Lock suite in `notes/`:
- `notes/FORENSIC_AUDIT_REPORT.md` (`10` verified bugs with verbatim code blocks & line numbers)
- `notes/BASELINE_FREEZE.md` (1:1 baseline-to-audit mapping)
- `notes/PRODUCTION_READINESS_SCOPE_LOCK.md` (Doc `001` — exact fix contracts & strict per-phase Scope Locks)
- `notes/PRODUCTION_ROADMAP.md` (Doc `002` — A–Z 3-phase execution roadmap: `V2-Audit-Phase-01` Auth/Email/Audit, `V2-Audit-Phase-02` Port 4000 Webhooks/IPv6 CIDR/Health/Python SDK, `V2-Audit-Phase-03` UI Disk Usage/Client Expiry/CSV.GZ Badges)
- `notes/PHASE_EXECUTION_LOG.md` (Doc `003` — `3 / 3` phases completed, `0 / 3` remaining, `10 / 10` bugs resolved)
- `notes/ERROR_HANDLING_MATRIX.md` (Doc `004` — active error catalog + `ERR-V2-AUDIT-01`..`05` all `5 / 5` completed)
- `notes/ACTUAL_IMPLEMENTATION_STATUS.md` (ground-truth working vs per-phase delta — `0` remaining items)
- `notes/README.md` (governance index & 6-step execution protocol)
- **`V2-Audit-Phase-01` Complete & Verified (`2026-11-20`)** (`BUG-V2-AUDIT-02`, `BUG-V2-AUDIT-03`, `BUG-V2-AUDIT-04`, `ERR-V2-AUDIT-01`):
- **`BUG-V2-AUDIT-04`**: Updated `revokeAdminSessions(target)` in `core/auth/adminUsers.ts` to accept explicit `{ memberId, email }` and passed `{ memberId: check.session.memberId, email: check.session.email }` from `POST /api/auth/logout` (`app/api/auth/logout/route.ts`), eliminating the multi-admin logout global race condition (`globals.__dataStackLastVerifiedMemberId`).
- **`BUG-V2-AUDIT-03` (`ERR-V2-AUDIT-01`)**: Enforced cross-table (`admin_users` ↔ `admin_team_members`) and in-memory peer duplicate email checks in `updateAdminAccount()` (`core/auth/adminUsers.ts`) returning `code: 'ADMIN_EMAIL_CONFLICT'`, mapped to HTTP `409 Conflict` in `PUT /api/auth/profile` (`app/api/auth/profile/route.ts`).
- **`BUG-V2-AUDIT-02`**: Wired all 4 missing audit actions (`ADMIN_LOGOUT` in `POST /api/auth/logout`, `ADMIN_SETUP` in `POST /api/auth/setup`, `STORAGE_FLUSH_TRIGGERED` in `POST /api/storage`, and `CLIENT_REVOKED` when `updates.status === 'revoked'` in `PATCH /api/clients/[id]`) and synchronized `ACTION_FILTER_OPTIONS` in `components/sections/AuditLogViewer.tsx` 1:1 with `AuditActionType` in `core/auth/auditLog.ts` (all 18/18 governance actions active).
- **Verified**: `103/103` TS tests (`5/5` in `tests/v2-phase03-rbac-audit.test.ts`), `10/10` Python SDK tests, `npm run build` clean, and `npm run lint` (`0` errors / `0` warnings).
- **`V2-Audit-Phase-02` Complete & Verified (`2026-11-20`)** (`BUG-V2-AUDIT-05`, `BUG-V2-AUDIT-06`, `BUG-V2-AUDIT-08`, `BUG-V2-AUDIT-09`, `ERR-V2-AUDIT-02`..`05`):
- **`BUG-V2-AUDIT-05` (`ERR-V2-AUDIT-02`)**: Wired outbound HMAC-SHA256 webhook alert dispatching (`triggerWebhookAlert` + `dispatchRejectionWebhook`) on Port 4000 Socket.IO (`server/index.js`, `server/socket/handlers.js`) for `CLIENT_TOKEN_EXPIRED`, `CLIENT_IP_FORBIDDEN`, `DISK_SPACE_LOW`, and `STORAGE_QUOTA_EXCEEDED` across handshake, `sync_data`, `batch_sync`, and `ping_verify`, with AES-256-GCM `decryptVaultSecret` in `syncSettingsFromDatabase()` and live `webhookSecret` propagation in `core/adminSettings/settingsManager.ts`.
- **`BUG-V2-AUDIT-06` (`ERR-V2-AUDIT-03`)**: Upgraded `isClientIpAllowed()` and `stripIpv6MappedPrefix()` in both `core/Security/ipAllowlist.ts` and `server/index.js` to use Node's built-in `net.BlockList` (`addSubnet`) for all IPv4 (`/0..32`) and IPv6 (`/0..128`) CIDR subnets and normalize `[ipv6]:port`, `ipv4:port`, `%zone`, and `::ffff:`.
- **`BUG-V2-AUDIT-08` (`ERR-V2-AUDIT-04`)**: Enforced `isClientTokenExpired(registeredClient.expiresAt)` (`401 CLIENT_TOKEN_EXPIRED`) and `isClientIpAllowed(extractClientIp(req), registeredClient.allowedIps)` (`403 CLIENT_IP_FORBIDDEN`) in `GET /api/health` (`app/api/health/route.ts`).
- **`BUG-V2-AUDIT-09` (`ERR-V2-AUDIT-05`)**: Added `"CLIENT_TOKEN_EXPIRED"` and `"CLIENT_IP_FORBIDDEN"` to `_is_non_retryable_auth_error()` and `_extract_auth_error_code()` in `sdk/python/datastack/datastack/client.py`, immediately raising `DataStackAuthError(msg, code=...)` matching the Node.js SDK 1:1.
- **Verified**: `104/104` TS tests (`5/5` in `tests/v2-phase02-alerts-ip-prometheus.test.ts`), `11/11` Python SDK tests (`tests/phase05_sdk_test.py`), `npm run build` clean, and `npm run lint` (`0` errors / `0` warnings).
- **`V2-Audit-Phase-03` Complete & Verified (`2026-11-20`)** (`BUG-V2-AUDIT-01`, `BUG-V2-AUDIT-07`, `BUG-V2-AUDIT-10`):
- **`BUG-V2-AUDIT-01`**: Aligned `DiskUsageSnapshot` (`core/Storage/csvStorage.ts`), `GET /api/storage` (`app/api/storage/route.ts`), and `SettingsWorkspace.tsx` (`components/sections/SettingsWorkspace.tsx`) with `available: boolean` (`totalBytes > 0`) and `freeBytes: number` (`availableBytes`) so `/vcon/settings` -> Storage -> "VPS disk usage" renders real disk utilization percentage, progress bar, and free/total GiB instead of `"Statfs unavailable"`.
- **`BUG-V2-AUDIT-07`**: Added `isClientExpired(expiresAt, nowMs)` in `app/vcon/clients/page.tsx` and excluded expired tokens from `onlineCount`, `pendingCount`, and `"online"`/`"pending"` filter tabs (classifying non-revoked expired clients under `"offline"`), matching `ClientsTable.tsx` `"expired"` badges 1:1.
- **`BUG-V2-AUDIT-10`**: Updated `components/sections/ExportWorkspace.tsx` and `app/vcon/storage/page.tsx` to render a `CSV.GZ` format badge whenever `file.compressed || file.filename.toLowerCase().endsWith('.csv.gz')` is true (`CSV` otherwise).
- **Verified**: `105/105` TS tests (`4/4` in `tests/v2-phase01-storage-quotas.test.ts`), `11/11` Python SDK tests (`tests/phase05_sdk_test.py`), `npm run build` clean, and `npm run lint` (`0` errors / `0` warnings). All `3 / 3` phases (`10 / 10` bugs) resolved.
- **R3 Full-Stack Production Remediation (`BASELINE-2026-11-20-R3` — `16` Verified Bugs across `4` Scope-Locked Phases)**:
- **`R3-Phase-01` Complete & Verified (`2026-11-20`)** (`BUG-01`, `BUG-03`, `BUG-04`, `BUG-06`, `BUG-10`, `BUG-13`, `BUG-15`; `ERR-R3-01`..`ERR-R3-04`):
- **`BUG-01`**: Aligned Port 4000 `decryptVaultSecret()` (`server/index.js`) 1:1 with `core/Security/secretVault.ts` (`v1:` prefix, `DATASTACK_ADMIN_SESSION_SECRET`, `base64url` decoding, fail-closed `""` + warning) and injected `DATASTACK_ADMIN_SESSION_SECRET` (`SERVICE_REALBASE64_64_ADMINSESSION`) into `docker/docker-compose.yml` `services.ingestion.environment`.
- **`BUG-03`**: Updated `AuthService.generateClientToken(payload, expiresIn?)` (`core/auth/jwt.ts`) to omit JWT `exp` when `expiresIn` is `null`/`undefined`/`""`, and passed `null` across all 6 minting sites in `core/sdk/sdkManager.ts` so DB `client_sdks.expires_at` (`Never expires`, `>30d`, live extension) is authoritative.
- **`BUG-04`**: Enforced `requireAdminApiSession(req, { minRole: 'operator' })` on admin-session calls to `POST /api/events` (`app/api/events/route.ts`), returning `403 INSUFFICIENT_ROLE_PERMISSION` for read-only `viewer` accounts.
- **`BUG-06`**: Wrapped `decryptSecret(settings.webhookSecretEnc)` in `SettingsManager.notifyIngestionSettingsSync()` (`core/adminSettings/settingsManager.ts`) in `try/catch` (`""` fallback + warning) and deferred `decryptSecret` in `POST /api/settings/webhook-test` (`app/api/settings/webhook-test/route.ts`) until neither draft `webhookSecret` nor `clearWebhookSecret` is provided.
- **`BUG-10`**: Wired non-blocking `ADMIN_LOGIN_LOCKOUT` webhook alert dispatch inside the `429 TOO_MANY_ATTEMPTS` lockout branch of `POST /api/auth/login` (`app/api/auth/login/route.ts`).
- **`BUG-13`**: Unified Port 4000 `formatOutboundWebhookBody()` (`server/index.js`) with Port 3000 `core/alerts/webhookNotifier.ts` (`instance`, `event`, `severity`, `title`, `message`, `clientId`, `details`, `timestamp`, `text`, `content`).
- **`BUG-15`**: Tightened `POST /api/storage/s3-test` (`app/api/storage/s3-test/route.ts`) from `operator` to `owner`-only RBAC.
- **Verified**: `6/6` in `tests/r3-phase01-security-vault-webhooks.test.ts`, `15/15` in `tests/v2-phase01..03`, `40/40` in `tests/phase01/02/07`, `npm run build` clean, `npm run lint` (`0` errors). `0` UI/SDK/CSV files modified.
- **`R3-Phase-02` Complete & Verified (`2026-11-20`)** (`BUG-05`, `BUG-09`):
- **`BUG-05`**: Added singleton unref'd periodic maintenance scheduler (`ensureMaintenanceScheduler`, `runScheduledMaintenanceTick`, `stopMaintenanceScheduler`, `stopMaintenanceSchedulerForTests`, `isMaintenanceSchedulerActive`) in `core/adminSettings/settingsManager.ts` with overlap locks (`__dataStackMaintenanceRunning`, `__dataStackArchivePruneInFlight`, `__dataStackCompressInFlight`), cooldown guards, and unhydrated-DB protection so S3 off-site archival + retention pruning and partition compression run unattended on schedule when `enableS3Backup === true`.
- **`BUG-09`**: Updated `extractDateFromFilename()` in both `core/Storage/csvStorage.ts` and `server/workers/csvWriter.js` to strip `.csv(.gz)` and trailing `-(?:part|gz)\d+` suffixes so `-gzN.csv.gz` and `-partN-gzM.csv.gz` collision files work identically across Port 3000/4000 pruning, S3 archiving, `/api/data`, and `/api/export`.
- **Verified**: `3/3` in `tests/r3-phase02-storage-scheduler-gzn.test.ts`, `40/40` in `tests/phase03/v2-phase01/phase07/r3-phase01`, `npm run build` clean, `npm run lint` (`0` errors). `0` UI/SDK/Auth/Docker files modified.
- **`R3-Phase-03` Complete & Verified (`2026-11-20`)** (`BUG-02`, `BUG-07`, `BUG-08`; `ERR-R3-05`, `ERR-R3-06`):
- **`BUG-02`**: Fixed Node.js SDK (`sdk/node/datastack/index.ts`) WS `batch_sync` to emit a raw `Array` directly (1:1 with Port 4000 `server/socket/handlers.js:595`), parse `ack.processed ?? ack.processedCount ?? ack.accepted` and `ack.duplicates ?? 0` across WS & HTTP batch responses, and propagate `duplicates` through `sendBatch()`, `batchSync()`, and `sendFile()`.
- **`BUG-07` (`ERR-R3-05`)**: Implemented real `heartbeatIntervalMs` monitor in Node.js SDK (`startHeartbeatMonitor` with `.unref?.()`, zero-CPU `IDLE_WS` pause, automatic `ping_verify` recovery & spool drain, clean timer teardown, `status().heartbeatActive`); fixed single HTTP/WS `DUPLICATE_DROPPED` reporting (`accepted: 0, duplicates: 1`); wired `TOKEN_ROTATED` (`401`) and `CLIENT_REVOKED` (`403`) fast-fail `DataStackAuthError` across WS `connect_error`, `ping_verify`, `sync_data`, `batch_sync`, and HTTP fallback; compiled CommonJS `sdk/node/datastack/dist/{index,types,spool}.{js,d.ts}` via `tsconfig.json` and pointed `package.json` `main`/`types`/`exports` to `./dist/index.js` & `./dist/index.d.ts`.
- **`BUG-08` (`ERR-R3-06`)**: Added non-retryable `DataStackQuotaExceededError(DataStackError, retryable=False)` in `sdk/python/datastack/datastack/exceptions.py`, re-exported it in both `datastack` and `datastack_sdk`, and wired immediate fail-fast handling for HTTP `507` and WS `STORAGE_QUOTA_EXCEEDED` / `DISK_SPACE_LOW` in `sdk/python/datastack/datastack/client.py` with `0` retries, `0` HTTP fallback after WS quota rejection, and `0` offline spooling.
- **Verified**: `13/13` Node SDK/packaging tests (`5/5` `tests/r3-phase03-sdk-parity.test.ts` against live Socket.IO handlers + `4/4` `v2-phase04-node-sdk.test.ts` + `4/4` `phase05-packaging.test.ts`), `14/14` Python SDK tests (`3/3` `tests/r3_phase03_python_sdk_test.py` + `11/11` `phase05_sdk_test.py`), `py_compile`, `tsc`, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings). `0` Port 3000/4000/UI/Docker files modified.
---