diff --git a/MEMORY.md b/MEMORY.md index 897b9b0..5ea198e 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -1,7 +1,7 @@ # MEMORY.md — DataStack Cross-IDE Persistent Project Memory > **Last Synchronized**: 2026-11-19 -> **Status**: **Production & Deployment Hardening Complete (`BASELINE-2026-11-19-R2` + Deployment Hardening) — `4 / 4` R2 Phases (`18/18` bugs) + `5/5` Docker/Coolify Redeploy Fixes (`DEPLOY-BUG-01`–`05`) Verified (`85/85` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).** +> **Status**: **V2-Phase-01 Complete & Verified (`BASELINE-2026-11-19-V2`) — `1 / 4` V2 Phases Done (Storage `.csv.gz` Auto-Compression, S3/R2 Off-Site Archive with AES-256-GCM Secret Vault, Per-Client `storageQuotaMb` & VPS Disk Pause Guard `HTTP 507`) — Verified (`89/89` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).** > **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`). --- @@ -300,6 +300,21 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are ** - **`DEPLOY-BUG-05`**: Clarified Coolify `Base Directory: /` and `Docker Compose Location: /docker/docker-compose.yml` in `docs/users/installation.md` and `README.md`. - Verified: `85/85` full TS suite pass (`11/11` in `tests/phase07-deployment.test.ts`), `10/10` Python SDK tests pass, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings). + - **V2 Baseline Freeze & Governance Planning Complete (`BASELINE-2026-11-19-V2`)**: + - Froze the clean, Deployment-Hardened runtime state (`85/85` TS tests, `10/10` Python tests, clean `npm run build` & `eslint .`, zero runtime code changes) as **`BASELINE-2026-11-19-V2`**. + - Created raw-source verified V2 planning & contract documents in `notes/`: + - `notes/V2_FEATURE_GAP_AUDIT.md` (`V2-FEAT-01`–`V2-FEAT-07` gap analysis + full `/vcon/*` control mapping). + - `notes/V2_DATABASE_AND_SETTINGS_CONTRACT.md` (zero-loss additive PostgreSQL DDL rules preserving `admin_users` `id = 1 CHECK (id = 1)`, AES-256-GCM DB secret encryption + masked `GET /api/settings`, Port `3000`↔`4000` live sync). + - `notes/BASELINE_FREEZE.md`, `notes/PRODUCTION_READINESS_SCOPE_LOCK.md` (Doc `001` Scope Locks for `V2-Phase-01`–`04`), `notes/PRODUCTION_ROADMAP.md` (Doc `002`), `notes/PHASE_EXECUTION_LOG.md` (Doc `003`, `0 / 4` V2 phases done), `notes/ERROR_HANDLING_MATRIX.md` (Doc `004`, including `V2-E01`–`V2-E12`), `notes/ACTUAL_IMPLEMENTATION_STATUS.md`, and `notes/README.md`. + - Roadmap grouped into 4 Scope-Locked V2 Phases (`V2-Phase-01` Archival/Gzip + Quota/Disk Guard; `V2-Phase-02` Webhooks/Alerts + IP Allowlist/Expiry + Prometheus/NDJSON; `V2-Phase-03` Multi-Admin RBAC + Audit Log; `V2-Phase-04` Node.js/TypeScript SDK + `/vcon/sdk` guide). + - **`V2-Phase-01` Complete & Verified (`2026-11-19`)** (`V2-FEAT-01`, `V2-FEAT-03`, `V2-E01`–`V2-E05`): + - **AES-256-GCM Secret Vault (`core/Security/secretVault.ts`)**: Deterministic HKDF-SHA256 key derivation (`encryptSecret`, `decryptSecret`, `getMaskedHintFromEncrypted`, `SecretVaultError`) so `s3_secret_access_key_enc` is encrypted at rest in PostgreSQL and never leaked in plaintext by `GET /api/settings` (`hasS3SecretKey` + `s3SecretKeyMask: "••••xxxx"`). + - **S3 / Cloudflare R2 / MinIO SigV4 Client (`core/Storage/s3Archive.ts`)**: Pure Node `node:crypto` + `fetch` AWS Signature v4 client (`testS3Connection`, `uploadFileToS3`, `buildS3ObjectKey`, `normalizeS3EndpointUrl`) with `POST /api/storage/s3-test` and `POST /api/storage/archive`. + - **Transparent `.csv.gz` Auto-Compression & Safe Archive-Before-Prune (`core/Storage/csvStorage.ts`, `server/workers/csvWriter.js`)**: Cooperatively locked atomic `.csv` → `.csv.gz` compression (`compressOldPartitionsAsync` / `compressOldPartitions`) excluding today's active UTC partition; transparent `.csv` + `.csv.gz` reading/streaming across `GET /api/data` (surfacing `500 Corrupt .csv.gz partition` on corrupt gzip — `V2-E03`), `GET /api/events`, and `GET /api/export` (`application/gzip` for raw single `.csv.gz` download); fail-safe `beforeDeleteFile` S3 upload hook (`V2-E02`) that never deletes local partitions if S3 upload fails (`502 S3_ARCHIVE_FAILED`). + - **Per-Client Storage Quota (`storageQuotaMb`) & VPS Disk Pause Guard (`diskPauseThresholdPct`)**: Additive PostgreSQL columns (`client_sdks.storage_quota_mb` + 9 `admin_settings` columns in `core/Database/database.ts`, synced in `server/index.js` while preserving `ddlCount === 8`), `30s` TTL cached `statfs` + directory byte counters, and HTTP `507 Insufficient Storage` (`DISK_SPACE_LOW` / `STORAGE_QUOTA_EXCEEDED` — `V2-E04`, `V2-E05`) on `POST /api/events` and Port `4000` `sync_data`/`batch_sync` while keeping all Admin read/export/settings routes online. + - **Admin UI Controls (`SettingsWorkspace.tsx`, `NewClientModal.tsx`, `ClientsTable.tsx`, `lib/vconUiHelpers.ts`)**: Added live VPS disk usage bar, `Auto-compress (.csv → .csv.gz)`, `Disk pause threshold (0 or 50–99%)`, full `S3 / R2 off-site archive` configuration with `Test S3 Connection` & `Archive Now`, and per-client `Storage quota (MB, 0 = unlimited)` in modal and table. + - **Verified**: `89/89` TS tests (`4/4` in `tests/v2-phase01-storage-quotas.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/settings` (Storage tab) and `/vcon/clients` (`0` console errors). + --- ## 5. Known Constraints & Operational Notes for Future Contributors diff --git a/app/api/clients/[id]/route.ts b/app/api/clients/[id]/route.ts index 2ce6929..284a57c 100644 --- a/app/api/clients/[id]/route.ts +++ b/app/api/clients/[id]/route.ts @@ -1,9 +1,21 @@ import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; import { SDKManager } from "@/core/sdk/sdkManager"; +import { + validateAllowedIpsInput, + validateClientExpiresAtInput, +} from "@/core/Security/ipAllowlist"; import { NextRequest, NextResponse } from "next/server"; const SESSION_COOKIE = "datastack_admin_session"; -const ALLOWED_PATCH_KEYS = new Set(["action", "name", "rateLimit", "status"]); +const ALLOWED_PATCH_KEYS = new Set([ + "action", + "name", + "rateLimit", + "storageQuotaMb", + "allowedIps", + "expiresAt", + "status", +]); export async function GET( req: NextRequest, @@ -125,7 +137,14 @@ export async function PATCH( } } - const updates: { name?: string; rateLimit?: number; status?: "active" | "revoked" } = {}; + const updates: { + name?: string; + rateLimit?: number; + storageQuotaMb?: number; + allowedIps?: string; + expiresAt?: string | null; + status?: "active" | "revoked"; + } = {}; if (body.name !== undefined) { if ( typeof body.name !== "string" || @@ -153,6 +172,46 @@ export async function PATCH( } updates.rateLimit = body.rateLimit; } + if (body.storageQuotaMb !== undefined) { + if ( + typeof body.storageQuotaMb !== "number" || + !Number.isInteger(body.storageQuotaMb) || + body.storageQuotaMb < 0 || + body.storageQuotaMb > 1048576 + ) { + return NextResponse.json( + { error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + updates.storageQuotaMb = body.storageQuotaMb; + } + if (body.allowedIps !== undefined) { + if (typeof body.allowedIps !== "string") { + return NextResponse.json( + { error: "allowedIps must be a string.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + const ipValidation = validateAllowedIpsInput(body.allowedIps); + if (!ipValidation.valid) { + return NextResponse.json( + { error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + updates.allowedIps = ipValidation.normalized; + } + if (body.expiresAt !== undefined) { + const expValidation = validateClientExpiresAtInput(body.expiresAt); + if (!expValidation.valid) { + return NextResponse.json( + { error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + updates.expiresAt = expValidation.normalized; + } if (body.status !== undefined) { if (body.status !== "active" && body.status !== "revoked") { return NextResponse.json( diff --git a/app/api/clients/route.ts b/app/api/clients/route.ts index d3ee91a..bca320a 100644 --- a/app/api/clients/route.ts +++ b/app/api/clients/route.ts @@ -1,5 +1,9 @@ import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; import { SDKManager } from "@/core/sdk/sdkManager"; +import { + validateAllowedIpsInput, + validateClientExpiresAtInput, +} from "@/core/Security/ipAllowlist"; import { NextRequest, NextResponse } from "next/server"; const SESSION_COOKIE = "datastack_admin_session"; @@ -58,7 +62,14 @@ export async function POST(req: NextRequest) { } try { - const { clientId, name, rateLimit } = body; + const { + clientId, + name, + rateLimit, + storageQuotaMb, + allowedIps, + expiresAt, + } = body; if (typeof clientId !== "string" || typeof name !== "string" || !clientId.trim() || !name.trim()) { return NextResponse.json( @@ -83,6 +94,44 @@ export async function POST(req: NextRequest) { ); } + const parsedQuotaMb = storageQuotaMb === undefined ? 0 : Number(storageQuotaMb); + if (!Number.isInteger(parsedQuotaMb) || parsedQuotaMb < 0 || parsedQuotaMb > 1048576) { + return NextResponse.json( + { error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + + let normalizedAllowedIps = ""; + if (allowedIps !== undefined && allowedIps !== null) { + if (typeof allowedIps !== "string") { + return NextResponse.json( + { error: "allowedIps must be a string.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + const ipValidation = validateAllowedIpsInput(allowedIps); + if (!ipValidation.valid) { + return NextResponse.json( + { error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + normalizedAllowedIps = ipValidation.normalized; + } + + let normalizedExpiresAt: string | null = null; + if (expiresAt !== undefined) { + const expValidation = validateClientExpiresAtInput(expiresAt); + if (!expValidation.valid) { + return NextResponse.json( + { error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" }, + { status: 400 }, + ); + } + normalizedExpiresAt = expValidation.normalized; + } + const existing = await SDKManager.getClientAsync(safeClientId); if (existing) { return NextResponse.json( @@ -95,6 +144,9 @@ export async function POST(req: NextRequest) { clientId: safeClientId, name: name.trim().slice(0, 120), rateLimit: parsedRateLimit, + storageQuotaMb: parsedQuotaMb, + allowedIps: normalizedAllowedIps, + expiresAt: normalizedExpiresAt, }); return NextResponse.json({ client: newClient }, { status: 201 }); diff --git a/app/api/data/route.ts b/app/api/data/route.ts index aba149d..f8dbdbb 100644 --- a/app/api/data/route.ts +++ b/app/api/data/route.ts @@ -6,7 +6,6 @@ import { CSVStorageManager } from "@/core/Storage/csvStorage"; import { requireAdminApiSession } from "@/lib/adminAuth"; import { parse } from "csv-parse"; import { NextRequest, NextResponse } from "next/server"; -import { createReadStream } from "node:fs"; import { lstat, readdir } from "node:fs/promises"; import path from "node:path"; @@ -135,10 +134,10 @@ export async function GET(request: NextRequest) { try { filenames = (await readdir(clientDir)) .filter((filename) => { - if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?\.csv$/.test(filename)) { + if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?(?:-gz\d+)?\.csv(?:\.gz)?$/.test(filename)) { return false; } - const fileDate = filename.slice(0, 10); + const fileDate = CSVStorageManager.extractDateFromFilename(filename); if ((from && fileDate < from) || (to && fileDate > to)) { return false; } @@ -150,7 +149,7 @@ export async function GET(request: NextRequest) { } for (const filename of filenames) { - const fileDate = filename.slice(0, 10); + const fileDate = CSVStorageManager.extractDateFromFilename(filename); const filePath = path.join(clientDir, filename); let fileLstat; @@ -161,10 +160,13 @@ export async function GET(request: NextRequest) { } if (fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue; - const fileStream = createReadStream(filePath); + const fileStream = CSVStorageManager.openPartitionTextStream(filePath); const parser = fileStream.pipe( parse({ columns: true, skip_empty_lines: true, bom: true }), ); + fileStream.on("error", (err) => { + if (!parser.destroyed) parser.destroy(err); + }); try { for await (const record of parser as AsyncIterable< @@ -180,7 +182,11 @@ export async function GET(request: NextRequest) { if (event.eventId) addIfMatching(event); } } catch (err) { - console.warn(`[API /data] Failed reading CSV partition ${filePath}:`, err); + console.warn(`[API /data] Failed reading partition ${filePath}:`, err); + if (filename.endsWith(".csv.gz")) { + const detail = err instanceof Error ? err.message : String(err); + throw new Error(`Corrupt .csv.gz partition (${filename}): ${detail}`); + } } finally { if (!parser.destroyed) parser.destroy(); if (!fileStream.destroyed) fileStream.destroy(); diff --git a/app/api/events/route.ts b/app/api/events/route.ts index 4944db9..bc15d2c 100644 --- a/app/api/events/route.ts +++ b/app/api/events/route.ts @@ -3,6 +3,10 @@ import { verifyRequestAdminSession } from '@/core/auth/adminUsers'; import { AuthService } from '@/core/auth/jwt'; import { DataBufferEngine, REDIS_BUFFER_KEY } from '@/core/data/buffer'; import { SDKManager } from '@/core/sdk/sdkManager'; +import { + isClientIpAllowed, + isClientTokenExpired, +} from '@/core/Security/ipAllowlist'; import { SecurityEngine, getInternalIngestionHeaders, @@ -15,8 +19,7 @@ import { import { parse } from 'csv-parse'; import { NextRequest, NextResponse } from 'next/server'; import crypto from 'node:crypto'; -import { createReadStream } from 'node:fs'; -import { requireAdminApiSession } from '@/lib/adminAuth'; +import { extractClientIp, requireAdminApiSession } from '@/lib/adminAuth'; import type { BufferedBucket, BufferedSummary } from '@/lib/vconUiHelpers'; const SESSION_COOKIE = 'datastack_admin_session'; @@ -233,10 +236,13 @@ async function getRecentStoredEvents( for (const file of files) { const fileTail: DataEvent[] = []; - const fileStream = createReadStream(file.filePath); + const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath); const parser = fileStream.pipe( parse({ columns: true, skip_empty_lines: true, bom: true }), ); + fileStream.on('error', (err) => { + if (!parser.destroyed) parser.destroy(err); + }); try { for await (const record of parser as AsyncIterable< @@ -254,7 +260,7 @@ async function getRecentStoredEvents( }); } } catch (err) { - console.warn(`[API /events] Failed reading CSV partition ${file.filePath}:`, err); + console.warn(`[API /events] Failed reading partition ${file.filePath}:`, err); } finally { if (!parser.destroyed) parser.destroy(); if (!fileStream.destroyed) fileStream.destroy(); @@ -476,6 +482,77 @@ export async function POST(req: NextRequest) { { status: 401 }, ); } + if (isClientTokenExpired(registeredClient.expiresAt)) { + SettingsManager.triggerWebhookAlert({ + event: 'CLIENT_TOKEN_EXPIRED', + severity: 'warning', + title: 'Expired Client Token Rejected', + message: `Client '${clientId}' attempted HTTP ingestion with an expired token (expiresAt=${registeredClient.expiresAt}).`, + clientId, + details: { expiresAt: registeredClient.expiresAt }, + }); + return NextResponse.json( + { + error: `CLIENT_TOKEN_EXPIRED: Client '${clientId}' token expired at ${registeredClient.expiresAt}. Rotate or extend client expiration.`, + code: 'CLIENT_TOKEN_EXPIRED', + }, + { status: 401 }, + ); + } + if (registeredClient.allowedIps && registeredClient.allowedIps.trim()) { + const clientIp = extractClientIp(req); + if (!isClientIpAllowed(clientIp, registeredClient.allowedIps)) { + SettingsManager.triggerWebhookAlert({ + event: 'CLIENT_IP_FORBIDDEN', + severity: 'warning', + title: 'Unauthorized Client IP Blocked', + message: `Client '${clientId}' request from IP '${clientIp}' was blocked by allowedIps policy.`, + clientId, + details: { + clientIp, + allowedIps: registeredClient.allowedIps, + }, + }); + return NextResponse.json( + { + error: `CLIENT_IP_FORBIDDEN: Request IP '${clientIp}' is not authorized for client '${clientId}'.`, + code: 'CLIENT_IP_FORBIDDEN', + }, + { status: 403 }, + ); + } + } + } + + const storageAdmission = await CSVStorageManager.checkStorageAdmissionAsync( + clientId, + registeredClient.storageQuotaMb, + settings.diskPauseThresholdPct, + ); + if (!storageAdmission.allowed) { + if (storageAdmission.code) { + SettingsManager.triggerWebhookAlert({ + event: storageAdmission.code, + severity: + storageAdmission.code === 'DISK_SPACE_LOW' ? 'critical' : 'warning', + title: + storageAdmission.code === 'DISK_SPACE_LOW' + ? 'VPS Disk Pause Threshold Reached' + : 'Client Storage Quota Exceeded', + message: + storageAdmission.reason || + `Storage admission paused (${storageAdmission.code}) for client '${clientId}'.`, + clientId, + }); + } + return NextResponse.json( + { + success: false, + error: storageAdmission.reason, + code: storageAdmission.code, + }, + { status: 507 }, + ); } const effectiveClientRate = diff --git a/app/api/export/route.ts b/app/api/export/route.ts index 9710b40..1ad6bfe 100644 --- a/app/api/export/route.ts +++ b/app/api/export/route.ts @@ -12,7 +12,7 @@ import { NextRequest, NextResponse } from "next/server"; import { createReadStream } from "node:fs"; import path from "node:path"; -type ExportFormat = "csv" | "json"; +type ExportFormat = "csv" | "json" | "ndjson"; type CsvRecord = Record; type StreamChunk = string | Uint8Array; @@ -46,10 +46,13 @@ function toByteStream(chunks: AsyncIterable) { async function* readFileRecords( file: StorageFileMeta, ): AsyncGenerator { - const fileStream = createReadStream(file.filePath); + const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath); const parser = fileStream.pipe( parse({ columns: true, skip_empty_lines: true, bom: true }), ); + fileStream.on("error", (err) => { + if (!parser.destroyed) parser.destroy(err); + }); try { for await (const record of parser as AsyncIterable) { @@ -72,13 +75,13 @@ function makeExportStream( if (format === "json") { yield "["; - } else if (includeHeader) { + } else if (format === "csv" && includeHeader) { yield `${fields.join(",")}\n`; } for (const file of files) { for await (const record of readFileRecords(file)) { - if (format === "json") { + if (format === "json" || format === "ndjson") { const payload = record.payload ?? ""; let parsedPayload: unknown = payload; try { @@ -86,7 +89,15 @@ function makeExportStream( } catch { // Keep non-JSON payloads as strings. } - yield `${isFirst ? "" : ","}${JSON.stringify({ ...record, payload: parsedPayload })}`; + const serialized = JSON.stringify({ + ...record, + payload: parsedPayload, + }); + if (format === "ndjson") { + yield `${serialized}\n`; + } else { + yield `${isFirst ? "" : ","}${serialized}`; + } } else { const line = fields .map((field) => `"${String(record[field] ?? "").replaceAll('"', '""')}"`) @@ -224,9 +235,14 @@ export async function GET(request: NextRequest) { : SDKManager.normalizeClientId(rawClientId) || "all"; const sanitizedClientId = clientId === "all" ? "all" : CSVStorageManager.sanitizeClientId(clientId); - const filename = searchParams.get("filename"); - const rawFormat = searchParams.get("format"); - const format: ExportFormat = rawFormat === "json" ? "json" : "csv"; + const filename = searchParams.get("filename") || searchParams.get("file"); + const rawFormat = (searchParams.get("format") || "csv").toLowerCase(); + const format: ExportFormat = + rawFormat === "json" + ? "json" + : rawFormat === "ndjson" + ? "ndjson" + : "csv"; const includeHeader = searchParams.get("includeHeader") !== "false"; let files = await CSVStorageManager.listAllFilesAsync(); @@ -259,21 +275,31 @@ export async function GET(request: NextRequest) { ); } const file = matches[0]; + const baseStem = file.filename.replace(/\.csv(?:\.gz)?$/i, ""); const downloadName = format === "json" - ? `${file.filename.replace(/\.csv$/i, "")}.json` - : file.filename; + ? `${baseStem}.json` + : format === "ndjson" + ? `${baseStem}.ndjson` + : !includeHeader && file.compressed + ? `${baseStem}.csv` + : file.filename; const bodyStream = - format === "json" || !includeHeader + format === "json" || format === "ndjson" || !includeHeader ? makeExportStream([file], format, includeHeader) : streamFile(file); + const contentType = + format === "json" + ? "application/json; charset=utf-8" + : format === "ndjson" + ? "application/x-ndjson; charset=utf-8" + : file.compressed && includeHeader + ? "application/gzip" + : "text/csv; charset=utf-8"; return new NextResponse(bodyStream, { status: 200, headers: { - "Content-Type": - format === "json" - ? "application/json; charset=utf-8" - : "text/csv; charset=utf-8", + "Content-Type": contentType, "Content-Disposition": `attachment; filename="${downloadName.replace(/"/g, "")}"`, "Cache-Control": "no-store", }, @@ -294,7 +320,9 @@ export async function GET(request: NextRequest) { "Content-Type": format === "json" ? "application/json; charset=utf-8" - : "text/csv; charset=utf-8", + : format === "ndjson" + ? "application/x-ndjson; charset=utf-8" + : "text/csv; charset=utf-8", "Content-Disposition": `attachment; filename="${baseName}_${datePart}.${format}"`, "Cache-Control": "no-store", }, diff --git a/app/api/prometheus/route.ts b/app/api/prometheus/route.ts new file mode 100644 index 0000000..25a75aa --- /dev/null +++ b/app/api/prometheus/route.ts @@ -0,0 +1,207 @@ +import { SettingsManager } from "@/core/adminSettings/settingsManager"; +import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; +import { DataBufferEngine } from "@/core/data/buffer"; +import { SDKManager } from "@/core/sdk/sdkManager"; +import { getInternalIngestionHeaders } from "@/core/Security/security"; +import { CSVStorageManager } from "@/core/Storage/csvStorage"; +import { ADMIN_SESSION_COOKIE } from "@/lib/adminAuth"; +import { NextRequest, NextResponse } from "next/server"; +import { createHash, timingSafeEqual } from "node:crypto"; + +function verifyBearerTokenConstantTime( + expectedToken: string, + providedToken: string, +): boolean { + const expected = expectedToken.trim(); + const provided = providedToken.trim(); + if (!expected || !provided) return false; + const expectedDigest = createHash("sha256") + .update(expected, "utf8") + .digest(); + const providedDigest = createHash("sha256") + .update(provided, "utf8") + .digest(); + return timingSafeEqual(expectedDigest, providedDigest); +} + +function escapePrometheusLabel(raw: string): string { + return String(raw || "") + .replace(/\\/g, "\\\\") + .replace(/\n/g, "\\n") + .replace(/"/g, '\\"'); +} + +export async function GET(req: NextRequest) { + try { + let authorized = false; + + // 1. Check Bearer token (or ?token= query param) against encrypted Prometheus token + const authHeader = req.headers.get("authorization") || ""; + const bearerCandidate = authHeader.toLowerCase().startsWith("bearer ") + ? authHeader.slice(7).trim() + : req.nextUrl.searchParams.get("token")?.trim() || ""; + + const promConfig = await SettingsManager.getDecryptedPrometheusConfigAsync(); + if ( + promConfig.enabled && + promConfig.token.trim().length > 0 && + bearerCandidate.length > 0 && + verifyBearerTokenConstantTime(promConfig.token, bearerCandidate) + ) { + authorized = true; + } + + // 2. Fallback: valid Admin Console session cookie + if (!authorized) { + const sessionCookie = req.cookies.get(ADMIN_SESSION_COOKIE)?.value; + if (sessionCookie) { + const adminAuth = await verifyRequestAdminSession(sessionCookie); + if (adminAuth.valid) { + authorized = true; + } + } + } + + if (!authorized) { + return NextResponse.json( + { + error: + "PROMETHEUS_UNAUTHORIZED: Valid bearer scrape token or admin session required.", + code: "PROMETHEUS_UNAUTHORIZED", + }, + { status: 401 }, + ); + } + + const [clients, storageStats, diskHealth, bufferStats] = await Promise.all([ + SDKManager.listClientsAsync(), + CSVStorageManager.getStatsAsync(), + CSVStorageManager.getDiskSpaceHealthAsync(), + DataBufferEngine.getBufferStatsAsync(), + ]); + + let activeSockets = 0; + let bridgeUp = 0; + let remoteBufferSize = 0; + + const headers = getInternalIngestionHeaders(); + if (headers["x-datastack-internal-secret"]) { + const ingestionBaseUrl = ( + process.env.INGESTION_INTERNAL_URL || "http://127.0.0.1:4000" + ).replace(/\/$/, ""); + try { + const res = await fetch(`${ingestionBaseUrl}/internal/metrics`, { + headers, + cache: "no-store", + signal: AbortSignal.timeout(1000), + }); + if (res.ok) { + bridgeUp = 1; + const data = (await res.json()) as { + activeSockets?: number; + bufferSize?: number; + }; + activeSockets = Math.max(0, Number(data.activeSockets) || 0); + remoteBufferSize = Math.max(0, Number(data.bufferSize) || 0); + } + } catch { + bridgeUp = 0; + } + } + + const totalBufferedEvents = + bufferStats.localBufferSize + + (bufferStats.redisEnabled && bufferStats.redisConnected + ? bufferStats.redisBufferSize + : remoteBufferSize); + + let activeClientsCount = 0; + let revokedClientsCount = 0; + for (const c of clients) { + if (c.status === "revoked") { + revokedClientsCount += 1; + } else { + activeClientsCount += 1; + } + } + + const lines: string[] = [ + "# HELP datastack_up Whether the DataStack control plane is up (1 = healthy).", + "# TYPE datastack_up gauge", + "datastack_up 1", + "# HELP datastack_ingestion_bridge_up Whether the Port 4000 Socket.IO ingestion engine is reachable (1 = reachable, 0 = unreachable).", + "# TYPE datastack_ingestion_bridge_up gauge", + `datastack_ingestion_bridge_up ${bridgeUp}`, + "# HELP datastack_active_sockets Currently connected Socket.IO SDK clients on Port 4000.", + "# TYPE datastack_active_sockets gauge", + `datastack_active_sockets ${activeSockets}`, + "# HELP datastack_buffer_events_total Total events currently queued in memory or Redis awaiting CSV flush.", + "# TYPE datastack_buffer_events_total gauge", + `datastack_buffer_events_total ${totalBufferedEvents}`, + "# HELP datastack_storage_files_total Total CSV and gzipped CSV partition files on disk.", + "# TYPE datastack_storage_files_total gauge", + `datastack_storage_files_total ${storageStats.totalFiles}`, + "# HELP datastack_storage_bytes_total Total bytes occupied by CSV and gzipped CSV partition files on disk.", + "# TYPE datastack_storage_bytes_total gauge", + `datastack_storage_bytes_total ${storageStats.totalSizeBytes}`, + "# HELP datastack_storage_rows_total Total telemetry rows persisted across all partition files.", + "# TYPE datastack_storage_rows_total gauge", + `datastack_storage_rows_total ${storageStats.totalRows}`, + "# HELP datastack_disk_used_percent Percentage of underlying filesystem capacity currently used.", + "# TYPE datastack_disk_used_percent gauge", + `datastack_disk_used_percent ${diskHealth.usedPct}`, + "# HELP datastack_disk_free_bytes Free bytes available on the underlying storage volume.", + "# TYPE datastack_disk_free_bytes gauge", + `datastack_disk_free_bytes ${diskHealth.freeBytes}`, + "# HELP datastack_disk_total_bytes Total capacity in bytes of the underlying storage volume.", + "# TYPE datastack_disk_total_bytes gauge", + `datastack_disk_total_bytes ${diskHealth.totalBytes}`, + "# HELP datastack_clients_total Number of registered SDK clients partitioned by status.", + "# TYPE datastack_clients_total gauge", + `datastack_clients_total{status="active"} ${activeClientsCount}`, + `datastack_clients_total{status="revoked"} ${revokedClientsCount}`, + "# HELP datastack_client_events_total Cumulative ingested events per registered client.", + "# TYPE datastack_client_events_total counter", + ]; + + for (const c of clients) { + const safeId = escapePrometheusLabel(c.clientId); + lines.push( + `datastack_client_events_total{client_id="${safeId}"} ${Math.max(0, Number(c.totalEvents) || 0)}`, + ); + } + + lines.push( + "# HELP datastack_client_storage_bytes Disk usage in bytes per registered client.", + "# TYPE datastack_client_storage_bytes gauge", + ); + + const perClientBytes = new Map(); + for (const f of storageStats.files) { + perClientBytes.set( + f.clientId, + (perClientBytes.get(f.clientId) || 0) + f.sizeBytes, + ); + } + for (const c of clients) { + const safeId = escapePrometheusLabel(c.clientId); + const usedBytes = perClientBytes.get(c.clientId) || 0; + lines.push( + `datastack_client_storage_bytes{client_id="${safeId}"} ${usedBytes}`, + ); + } + + const body = `${lines.join("\n")}\n`; + return new NextResponse(body, { + status: 200, + headers: { + "Content-Type": "text/plain; version=0.0.4; charset=utf-8", + "Cache-Control": "no-store, no-cache, must-revalidate", + }, + }); + } catch (err: unknown) { + const message = + err instanceof Error ? err.message : "Failed to generate Prometheus metrics"; + return NextResponse.json({ error: message }, { status: 500 }); + } +} diff --git a/app/api/settings/route.ts b/app/api/settings/route.ts index 27a2c32..fdaea49 100644 --- a/app/api/settings/route.ts +++ b/app/api/settings/route.ts @@ -1,8 +1,9 @@ import { SettingsManager, - type AdminSettings, + type AdminSettingsUpdateInput, } from "@/core/adminSettings/settingsManager"; import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; +import { SecretVaultError } from "@/core/Security/secretVault"; import { NextRequest, NextResponse } from "next/server"; const SESSION_COOKIE = "datastack_admin_session"; @@ -16,6 +17,24 @@ const ALLOWED_SETTING_KEYS = new Set([ "enableIdempotency", "autoPruneDays", "jwtSecret", + "compressAfterDays", + "enableS3Backup", + "s3Endpoint", + "s3Region", + "s3Bucket", + "s3Prefix", + "s3AccessKeyId", + "s3SecretAccessKey", + "clearS3SecretKey", + "diskPauseThresholdPct", + "enableWebhookAlerts", + "webhookUrl", + "webhookSecret", + "clearWebhookSecret", + "webhookCooldownSec", + "enablePrometheusMetrics", + "prometheusMetricsToken", + "clearPrometheusToken", ]); function parseBoundedInt( @@ -33,6 +52,21 @@ function parseBoundedInt( return raw; } +function parseOptionalString( + raw: unknown, + field: string, + maxLength: number, +): string { + if (typeof raw !== "string") { + throw new Error(`${field} must be a string.`); + } + const trimmed = raw.trim(); + if (trimmed.length > maxLength) { + throw new Error(`${field} must be at most ${maxLength} characters.`); + } + return trimmed; +} + export async function GET(req: NextRequest) { try { const auth = await verifyRequestAdminSession( @@ -109,7 +143,7 @@ export async function POST(req: NextRequest) { ); } - const validated: Partial = {}; + const validated: AdminSettingsUpdateInput = {}; try { if (body.maxEventsPerSec !== undefined) { validated.maxEventsPerSec = parseBoundedInt( @@ -159,14 +193,161 @@ export async function POST(req: NextRequest) { 3650, ); } + if (body.compressAfterDays !== undefined) { + validated.compressAfterDays = parseBoundedInt( + body.compressAfterDays, + "compressAfterDays", + 0, + 3650, + ); + } + if (body.diskPauseThresholdPct !== undefined) { + const pct = parseBoundedInt( + body.diskPauseThresholdPct, + "diskPauseThresholdPct", + 0, + 99, + ); + if (pct !== 0 && pct < 50) { + throw new Error( + "diskPauseThresholdPct must be 0 (disabled) or between 50 and 99.", + ); + } + validated.diskPauseThresholdPct = pct; + } if (body.enableIdempotency !== undefined) { if (typeof body.enableIdempotency !== "boolean") { throw new Error("enableIdempotency must be a boolean."); } validated.enableIdempotency = body.enableIdempotency; } + if (body.enableS3Backup !== undefined) { + if (typeof body.enableS3Backup !== "boolean") { + throw new Error("enableS3Backup must be a boolean."); + } + validated.enableS3Backup = body.enableS3Backup; + } + if (body.s3Endpoint !== undefined) { + const ep = parseOptionalString(body.s3Endpoint, "s3Endpoint", 512); + if (ep) { + let parsedEp: URL; + try { + parsedEp = new URL( + /^https?:\/\//i.test(ep) ? ep : `https://${ep}`, + ); + } catch { + throw new Error("s3Endpoint must be a valid http(s) URL."); + } + if (parsedEp.protocol !== "http:" && parsedEp.protocol !== "https:") { + throw new Error("s3Endpoint must use http:// or https://."); + } + } + validated.s3Endpoint = ep; + } + if (body.s3Region !== undefined) { + validated.s3Region = + parseOptionalString(body.s3Region, "s3Region", 64) || "auto"; + } + if (body.s3Bucket !== undefined) { + validated.s3Bucket = parseOptionalString( + body.s3Bucket, + "s3Bucket", + 128, + ); + } + if (body.s3Prefix !== undefined) { + validated.s3Prefix = parseOptionalString( + body.s3Prefix, + "s3Prefix", + 256, + ); + } + if (body.s3AccessKeyId !== undefined) { + validated.s3AccessKeyId = parseOptionalString( + body.s3AccessKeyId, + "s3AccessKeyId", + 256, + ); + } + if (body.s3SecretAccessKey !== undefined) { + validated.s3SecretAccessKey = parseOptionalString( + body.s3SecretAccessKey, + "s3SecretAccessKey", + 512, + ); + } + if (body.clearS3SecretKey !== undefined) { + if (typeof body.clearS3SecretKey !== "boolean") { + throw new Error("clearS3SecretKey must be a boolean."); + } + validated.clearS3SecretKey = body.clearS3SecretKey; + } + if (body.enableWebhookAlerts !== undefined) { + if (typeof body.enableWebhookAlerts !== "boolean") { + throw new Error("enableWebhookAlerts must be a boolean."); + } + validated.enableWebhookAlerts = body.enableWebhookAlerts; + } + if (body.webhookUrl !== undefined) { + const wh = parseOptionalString(body.webhookUrl, "webhookUrl", 1024); + if (wh) { + let parsedWh: URL; + try { + parsedWh = new URL(wh); + } catch { + throw new Error("webhookUrl must be a valid http(s) URL."); + } + if (parsedWh.protocol !== "http:" && parsedWh.protocol !== "https:") { + throw new Error("webhookUrl must use http:// or https://."); + } + } + validated.webhookUrl = wh; + } + if (body.webhookSecret !== undefined) { + validated.webhookSecret = parseOptionalString( + body.webhookSecret, + "webhookSecret", + 512, + ); + } + if (body.clearWebhookSecret !== undefined) { + if (typeof body.clearWebhookSecret !== "boolean") { + throw new Error("clearWebhookSecret must be a boolean."); + } + validated.clearWebhookSecret = body.clearWebhookSecret; + } + if (body.webhookCooldownSec !== undefined) { + validated.webhookCooldownSec = parseBoundedInt( + body.webhookCooldownSec, + "webhookCooldownSec", + 10, + 86400, + ); + } + if (body.enablePrometheusMetrics !== undefined) { + if (typeof body.enablePrometheusMetrics !== "boolean") { + throw new Error("enablePrometheusMetrics must be a boolean."); + } + validated.enablePrometheusMetrics = body.enablePrometheusMetrics; + } + if (body.prometheusMetricsToken !== undefined) { + validated.prometheusMetricsToken = parseOptionalString( + body.prometheusMetricsToken, + "prometheusMetricsToken", + 512, + ); + } + if (body.clearPrometheusToken !== undefined) { + if (typeof body.clearPrometheusToken !== "boolean") { + throw new Error("clearPrometheusToken must be a boolean."); + } + validated.clearPrometheusToken = body.clearPrometheusToken; + } if (body.jwtSecret !== undefined) { - if (typeof body.jwtSecret !== "string" || body.jwtSecret.trim().length < 32) { + if ( + typeof body.jwtSecret !== "string" || + body.jwtSecret.trim().length < 32 + ) { throw new Error( "jwtSecret must be a string with at least 32 characters.", ); @@ -208,6 +389,12 @@ export async function POST(req: NextRequest) { : {}), }); } catch (err: unknown) { + if (err instanceof SecretVaultError) { + return NextResponse.json( + { error: err.message, code: err.code }, + { status: 500 }, + ); + } const message = err instanceof Error ? err.message : "Settings update failed"; return NextResponse.json({ error: message }, { status: 500 }); diff --git a/app/api/settings/webhook-test/route.ts b/app/api/settings/webhook-test/route.ts new file mode 100644 index 0000000..e038c9c --- /dev/null +++ b/app/api/settings/webhook-test/route.ts @@ -0,0 +1,119 @@ +import { SettingsManager } from "@/core/adminSettings/settingsManager"; +import { dispatchWebhookAlertAsync } from "@/core/alerts/webhookNotifier"; +import { SecretVaultError } from "@/core/Security/secretVault"; +import { requireAdminApiSession } from "@/lib/adminAuth"; +import { NextRequest, NextResponse } from "next/server"; + +export async function POST(req: NextRequest) { + const auth = await requireAdminApiSession(req); + if (!auth.valid) { + return auth.response; + } + + let body: Record = {}; + try { + const text = await req.text(); + if (text.trim()) { + const parsed = JSON.parse(text); + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + body = parsed as Record; + } + } + } catch { + return NextResponse.json( + { error: "Invalid JSON request payload.", code: "INVALID_JSON" }, + { status: 400 }, + ); + } + + try { + const savedConfig = await SettingsManager.getDecryptedWebhookConfigAsync(); + const webhookUrl = + typeof body.webhookUrl === "string" + ? body.webhookUrl.trim() + : savedConfig.webhookUrl.trim(); + const clearWebhookSecret = body.clearWebhookSecret === true; + const webhookSecret = clearWebhookSecret + ? "" + : typeof body.webhookSecret === "string" && body.webhookSecret.trim() + ? body.webhookSecret.trim() + : savedConfig.webhookSecret; + + if (!webhookUrl) { + return NextResponse.json( + { + success: false, + code: "VALIDATION_ERROR", + error: "Webhook URL is required to send a test alert.", + }, + { status: 400 }, + ); + } + + try { + const parsedUrl = new URL(webhookUrl); + if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") { + throw new Error("Webhook URL must use http:// or https://."); + } + } catch { + return NextResponse.json( + { + success: false, + code: "VALIDATION_ERROR", + error: "Webhook URL must be a valid http:// or https:// endpoint.", + }, + { status: 400 }, + ); + } + + const result = await dispatchWebhookAlertAsync( + { + event: "TEST_ALERT", + severity: "info", + title: "Webhook Verification", + message: "Test alert dispatched from DataStack Admin Console.", + details: { + triggeredBy: auth.auth.user?.email || "admin", + }, + }, + { + enabled: true, + webhookUrl, + webhookSecret, + cooldownSec: savedConfig.cooldownSec, + }, + { bypassCooldown: true }, + ); + + if (!result.ok) { + return NextResponse.json( + { + success: false, + code: "WEBHOOK_DELIVERY_FAILED", + error: result.error || "Webhook target failed to accept test alert.", + status: result.status, + }, + { status: 502 }, + ); + } + + return NextResponse.json({ + success: true, + status: result.status, + message: `Webhook test alert delivered (HTTP ${result.status ?? 200}).`, + }); + } catch (err: unknown) { + if (err instanceof SecretVaultError) { + return NextResponse.json( + { success: false, code: err.code, error: err.message }, + { status: 400 }, + ); + } + const message = + err instanceof Error ? err.message : "Webhook test alert failed"; + return NextResponse.json( + { success: false, code: "WEBHOOK_DELIVERY_FAILED", error: message }, + { status: 502 }, + ); + } +} diff --git a/app/api/storage/archive/route.ts b/app/api/storage/archive/route.ts new file mode 100644 index 0000000..c0daaf6 --- /dev/null +++ b/app/api/storage/archive/route.ts @@ -0,0 +1,163 @@ +import { SettingsManager } from "@/core/adminSettings/settingsManager"; +import { SecretVaultError } from "@/core/Security/secretVault"; +import { CSVStorageManager } from "@/core/Storage/csvStorage"; +import { + buildS3ObjectKey, + normalizeS3EndpointUrl, + uploadFileToS3, +} from "@/core/Storage/s3Archive"; +import { requireAdminApiSession } from "@/lib/adminAuth"; +import { NextRequest, NextResponse } from "next/server"; + +export async function POST(req: NextRequest) { + const auth = await requireAdminApiSession(req); + if (!auth.valid) { + return auth.response; + } + + let body: Record = {}; + try { + const text = await req.text(); + if (text.trim()) { + const parsed = JSON.parse(text); + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + body = parsed as Record; + } + } + } catch { + return NextResponse.json( + { error: "Invalid JSON request payload.", code: "INVALID_JSON" }, + { status: 400 }, + ); + } + + try { + const settings = await SettingsManager.getSettingsAsync(); + const s3Config = await SettingsManager.getDecryptedS3ConfigAsync(); + + if (!s3Config.bucket.trim() || !s3Config.accessKeyId.trim() || !s3Config.secretAccessKey.trim()) { + return NextResponse.json( + { + success: false, + code: "S3_INVALID_CONFIG", + error: "Configure S3 Bucket, Access Key ID, and Secret Access Key before archiving.", + }, + { status: 400 }, + ); + } + + try { + normalizeS3EndpointUrl(s3Config.endpoint, s3Config.region); + } catch (urlErr: unknown) { + return NextResponse.json( + { + success: false, + code: "S3_INVALID_CONFIG", + error: urlErr instanceof Error ? urlErr.message : "Invalid S3 endpoint URL.", + }, + { status: 400 }, + ); + } + + const pruneAfterUpload = body.pruneAfterUpload === true; + const rawOlderThanDays = + typeof body.olderThanDays === "number" && Number.isFinite(body.olderThanDays) + ? Math.max(0, Math.floor(body.olderThanDays)) + : settings.autoPruneDays > 0 + ? settings.autoPruneDays + : 0; + const shouldCompressFirst = + body.compressFirst !== false && settings.compressAfterDays > 0; + + let compressedCount = 0; + if (shouldCompressFirst) { + compressedCount = await CSVStorageManager.compressOldPartitionsAsync( + settings.compressAfterDays, + ); + } + + const archiveResult = + await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync( + settings.autoPruneDays, + { + archiveOnlyWithoutDelete: !pruneAfterUpload, + maxCutoffDays: rawOlderThanDays, + beforeDeleteFile: async (file) => { + const key = buildS3ObjectKey( + s3Config.prefix, + file.clientId, + file.filename, + ); + const uploadRes = await uploadFileToS3( + s3Config, + file.filePath, + key, + ); + if (!uploadRes.ok) { + throw new Error( + uploadRes.error || `Failed to upload ${file.filename}`, + ); + } + }, + }, + ); + + if ( + archiveResult.errors.length > 0 && + archiveResult.archivedCount === 0 + ) { + SettingsManager.triggerWebhookAlert({ + event: "S3_ARCHIVE_FAILED", + severity: "critical", + title: "S3 Cold Archive Upload Failed", + message: `Manual S3 archive failed: ${archiveResult.errors[0]}`, + details: { errorCount: archiveResult.errors.length }, + }); + return NextResponse.json( + { + success: false, + code: "S3_ARCHIVE_FAILED", + error: archiveResult.errors[0], + compressedCount, + ...archiveResult, + }, + { status: 502 }, + ); + } + + if (archiveResult.errors.length > 0) { + SettingsManager.triggerWebhookAlert({ + event: "S3_ARCHIVE_FAILED", + severity: "warning", + title: "Partial S3 Cold Archive Errors", + message: `S3 archive encountered ${archiveResult.errors.length} error(s): ${archiveResult.errors[0]}`, + details: { errorCount: archiveResult.errors.length }, + }); + } + + return NextResponse.json({ + success: archiveResult.errors.length === 0, + compressedCount, + ...archiveResult, + }); + } catch (err: unknown) { + if (err instanceof SecretVaultError) { + return NextResponse.json( + { success: false, code: err.code, error: err.message }, + { status: 400 }, + ); + } + const message = + err instanceof Error ? err.message : "Manual S3 archive failed"; + SettingsManager.triggerWebhookAlert({ + event: "S3_ARCHIVE_FAILED", + severity: "critical", + title: "S3 Cold Archive Upload Failed", + message, + }); + return NextResponse.json( + { success: false, code: "S3_ARCHIVE_FAILED", error: message }, + { status: 500 }, + ); + } +} diff --git a/app/api/storage/route.ts b/app/api/storage/route.ts index b058f88..0fbdc9a 100644 --- a/app/api/storage/route.ts +++ b/app/api/storage/route.ts @@ -23,7 +23,10 @@ export async function GET(req: NextRequest) { ? CSVStorageManager.sanitizeClientId(clientId) : undefined; - const allFiles = await CSVStorageManager.listAllFilesAsync(); + const [allFiles, diskUsage] = await Promise.all([ + CSVStorageManager.listAllFilesAsync(), + CSVStorageManager.getDiskUsageAsync(), + ]); let files = allFiles; if (clientId) { @@ -39,7 +42,7 @@ export async function GET(req: NextRequest) { const stats = CSVStorageManager.formatStorageStats(files); - return NextResponse.json({ stats, files }); + return NextResponse.json({ stats, files, diskUsage }); } catch (err: unknown) { const message = err instanceof Error ? err.message : 'Storage stats failed'; return NextResponse.json({ error: message }, { status: 500 }); diff --git a/app/api/storage/s3-test/route.ts b/app/api/storage/s3-test/route.ts new file mode 100644 index 0000000..2ae5643 --- /dev/null +++ b/app/api/storage/s3-test/route.ts @@ -0,0 +1,105 @@ +import { SettingsManager } from "@/core/adminSettings/settingsManager"; +import { + decryptSecret, + SecretVaultError, +} from "@/core/Security/secretVault"; +import { + testS3Connection, + type S3ArchiveConfig, +} from "@/core/Storage/s3Archive"; +import { requireAdminApiSession } from "@/lib/adminAuth"; +import { NextRequest, NextResponse } from "next/server"; + +export async function POST(req: NextRequest) { + const auth = await requireAdminApiSession(req); + if (!auth.valid) { + return auth.response; + } + + let body: Record = {}; + try { + const text = await req.text(); + if (text.trim()) { + const parsed = JSON.parse(text); + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + body = parsed as Record; + } + } + } catch { + return NextResponse.json( + { error: "Invalid JSON request payload.", code: "INVALID_JSON" }, + { status: 400 }, + ); + } + + try { + const saved = await SettingsManager.getSettingsAsync(); + const draftSecret = + typeof body.s3SecretAccessKey === "string" && + body.s3SecretAccessKey.trim().length > 0 + ? body.s3SecretAccessKey.trim() + : ""; + + let effectiveSecret = draftSecret; + if (!effectiveSecret && !body.clearS3SecretKey) { + effectiveSecret = decryptSecret(saved.s3SecretAccessKeyEnc); + } + + const config: S3ArchiveConfig = { + endpoint: + typeof body.s3Endpoint === "string" + ? body.s3Endpoint.trim() + : saved.s3Endpoint, + region: + typeof body.s3Region === "string" && body.s3Region.trim() + ? body.s3Region.trim() + : saved.s3Region || "auto", + bucket: + typeof body.s3Bucket === "string" + ? body.s3Bucket.trim() + : saved.s3Bucket, + prefix: + typeof body.s3Prefix === "string" + ? body.s3Prefix.trim() + : saved.s3Prefix, + accessKeyId: + typeof body.s3AccessKeyId === "string" + ? body.s3AccessKeyId.trim() + : saved.s3AccessKeyId, + secretAccessKey: effectiveSecret, + }; + + const result = await testS3Connection(config); + if (!result.ok) { + return NextResponse.json( + { + success: false, + code: result.code || "S3_CONNECTION_FAILED", + error: result.error || "S3 connectivity check failed.", + ...(result.status !== undefined ? { s3Status: result.status } : {}), + }, + { status: result.code === "S3_INVALID_CONFIG" ? 400 : 502 }, + ); + } + + return NextResponse.json({ + success: true, + endpoint: result.endpoint, + bucket: result.bucket, + status: result.status, + }); + } catch (err: unknown) { + if (err instanceof SecretVaultError) { + return NextResponse.json( + { success: false, code: err.code, error: err.message }, + { status: 400 }, + ); + } + const message = + err instanceof Error ? err.message : "S3 connectivity check failed"; + return NextResponse.json( + { success: false, code: "S3_CONNECTION_FAILED", error: message }, + { status: 500 }, + ); + } +} diff --git a/components/sections/ClientsTable.tsx b/components/sections/ClientsTable.tsx index 24f2b7e..fa71a40 100644 --- a/components/sections/ClientsTable.tsx +++ b/components/sections/ClientsTable.tsx @@ -47,6 +47,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps const [editClient, setEditClient] = useState(null); const [editName, setEditName] = useState(''); const [editRateLimit, setEditRateLimit] = useState('100'); + const [editStorageQuotaMb, setEditStorageQuotaMb] = useState('0'); + const [editAllowedIps, setEditAllowedIps] = useState(''); + const [editExpiresAt, setEditExpiresAt] = useState(''); const [editError, setEditError] = useState(''); const [processing, setProcessing] = useState(false); @@ -59,6 +62,18 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps setEditClient(client); setEditName(client.name); setEditRateLimit(String(client.rateLimit || 100)); + setEditStorageQuotaMb(String(client.storageQuotaMb ?? 0)); + setEditAllowedIps(client.allowedIps || ''); + if (client.expiresAt) { + const parsedDate = new Date(client.expiresAt); + setEditExpiresAt( + !Number.isNaN(parsedDate.getTime()) + ? parsedDate.toISOString().slice(0, 16) + : '', + ); + } else { + setEditExpiresAt(''); + } setEditError(''); }; @@ -68,6 +83,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps const trimmedName = editName.trim(); const numericRateLimit = Number(editRateLimit); + const numericQuotaMb = Number( + editStorageQuotaMb.trim() === '' ? '0' : editStorageQuotaMb, + ); if (!trimmedName) { setEditError('Display name is required.'); return; @@ -76,6 +94,19 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps setEditError('Rate limit must be an integer between 1 and 100,000 events/s.'); return; } + if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) { + setEditError('Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).'); + return; + } + let normalizedExpiresAt: string | null = null; + if (editExpiresAt.trim()) { + const parsedDate = new Date(editExpiresAt.trim()); + if (Number.isNaN(parsedDate.getTime())) { + setEditError('Expiration date/time is invalid.'); + return; + } + normalizedExpiresAt = parsedDate.toISOString(); + } setEditError(''); setProcessing(true); @@ -86,6 +117,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps body: JSON.stringify({ name: trimmedName, rateLimit: Math.floor(numericRateLimit), + storageQuotaMb: Math.floor(numericQuotaMb), + allowedIps: editAllowedIps.trim(), + expiresAt: normalizedExpiresAt, }), }); if (response.status === 401) { @@ -224,17 +258,32 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps ) : ( clients.map((client) => { const revoked = client.status === 'revoked'; - const online = client.status === 'active' && Boolean(client.lastActive) && + const expired = + Boolean(client.expiresAt) && + now >= new Date(client.expiresAt!).getTime(); + const online = + client.status === 'active' && + !expired && + Boolean(client.lastActive) && now - new Date(client.lastActive!).getTime() < 60_000; - const pending = client.status === 'active' && !client.lastActive; - const label = revoked ? 'revoked' : pending ? 'pending' : online ? 'online' : 'offline'; - const dotClass = revoked - ? 'bg-[var(--danger)]' - : online - ? 'bg-[var(--accent)]' + const pending = client.status === 'active' && !expired && !client.lastActive; + const label = revoked + ? 'revoked' + : expired + ? 'expired' : pending - ? 'bg-[var(--warning)]' - : 'bg-[var(--text-faint)]'; + ? 'pending' + : online + ? 'online' + : 'offline'; + const dotClass = + revoked || expired + ? 'bg-[var(--danger)]' + : online + ? 'bg-[var(--accent)]' + : pending + ? 'bg-[var(--warning)]' + : 'bg-[var(--text-faint)]'; return ( @@ -249,6 +298,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps revoked )} + {expired && ( + + expired + + )}
{client.clientId}
@@ -259,7 +313,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps {client.tokenPreview || 'JWT Active'}
- {client.rateLimit}/s limit + {client.rateLimit}/s · {(client.storageQuotaMb ?? 0) > 0 ? `${client.storageQuotaMb}MB quota` : 'Unlimited'} + {client.allowedIps ? ` · IP: ${client.allowedIps}` : ''} + {client.expiresAt + ? ` · Exp: ${new Date(client.expiresAt).toISOString().slice(0, 10)}` + : ''}
@@ -375,6 +433,46 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps onChange={(event) => setEditRateLimit(event.target.value)} required /> + setEditStorageQuotaMb(event.target.value)} + required + /> + setEditAllowedIps(event.target.value)} + placeholder="203.0.113.10, 10.0.0.0/8" + className="font-mono text-[11px]" + /> +
+
+ setEditExpiresAt(event.target.value)} + className="font-mono text-[11px]" + /> +
+ {editExpiresAt && ( + + )} +
{editError && (

{editError} diff --git a/components/sections/ExportWorkspace.tsx b/components/sections/ExportWorkspace.tsx index f42ee6b..257b832 100644 --- a/components/sections/ExportWorkspace.tsx +++ b/components/sections/ExportWorkspace.tsx @@ -33,7 +33,7 @@ interface StorageFile { modifiedAt: string; } -type ExportFormat = 'csv' | 'json'; +type ExportFormat = 'csv' | 'json' | 'ndjson'; type RangePreset = 'today' | '7d' | '30d' | 'all' | 'custom'; function formatBytes(bytes: number) { @@ -368,7 +368,7 @@ export function ExportWorkspace() { Format

@@ -384,6 +384,12 @@ export function ExportWorkspace() { label="JSON" onSelect={setFormat} /> +
diff --git a/components/sections/NewClientModal.tsx b/components/sections/NewClientModal.tsx index 892d782..16f417f 100644 --- a/components/sections/NewClientModal.tsx +++ b/components/sections/NewClientModal.tsx @@ -19,6 +19,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps const [clientId, setClientId] = useState(''); const [name, setName] = useState(''); const [rateLimit, setRateLimit] = useState('100'); + const [storageQuotaMb, setStorageQuotaMb] = useState('0'); + const [allowedIps, setAllowedIps] = useState(''); + const [expiresAt, setExpiresAt] = useState(''); const [submitting, setSubmitting] = useState(false); const [credentials, setCredentials] = useState(null); @@ -35,12 +38,33 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps error('Validation error', 'Rate limit must be an integer between 1 and 100,000 events/s.'); return; } + const numericQuotaMb = Number(storageQuotaMb.trim() === '' ? '0' : storageQuotaMb); + if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) { + error('Validation error', 'Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).'); + return; + } + let normalizedExpiresAt: string | null = null; + if (expiresAt.trim()) { + const parsedDate = new Date(expiresAt.trim()); + if (Number.isNaN(parsedDate.getTime())) { + error('Validation error', 'Token expiration must be a valid date/time.'); + return; + } + normalizedExpiresAt = parsedDate.toISOString(); + } setSubmitting(true); try { const response = await fetch('/api/clients', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ clientId: trimmedId, name: trimmedName, rateLimit: numericRate }), + body: JSON.stringify({ + clientId: trimmedId, + name: trimmedName, + rateLimit: numericRate, + storageQuotaMb: numericQuotaMb, + allowedIps: allowedIps.trim(), + expiresAt: normalizedExpiresAt, + }), }); if (response.status === 401) { window.location.assign('/vcon/login'); @@ -53,6 +77,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps setClientId(''); setName(''); setRateLimit('100'); + setStorageQuotaMb('0'); + setAllowedIps(''); + setExpiresAt(''); onClose(); success('Client registered', `${data.client.clientId} is ready to connect.`); await onCreated(); @@ -103,6 +130,31 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps value={rateLimit} onChange={(event) => setRateLimit(event.target.value)} /> + setStorageQuotaMb(event.target.value)} + /> + setAllowedIps(event.target.value)} + className="font-mono text-[11px]" + /> + setExpiresAt(event.target.value)} + className="font-mono text-[11px]" + /> diff --git a/components/sections/SettingsWorkspace.tsx b/components/sections/SettingsWorkspace.tsx index 21e3018..27aa55f 100644 --- a/components/sections/SettingsWorkspace.tsx +++ b/components/sections/SettingsWorkspace.tsx @@ -16,15 +16,21 @@ import { StatStrip } from '@/components/ui/StatStrip'; import { useToast } from '@/components/ui/Toast'; import { getCached, setCached } from '@/lib/vconCache'; import { + type AlertingAndMetricsSettingsFields, + DEFAULT_ALERTING_SETTINGS, DEFAULT_EDITABLE_SETTINGS, + DEFAULT_S3_SETTINGS, describeSettingsBridgeSync, type EditableSettings, mapBridgeStatusToSocketUiState, + normalizeAlertingSettings, normalizeEditableSettings, + normalizeS3Settings, NUMERIC_SETTING_KEYS, type NumericSettingDrafts, type NumericSettingKey, resolvePublicEndpoints, + type S3SettingsFields, SETTINGS_LIMITS, settingsToNumericDrafts, type SettingsValidationErrors, @@ -34,12 +40,29 @@ import { } from '@/lib/vconUiHelpers'; import { useCallback, useEffect, useMemo, useState } from 'react'; -interface PublicSettings extends EditableSettings { +interface PublicSettings + extends EditableSettings, + S3SettingsFields, + AlertingAndMetricsSettingsFields { serverPort: number; webPort: number; + hasS3SecretKey?: boolean; + s3SecretKeyMask?: string; + hasWebhookSecret?: boolean; + webhookSecretMask?: string; + hasPrometheusToken?: boolean; + prometheusTokenMask?: string; updatedAt: string; } +interface DiskUsageSummary { + available: boolean; + usedPct: number; + freeBytes: number; + totalBytes: number; + checkedAt?: string; +} + const TABS = [ { id: 'general', label: 'General' }, { id: 'connections', label: 'Connections' }, @@ -55,24 +78,89 @@ const FIELD_TO_TAB: Record = { batchFlushIntervalMs: 'connections', fileRotationMb: 'storage', autoPruneDays: 'storage', + compressAfterDays: 'storage', + diskPauseThresholdPct: 'storage', maxPayloadSizeKb: 'security', + webhookCooldownSec: 'security', }; +function formatGiB(bytes: number): string { + if (!Number.isFinite(bytes) || bytes <= 0) return '0.0 GB'; + return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)} GB`; +} + export function SettingsWorkspace() { const { success, warning, error } = useToast(); const [savedSettings, setSavedSettings] = useState(() => - getCached('vcon:settings', DEFAULT_EDITABLE_SETTINGS) + normalizeEditableSettings( + getCached>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) + ) ); const [numericDrafts, setNumericDrafts] = useState(() => settingsToNumericDrafts( - getCached('vcon:settings', DEFAULT_EDITABLE_SETTINGS) + normalizeEditableSettings( + getCached>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) + ) ) ); const [enableIdempotency, setEnableIdempotency] = useState( () => - getCached('vcon:settings', DEFAULT_EDITABLE_SETTINGS) - .enableIdempotency + normalizeEditableSettings( + getCached>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) + ).enableIdempotency ); + const [savedS3Settings, setSavedS3Settings] = useState(() => + normalizeS3Settings( + getCached>('vcon:s3-settings', DEFAULT_S3_SETTINGS) + ) + ); + const [s3Draft, setS3Draft] = useState(() => + normalizeS3Settings( + getCached>('vcon:s3-settings', DEFAULT_S3_SETTINGS) + ) + ); + const [hasS3SecretKey, setHasS3SecretKey] = useState(false); + const [s3SecretKeyMask, setS3SecretKeyMask] = useState(''); + const [s3SecretAccessKey, setS3SecretAccessKey] = useState(''); + const [clearS3SecretKey, setClearS3SecretKey] = useState(false); + const [savedAlertingSettings, setSavedAlertingSettings] = + useState(() => + normalizeAlertingSettings( + getCached>( + 'vcon:alerting-settings', + DEFAULT_ALERTING_SETTINGS + ) + ) + ); + const [alertingDraft, setAlertingDraft] = + useState(() => + normalizeAlertingSettings( + getCached>( + 'vcon:alerting-settings', + DEFAULT_ALERTING_SETTINGS + ) + ) + ); + const [hasWebhookSecret, setHasWebhookSecret] = useState(false); + const [webhookSecretMask, setWebhookSecretMask] = useState(''); + const [webhookSecret, setWebhookSecret] = useState(''); + const [clearWebhookSecret, setClearWebhookSecret] = useState(false); + const [testingWebhook, setTestingWebhook] = useState(false); + const [webhookTestStatus, setWebhookTestStatus] = useState<{ + ok: boolean; + text: string; + } | null>(null); + const [hasPrometheusToken, setHasPrometheusToken] = useState(false); + const [prometheusTokenMask, setPrometheusTokenMask] = useState(''); + const [prometheusMetricsToken, setPrometheusMetricsToken] = useState(''); + const [clearPrometheusToken, setClearPrometheusToken] = useState(false); + const [diskUsage, setDiskUsage] = useState(null); + const [testingS3, setTestingS3] = useState(false); + const [archivingNow, setArchivingNow] = useState(false); + const [s3ActionStatus, setS3ActionStatus] = useState<{ + ok: boolean; + text: string; + } | null>(null); const [fieldErrors, setFieldErrors] = useState({}); const [activeTab, setActiveTab] = useState('general'); const [jwtSecret, setJwtSecret] = useState(''); @@ -169,40 +257,73 @@ export function SettingsWorkspace() { ); const loadedPort = Number.isFinite(runtimePort) && runtimePort > 0 ? runtimePort : settingsPort; + const rawSettings = settingsData.settings as PublicSettings; const loaded: EditableSettings = { maxEventsPerSec: Number( - settingsData.settings.maxEventsPerSec ?? + rawSettings.maxEventsPerSec ?? DEFAULT_EDITABLE_SETTINGS.maxEventsPerSec ), maxPayloadSizeKb: Number( - settingsData.settings.maxPayloadSizeKb ?? + rawSettings.maxPayloadSizeKb ?? DEFAULT_EDITABLE_SETTINGS.maxPayloadSizeKb ), batchFlushThreshold: Number( - settingsData.settings.batchFlushThreshold ?? + rawSettings.batchFlushThreshold ?? DEFAULT_EDITABLE_SETTINGS.batchFlushThreshold ), batchFlushIntervalMs: Number( - settingsData.settings.batchFlushIntervalMs ?? + rawSettings.batchFlushIntervalMs ?? DEFAULT_EDITABLE_SETTINGS.batchFlushIntervalMs ), fileRotationMb: Number( - settingsData.settings.fileRotationMb ?? + rawSettings.fileRotationMb ?? DEFAULT_EDITABLE_SETTINGS.fileRotationMb ), enableIdempotency: Boolean( - settingsData.settings.enableIdempotency ?? + rawSettings.enableIdempotency ?? DEFAULT_EDITABLE_SETTINGS.enableIdempotency ), autoPruneDays: Number( - settingsData.settings.autoPruneDays ?? + rawSettings.autoPruneDays ?? DEFAULT_EDITABLE_SETTINGS.autoPruneDays ), + compressAfterDays: Number( + rawSettings.compressAfterDays ?? + DEFAULT_EDITABLE_SETTINGS.compressAfterDays + ), + diskPauseThresholdPct: Number( + rawSettings.diskPauseThresholdPct ?? + DEFAULT_EDITABLE_SETTINGS.diskPauseThresholdPct + ), + webhookCooldownSec: Number( + rawSettings.webhookCooldownSec ?? + DEFAULT_EDITABLE_SETTINGS.webhookCooldownSec + ), }; - const cachedLoaded = setCached('vcon:settings', loaded); + const loadedS3 = normalizeS3Settings(rawSettings); + const loadedAlerting = normalizeAlertingSettings(rawSettings); + const cachedLoaded = setCached('vcon:settings', normalizeEditableSettings(loaded)); + const cachedS3 = setCached('vcon:s3-settings', loadedS3); + const cachedAlerting = setCached('vcon:alerting-settings', loadedAlerting); setSavedSettings(cachedLoaded); setNumericDrafts(settingsToNumericDrafts(cachedLoaded)); setEnableIdempotency(cachedLoaded.enableIdempotency); + setSavedS3Settings(cachedS3); + setS3Draft(cachedS3); + setHasS3SecretKey(Boolean(rawSettings.hasS3SecretKey)); + setS3SecretKeyMask(String(rawSettings.s3SecretKeyMask || '')); + setS3SecretAccessKey(''); + setClearS3SecretKey(false); + setSavedAlertingSettings(cachedAlerting); + setAlertingDraft(cachedAlerting); + setHasWebhookSecret(Boolean(rawSettings.hasWebhookSecret)); + setWebhookSecretMask(String(rawSettings.webhookSecretMask || '')); + setWebhookSecret(''); + setClearWebhookSecret(false); + setHasPrometheusToken(Boolean(rawSettings.hasPrometheusToken)); + setPrometheusTokenMask(String(rawSettings.prometheusTokenMask || '')); + setPrometheusMetricsToken(''); + setClearPrometheusToken(false); setFieldErrors({}); setServerPort(setCached('vcon:server-port', loadedPort)); setStoragePath( @@ -211,6 +332,9 @@ export function SettingsWorkspace() { String(storageData.stats?.storageDirectory ?? '') ) ); + if (storageData.diskUsage && typeof storageData.diskUsage === 'object') { + setDiskUsage(storageData.diskUsage as DiskUsageSummary); + } if (metricsData) { setActiveSockets(Number(metricsData.activeSockets) || 0); setSocketStatus( @@ -228,14 +352,57 @@ export function SettingsWorkspace() { .finally(() => setLoading(false)); }, [error]); + const s3ConfigDirty = useMemo( + () => + s3Draft.enableS3Backup !== savedS3Settings.enableS3Backup || + s3Draft.s3Endpoint.trim() !== savedS3Settings.s3Endpoint || + (s3Draft.s3Region.trim() || 'auto') !== (savedS3Settings.s3Region || 'auto') || + s3Draft.s3Bucket.trim() !== savedS3Settings.s3Bucket || + s3Draft.s3Prefix.trim() !== savedS3Settings.s3Prefix || + s3Draft.s3AccessKeyId.trim() !== savedS3Settings.s3AccessKeyId || + s3SecretAccessKey.trim().length > 0 || + clearS3SecretKey, + [clearS3SecretKey, s3Draft, s3SecretAccessKey, savedS3Settings] + ); + + const alertingConfigDirty = useMemo( + () => + alertingDraft.enableWebhookAlerts !== + savedAlertingSettings.enableWebhookAlerts || + alertingDraft.webhookUrl.trim() !== savedAlertingSettings.webhookUrl || + webhookSecret.trim().length > 0 || + clearWebhookSecret || + alertingDraft.enablePrometheusMetrics !== + savedAlertingSettings.enablePrometheusMetrics || + prometheusMetricsToken.trim().length > 0 || + clearPrometheusToken, + [ + alertingDraft, + clearPrometheusToken, + clearWebhookSecret, + prometheusMetricsToken, + savedAlertingSettings, + webhookSecret, + ] + ); + const isDirty = useMemo( () => jwtSecret.trim().length > 0 || enableIdempotency !== savedSettings.enableIdempotency || + s3ConfigDirty || + alertingConfigDirty || NUMERIC_SETTING_KEYS.some( (key) => numericDrafts[key].trim() !== String(savedSettings[key]) ), - [enableIdempotency, jwtSecret, numericDrafts, savedSettings] + [ + alertingConfigDirty, + enableIdempotency, + jwtSecret, + numericDrafts, + s3ConfigDirty, + savedSettings, + ] ); const updateNumberDraft = (key: NumericSettingKey, value: string) => { @@ -257,6 +424,11 @@ export function SettingsWorkspace() { const executeSave = async (validated: EditableSettings) => { const trimmedSecret = jwtSecret.trim(); + const normalizedS3 = normalizeS3Settings(s3Draft); + const trimmedS3Secret = s3SecretAccessKey.trim(); + const normalizedAlerting = normalizeAlertingSettings(alertingDraft); + const trimmedWebhookSecret = webhookSecret.trim(); + const trimmedPrometheusToken = prometheusMetricsToken.trim(); setConfirmRotateOpen(false); setSaving(true); try { @@ -265,6 +437,30 @@ export function SettingsWorkspace() { headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ ...validated, + enableS3Backup: normalizedS3.enableS3Backup, + s3Endpoint: normalizedS3.s3Endpoint, + s3Region: normalizedS3.s3Region, + s3Bucket: normalizedS3.s3Bucket, + s3Prefix: normalizedS3.s3Prefix, + s3AccessKeyId: normalizedS3.s3AccessKeyId, + ...(trimmedS3Secret + ? { s3SecretAccessKey: trimmedS3Secret } + : clearS3SecretKey + ? { clearS3SecretKey: true } + : {}), + enableWebhookAlerts: normalizedAlerting.enableWebhookAlerts, + webhookUrl: normalizedAlerting.webhookUrl, + ...(trimmedWebhookSecret + ? { webhookSecret: trimmedWebhookSecret } + : clearWebhookSecret + ? { clearWebhookSecret: true } + : {}), + enablePrometheusMetrics: normalizedAlerting.enablePrometheusMetrics, + ...(trimmedPrometheusToken + ? { prometheusMetricsToken: trimmedPrometheusToken } + : clearPrometheusToken + ? { clearPrometheusToken: true } + : {}), ...(trimmedSecret ? { jwtSecret: trimmedSecret } : {}), }), }); @@ -275,10 +471,32 @@ export function SettingsWorkspace() { const data = await response.json(); if (!response.ok) throw new Error(data.error || 'Settings update failed'); const synced = normalizeEditableSettings(data.settings ?? validated); + const syncedS3 = normalizeS3Settings(data.settings ?? normalizedS3); + const syncedAlerting = normalizeAlertingSettings( + data.settings ?? normalizedAlerting + ); setCached('vcon:settings', synced); + setCached('vcon:s3-settings', syncedS3); + setCached('vcon:alerting-settings', syncedAlerting); setSavedSettings(synced); setNumericDrafts(settingsToNumericDrafts(synced)); setEnableIdempotency(synced.enableIdempotency); + setSavedS3Settings(syncedS3); + setS3Draft(syncedS3); + setHasS3SecretKey(Boolean(data.settings?.hasS3SecretKey)); + setS3SecretKeyMask(String(data.settings?.s3SecretKeyMask || '')); + setS3SecretAccessKey(''); + setClearS3SecretKey(false); + setSavedAlertingSettings(syncedAlerting); + setAlertingDraft(syncedAlerting); + setHasWebhookSecret(Boolean(data.settings?.hasWebhookSecret)); + setWebhookSecretMask(String(data.settings?.webhookSecretMask || '')); + setWebhookSecret(''); + setClearWebhookSecret(false); + setHasPrometheusToken(Boolean(data.settings?.hasPrometheusToken)); + setPrometheusTokenMask(String(data.settings?.prometheusTokenMask || '')); + setPrometheusMetricsToken(''); + setClearPrometheusToken(false); setFieldErrors({}); setJwtSecret(''); setJwtError(''); @@ -316,6 +534,127 @@ export function SettingsWorkspace() { } }; + const handleTestWebhookAlert = async () => { + setTestingWebhook(true); + setWebhookTestStatus(null); + try { + const normalizedAlerting = normalizeAlertingSettings(alertingDraft); + const trimmedWebhookSecret = webhookSecret.trim(); + const response = await fetch('/api/settings/webhook-test', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + webhookUrl: normalizedAlerting.webhookUrl, + ...(trimmedWebhookSecret ? { webhookSecret: trimmedWebhookSecret } : {}), + ...(clearWebhookSecret ? { clearWebhookSecret: true } : {}), + }), + }); + if (response.status === 401) { + window.location.assign('/vcon/login'); + return; + } + const data = await response.json().catch(() => ({})); + if (!response.ok || !data.success) { + const msg = String(data.error || 'Webhook test alert failed.'); + setWebhookTestStatus({ ok: false, text: msg }); + error('Webhook test failed', msg); + return; + } + const msg = `Test alert delivered (HTTP ${data.status}).`; + setWebhookTestStatus({ ok: true, text: msg }); + success('Webhook alert delivered', msg); + } catch (testErr) { + const msg = + testErr instanceof Error ? testErr.message : 'Webhook test alert failed.'; + setWebhookTestStatus({ ok: false, text: msg }); + error('Webhook test error', msg); + } finally { + setTestingWebhook(false); + } + }; + + const handleTestS3Connection = async () => { + setTestingS3(true); + setS3ActionStatus(null); + try { + const normalizedS3 = normalizeS3Settings(s3Draft); + const trimmedS3Secret = s3SecretAccessKey.trim(); + const response = await fetch('/api/storage/s3-test', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + s3Endpoint: normalizedS3.s3Endpoint, + s3Region: normalizedS3.s3Region, + s3Bucket: normalizedS3.s3Bucket, + s3Prefix: normalizedS3.s3Prefix, + s3AccessKeyId: normalizedS3.s3AccessKeyId, + ...(trimmedS3Secret ? { s3SecretAccessKey: trimmedS3Secret } : {}), + ...(clearS3SecretKey ? { clearS3SecretKey: true } : {}), + }), + }); + if (response.status === 401) { + window.location.assign('/vcon/login'); + return; + } + const data = await response.json().catch(() => ({})); + if (!response.ok || !data.success) { + const msg = String(data.error || 'S3 connection check failed.'); + setS3ActionStatus({ ok: false, text: msg }); + error('S3 test failed', msg); + return; + } + const msg = `Connected to bucket "${data.bucket}" (HTTP ${data.status}).`; + setS3ActionStatus({ ok: true, text: msg }); + success('S3 connection verified', msg); + } catch (testErr) { + const msg = + testErr instanceof Error ? testErr.message : 'S3 connection check failed.'; + setS3ActionStatus({ ok: false, text: msg }); + error('S3 test error', msg); + } finally { + setTestingS3(false); + } + }; + + const handleArchiveNow = async () => { + if (s3ConfigDirty) { + warning('Unsaved S3 settings', 'Save your S3 credentials before archiving.'); + return; + } + setArchivingNow(true); + setS3ActionStatus(null); + try { + const response = await fetch('/api/storage/archive', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ pruneAfterUpload: false }), + }); + if (response.status === 401) { + window.location.assign('/vcon/login'); + return; + } + const data = await response.json().catch(() => ({})); + if (!response.ok || !data.success) { + const msg = String(data.error || 'Off-site S3 archive failed.'); + setS3ActionStatus({ ok: false, text: msg }); + error('Archive failed', msg); + return; + } + const compressed = Number(data.compressedCount) || 0; + const archived = Number(data.archivedCount) || 0; + const msg = `Compressed ${compressed} file(s) and archived ${archived} partition(s) to S3 (local copies retained).`; + setS3ActionStatus({ ok: true, text: msg }); + success('Archive completed', msg); + } catch (archiveErr) { + const msg = + archiveErr instanceof Error ? archiveErr.message : 'Off-site S3 archive failed.'; + setS3ActionStatus({ ok: false, text: msg }); + error('Archive error', msg); + } finally { + setArchivingNow(false); + } + }; + const handleFormSubmit = async (event: React.FormEvent) => { event.preventDefault(); const validation = validateEditableSettingsDraft( @@ -335,6 +674,38 @@ export function SettingsWorkspace() { return; } + const trimmedEndpoint = s3Draft.s3Endpoint.trim(); + if (trimmedEndpoint) { + try { + const parsed = new URL( + /^https?:\/\//i.test(trimmedEndpoint) + ? trimmedEndpoint + : `https://${trimmedEndpoint}` + ); + if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { + throw new Error('Invalid protocol'); + } + } catch { + setActiveTab('storage'); + error('Invalid S3 Endpoint', 'S3 Endpoint must be a valid http(s) URL.'); + return; + } + } + + const trimmedWebhookUrl = alertingDraft.webhookUrl.trim(); + if (trimmedWebhookUrl) { + try { + const parsed = new URL(trimmedWebhookUrl); + if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { + throw new Error('Invalid protocol'); + } + } catch { + setActiveTab('security'); + error('Invalid Webhook URL', 'Webhook URL must be a valid http(s) URL.'); + return; + } + } + setFieldErrors({}); const trimmedSecret = jwtSecret.trim(); if (trimmedSecret.length > 0 && trimmedSecret.length < 32) { @@ -373,6 +744,16 @@ export function SettingsWorkspace() { const discardChanges = () => { setNumericDrafts(settingsToNumericDrafts(savedSettings)); setEnableIdempotency(savedSettings.enableIdempotency); + setS3Draft(savedS3Settings); + setS3SecretAccessKey(''); + setClearS3SecretKey(false); + setS3ActionStatus(null); + setAlertingDraft(savedAlertingSettings); + setWebhookSecret(''); + setClearWebhookSecret(false); + setWebhookTestStatus(null); + setPrometheusMetricsToken(''); + setClearPrometheusToken(false); setFieldErrors({}); setJwtSecret(''); setJwtError(''); @@ -563,7 +944,7 @@ export function SettingsWorkspace() { {activeTab === 'storage' && ( <> - + + +
+
+ + {diskUsage?.available + ? `${diskUsage.usedPct.toFixed(1)}% used` + : 'Statfs unavailable'} + + {diskUsage?.available && ( + + {formatGiB(diskUsage.freeBytes)} free /{' '} + {formatGiB(diskUsage.totalBytes)} + + )} +
+ {diskUsage?.available && ( +
+
0 && + diskUsage.usedPct >= + Number(numericDrafts.diskPauseThresholdPct) + ? 'bg-[var(--danger)]' + : diskUsage.usedPct >= 80 + ? 'bg-[var(--warning)]' + : 'bg-[var(--accent)]' + }`} + style={{ + width: `${Math.min(100, Math.max(0, diskUsage.usedPct))}%`, + }} + /> +
+ )} +
+ + + + updateNumberDraft('compressAfterDays', event.target.value) + } + className="font-mono" + /> + + + + updateNumberDraft('diskPauseThresholdPct', event.target.value) + } + className="font-mono" + /> + + + + + + setS3Draft((prev) => ({ ...prev, enableS3Backup: checked })) + } + /> + + + + setS3Draft((prev) => ({ + ...prev, + s3Endpoint: event.target.value, + })) + } + className="font-mono text-[11px]" + /> + + + + setS3Draft((prev) => ({ + ...prev, + s3Region: event.target.value, + })) + } + className="font-mono" + /> + + + + setS3Draft((prev) => ({ + ...prev, + s3Bucket: event.target.value, + })) + } + className="font-mono" + /> + + + + setS3Draft((prev) => ({ + ...prev, + s3Prefix: event.target.value, + })) + } + className="font-mono" + /> + + + + setS3Draft((prev) => ({ + ...prev, + s3AccessKeyId: event.target.value, + })) + } + className="font-mono text-[11px]" + /> + + +
+ { + setS3SecretAccessKey(event.target.value); + if (clearS3SecretKey) setClearS3SecretKey(false); + }} + placeholder={ + clearS3SecretKey + ? 'Saved key will be cleared on Save' + : hasS3SecretKey + ? `Saved (${s3SecretKeyMask || 'encrypted'}) — blank keeps current` + : 'Enter Secret Access Key...' + } + className="font-mono text-[11px]" + /> + {hasS3SecretKey && ( + + )} +
+
+ +
+
+ + +
+ {s3ConfigDirty && ( + + Save S3 changes before running Archive Now. + + )} + {s3ActionStatus && ( +
+ {s3ActionStatus.text} +
+ )} +
+
+
+ - - { - const val = event.target.value; - setJwtSecret(val); - if (!val.trim() || val.trim().length >= 32) { - setJwtError(''); + <> + + + { + const val = event.target.value; + setJwtSecret(val); + if (!val.trim() || val.trim().length >= 32) { + setJwtError(''); + } + }} + placeholder="Minimum 32 characters to rotate..." + /> + + + + updateNumberDraft('maxPayloadSizeKb', event.target.value) } - }} - placeholder="Minimum 32 characters to rotate..." - /> - - - - updateNumberDraft('maxPayloadSizeKb', event.target.value) - } - className="font-mono" - /> - - + className="font-mono" + /> + + + + + + + setAlertingDraft((prev) => ({ + ...prev, + enableWebhookAlerts: checked, + })) + } + /> + + + + setAlertingDraft((prev) => ({ + ...prev, + webhookUrl: event.target.value, + })) + } + className="font-mono text-[11px]" + /> + + +
+ { + setWebhookSecret(event.target.value); + if (clearWebhookSecret) setClearWebhookSecret(false); + }} + placeholder={ + clearWebhookSecret + ? 'Saved secret will be cleared on Save' + : hasWebhookSecret + ? `Saved (${webhookSecretMask || 'encrypted'}) — blank keeps current` + : 'Optional HMAC-SHA256 signing secret...' + } + className="font-mono text-[11px]" + /> + {hasWebhookSecret && ( + + )} +
+
+ + + updateNumberDraft('webhookCooldownSec', event.target.value) + } + className="font-mono" + /> + + +
+
+ +
+ {webhookTestStatus && ( +
+ {webhookTestStatus.text} +
+ )} +
+
+
+ + + + + setAlertingDraft((prev) => ({ + ...prev, + enablePrometheusMetrics: checked, + })) + } + /> + + + + + +
+ { + setPrometheusMetricsToken(event.target.value); + if (clearPrometheusToken) setClearPrometheusToken(false); + }} + placeholder={ + clearPrometheusToken + ? 'Saved token will be cleared on Save' + : hasPrometheusToken + ? `Saved (${prometheusTokenMask || 'encrypted'}) — blank keeps current` + : 'Enter Bearer token for external Prometheus scrape...' + } + className="font-mono text-[11px]" + /> + {hasPrometheusToken && ( + + )} +
+
+
+ )} Number(octet)); + if ( + parts.length !== 4 || + parts.some((n) => !Number.isInteger(n) || n < 0 || n > 255) + ) { + return null; + } + return ( + (((parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3]) >>> 0) + ); +} + +export function parseAllowedIpsList(raw?: string | null): string[] { + if (!raw || typeof raw !== "string") return []; + return raw + .split(/[\s,;]+/) + .map((entry) => entry.trim()) + .filter(Boolean); +} + +export function validateAllowedIpsInput( + raw: unknown, +): { valid: true; normalized: string } | { valid: false; error: string } { + if (raw === undefined || raw === null || raw === "") { + return { valid: true, normalized: "" }; + } + if (typeof raw !== "string") { + return { valid: false, error: "allowedIps must be a string." }; + } + if (raw.length > 2048) { + return { + valid: false, + error: "allowedIps cannot exceed 2048 characters.", + }; + } + + const entries = parseAllowedIpsList(raw); + if (entries.length === 0) { + return { valid: true, normalized: "" }; + } + if (entries.length > 64) { + return { + valid: false, + error: "allowedIps cannot contain more than 64 IP/CIDR entries.", + }; + } + + const normalizedEntries: string[] = []; + for (const entry of entries) { + if (entry === "*") { + normalizedEntries.push("*"); + continue; + } + + if (entry.includes("/")) { + const [ipPart, prefixPart, ...extra] = entry.split("/"); + if (extra.length > 0 || !ipPart || !prefixPart || !/^\d+$/.test(prefixPart)) { + return { + valid: false, + error: `Invalid CIDR entry '${entry}'. Expected format like 203.0.113.0/24.`, + }; + } + const prefix = Number(prefixPart); + const cleanIp = stripIpv6MappedPrefix(ipPart); + if (net.isIPv4(cleanIp)) { + if (prefix < 0 || prefix > 32) { + return { + valid: false, + error: `Invalid IPv4 CIDR prefix in '${entry}' (must be 0..32).`, + }; + } + normalizedEntries.push(`${cleanIp}/${prefix}`); + continue; + } + if (net.isIPv6(cleanIp)) { + if (prefix < 0 || prefix > 128) { + return { + valid: false, + error: `Invalid IPv6 CIDR prefix in '${entry}' (must be 0..128).`, + }; + } + normalizedEntries.push(`${cleanIp.toLowerCase()}/${prefix}`); + continue; + } + return { + valid: false, + error: `Invalid IP address in CIDR entry '${entry}'.`, + }; + } + + const cleanIp = stripIpv6MappedPrefix(entry); + const ipVersion = net.isIP(cleanIp); + if (ipVersion === 4) { + normalizedEntries.push(cleanIp); + } else if (ipVersion === 6) { + normalizedEntries.push(cleanIp.toLowerCase()); + } else { + return { + valid: false, + error: `Invalid IP or CIDR entry '${entry}'.`, + }; + } + } + + return { + valid: true, + normalized: Array.from(new Set(normalizedEntries)).join(", "), + }; +} + +export function isClientIpAllowed( + clientIp: string, + allowedIpsRaw?: string | null, +): boolean { + const entries = parseAllowedIpsList(allowedIpsRaw); + if (entries.length === 0) { + return true; + } + + const cleanClientIp = stripIpv6MappedPrefix(clientIp); + if (!cleanClientIp || cleanClientIp === "unknown") { + return entries.includes("*"); + } + + const clientIpv4Int = ipv4ToUint32(cleanClientIp); + const isLoopbackClient = + cleanClientIp === "127.0.0.1" || cleanClientIp.toLowerCase() === "::1"; + + for (const rule of entries) { + if (rule === "*") return true; + + if (rule.includes("/")) { + const [ruleIpRaw, prefixRaw] = rule.split("/"); + const ruleIp = stripIpv6MappedPrefix(ruleIpRaw); + const prefix = Number(prefixRaw); + const ruleIpv4Int = ipv4ToUint32(ruleIp); + + if (clientIpv4Int !== null && ruleIpv4Int !== null && prefix >= 0 && prefix <= 32) { + if (prefix === 0) return true; + const mask = prefix === 32 ? 0xffffffff : (~((1 << (32 - prefix)) - 1)) >>> 0; + if ((clientIpv4Int & mask) >>> 0 === (ruleIpv4Int & mask) >>> 0) { + return true; + } + } + continue; + } + + const cleanRule = stripIpv6MappedPrefix(rule); + if (cleanRule.toLowerCase() === cleanClientIp.toLowerCase()) { + return true; + } + if ( + isLoopbackClient && + (cleanRule === "127.0.0.1" || cleanRule.toLowerCase() === "::1") + ) { + return true; + } + } + + return false; +} + +export function validateClientExpiresAtInput( + raw: unknown, +): { valid: true; iso: string | null } | { valid: false; error: string } { + if (raw === undefined || raw === null || raw === "") { + return { valid: true, iso: null }; + } + if (typeof raw !== "string") { + return { + valid: false, + error: "expiresAt must be an ISO timestamp string or null.", + }; + } + const trimmed = raw.trim(); + if (!trimmed) { + return { valid: true, iso: null }; + } + const parsedMs = Date.parse(trimmed); + if (!Number.isFinite(parsedMs)) { + return { + valid: false, + error: "expiresAt must be a valid ISO-8601 date or timestamp.", + }; + } + return { valid: true, iso: new Date(parsedMs).toISOString() }; +} + +export function isClientTokenExpired( + expiresAt?: string | null, + nowMs = Date.now(), +): boolean { + if (!expiresAt || typeof expiresAt !== "string") return false; + const trimmed = expiresAt.trim(); + if (!trimmed) return false; + const parsedMs = Date.parse(trimmed); + if (!Number.isFinite(parsedMs)) return false; + return nowMs >= parsedMs; +} diff --git a/core/Security/secretVault.ts b/core/Security/secretVault.ts new file mode 100644 index 0000000..9d2baa1 --- /dev/null +++ b/core/Security/secretVault.ts @@ -0,0 +1,152 @@ +import { + createCipheriv, + createDecipheriv, + hkdfSync, + randomBytes, +} from "node:crypto"; + +const DEV_DEFAULT_SESSION_SECRET = + "datastack-admin-session-default-secret-32-chars"; +const VAULT_PREFIX = "v1"; +const HKDF_SALT = Buffer.from("datastack-vault-salt-v1", "utf8"); +const HKDF_INFO = Buffer.from("datastack-aes-256-gcm-v1", "utf8"); + +export type SecretVaultErrorCode = + | "SECRET_VAULT_KEY_UNAVAILABLE" + | "SECRET_VAULT_DECRYPT_FAILED" + | "SECRET_VAULT_INVALID_INPUT"; + +export class SecretVaultError extends Error { + public readonly code: SecretVaultErrorCode; + + constructor(code: SecretVaultErrorCode, message: string) { + super(`${code}: ${message}`); + this.name = "SecretVaultError"; + this.code = code; + } +} + +export function resolveVaultMasterSecret(secretOverride?: string): string | null { + const candidate = ( + secretOverride ?? + process.env.DATASTACK_ADMIN_SESSION_SECRET ?? + "" + ).trim(); + if (process.env.NODE_ENV === "production") { + return candidate.length >= 32 ? candidate : null; + } + if (candidate.length > 0) { + return candidate.length >= 32 ? candidate : null; + } + return DEV_DEFAULT_SESSION_SECRET; +} + +function deriveAesKey(secretOverride?: string): Buffer { + const master = resolveVaultMasterSecret(secretOverride); + if (!master) { + throw new SecretVaultError( + "SECRET_VAULT_KEY_UNAVAILABLE", + "DATASTACK_ADMIN_SESSION_SECRET (>= 32 characters) is required to encrypt or decrypt stored secrets.", + ); + } + const derived = hkdfSync( + "sha256", + Buffer.from(master, "utf8"), + HKDF_SALT, + HKDF_INFO, + 32, + ); + return Buffer.from(derived); +} + +export function encryptSecret( + plaintext: string, + secretOverride?: string, +): string { + if (typeof plaintext !== "string") { + throw new SecretVaultError( + "SECRET_VAULT_INVALID_INPUT", + "Secret value must be a string.", + ); + } + if (!plaintext) { + return ""; + } + const key = deriveAesKey(secretOverride); + const iv = randomBytes(12); + const cipher = createCipheriv("aes-256-gcm", key, iv); + const encrypted = Buffer.concat([ + cipher.update(plaintext, "utf8"), + cipher.final(), + ]); + const authTag = cipher.getAuthTag(); + return [ + VAULT_PREFIX, + iv.toString("base64url"), + authTag.toString("base64url"), + encrypted.toString("base64url"), + ].join(":"); +} + +export function decryptSecret( + ciphertext: string, + secretOverride?: string, +): string { + if (!ciphertext || typeof ciphertext !== "string") { + return ""; + } + const parts = ciphertext.trim().split(":"); + if (parts.length !== 4 || parts[0] !== VAULT_PREFIX) { + throw new SecretVaultError( + "SECRET_VAULT_DECRYPT_FAILED", + "Stored secret ciphertext has an unrecognized format.", + ); + } + + const key = deriveAesKey(secretOverride); + try { + const iv = Buffer.from(parts[1], "base64url"); + const authTag = Buffer.from(parts[2], "base64url"); + const payload = Buffer.from(parts[3], "base64url"); + if (iv.length !== 12 || authTag.length !== 16) { + throw new Error("Invalid IV or auth tag length"); + } + const decipher = createDecipheriv("aes-256-gcm", key, iv); + decipher.setAuthTag(authTag); + const decrypted = Buffer.concat([ + decipher.update(payload), + decipher.final(), + ]); + return decrypted.toString("utf8"); + } catch (err: unknown) { + if (err instanceof SecretVaultError) { + throw err; + } + throw new SecretVaultError( + "SECRET_VAULT_DECRYPT_FAILED", + "Unable to decrypt stored secret. Verify DATASTACK_ADMIN_SESSION_SECRET matches the key used to encrypt it.", + ); + } +} + +export function maskSecretHint(plaintext: string): string { + const trimmed = String(plaintext || "").trim(); + if (!trimmed) return ""; + if (trimmed.length >= 8) { + return `••••${trimmed.slice(-4)}`; + } + return "••••••••"; +} + +export function getMaskedHintFromEncrypted( + ciphertext: string, + secretOverride?: string, +): string { + if (!ciphertext || !ciphertext.trim()) return ""; + try { + const plain = decryptSecret(ciphertext, secretOverride); + return maskSecretHint(plain); + } catch { + return "••••(locked)"; + } +} diff --git a/core/Storage/csvStorage.ts b/core/Storage/csvStorage.ts index ada1ddb..0243534 100644 --- a/core/Storage/csvStorage.ts +++ b/core/Storage/csvStorage.ts @@ -1,7 +1,10 @@ import crypto from 'node:crypto'; import fs from 'fs'; import path from 'path'; +import { Readable } from 'node:stream'; +import { pipeline } from 'node:stream/promises'; import { StringDecoder } from 'node:string_decoder'; +import zlib from 'node:zlib'; import { Parser } from 'json2csv'; export interface DataEvent { @@ -19,8 +22,28 @@ export interface StorageFileMeta { sizeBytes: number; linesCount: number; modifiedAt: Date; + compressed?: boolean; } +export interface DiskUsageSnapshot { + usedPct: number; + availableBytes: number; + totalBytes: number; + checkedAt: number; +} + +export interface StorageAdmissionResult { + allowed: boolean; + code?: 'DISK_SPACE_LOW' | 'STORAGE_QUOTA_EXCEEDED'; + reason?: string; +} + +export type StatfsProvider = (dir: string) => Promise<{ + blocks: number; + bavail: number; + bsize: number; +}>; + interface RowCountCacheEntry { sizeBytes: number; mtimeMs: number; @@ -31,6 +54,7 @@ const LOCK_RETRY_MIN_MS = 5; const LOCK_RETRY_MAX_MS = 18; const LOCK_TIMEOUT_MS = 10_000; const STALE_LOCK_AGE_MS = 30_000; +const QUOTA_CACHE_TTL_MS = 30_000; function sleepMs(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); @@ -49,11 +73,39 @@ export class CSVStorageManager { public static baseStorageDir = resolveDefaultStorageDir(); public static maxFileSizeBytes = 100 * 1024 * 1024; // 100MB rotation limit private static rowCountCache = new Map(); + private static diskUsageCache: DiskUsageSnapshot | null = null; + private static clientBytesCache = new Map< + string, + { checkedAt: number; bytes: number } + >(); + private static statfsOverrideForTests: StatfsProvider | null = null; + + public static isPartitionFilename(filename: string): boolean { + return filename.endsWith('.csv') || filename.endsWith('.csv.gz'); + } + + public static extractDateFromFilename(filename: string): string { + return filename.replace(/\.csv(?:\.gz)?$/, '').split('-part')[0]; + } public static clearRowCountCache(): void { this.rowCountCache.clear(); } + public static clearQuotaCaches(clientId?: string): void { + this.diskUsageCache = null; + if (clientId) { + this.clientBytesCache.delete(this.sanitizeClientId(clientId)); + } else { + this.clientBytesCache.clear(); + } + } + + public static setStatfsOverrideForTests(provider: StatfsProvider | null): void { + this.statfsOverrideForTests = provider; + this.diskUsageCache = null; + } + public static sanitizeClientId(clientId: string): string { const raw = String(clientId || 'unknown').trim().replace(/[^a-zA-Z0-9_-]/g, '_'); return raw || 'unknown'; @@ -299,6 +351,12 @@ export class CSVStorageManager { private static countCsvDataRowsChunkedSync(filePath: string): number { const counter = this.createChunkRowCounter(); + if (filePath.endsWith('.gz')) { + const compressed = fs.readFileSync(filePath); + const decompressed = zlib.gunzipSync(compressed); + counter.pushChunk(decompressed); + return counter.finish(); + } const buf = Buffer.allocUnsafe(64 * 1024); const fd = fs.openSync(filePath, 'r'); try { @@ -312,14 +370,34 @@ export class CSVStorageManager { } } + public static openPartitionTextStream(filePath: string): Readable { + const fileStream = fs.createReadStream(filePath, { + highWaterMark: 64 * 1024, + }); + if (!filePath.endsWith('.gz')) { + return fileStream; + } + const gunzip = zlib.createGunzip(); + fileStream.on('error', (err) => { + if (!gunzip.destroyed) { + gunzip.destroy(err); + } + }); + gunzip.on('close', () => { + if (!fileStream.destroyed) { + fileStream.destroy(); + } + }); + fileStream.pipe(gunzip); + return gunzip; + } + private static async countCsvDataRowsStreamAsync( filePath: string, ): Promise { const counter = this.createChunkRowCounter(); return new Promise((resolve, reject) => { - const stream = fs.createReadStream(filePath, { - highWaterMark: 64 * 1024, - }); + const stream = this.openPartitionTextStream(filePath); stream.on('data', (chunk) => counter.pushChunk(chunk)); stream.on('end', () => resolve(counter.finish())); stream.on('error', (err) => reject(err)); @@ -437,8 +515,9 @@ export class CSVStorageManager { fs.writeFileSync(resolvedPath, fullCsv, 'utf8'); } - // Invalidate row cache for modified file immediately + // Invalidate row & quota caches for modified file immediately this.rowCountCache.delete(resolvedPath); + this.clearQuotaCaches(clientId); return resolvedPath; }); @@ -449,7 +528,7 @@ export class CSVStorageManager { } /** - * List all stored CSV files with cached row counts (invalidated on mtime/size changes) + * List all stored CSV and gzipped CSV (.csv / .csv.gz) files with cached row counts */ public static listAllFiles(): StorageFileMeta[] { const results: StorageFileMeta[] = []; @@ -473,7 +552,7 @@ export class CSVStorageManager { const files = fs.readdirSync(fullClientPath); for (const file of files) { - if (!file.endsWith('.csv')) continue; + if (!this.isPartitionFilename(file)) continue; const fullFilePath = path.join(fullClientPath, file); let stats: fs.Stats; try { @@ -505,7 +584,7 @@ export class CSVStorageManager { } } - const datePart = file.replace('.csv', '').split('-part')[0]; + const datePart = this.extractDateFromFilename(file); results.push({ clientId: clientDir, @@ -515,6 +594,7 @@ export class CSVStorageManager { sizeBytes: stats.size, linesCount, modifiedAt: stats.mtime, + compressed: file.endsWith('.csv.gz'), }); } } @@ -551,7 +631,7 @@ export class CSVStorageManager { const files = await fs.promises.readdir(fullClientPath); for (const file of files) { - if (!file.endsWith('.csv')) continue; + if (!this.isPartitionFilename(file)) continue; const fullFilePath = path.join(fullClientPath, file); const stats = await fs.promises.lstat(fullFilePath).catch(() => null); if (!stats || stats.isSymbolicLink() || !stats.isFile()) continue; @@ -578,7 +658,7 @@ export class CSVStorageManager { } } - const datePart = file.replace('.csv', '').split('-part')[0]; + const datePart = this.extractDateFromFilename(file); results.push({ clientId: clientDir, @@ -588,6 +668,7 @@ export class CSVStorageManager { sizeBytes: stats.size, linesCount, modifiedAt: stats.mtime, + compressed: file.endsWith('.csv.gz'), }); } } @@ -605,7 +686,121 @@ export class CSVStorageManager { } /** - * Prune CSV partitions older than autoPruneDays under cooperative partition locks + * Compress uncompressed .csv partitions older than compressAfterDays into .csv.gz under cooperative partition locks + */ + public static async compressOldPartitionsAsync( + compressAfterDays: number, + ): Promise { + if (!Number.isFinite(compressAfterDays) || compressAfterDays <= 0) { + return 0; + } + + const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null); + if (!baseStat || !baseStat.isDirectory()) { + return 0; + } + + const cutoffDate = new Date( + Date.now() - Math.floor(compressAfterDays) * 86_400_000, + ) + .toISOString() + .split('T')[0]; + + let compressedCount = 0; + try { + const clientDirs = await fs.promises.readdir(this.baseStorageDir); + for (const clientDir of clientDirs) { + const fullClientPath = path.join(this.baseStorageDir, clientDir); + const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null); + if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) { + continue; + } + + const files = await fs.promises + .readdir(fullClientPath) + .catch(() => [] as string[]); + for (const file of files) { + if (!file.endsWith('.csv')) continue; + const datePart = this.extractDateFromFilename(file); + if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) { + continue; + } + const fullPath = path.join(fullClientPath, file); + const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); + if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) { + continue; + } + + const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`); + const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`; + let acquired = false; + const tryAcquireLock = async (): Promise => { + const handle = await fs.promises.open(lockPath, 'wx'); + try { + await handle.writeFile(lockValue, 'utf8'); + } finally { + await handle.close(); + } + return true; + }; + + try { + acquired = await tryAcquireLock(); + } catch (err: unknown) { + if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') { + await this.removeStaleLockAtomicallyAsync(lockPath); + try { + acquired = await tryAcquireLock(); + } catch { + acquired = false; + } + } + } + + if (!acquired) continue; + + let targetGzPath = `${fullPath}.gz`; + if (fs.existsSync(targetGzPath)) { + const stem = fullPath.replace(/\.csv$/, ''); + let suffix = 2; + while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) { + suffix += 1; + } + targetGzPath = `${stem}-gz${suffix}.csv.gz`; + } + const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`; + + try { + await pipeline( + fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }), + zlib.createGzip({ level: 6 }), + fs.createWriteStream(tmpGzPath), + ); + await fs.promises + .utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime) + .catch(() => undefined); + await fs.promises.rename(tmpGzPath, targetGzPath); + await fs.promises.unlink(fullPath); + this.rowCountCache.delete(fullPath); + this.rowCountCache.delete(targetGzPath); + this.clearQuotaCaches(clientDir); + compressedCount += 1; + } catch { + await fs.promises.unlink(tmpGzPath).catch(() => undefined); + } finally { + await this.releaseLockIfOwnedAsync(lockPath, lockValue); + } + } + } + } catch { + // Ignore directory scan failure + } + + return compressedCount; + } + + /** + * Prune CSV / CSV.GZ partitions older than autoPruneDays under cooperative partition locks */ public static pruneExpiredPartitions(autoPruneDays: number): number { if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) { @@ -636,8 +831,8 @@ export class CSVStorageManager { const files = fs.readdirSync(fullClientPath); for (const file of files) { - if (!file.endsWith('.csv')) continue; - const datePart = file.replace('.csv', '').split('-part')[0]; + if (!this.isPartitionFilename(file)) continue; + const datePart = this.extractDateFromFilename(file); if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { const fullPath = path.join(fullClientPath, file); let fileLstat: fs.Stats; @@ -683,6 +878,7 @@ export class CSVStorageManager { if (fs.existsSync(fullPath)) { fs.unlinkSync(fullPath); this.rowCountCache.delete(fullPath); + this.clearQuotaCaches(clientDir); prunedCount += 1; } } catch { @@ -701,27 +897,54 @@ export class CSVStorageManager { } /** - * Non-blocking async partition pruner using fs.promises under cooperative partition locks + * Archive and/or prune expired partitions under cooperative partition locks. + * Never deletes a local partition if options.beforeDeleteFile throws or rejects (V2-E02). */ - public static async pruneExpiredPartitionsAsync( + public static async pruneOrArchiveExpiredPartitionsAsync( autoPruneDays: number, - ): Promise { - if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) { - return 0; + options?: { + beforeDeleteFile?: (file: { + clientId: string; + filename: string; + filePath: string; + date: string; + }) => Promise; + archiveOnlyWithoutDelete?: boolean; + maxCutoffDays?: number; + }, + ): Promise<{ + scannedCount: number; + archivedCount: number; + prunedCount: number; + errors: string[]; + }> { + const targetDays = + options?.maxCutoffDays !== undefined + ? options.maxCutoffDays + : autoPruneDays; + if (!Number.isFinite(targetDays) || targetDays < 0) { + return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] }; + } + if (options?.maxCutoffDays === undefined && targetDays <= 0) { + return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] }; } const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null); if (!baseStat || !baseStat.isDirectory()) { - return 0; + return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] }; } const cutoffDate = new Date( - Date.now() - Math.floor(autoPruneDays) * 86_400_000, + Date.now() - Math.floor(targetDays) * 86_400_000, ) .toISOString() .split('T')[0]; + let scannedCount = 0; + let archivedCount = 0; let prunedCount = 0; + const errors: string[] = []; + try { const clientDirs = await fs.promises.readdir(this.baseStorageDir); for (const clientDir of clientDirs) { @@ -731,54 +954,71 @@ export class CSVStorageManager { continue; } - const files = await fs.promises.readdir(fullClientPath).catch(() => [] as string[]); + const files = await fs.promises + .readdir(fullClientPath) + .catch(() => [] as string[]); for (const file of files) { - if (!file.endsWith('.csv')) continue; - const datePart = file.replace('.csv', '').split('-part')[0]; - if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { - const fullPath = path.join(fullClientPath, file); - const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); - if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) { - continue; - } + if (!this.isPartitionFilename(file)) continue; + const datePart = this.extractDateFromFilename(file); + if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) { + continue; + } + const fullPath = path.join(fullClientPath, file); + const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); + if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) { + continue; + } + scannedCount += 1; - const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`); - const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`; - let acquired = false; - const tryAcquireLock = async (): Promise => { - const handle = await fs.promises.open(lockPath, 'wx'); + const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`); + const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`; + let acquired = false; + const tryAcquireLock = async (): Promise => { + const handle = await fs.promises.open(lockPath, 'wx'); + try { + await handle.writeFile(lockValue, 'utf8'); + } finally { + await handle.close(); + } + return true; + }; + + try { + acquired = await tryAcquireLock(); + } catch (err: unknown) { + if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') { + await this.removeStaleLockAtomicallyAsync(lockPath); try { - await handle.writeFile(lockValue, 'utf8'); - } finally { - await handle.close(); - } - return true; - }; - - try { - acquired = await tryAcquireLock(); - } catch (err: unknown) { - if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') { - await this.removeStaleLockAtomicallyAsync(lockPath); - try { - acquired = await tryAcquireLock(); - } catch { - acquired = false; - } + acquired = await tryAcquireLock(); + } catch { + acquired = false; } } + } - if (!acquired) continue; + if (!acquired) continue; - try { + try { + if (options?.beforeDeleteFile) { + await options.beforeDeleteFile({ + clientId: clientDir, + filename: file, + filePath: fullPath, + date: datePart, + }); + archivedCount += 1; + } + if (!options?.archiveOnlyWithoutDelete) { await fs.promises.unlink(fullPath); this.rowCountCache.delete(fullPath); + this.clearQuotaCaches(clientDir); prunedCount += 1; - } catch { - // Ignore individual unlink error or concurrent removal - } finally { - await this.releaseLockIfOwnedAsync(lockPath, lockValue); } + } catch (err: unknown) { + const msg = err instanceof Error ? err.message : 'Archive/prune error'; + errors.push(`${clientDir}/${file}: ${msg}`); + } finally { + await this.releaseLockIfOwnedAsync(lockPath, lockValue); } } } @@ -786,7 +1026,149 @@ export class CSVStorageManager { // Ignore directory scan failure } - return prunedCount; + return { scannedCount, archivedCount, prunedCount, errors }; + } + + /** + * Non-blocking async partition pruner using fs.promises under cooperative partition locks + */ + public static async pruneExpiredPartitionsAsync( + autoPruneDays: number, + ): Promise { + const res = await this.pruneOrArchiveExpiredPartitionsAsync(autoPruneDays); + return res.prunedCount; + } + + /** + * Cached VPS filesystem disk usage snapshot (30s TTL) for disk-full protection + */ + public static async getDiskUsageAsync(): Promise { + const now = Date.now(); + if ( + this.diskUsageCache && + now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS + ) { + return this.diskUsageCache; + } + + let probeDir = path.resolve(this.baseStorageDir); + while (!fs.existsSync(probeDir)) { + const parent = path.dirname(probeDir); + if (parent === probeDir) break; + probeDir = parent; + } + + try { + const statfsRes = this.statfsOverrideForTests + ? await this.statfsOverrideForTests(probeDir) + : await fs.promises.statfs(probeDir); + const bsize = Math.max(1, Number(statfsRes.bsize) || 4096); + const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize); + const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize); + const usedBytes = Math.max(0, totalBytes - availableBytes); + const usedPct = + totalBytes > 0 + ? Number( + Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2), + ) + : 0; + const snapshot: DiskUsageSnapshot = { + usedPct, + availableBytes, + totalBytes, + checkedAt: now, + }; + this.diskUsageCache = snapshot; + return snapshot; + } catch { + const fallback: DiskUsageSnapshot = { + usedPct: 0, + availableBytes: 0, + totalBytes: 0, + checkedAt: now, + }; + this.diskUsageCache = fallback; + return fallback; + } + } + + /** + * Cached per-client directory byte count (30s TTL) + */ + public static async getClientStorageBytesAsync( + clientId: string, + ): Promise { + const safeClientId = this.sanitizeClientId(clientId); + const now = Date.now(); + const cached = this.clientBytesCache.get(safeClientId); + if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) { + return cached.bytes; + } + + const resolvedBase = path.resolve(this.baseStorageDir); + const clientDir = path.resolve(resolvedBase, safeClientId); + if (!clientDir.startsWith(resolvedBase + path.sep)) { + return 0; + } + + let totalBytes = 0; + try { + const dirStat = await fs.promises.lstat(clientDir).catch(() => null); + if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) { + const entries = await fs.promises + .readdir(clientDir) + .catch(() => [] as string[]); + for (const entry of entries) { + if (!this.isPartitionFilename(entry)) continue; + const filePath = path.join(clientDir, entry); + const stats = await fs.promises.lstat(filePath).catch(() => null); + if (stats && !stats.isSymbolicLink() && stats.isFile()) { + totalBytes += stats.size; + } + } + } + } catch { + // Fall back to 0 on unreadable directory + } + + this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes }); + return totalBytes; + } + + /** + * Checks global VPS disk threshold and per-client MB quota before admitting new events + */ + public static async checkStorageAdmissionAsync( + clientId: string, + clientQuotaMb: number, + diskPauseThresholdPct: number, + ): Promise { + const threshold = Number(diskPauseThresholdPct); + if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) { + const disk = await this.getDiskUsageAsync(); + if (disk.totalBytes > 0 && disk.usedPct >= threshold) { + return { + allowed: false, + code: 'DISK_SPACE_LOW', + reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`, + }; + } + } + + const quotaMb = Number(clientQuotaMb); + if (Number.isFinite(quotaMb) && quotaMb > 0) { + const currentBytes = await this.getClientStorageBytesAsync(clientId); + const quotaBytes = Math.floor(quotaMb) * 1024 * 1024; + if (currentBytes >= quotaBytes) { + return { + allowed: false, + code: 'STORAGE_QUOTA_EXCEEDED', + reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`, + }; + } + } + + return { allowed: true }; } public static formatStorageStats(files: StorageFileMeta[]) { @@ -826,12 +1208,12 @@ export class CSVStorageManager { } /** - * Reads file content for export / download with path traversal protection + * Reads file content for export / download with path traversal protection (supports .csv & .csv.gz) */ public static getFileContent(clientId: string, filename: string): string | null { const safeClientId = this.sanitizeClientId(clientId); const safeFilename = path.basename(String(filename || '')); - if (!safeClientId || !safeFilename || !safeFilename.endsWith('.csv')) { + if (!safeClientId || !safeFilename || !this.isPartitionFilename(safeFilename)) { return null; } const resolvedBase = path.resolve(this.baseStorageDir); @@ -844,6 +1226,9 @@ export class CSVStorageManager { if (stats.isSymbolicLink() || !stats.isFile()) { return null; } + if (safeFilename.endsWith('.gz')) { + return zlib.gunzipSync(fs.readFileSync(fullPath)).toString('utf8'); + } return fs.readFileSync(fullPath, 'utf8'); } return null; diff --git a/core/Storage/s3Archive.ts b/core/Storage/s3Archive.ts new file mode 100644 index 0000000..744f3e0 --- /dev/null +++ b/core/Storage/s3Archive.ts @@ -0,0 +1,335 @@ +import crypto from "node:crypto"; +import fs from "node:fs"; + +export interface S3ArchiveConfig { + endpoint: string; + region: string; + bucket: string; + prefix: string; + accessKeyId: string; + secretAccessKey: string; +} + +export interface S3OperationResult { + ok: boolean; + status?: number; + endpoint?: string; + bucket?: string; + key?: string; + etag?: string; + code?: "S3_INVALID_CONFIG" | "S3_CONNECTION_FAILED" | "S3_ARCHIVE_FAILED"; + error?: string; +} + +function sha256Hex(data: Buffer | string): string { + return crypto.createHash("sha256").update(data).digest("hex"); +} + +function hmacSha256(key: Buffer | string, data: string): Buffer { + return crypto.createHmac("sha256", key).update(data, "utf8").digest(); +} + +function encodeRfc3986(value: string): string { + return encodeURIComponent(value).replace( + /[!'()*]/g, + (ch) => `%${ch.charCodeAt(0).toString(16).toUpperCase()}`, + ); +} + +function encodeS3Path(rawPath: string): string { + return rawPath + .split("/") + .map((segment) => encodeRfc3986(segment)) + .join("/"); +} + +export function buildS3ObjectKey( + prefix: string, + clientId: string, + filename: string, +): string { + const cleanPrefix = String(prefix || "") + .trim() + .replace(/^\/+|\/+$/g, ""); + const cleanClient = String(clientId || "unknown") + .trim() + .replace(/^\/+|\/+$/g, ""); + const cleanFile = String(filename || "") + .trim() + .replace(/^\/+/, ""); + return [cleanPrefix, cleanClient, cleanFile].filter(Boolean).join("/"); +} + +export function normalizeS3EndpointUrl( + endpoint: string, + region: string, +): { url: URL; region: string } { + const trimmedRegion = String(region || "").trim() || "auto"; + const trimmedEndpoint = String(endpoint || "").trim().replace(/\/+$/, ""); + + if (trimmedEndpoint) { + const withProtocol = /^https?:\/\//i.test(trimmedEndpoint) + ? trimmedEndpoint + : `https://${trimmedEndpoint}`; + const parsed = new URL(withProtocol); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") { + throw new Error("S3 endpoint must use http:// or https://"); + } + return { url: parsed, region: trimmedRegion }; + } + + if (trimmedRegion === "auto") { + throw new Error( + "S3 Endpoint URL is required when Region is set to 'auto' (e.g. Cloudflare R2 or MinIO).", + ); + } + return { + url: new URL(`https://s3.${trimmedRegion}.amazonaws.com`), + region: trimmedRegion, + }; +} + +export function signS3Request(params: { + method: "GET" | "HEAD" | "PUT"; + endpointUrl: URL; + bucket: string; + objectKey?: string; + queryParams?: Record; + region: string; + accessKeyId: string; + secretAccessKey: string; + payloadHashHex: string; + contentType?: string; + now?: Date; +}): { targetUrl: string; headers: Record } { + const now = params.now ?? new Date(); + const amzDate = now.toISOString().replace(/[:-]|\.\d{3}/g, ""); + const dateStamp = amzDate.slice(0, 8); + + const basePath = params.endpointUrl.pathname.replace(/\/+$/, ""); + const bucketPart = encodeRfc3986(params.bucket.trim()); + const keyPart = params.objectKey ? `/${encodeS3Path(params.objectKey)}` : ""; + const canonicalUri = `${basePath}/${bucketPart}${keyPart}`.replace( + /\/{2,}/g, + "/", + ); + + const queryEntries = Object.entries(params.queryParams ?? {}).sort(([a], [b]) => + a < b ? -1 : a > b ? 1 : 0, + ); + const canonicalQueryString = queryEntries + .map(([k, v]) => `${encodeRfc3986(k)}=${encodeRfc3986(v)}`) + .join("&"); + + const hostHeader = params.endpointUrl.host; + const headerMap: Record = { + host: hostHeader, + "x-amz-content-sha256": params.payloadHashHex, + "x-amz-date": amzDate, + }; + if (params.contentType) { + headerMap["content-type"] = params.contentType; + } + + const sortedHeaderKeys = Object.keys(headerMap).sort(); + const canonicalHeaders = sortedHeaderKeys + .map((key) => `${key}:${headerMap[key].trim()}\n`) + .join(""); + const signedHeaders = sortedHeaderKeys.join(";"); + + const canonicalRequest = [ + params.method, + canonicalUri, + canonicalQueryString, + canonicalHeaders, + signedHeaders, + params.payloadHashHex, + ].join("\n"); + + const credentialScope = `${dateStamp}/${params.region}/s3/aws4_request`; + const stringToSign = [ + "AWS4-HMAC-SHA256", + amzDate, + credentialScope, + sha256Hex(canonicalRequest), + ].join("\n"); + + const kDate = hmacSha256(`AWS4${params.secretAccessKey}`, dateStamp); + const kRegion = hmacSha256(kDate, params.region); + const kService = hmacSha256(kRegion, "s3"); + const kSigning = hmacSha256(kService, "aws4_request"); + const signature = crypto + .createHmac("sha256", kSigning) + .update(stringToSign, "utf8") + .digest("hex"); + + const authorization = `AWS4-HMAC-SHA256 Credential=${params.accessKeyId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}`; + + const targetUrl = `${params.endpointUrl.origin}${canonicalUri}${ + canonicalQueryString ? `?${canonicalQueryString}` : "" + }`; + + return { + targetUrl, + headers: { + "x-amz-date": amzDate, + "x-amz-content-sha256": params.payloadHashHex, + ...(params.contentType ? { "content-type": params.contentType } : {}), + Authorization: authorization, + }, + }; +} + +function validateS3Config(config: S3ArchiveConfig): string | null { + if (!config.bucket?.trim()) return "S3 Bucket name is required."; + if (!config.accessKeyId?.trim()) return "S3 Access Key ID is required."; + if (!config.secretAccessKey?.trim()) return "S3 Secret Access Key is required."; + return null; +} + +export async function testS3Connection( + config: S3ArchiveConfig, +): Promise { + const validationError = validateS3Config(config); + if (validationError) { + return { + ok: false, + code: "S3_INVALID_CONFIG", + error: validationError, + }; + } + + try { + const { url, region } = normalizeS3EndpointUrl( + config.endpoint, + config.region, + ); + const signed = signS3Request({ + method: "GET", + endpointUrl: url, + bucket: config.bucket.trim(), + queryParams: { "list-type": "2", "max-keys": "1" }, + region, + accessKeyId: config.accessKeyId.trim(), + secretAccessKey: config.secretAccessKey, + payloadHashHex: sha256Hex(""), + }); + + const response = await fetch(signed.targetUrl, { + method: "GET", + headers: signed.headers, + cache: "no-store", + signal: AbortSignal.timeout(8000), + }); + + if (!response.ok) { + const bodyPreview = await response.text().catch(() => ""); + return { + ok: false, + status: response.status, + endpoint: url.origin, + bucket: config.bucket.trim(), + code: "S3_CONNECTION_FAILED", + error: `S3 endpoint returned HTTP ${response.status}${ + bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : "" + }`, + }; + } + + return { + ok: true, + status: response.status, + endpoint: url.origin, + bucket: config.bucket.trim(), + }; + } catch (err: unknown) { + return { + ok: false, + code: "S3_CONNECTION_FAILED", + error: + err instanceof Error ? err.message : "Unable to reach S3 endpoint", + }; + } +} + +export async function uploadFileToS3( + config: S3ArchiveConfig, + localFilePath: string, + objectKey: string, +): Promise { + const validationError = validateS3Config(config); + if (validationError) { + return { + ok: false, + code: "S3_INVALID_CONFIG", + error: validationError, + key: objectKey, + }; + } + + try { + const { url, region } = normalizeS3EndpointUrl( + config.endpoint, + config.region, + ); + const bodyBuffer = await fs.promises.readFile(localFilePath); + const payloadHashHex = sha256Hex(bodyBuffer); + const contentType = localFilePath.endsWith(".gz") + ? "application/gzip" + : "text/csv; charset=utf-8"; + + const signed = signS3Request({ + method: "PUT", + endpointUrl: url, + bucket: config.bucket.trim(), + objectKey, + region, + accessKeyId: config.accessKeyId.trim(), + secretAccessKey: config.secretAccessKey, + payloadHashHex, + contentType, + }); + + const response = await fetch(signed.targetUrl, { + method: "PUT", + headers: { + ...signed.headers, + "content-length": String(bodyBuffer.byteLength), + }, + body: new Uint8Array(bodyBuffer), + cache: "no-store", + signal: AbortSignal.timeout(30000), + }); + + if (!response.ok) { + const bodyPreview = await response.text().catch(() => ""); + return { + ok: false, + status: response.status, + endpoint: url.origin, + bucket: config.bucket.trim(), + key: objectKey, + code: "S3_ARCHIVE_FAILED", + error: `S3 upload failed with HTTP ${response.status}${ + bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : "" + }`, + }; + } + + return { + ok: true, + status: response.status, + endpoint: url.origin, + bucket: config.bucket.trim(), + key: objectKey, + etag: response.headers.get("etag") ?? undefined, + }; + } catch (err: unknown) { + return { + ok: false, + key: objectKey, + code: "S3_ARCHIVE_FAILED", + error: err instanceof Error ? err.message : "S3 upload request failed", + }; + } +} diff --git a/core/adminSettings/settingsManager.ts b/core/adminSettings/settingsManager.ts index e7d5a12..9af6800 100644 --- a/core/adminSettings/settingsManager.ts +++ b/core/adminSettings/settingsManager.ts @@ -1,12 +1,27 @@ +import { + dispatchWebhookAlertAsync, + type WebhookAlertPayload, + type WebhookRuntimeConfig, +} from "../alerts/webhookNotifier"; import { AuthService, resolveDefaultJwtSecret } from "../auth/jwt"; import { DataBufferEngine } from "../data/buffer"; import { DatabaseEngine } from "../Database/database"; import { SDKManager } from "../sdk/sdkManager"; +import { + decryptSecret, + encryptSecret, + getMaskedHintFromEncrypted, +} from "../Security/secretVault"; import { SecurityEngine, getInternalIngestionHeaders, } from "../Security/security"; import { CSVStorageManager } from "../Storage/csvStorage"; +import { + buildS3ObjectKey, + type S3ArchiveConfig, + uploadFileToS3, +} from "../Storage/s3Archive"; export interface AdminSettings { serverPort: number; @@ -19,9 +34,49 @@ export interface AdminSettings { jwtSecret: string; enableIdempotency: boolean; autoPruneDays: number; + compressAfterDays: number; + enableS3Backup: boolean; + s3Endpoint: string; + s3Region: string; + s3Bucket: string; + s3Prefix: string; + s3AccessKeyId: string; + s3SecretAccessKeyEnc: string; + diskPauseThresholdPct: number; + enableWebhookAlerts: boolean; + webhookUrl: string; + webhookSecretEnc: string; + webhookCooldownSec: number; + enablePrometheusMetrics: boolean; + prometheusMetricsTokenEnc: string; updatedAt: string; } +export interface PublicAdminSettings + extends Omit< + AdminSettings, + | "jwtSecret" + | "s3SecretAccessKeyEnc" + | "webhookSecretEnc" + | "prometheusMetricsTokenEnc" + > { + hasS3SecretKey: boolean; + s3SecretKeyMask: string; + hasWebhookSecret: boolean; + webhookSecretMask: string; + hasPrometheusToken: boolean; + prometheusTokenMask: string; +} + +export type AdminSettingsUpdateInput = Partial & { + s3SecretAccessKey?: string; + clearS3SecretKey?: boolean; + webhookSecret?: string; + clearWebhookSecret?: boolean; + prometheusMetricsToken?: string; + clearPrometheusToken?: boolean; +}; + export interface SettingsBridgeSyncStatus { attempted: boolean; ok: boolean; @@ -36,9 +91,11 @@ type SettingsGlobals = typeof globalThis & { __dataStackSettingsHydrated?: boolean; __dataStackLastSettingsBridgeSync?: SettingsBridgeSyncStatus; __dataStackLastPruneMeta?: { days: number; ranAtMs: number }; + __dataStackLastCompressMeta?: { days: number; ranAtMs: number }; }; const PRUNE_COOLDOWN_MS = 60 * 60 * 1000; +const COMPRESS_COOLDOWN_MS = 60 * 60 * 1000; const globals = globalThis as SettingsGlobals; function getDefaultSettings(): AdminSettings { @@ -53,6 +110,21 @@ function getDefaultSettings(): AdminSettings { jwtSecret: resolveDefaultJwtSecret(), enableIdempotency: true, autoPruneDays: 30, + compressAfterDays: 0, + enableS3Backup: false, + s3Endpoint: "", + s3Region: "auto", + s3Bucket: "", + s3Prefix: "datastack/", + s3AccessKeyId: "", + s3SecretAccessKeyEnc: "", + diskPauseThresholdPct: 95, + enableWebhookAlerts: false, + webhookUrl: "", + webhookSecretEnc: "", + webhookCooldownSec: 300, + enablePrometheusMetrics: false, + prometheusMetricsTokenEnc: "", updatedAt: new Date().toISOString(), }; } @@ -64,8 +136,132 @@ function getSettingsState(): AdminSettings { return globals.__dataStackAdminSettings; } +function toPublicSettings(settings: AdminSettings): PublicAdminSettings { + const s3SecretKeyMask = getMaskedHintFromEncrypted( + settings.s3SecretAccessKeyEnc, + ); + const webhookSecretMask = getMaskedHintFromEncrypted( + settings.webhookSecretEnc, + ); + const prometheusTokenMask = getMaskedHintFromEncrypted( + settings.prometheusMetricsTokenEnc, + ); + return { + serverPort: settings.serverPort, + webPort: settings.webPort, + maxEventsPerSec: settings.maxEventsPerSec, + maxPayloadSizeKb: settings.maxPayloadSizeKb, + batchFlushThreshold: settings.batchFlushThreshold, + batchFlushIntervalMs: settings.batchFlushIntervalMs, + fileRotationMb: settings.fileRotationMb, + enableIdempotency: settings.enableIdempotency, + autoPruneDays: settings.autoPruneDays, + compressAfterDays: settings.compressAfterDays, + enableS3Backup: settings.enableS3Backup, + s3Endpoint: settings.s3Endpoint, + s3Region: settings.s3Region, + s3Bucket: settings.s3Bucket, + s3Prefix: settings.s3Prefix, + s3AccessKeyId: settings.s3AccessKeyId, + hasS3SecretKey: Boolean(settings.s3SecretAccessKeyEnc), + s3SecretKeyMask, + diskPauseThresholdPct: settings.diskPauseThresholdPct, + enableWebhookAlerts: settings.enableWebhookAlerts, + webhookUrl: settings.webhookUrl, + hasWebhookSecret: Boolean(settings.webhookSecretEnc), + webhookSecretMask, + webhookCooldownSec: settings.webhookCooldownSec, + enablePrometheusMetrics: settings.enablePrometheusMetrics, + hasPrometheusToken: Boolean(settings.prometheusMetricsTokenEnc), + prometheusTokenMask, + updatedAt: settings.updatedAt, + }; +} + export class SettingsManager { - private static applyRuntimeSettings(settings: AdminSettings, forcePrune = false) { + public static triggerWebhookAlert(payload: WebhookAlertPayload): void { + const current = getSettingsState(); + if (!current.enableWebhookAlerts || !current.webhookUrl.trim()) { + return; + } + void (async () => { + try { + let secret = ""; + if (current.webhookSecretEnc) { + try { + secret = decryptSecret(current.webhookSecretEnc); + } catch { + secret = ""; + } + } + await dispatchWebhookAlertAsync(payload, { + enabled: current.enableWebhookAlerts, + webhookUrl: current.webhookUrl, + webhookSecret: secret, + cooldownSec: current.webhookCooldownSec, + }); + } catch { + // Never allow background webhook dispatch to throw into callers + } + })(); + } + + private static async executeArchiveAndPrune( + settings: AdminSettings, + ): Promise { + if (!settings.enableS3Backup) { + await CSVStorageManager.pruneExpiredPartitionsAsync( + settings.autoPruneDays, + ); + return; + } + + const s3Config: S3ArchiveConfig = { + endpoint: settings.s3Endpoint, + region: settings.s3Region, + bucket: settings.s3Bucket, + prefix: settings.s3Prefix, + accessKeyId: settings.s3AccessKeyId, + secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc), + }; + + await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync( + settings.autoPruneDays, + { + beforeDeleteFile: async (file) => { + const objectKey = buildS3ObjectKey( + s3Config.prefix, + file.clientId, + file.filename, + ); + const result = await uploadFileToS3( + s3Config, + file.filePath, + objectKey, + ); + if (!result.ok) { + const errMsg = + result.error || `Failed to upload ${file.filename} to S3`; + this.triggerWebhookAlert({ + event: "S3_ARCHIVE_FAILED", + severity: "critical", + title: "Off-Site S3 Archive Upload Failed", + message: errMsg, + clientId: file.clientId, + details: { filename: file.filename, bucket: s3Config.bucket }, + }); + throw new Error(errMsg); + } + }, + }, + ); + } + + private static applyRuntimeSettings( + settings: AdminSettings, + forcePrune = false, + forceCompress = false, + ) { SecurityEngine.maxEventsPerSec = settings.maxEventsPerSec; SecurityEngine.enableIdempotency = settings.enableIdempotency; SecurityEngine.maxPayloadSizeBytes = settings.maxPayloadSizeKb * 1024; @@ -76,10 +272,34 @@ export class SettingsManager { } CSVStorageManager.maxFileSizeBytes = settings.fileRotationMb * 1024 * 1024; const hasConfiguredDb = Boolean(process.env.DATABASE_URL?.trim()); - if ( - settings.autoPruneDays > 0 && - (!hasConfiguredDb || globals.__dataStackSettingsHydrated === true) - ) { + const dbReadyForMaintenance = + !hasConfiguredDb || globals.__dataStackSettingsHydrated === true; + + if (settings.compressAfterDays > 0 && dbReadyForMaintenance) { + const prevComp = globals.__dataStackLastCompressMeta; + const now = Date.now(); + const shouldCompress = + forceCompress || + !prevComp || + prevComp.days !== settings.compressAfterDays || + now - prevComp.ranAtMs >= COMPRESS_COOLDOWN_MS; + if (shouldCompress) { + globals.__dataStackLastCompressMeta = { + days: settings.compressAfterDays, + ranAtMs: now, + }; + void CSVStorageManager.compressOldPartitionsAsync( + settings.compressAfterDays, + ).catch((err) => { + console.warn( + "[DataStack Storage] Background partition compression failed:", + err, + ); + }); + } + } + + if (settings.autoPruneDays > 0 && dbReadyForMaintenance) { const prev = globals.__dataStackLastPruneMeta; const now = Date.now(); const shouldPrune = @@ -92,11 +312,9 @@ export class SettingsManager { days: settings.autoPruneDays, ranAtMs: now, }; - void CSVStorageManager.pruneExpiredPartitionsAsync( - settings.autoPruneDays, - ).catch((err) => { + void this.executeArchiveAndPrune(settings).catch((err) => { console.warn( - "[DataStack Storage] Background async partition prune failed:", + "[DataStack Storage] Background async partition prune/archive failed:", err, ); }); @@ -115,7 +333,12 @@ export class SettingsManager { const result = await pool.query( `SELECT server_port, web_port, max_events_per_sec, max_payload_size_kb, batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb, - jwt_secret, enable_idempotency, auto_prune_days, updated_at + jwt_secret, enable_idempotency, auto_prune_days, + compress_after_days, enable_s3_backup, s3_endpoint, s3_region, + s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc, + disk_pause_threshold_pct, enable_webhook_alerts, webhook_url, + webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics, + prometheus_metrics_token_enc, updated_at FROM admin_settings WHERE id = 'default'`, ); @@ -126,6 +349,21 @@ export class SettingsManager { row.auto_prune_days !== undefined && row.auto_prune_days !== null ? Number(row.auto_prune_days) : NaN; + const parsedCompressDays = + row.compress_after_days !== undefined && + row.compress_after_days !== null + ? Number(row.compress_after_days) + : NaN; + const parsedDiskThreshold = + row.disk_pause_threshold_pct !== undefined && + row.disk_pause_threshold_pct !== null + ? Number(row.disk_pause_threshold_pct) + : NaN; + const parsedWebhookCooldown = + row.webhook_cooldown_sec !== undefined && + row.webhook_cooldown_sec !== null + ? Number(row.webhook_cooldown_sec) + : NaN; const envSocketPort = Number(process.env.PORT_SOCKET); const envWebPort = Number(process.env.PORT); const loaded: AdminSettings = { @@ -142,9 +380,34 @@ export class SettingsManager { batchFlushThreshold: Number(row.batch_flush_threshold) || 500, batchFlushIntervalMs: Number(row.batch_flush_interval_ms) || 2000, fileRotationMb: Number(row.file_rotation_mb) || 100, - jwtSecret: String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(), + jwtSecret: + String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(), enableIdempotency: Boolean(row.enable_idempotency), autoPruneDays: Number.isFinite(parsedPruneDays) ? parsedPruneDays : 30, + compressAfterDays: Number.isFinite(parsedCompressDays) + ? Math.max(0, parsedCompressDays) + : 0, + enableS3Backup: Boolean(row.enable_s3_backup), + s3Endpoint: String(row.s3_endpoint ?? ""), + s3Region: String(row.s3_region || "auto"), + s3Bucket: String(row.s3_bucket ?? ""), + s3Prefix: String(row.s3_prefix ?? "datastack/"), + s3AccessKeyId: String(row.s3_access_key_id ?? ""), + s3SecretAccessKeyEnc: String(row.s3_secret_access_key_enc ?? ""), + diskPauseThresholdPct: Number.isFinite(parsedDiskThreshold) + ? parsedDiskThreshold + : 95, + enableWebhookAlerts: Boolean(row.enable_webhook_alerts), + webhookUrl: String(row.webhook_url ?? ""), + webhookSecretEnc: String(row.webhook_secret_enc ?? ""), + webhookCooldownSec: + Number.isFinite(parsedWebhookCooldown) && parsedWebhookCooldown >= 10 + ? Math.floor(parsedWebhookCooldown) + : 300, + enablePrometheusMetrics: Boolean(row.enable_prometheus_metrics), + prometheusMetricsTokenEnc: String( + row.prometheus_metrics_token_enc ?? "", + ), updatedAt: row.updated_at ? new Date(row.updated_at as string | Date).toISOString() : new Date().toISOString(), @@ -198,8 +461,17 @@ export class SettingsManager { `INSERT INTO admin_settings ( id, server_port, web_port, max_events_per_sec, max_payload_size_kb, batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb, - jwt_secret, enable_idempotency, auto_prune_days, updated_at - ) VALUES ('default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, NOW()) + jwt_secret, enable_idempotency, auto_prune_days, + compress_after_days, enable_s3_backup, s3_endpoint, s3_region, + s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc, + disk_pause_threshold_pct, enable_webhook_alerts, webhook_url, + webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics, + prometheus_metrics_token_enc, updated_at + ) VALUES ( + 'default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, + $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, + $22, $23, $24, $25, NOW() + ) ON CONFLICT (id) DO UPDATE SET server_port = EXCLUDED.server_port, web_port = EXCLUDED.web_port, @@ -211,6 +483,21 @@ export class SettingsManager { jwt_secret = EXCLUDED.jwt_secret, enable_idempotency = EXCLUDED.enable_idempotency, auto_prune_days = EXCLUDED.auto_prune_days, + compress_after_days = EXCLUDED.compress_after_days, + enable_s3_backup = EXCLUDED.enable_s3_backup, + s3_endpoint = EXCLUDED.s3_endpoint, + s3_region = EXCLUDED.s3_region, + s3_bucket = EXCLUDED.s3_bucket, + s3_prefix = EXCLUDED.s3_prefix, + s3_access_key_id = EXCLUDED.s3_access_key_id, + s3_secret_access_key_enc = EXCLUDED.s3_secret_access_key_enc, + disk_pause_threshold_pct = EXCLUDED.disk_pause_threshold_pct, + enable_webhook_alerts = EXCLUDED.enable_webhook_alerts, + webhook_url = EXCLUDED.webhook_url, + webhook_secret_enc = EXCLUDED.webhook_secret_enc, + webhook_cooldown_sec = EXCLUDED.webhook_cooldown_sec, + enable_prometheus_metrics = EXCLUDED.enable_prometheus_metrics, + prometheus_metrics_token_enc = EXCLUDED.prometheus_metrics_token_enc, updated_at = NOW()`, [ settings.serverPort, @@ -223,6 +510,21 @@ export class SettingsManager { settings.jwtSecret, settings.enableIdempotency, settings.autoPruneDays, + settings.compressAfterDays, + settings.enableS3Backup, + settings.s3Endpoint, + settings.s3Region, + settings.s3Bucket, + settings.s3Prefix, + settings.s3AccessKeyId, + settings.s3SecretAccessKeyEnc, + settings.diskPauseThresholdPct, + settings.enableWebhookAlerts, + settings.webhookUrl, + settings.webhookSecretEnc, + settings.webhookCooldownSec, + settings.enablePrometheusMetrics, + settings.prometheusMetricsTokenEnc, ], ); } catch (err: unknown) { @@ -250,37 +552,49 @@ export class SettingsManager { return { ...current }; } - public static getPublicSettings(): Omit { - const settings = this.getSettings(); + public static getPublicSettings(): PublicAdminSettings { + return toPublicSettings(this.getSettings()); + } + + public static async getPublicSettingsAsync(): Promise { + const settings = await this.getSettingsAsync(); + return toPublicSettings(settings); + } + + public static async getDecryptedS3ConfigAsync(): Promise { + const settings = await this.getSettingsAsync(); return { - serverPort: settings.serverPort, - webPort: settings.webPort, - maxEventsPerSec: settings.maxEventsPerSec, - maxPayloadSizeKb: settings.maxPayloadSizeKb, - batchFlushThreshold: settings.batchFlushThreshold, - batchFlushIntervalMs: settings.batchFlushIntervalMs, - fileRotationMb: settings.fileRotationMb, - enableIdempotency: settings.enableIdempotency, - autoPruneDays: settings.autoPruneDays, - updatedAt: settings.updatedAt, + endpoint: settings.s3Endpoint, + region: settings.s3Region, + bucket: settings.s3Bucket, + prefix: settings.s3Prefix, + accessKeyId: settings.s3AccessKeyId, + secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc), }; } - public static async getPublicSettingsAsync(): Promise< - Omit - > { + public static async getDecryptedWebhookConfigAsync(): Promise { const settings = await this.getSettingsAsync(); return { - serverPort: settings.serverPort, - webPort: settings.webPort, - maxEventsPerSec: settings.maxEventsPerSec, - maxPayloadSizeKb: settings.maxPayloadSizeKb, - batchFlushThreshold: settings.batchFlushThreshold, - batchFlushIntervalMs: settings.batchFlushIntervalMs, - fileRotationMb: settings.fileRotationMb, - enableIdempotency: settings.enableIdempotency, - autoPruneDays: settings.autoPruneDays, - updatedAt: settings.updatedAt, + enabled: settings.enableWebhookAlerts, + webhookUrl: settings.webhookUrl, + webhookSecret: settings.webhookSecretEnc + ? decryptSecret(settings.webhookSecretEnc) + : "", + cooldownSec: settings.webhookCooldownSec, + }; + } + + public static async getDecryptedPrometheusConfigAsync(): Promise<{ + enabled: boolean; + token: string; + }> { + const settings = await this.getSettingsAsync(); + return { + enabled: settings.enablePrometheusMetrics, + token: settings.prometheusMetricsTokenEnc + ? decryptSecret(settings.prometheusMetricsTokenEnc) + : "", }; } @@ -326,6 +640,12 @@ export class SettingsManager { fileRotationMb: settings.fileRotationMb, enableIdempotency: settings.enableIdempotency, autoPruneDays: settings.autoPruneDays, + compressAfterDays: settings.compressAfterDays, + enableS3Backup: settings.enableS3Backup, + diskPauseThresholdPct: settings.diskPauseThresholdPct, + enableWebhookAlerts: settings.enableWebhookAlerts, + webhookUrl: settings.webhookUrl, + webhookCooldownSec: settings.webhookCooldownSec, ...(jwtSecretChanged ? { jwtSecret: settings.jwtSecret } : {}), }), cache: "no-store", @@ -355,7 +675,10 @@ export class SettingsManager { globals.__dataStackLastSettingsBridgeSync = result; return result; } catch (err: unknown) { - const message = err instanceof Error ? err.message : "Port 4000 settings-sync unreachable"; + const message = + err instanceof Error + ? err.message + : "Port 4000 settings-sync unreachable"; if ( process.env.NODE_ENV === "production" || Boolean(process.env.INGESTION_INTERNAL_URL) @@ -374,18 +697,79 @@ export class SettingsManager { } } - public static updateSettings(updates: Partial): AdminSettings { + private static mergeSettingsUpdate( + current: AdminSettings, + updates: AdminSettingsUpdateInput, + ): AdminSettings { + const { + s3SecretAccessKey, + clearS3SecretKey, + webhookSecret, + clearWebhookSecret, + prometheusMetricsToken, + clearPrometheusToken, + ...rest + } = updates; + let nextEncryptedSecret = current.s3SecretAccessKeyEnc; + if (clearS3SecretKey) { + nextEncryptedSecret = ""; + } else if ( + typeof s3SecretAccessKey === "string" && + s3SecretAccessKey.trim().length > 0 + ) { + nextEncryptedSecret = encryptSecret(s3SecretAccessKey.trim()); + } else if (rest.s3SecretAccessKeyEnc !== undefined) { + nextEncryptedSecret = rest.s3SecretAccessKeyEnc; + } + + let nextWebhookSecretEnc = current.webhookSecretEnc; + if (clearWebhookSecret) { + nextWebhookSecretEnc = ""; + } else if ( + typeof webhookSecret === "string" && + webhookSecret.trim().length > 0 + ) { + nextWebhookSecretEnc = encryptSecret(webhookSecret.trim()); + } else if (rest.webhookSecretEnc !== undefined) { + nextWebhookSecretEnc = rest.webhookSecretEnc; + } + + let nextPrometheusTokenEnc = current.prometheusMetricsTokenEnc; + if (clearPrometheusToken) { + nextPrometheusTokenEnc = ""; + } else if ( + typeof prometheusMetricsToken === "string" && + prometheusMetricsToken.trim().length > 0 + ) { + nextPrometheusTokenEnc = encryptSecret(prometheusMetricsToken.trim()); + } else if (rest.prometheusMetricsTokenEnc !== undefined) { + nextPrometheusTokenEnc = rest.prometheusMetricsTokenEnc; + } + + return { + ...current, + ...rest, + s3SecretAccessKeyEnc: nextEncryptedSecret, + webhookSecretEnc: nextWebhookSecretEnc, + prometheusMetricsTokenEnc: nextPrometheusTokenEnc, + updatedAt: new Date().toISOString(), + }; + } + + public static updateSettings( + updates: AdminSettingsUpdateInput, + ): AdminSettings { const current = getSettingsState(); const previousJwtSecret = current.jwtSecret; const jwtSecretChanged = Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret; - const nextSettings: AdminSettings = { - ...current, - ...updates, - updatedAt: new Date().toISOString(), - }; + const nextSettings = this.mergeSettingsUpdate(current, updates); globals.__dataStackAdminSettings = nextSettings; - this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); + this.applyRuntimeSettings( + nextSettings, + updates.autoPruneDays !== undefined, + updates.compressAfterDays !== undefined, + ); if (jwtSecretChanged) { SDKManager.resignAllClientTokens(); } @@ -403,17 +787,13 @@ export class SettingsManager { } public static async updateSettingsAsync( - updates: Partial, + updates: AdminSettingsUpdateInput, ): Promise { const current = await this.getSettingsAsync(); const previousJwtSecret = current.jwtSecret; const jwtSecretChanged = Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret; - const nextSettings: AdminSettings = { - ...current, - ...updates, - updatedAt: new Date().toISOString(), - }; + const nextSettings = this.mergeSettingsUpdate(current, updates); if ( jwtSecretChanged && @@ -450,11 +830,15 @@ export class SettingsManager { try { await this.persistSettingsToDb(nextSettings); globals.__dataStackAdminSettings = nextSettings; - this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); + this.applyRuntimeSettings( + nextSettings, + updates.autoPruneDays !== undefined, + updates.compressAfterDays !== undefined, + ); await SDKManager.resignAllClientTokensAsync(); } catch (err) { globals.__dataStackAdminSettings = current; - this.applyRuntimeSettings(current, false); + this.applyRuntimeSettings(current, false, false); await this.persistSettingsToDb(current).catch(() => undefined); if (preSyncStatus.synced) { await this.notifyIngestionSettingsSync( @@ -482,7 +866,11 @@ export class SettingsManager { await this.persistSettingsToDb(nextSettings); globals.__dataStackAdminSettings = nextSettings; - this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); + this.applyRuntimeSettings( + nextSettings, + updates.autoPruneDays !== undefined, + updates.compressAfterDays !== undefined, + ); await this.notifyIngestionSettingsSync( nextSettings, previousJwtSecret, diff --git a/core/alerts/webhookNotifier.ts b/core/alerts/webhookNotifier.ts new file mode 100644 index 0000000..e3d4fbc --- /dev/null +++ b/core/alerts/webhookNotifier.ts @@ -0,0 +1,183 @@ +import crypto from "node:crypto"; + +export type WebhookAlertEventType = + | "DISK_SPACE_LOW" + | "STORAGE_QUOTA_EXCEEDED" + | "S3_ARCHIVE_FAILED" + | "CLIENT_IP_FORBIDDEN" + | "CLIENT_TOKEN_EXPIRED" + | "ADMIN_LOGIN_LOCKOUT" + | "TEST_ALERT"; + +export interface WebhookAlertPayload { + event: WebhookAlertEventType; + severity: "info" | "warning" | "critical"; + title: string; + message: string; + clientId?: string; + details?: Record; + timestamp?: string; +} + +export interface WebhookRuntimeConfig { + enabled: boolean; + webhookUrl: string; + webhookSecret: string; + cooldownSec: number; +} + +export interface WebhookDispatchResult { + ok: boolean; + skipped?: boolean; + reason?: string; + status?: number; + error?: string; +} + +type WebhookGlobals = typeof globalThis & { + __dataStackWebhookCooldownMap?: Map; +}; + +const globals = globalThis as WebhookGlobals; +const DEFAULT_WEBHOOK_TIMEOUT_MS = 5000; + +function getCooldownMap(): Map { + if (!globals.__dataStackWebhookCooldownMap) { + globals.__dataStackWebhookCooldownMap = new Map(); + } + return globals.__dataStackWebhookCooldownMap; +} + +export function clearWebhookCooldownCacheForTests(): void { + getCooldownMap().clear(); +} + +export function buildWebhookSignature( + rawBody: string, + secret: string, +): string { + const hmac = crypto + .createHmac("sha256", secret) + .update(rawBody, "utf8") + .digest("hex"); + return `sha256=${hmac}`; +} + +export function formatOutboundWebhookBody(payload: WebhookAlertPayload): string { + const timestamp = payload.timestamp || new Date().toISOString(); + const prefix = + payload.severity === "critical" + ? "[CRITICAL]" + : payload.severity === "warning" + ? "[WARNING]" + : "[INFO]"; + const clientTag = payload.clientId ? ` (client: ${payload.clientId})` : ""; + const summaryLine = `${prefix} DataStack ${payload.event}${clientTag} — ${payload.title}: ${payload.message}`; + + return JSON.stringify({ + instance: "DataStack", + event: payload.event, + severity: payload.severity, + title: payload.title, + message: payload.message, + ...(payload.clientId ? { clientId: payload.clientId } : {}), + ...(payload.details ? { details: payload.details } : {}), + timestamp, + // Slack & Discord compatible top-level fields + text: summaryLine, + content: summaryLine, + }); +} + +export async function dispatchWebhookAlertAsync( + payload: WebhookAlertPayload, + config: WebhookRuntimeConfig, + options?: { bypassCooldown?: boolean; timeoutMs?: number }, +): Promise { + const webhookUrl = String(config.webhookUrl || "").trim(); + if (!options?.bypassCooldown && !config.enabled) { + return { ok: false, skipped: true, reason: "WEBHOOK_DISABLED" }; + } + if (!webhookUrl) { + return { + ok: false, + skipped: !options?.bypassCooldown, + error: "Webhook URL is not configured.", + }; + } + + let parsedUrl: URL; + try { + parsedUrl = new URL(webhookUrl); + if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") { + return { + ok: false, + error: "Webhook URL must use http:// or https://.", + }; + } + } catch { + return { + ok: false, + error: "Webhook URL is not a valid URL.", + }; + } + + const dedupeKey = `${payload.event}:${payload.clientId || "global"}`; + const cooldownMs = Math.max(0, (Number(config.cooldownSec) || 300) * 1000); + const now = Date.now(); + const cooldownMap = getCooldownMap(); + + if (!options?.bypassCooldown && cooldownMs > 0) { + const lastSentAt = cooldownMap.get(dedupeKey); + if (lastSentAt !== undefined && now - lastSentAt < cooldownMs) { + return { ok: true, skipped: true, reason: "COOLDOWN_ACTIVE" }; + } + cooldownMap.set(dedupeKey, now); + } + + const timestamp = payload.timestamp || new Date(now).toISOString(); + const bodyStr = formatOutboundWebhookBody({ ...payload, timestamp }); + const headers: Record = { + "Content-Type": "application/json", + "User-Agent": "DataStack-Webhook-Notifier/2.0", + "X-DataStack-Event": payload.event, + "X-DataStack-Timestamp": timestamp, + }; + + const secret = String(config.webhookSecret || "").trim(); + if (secret) { + headers["X-DataStack-Signature"] = buildWebhookSignature(bodyStr, secret); + } + + const timeoutMs = Math.max(500, options?.timeoutMs || DEFAULT_WEBHOOK_TIMEOUT_MS); + try { + const res = await fetch(parsedUrl.toString(), { + method: "POST", + headers, + body: bodyStr, + cache: "no-store", + signal: AbortSignal.timeout(timeoutMs), + }); + + if (!res.ok) { + if (!options?.bypassCooldown) { + // Keep cooldown entry so failing webhooks don't DDOS a broken upstream endpoint + } + const errText = await res.text().catch(() => ""); + return { + ok: false, + status: res.status, + error: `Webhook endpoint returned HTTP ${res.status}${errText ? `: ${errText.slice(0, 160)}` : ""}`, + }; + } + + return { ok: true, status: res.status }; + } catch (err: unknown) { + const msg = + err instanceof Error ? err.message : "Webhook delivery failed"; + return { + ok: false, + error: `Webhook delivery failed: ${msg}`, + }; + } +} diff --git a/core/apiEngine/apiService.ts b/core/apiEngine/apiService.ts index 189e0e0..8c93aaf 100644 --- a/core/apiEngine/apiService.ts +++ b/core/apiEngine/apiService.ts @@ -13,11 +13,13 @@ export class ApiEngine { */ public static async getDashboardMetrics() { const settings = await SettingsManager.getSettingsAsync(); - const [storageStats, localBufferStats, clients] = await Promise.all([ - CSVStorageManager.getStorageStatsAsync(), - DataBufferEngine.getBufferStatsAsync(), - SDKManager.listClientsAsync(), - ]); + const [storageStats, diskUsage, localBufferStats, clients] = + await Promise.all([ + CSVStorageManager.getStorageStatsAsync(), + CSVStorageManager.getDiskUsageAsync(), + DataBufferEngine.getBufferStatsAsync(), + SDKManager.listClientsAsync(), + ]); const activeClients = clients.filter((c) => c.status === 'active').length; let activeSockets = 0; let ingestionLocalBufferSize = 0; @@ -129,6 +131,7 @@ export class ApiEngine { activeClients, bufferStats, storageStats, + diskUsage, securityStats, runtime: { serverPort: liveServerPort, diff --git a/core/sdk/sdkManager.ts b/core/sdk/sdkManager.ts index b644127..7d4f9f1 100644 --- a/core/sdk/sdkManager.ts +++ b/core/sdk/sdkManager.ts @@ -23,6 +23,9 @@ export interface ClientSDK { secret: string; token: string; rateLimit: number; // max events per sec + storageQuotaMb: number; // max storage in MB per client (0 = unlimited) + allowedIps: string; // comma-separated IPv4/IPv6/CIDR list ("" = allow all) + expiresAt?: string | null; // ISO timestamp or null/undefined if never expires status: "active" | "revoked"; totalEvents: number; lastActive?: string; @@ -103,6 +106,9 @@ interface RemoteInMemoryClient { status?: string; token?: string; rateLimit?: number; + storageQuotaMb?: number; + allowedIps?: string; + expiresAt?: string | null; } function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): number { @@ -119,6 +125,9 @@ function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): numbe secret: "", token: String(item.token || ""), rateLimit: Math.max(1, Number(item.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)), + allowedIps: typeof item.allowedIps === "string" ? item.allowedIps : "", + expiresAt: item.expiresAt ? String(item.expiresAt) : null, status: item.status === "revoked" ? "revoked" : "active", totalEvents: 0, createdAt: new Date().toISOString(), @@ -135,6 +144,9 @@ function serializeSyncClient(client: ClientSDK) { status: client.status, token: client.token, rateLimit: client.rateLimit, + storageQuotaMb: client.storageQuotaMb, + allowedIps: client.allowedIps || "", + expiresAt: client.expiresAt ?? null, }; } @@ -224,12 +236,26 @@ function notifyIngestionClientSync( } function mapRowToClient(row: Record): ClientSDK { + const parsedQuota = Number(row.storage_quota_mb); + let expiresAt: string | null = null; + if (row.expires_at) { + const d = new Date(row.expires_at as string | Date); + if (!Number.isNaN(d.getTime())) { + expiresAt = d.toISOString(); + } + } return { clientId: String(row.client_id), name: String(row.name), secret: String(row.secret), token: String(row.token), rateLimit: Number(row.rate_limit) || 100, + storageQuotaMb: + Number.isFinite(parsedQuota) && parsedQuota >= 0 + ? Math.floor(parsedQuota) + : 0, + allowedIps: typeof row.allowed_ips === "string" ? row.allowed_ips : "", + expiresAt, status: row.status === "revoked" ? "revoked" : "active", totalEvents: Number(row.total_events) || 0, lastActive: row.last_active @@ -256,7 +282,7 @@ export class SDKManager { const [result, deletedResult] = await Promise.all([ pool.query( - `SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at + `SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at FROM client_sdks ORDER BY created_at DESC`, ), @@ -464,7 +490,7 @@ export class SDKManager { await this.flushPendingClientActivityAsync(); } const result = await pool.query( - `SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at + `SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at FROM client_sdks WHERE client_id = $1`, [normalizedId], @@ -500,10 +526,19 @@ export class SDKManager { clientId: string; name: string; rateLimit?: number; + storageQuotaMb?: number; + allowedIps?: string; + expiresAt?: string | null; }): ClientSDK { const safeClientId = this.normalizeClientId(data.clientId); const secret = this.generateClientSecret(safeClientId); const rateLimit = Math.max(1, Number(data.rateLimit) || 100); + const storageQuotaMb = Math.max( + 0, + Math.floor(Number(data.storageQuotaMb) || 0), + ); + const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : ""; + const expiresAt = data.expiresAt ? String(data.expiresAt) : null; const token = AuthService.generateClientToken({ clientId: safeClientId, @@ -517,6 +552,9 @@ export class SDKManager { secret, token, rateLimit, + storageQuotaMb, + allowedIps, + expiresAt, status: "active", totalEvents: 0, createdAt: new Date().toISOString(), @@ -548,13 +586,16 @@ export class SDKManager { [client.clientId], ); await pool.query( - `INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, status, total_events, created_at, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW()) + `INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW()) ON CONFLICT (client_id) DO UPDATE SET name = EXCLUDED.name, secret = EXCLUDED.secret, token = EXCLUDED.token, rate_limit = EXCLUDED.rate_limit, + storage_quota_mb = EXCLUDED.storage_quota_mb, + allowed_ips = EXCLUDED.allowed_ips, + expires_at = EXCLUDED.expires_at, status = EXCLUDED.status, updated_at = NOW()`, [ @@ -563,6 +604,9 @@ export class SDKManager { client.secret, client.token, client.rateLimit, + client.storageQuotaMb, + client.allowedIps || "", + client.expiresAt ?? null, client.status, client.totalEvents, client.createdAt, @@ -578,11 +622,20 @@ export class SDKManager { clientId: string; name: string; rateLimit?: number; + storageQuotaMb?: number; + allowedIps?: string; + expiresAt?: string | null; }): Promise { await ensureDbSettingsHydrated(); const safeClientId = this.normalizeClientId(data.clientId); const secret = this.generateClientSecret(safeClientId); const rateLimit = Math.max(1, Number(data.rateLimit) || 100); + const storageQuotaMb = Math.max( + 0, + Math.floor(Number(data.storageQuotaMb) || 0), + ); + const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : ""; + const expiresAt = data.expiresAt ? String(data.expiresAt) : null; const token = AuthService.generateClientToken({ clientId: safeClientId, @@ -596,6 +649,9 @@ export class SDKManager { secret, token, rateLimit, + storageQuotaMb, + allowedIps, + expiresAt, status: "active", totalEvents: 0, createdAt: new Date().toISOString(), @@ -683,7 +739,17 @@ export class SDKManager { public static updateClient( clientId: string, - updates: Partial>, + updates: Partial< + Pick< + ClientSDK, + | "name" + | "rateLimit" + | "storageQuotaMb" + | "allowedIps" + | "expiresAt" + | "status" + > + >, ): ClientSDK | null { const normalizedId = this.normalizeClientId(clientId); const client = getClientsMap().get(normalizedId); @@ -695,6 +761,20 @@ export class SDKManager { ...(updates.rateLimit !== undefined ? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) } : {}), + ...(updates.storageQuotaMb !== undefined + ? { + storageQuotaMb: Math.max( + 0, + Math.floor(Number(updates.storageQuotaMb) || 0), + ), + } + : {}), + ...(updates.allowedIps !== undefined + ? { allowedIps: String(updates.allowedIps).trim() } + : {}), + ...(updates.expiresAt !== undefined + ? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null } + : {}), ...(updates.status !== undefined ? { status: updates.status } : {}), }; @@ -722,9 +802,17 @@ export class SDKManager { try { await pool.query( `UPDATE client_sdks - SET name = $2, rate_limit = $3, status = $4, updated_at = NOW() + SET name = $2, rate_limit = $3, storage_quota_mb = $4, allowed_ips = $5, expires_at = $6, status = $7, updated_at = NOW() WHERE client_id = $1`, - [client.clientId, client.name, client.rateLimit, client.status], + [ + client.clientId, + client.name, + client.rateLimit, + client.storageQuotaMb, + client.allowedIps || "", + client.expiresAt ?? null, + client.status, + ], ); } catch (err: unknown) { const detail = err instanceof Error ? err.message : "Unknown DB error"; @@ -734,7 +822,17 @@ export class SDKManager { public static async updateClientAsync( clientId: string, - updates: Partial>, + updates: Partial< + Pick< + ClientSDK, + | "name" + | "rateLimit" + | "storageQuotaMb" + | "allowedIps" + | "expiresAt" + | "status" + > + >, ): Promise { await ensureDbSettingsHydrated(); const normalizedId = this.normalizeClientId(clientId); @@ -747,6 +845,20 @@ export class SDKManager { ...(updates.rateLimit !== undefined ? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) } : {}), + ...(updates.storageQuotaMb !== undefined + ? { + storageQuotaMb: Math.max( + 0, + Math.floor(Number(updates.storageQuotaMb) || 0), + ), + } + : {}), + ...(updates.allowedIps !== undefined + ? { allowedIps: String(updates.allowedIps).trim() } + : {}), + ...(updates.expiresAt !== undefined + ? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null } + : {}), ...(updates.status !== undefined ? { status: updates.status } : {}), }; @@ -853,7 +965,7 @@ export class SDKManager { const dirStat = lstatSync(targetDir); if (!dirStat.isSymbolicLink() && dirStat.isDirectory()) { for (const entry of readdirSync(targetDir)) { - if (!entry.endsWith(".csv")) continue; + if (!CSVStorageManager.isPartitionFilename(entry)) continue; const entryPath = resolve(targetDir, entry); try { const entryStat = lstatSync(entryPath); @@ -871,6 +983,7 @@ export class SDKManager { rmSync(targetDir, { recursive: true, force: true }); CSVStorageManager.clearRowCountCache(); + CSVStorageManager.clearQuotaCaches(safeId); return purgedFiles; } diff --git a/lib/vconUiHelpers.ts b/lib/vconUiHelpers.ts index 320f8d5..d642cb7 100644 --- a/lib/vconUiHelpers.ts +++ b/lib/vconUiHelpers.ts @@ -6,8 +6,41 @@ export interface EditableSettings { fileRotationMb: number; enableIdempotency: boolean; autoPruneDays: number; + compressAfterDays: number; + diskPauseThresholdPct: number; + webhookCooldownSec: number; } +export interface S3SettingsFields { + enableS3Backup: boolean; + s3Endpoint: string; + s3Region: string; + s3Bucket: string; + s3Prefix: string; + s3AccessKeyId: string; +} + +export interface AlertingAndMetricsSettingsFields { + enableWebhookAlerts: boolean; + webhookUrl: string; + enablePrometheusMetrics: boolean; +} + +export const DEFAULT_S3_SETTINGS: S3SettingsFields = { + enableS3Backup: false, + s3Endpoint: '', + s3Region: 'auto', + s3Bucket: '', + s3Prefix: 'datastack/', + s3AccessKeyId: '', +}; + +export const DEFAULT_ALERTING_SETTINGS: AlertingAndMetricsSettingsFields = { + enableWebhookAlerts: false, + webhookUrl: '', + enablePrometheusMetrics: false, +}; + export const SETTINGS_LIMITS = { maxEventsPerSec: { min: 1, max: 100000, defaultValue: 100 }, maxPayloadSizeKb: { min: 1, max: 10240, defaultValue: 100 }, @@ -15,6 +48,9 @@ export const SETTINGS_LIMITS = { batchFlushIntervalMs: { min: 100, max: 60000, defaultValue: 2000 }, fileRotationMb: { min: 1, max: 2048, defaultValue: 100 }, autoPruneDays: { min: 0, max: 3650, defaultValue: 30 }, + compressAfterDays: { min: 0, max: 3650, defaultValue: 0 }, + diskPauseThresholdPct: { min: 0, max: 99, defaultValue: 95 }, + webhookCooldownSec: { min: 10, max: 86400, defaultValue: 300 }, } as const; export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = { @@ -25,6 +61,9 @@ export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = { fileRotationMb: SETTINGS_LIMITS.fileRotationMb.defaultValue, enableIdempotency: true, autoPruneDays: SETTINGS_LIMITS.autoPruneDays.defaultValue, + compressAfterDays: SETTINGS_LIMITS.compressAfterDays.defaultValue, + diskPauseThresholdPct: SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue, + webhookCooldownSec: SETTINGS_LIMITS.webhookCooldownSec.defaultValue, }; function clampInt(raw: unknown, min: number, max: number, fallback: number): number { @@ -34,6 +73,57 @@ function clampInt(raw: unknown, min: number, max: number, fallback: number): num return Math.max(min, Math.min(max, rounded)); } +function normalizeDiskPauseThresholdPct(raw: unknown): number { + const clamped = clampInt( + raw, + SETTINGS_LIMITS.diskPauseThresholdPct.min, + SETTINGS_LIMITS.diskPauseThresholdPct.max, + SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue, + ); + if (clamped !== 0 && clamped < 50) { + return 50; + } + return clamped; +} + +export function normalizeS3Settings( + input: Partial | null | undefined, +): S3SettingsFields { + return { + enableS3Backup: + typeof input?.enableS3Backup === 'boolean' + ? input.enableS3Backup + : DEFAULT_S3_SETTINGS.enableS3Backup, + s3Endpoint: typeof input?.s3Endpoint === 'string' ? input.s3Endpoint.trim() : '', + s3Region: + typeof input?.s3Region === 'string' && input.s3Region.trim() + ? input.s3Region.trim() + : DEFAULT_S3_SETTINGS.s3Region, + s3Bucket: typeof input?.s3Bucket === 'string' ? input.s3Bucket.trim() : '', + s3Prefix: + typeof input?.s3Prefix === 'string' ? input.s3Prefix.trim() : DEFAULT_S3_SETTINGS.s3Prefix, + s3AccessKeyId: + typeof input?.s3AccessKeyId === 'string' ? input.s3AccessKeyId.trim() : '', + }; +} + +export function normalizeAlertingSettings( + input: Partial | null | undefined, +): AlertingAndMetricsSettingsFields { + return { + enableWebhookAlerts: + typeof input?.enableWebhookAlerts === 'boolean' + ? input.enableWebhookAlerts + : DEFAULT_ALERTING_SETTINGS.enableWebhookAlerts, + webhookUrl: + typeof input?.webhookUrl === 'string' ? input.webhookUrl.trim() : '', + enablePrometheusMetrics: + typeof input?.enablePrometheusMetrics === 'boolean' + ? input.enablePrometheusMetrics + : DEFAULT_ALERTING_SETTINGS.enablePrometheusMetrics, + }; +} + export function normalizeEditableSettings( input: Partial, ): EditableSettings { @@ -78,6 +168,21 @@ export function normalizeEditableSettings( SETTINGS_LIMITS.autoPruneDays.max, SETTINGS_LIMITS.autoPruneDays.defaultValue, ), + compressAfterDays: clampInt( + input.compressAfterDays, + SETTINGS_LIMITS.compressAfterDays.min, + SETTINGS_LIMITS.compressAfterDays.max, + SETTINGS_LIMITS.compressAfterDays.defaultValue, + ), + diskPauseThresholdPct: normalizeDiskPauseThresholdPct( + input.diskPauseThresholdPct, + ), + webhookCooldownSec: clampInt( + input.webhookCooldownSec, + SETTINGS_LIMITS.webhookCooldownSec.min, + SETTINGS_LIMITS.webhookCooldownSec.max, + SETTINGS_LIMITS.webhookCooldownSec.defaultValue, + ), }; } @@ -398,6 +503,9 @@ export const NUMERIC_SETTING_KEYS: readonly NumericSettingKey[] = [ 'batchFlushIntervalMs', 'fileRotationMb', 'autoPruneDays', + 'compressAfterDays', + 'diskPauseThresholdPct', + 'webhookCooldownSec', ] as const; export function settingsToNumericDrafts( @@ -410,6 +518,9 @@ export function settingsToNumericDrafts( batchFlushIntervalMs: String(settings.batchFlushIntervalMs), fileRotationMb: String(settings.fileRotationMb), autoPruneDays: String(settings.autoPruneDays), + compressAfterDays: String(settings.compressAfterDays), + diskPauseThresholdPct: String(settings.diskPauseThresholdPct), + webhookCooldownSec: String(settings.webhookCooldownSec), }; } @@ -429,6 +540,15 @@ export function validateSingleNumericSetting( if (!Number.isSafeInteger(parsed)) { return { valid: false, error: 'Enter a valid integer.' }; } + if (key === 'diskPauseThresholdPct') { + if (parsed !== 0 && (parsed < 50 || parsed > max)) { + return { + valid: false, + error: 'Must be 0 (disabled) or between 50 and 99.', + }; + } + return { valid: true, value: parsed }; + } if (parsed < min || parsed > max) { return { valid: false, @@ -475,6 +595,9 @@ export function validateEditableSettingsDraft( batchFlushIntervalMs: parsedValues.batchFlushIntervalMs!, fileRotationMb: parsedValues.fileRotationMb!, autoPruneDays: parsedValues.autoPruneDays!, + compressAfterDays: parsedValues.compressAfterDays!, + diskPauseThresholdPct: parsedValues.diskPauseThresholdPct!, + webhookCooldownSec: parsedValues.webhookCooldownSec!, enableIdempotency: Boolean(enableIdempotency), }, }; diff --git a/server/index.js b/server/index.js index b1e2512..8b65293 100644 --- a/server/index.js +++ b/server/index.js @@ -199,6 +199,9 @@ const runtimeConfig = { fileRotationMb: 100, enableIdempotency: true, autoPruneDays: 30, + compressAfterDays: 0, + enableS3Backup: false, + diskPauseThresholdPct: 95, }; let previousJwtSecret = ''; @@ -287,17 +290,113 @@ const inMemoryClientActivity = new Map(); let hasSyncedInMemoryClients = false; const CLIENT_CACHE_TTL_MS = 2000; -async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { +function stripIpv6MappedPrefix(rawIp) { + let ip = String(rawIp || '').trim(); + if (!ip) return ''; + if (ip.startsWith('[') && ip.includes(']')) { + ip = ip.slice(1, ip.indexOf(']')); + } else if (/^\d{1,3}(?:\.\d{1,3}){3}:\d+$/.test(ip)) { + ip = ip.slice(0, ip.lastIndexOf(':')); + } + const pctIndex = ip.indexOf('%'); + if (pctIndex !== -1) { + ip = ip.slice(0, pctIndex); + } + if (/^::ffff:\d{1,3}(?:\.\d{1,3}){3}$/i.test(ip)) { + return ip.slice(7); + } + return ip.toLowerCase(); +} + +function ipv4ToUint32(ip) { + const parts = ip.split('.'); + if (parts.length !== 4) return null; + let acc = 0; + for (const part of parts) { + if (!/^\d{1,3}$/.test(part)) return null; + if (part.length > 1 && part.startsWith('0')) return null; + const num = Number(part); + if (num < 0 || num > 255) return null; + acc = ((acc << 8) | num) >>> 0; + } + return acc >>> 0; +} + +function matchesIpv4Cidr(clientIp, cidrPattern) { + const [networkStr, prefixStr] = cidrPattern.split('/'); + if (!networkStr || prefixStr === undefined) return false; + const prefix = Number(prefixStr); + if (!Number.isInteger(prefix) || prefix < 0 || prefix > 32) return false; + const clientNum = ipv4ToUint32(clientIp); + const networkNum = ipv4ToUint32(networkStr); + if (clientNum === null || networkNum === null) return false; + if (prefix === 0) return true; + const mask = (0xffffffff << (32 - prefix)) >>> 0; + return (clientNum & mask) >>> 0 === (networkNum & mask) >>> 0; +} + +function isClientIpAllowed(clientIp, rawAllowedIps) { + if (!rawAllowedIps || !String(rawAllowedIps).trim()) return true; + const entries = String(rawAllowedIps) + .split(/[\s,]+/) + .map((entry) => entry.trim()) + .filter(Boolean); + if (entries.length === 0) return true; + if (entries.includes('*') || entries.includes('0.0.0.0/0') || entries.includes('::/0')) { + return true; + } + const normalizedClientIp = stripIpv6MappedPrefix(clientIp); + if (!normalizedClientIp || normalizedClientIp === 'unknown') { + return false; + } + for (const entry of entries) { + const normalizedEntry = stripIpv6MappedPrefix(entry); + if (normalizedEntry.includes('/')) { + if (matchesIpv4Cidr(normalizedClientIp, normalizedEntry)) { + return true; + } + const [netAddr, prefixStr] = normalizedEntry.split('/'); + if (Number(prefixStr) === 128 && normalizedClientIp === stripIpv6MappedPrefix(netAddr)) { + return true; + } + } else { + if (normalizedClientIp === normalizedEntry) { + return true; + } + if ( + (normalizedClientIp === '127.0.0.1' && normalizedEntry === '::1') || + (normalizedClientIp === '::1' && normalizedEntry === '127.0.0.1') + ) { + return true; + } + } + } + return false; +} + +function isClientTokenExpired(expiresAt, nowMs = Date.now()) { + if (!expiresAt) return false; + const ts = + expiresAt instanceof Date + ? expiresAt.getTime() + : new Date(String(expiresAt)).getTime(); + if (Number.isNaN(ts)) return false; + return ts <= nowMs; +} + +async function verifyClientInDb(clientId, presentedToken, bypassCache = false, clientIp = '') { const normalizedId = normalizeClientId(clientId); if (!normalizedId) { return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Missing client identifier', }; } if (deletedClientsSet.has(normalizedId)) { return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been deleted or revoked by admin', }; } @@ -313,6 +412,7 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { deletedClientsSet.add(normalizedId); return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been deleted by admin', }; } @@ -326,6 +426,9 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { status: parsed.status === 'revoked' ? 'revoked' : 'active', token: String(parsed.token || ''), rateLimit: Math.max(1, Number(parsed.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(parsed.storageQuotaMb) || 0)), + allowedIps: typeof parsed.allowedIps === 'string' ? parsed.allowedIps : '', + expiresAt: parsed.expiresAt ? String(parsed.expiresAt) : null, }; inMemoryClientRegistry.set(normalizedId, memClient); hasSyncedInMemoryClients = true; @@ -341,20 +444,41 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { if (memClient.status === 'revoked') { return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been revoked by admin', }; } if (memClient.token && presentedToken && memClient.token !== presentedToken) { return { allowed: false, + code: 'TOKEN_ROTATED', reason: 'TOKEN_ROTATED: Client credentials have been rotated', }; } - return { allowed: true, rateLimit: memClient.rateLimit || 100 }; + if (isClientTokenExpired(memClient.expiresAt)) { + return { + allowed: false, + code: 'CLIENT_TOKEN_EXPIRED', + reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`, + }; + } + if (memClient.allowedIps && memClient.allowedIps.trim() && !isClientIpAllowed(clientIp, memClient.allowedIps)) { + return { + allowed: false, + code: 'CLIENT_IP_FORBIDDEN', + reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`, + }; + } + return { + allowed: true, + rateLimit: memClient.rateLimit || 100, + storageQuotaMb: memClient.storageQuotaMb || 0, + }; } return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client registration not found or has been deleted', }; } @@ -365,16 +489,36 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { if (prevCached.status === 'revoked') { return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been revoked by admin', }; } if (prevCached.token && presentedToken && prevCached.token !== presentedToken) { return { allowed: false, + code: 'TOKEN_ROTATED', reason: 'TOKEN_ROTATED: Client credentials have been rotated', }; } - return { allowed: true, rateLimit: prevCached.rateLimit }; + if (isClientTokenExpired(prevCached.expiresAt, now)) { + return { + allowed: false, + code: 'CLIENT_TOKEN_EXPIRED', + reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`, + }; + } + if (prevCached.allowedIps && prevCached.allowedIps.trim() && !isClientIpAllowed(clientIp, prevCached.allowedIps)) { + return { + allowed: false, + code: 'CLIENT_IP_FORBIDDEN', + reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`, + }; + } + return { + allowed: true, + rateLimit: prevCached.rateLimit, + storageQuotaMb: prevCached.storageQuotaMb || 0, + }; } try { @@ -387,26 +531,39 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { deletedClientsSet.add(normalizedId); return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been deleted by admin', }; } const res = await dbPool.query( - `SELECT status, token, rate_limit FROM client_sdks WHERE client_id = $1`, + `SELECT status, token, rate_limit, storage_quota_mb, allowed_ips, expires_at FROM client_sdks WHERE client_id = $1`, [normalizedId], ); if (!res.rowCount || res.rowCount === 0) { clientAuthCache.delete(normalizedId); return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client registration not found or has been deleted', }; } const row = res.rows[0]; + const parsedQuota = Number(row.storage_quota_mb); + let expiresAtIso = null; + if (row.expires_at) { + const d = new Date(row.expires_at); + if (!Number.isNaN(d.getTime())) { + expiresAtIso = d.toISOString(); + } + } const entry = { status: row.status, token: row.token, rateLimit: Number(row.rate_limit) || 100, + storageQuotaMb: Number.isFinite(parsedQuota) && parsedQuota >= 0 ? Math.floor(parsedQuota) : 0, + allowedIps: typeof row.allowed_ips === 'string' ? row.allowed_ips : '', + expiresAt: expiresAtIso, checkedAt: now, }; clientAuthCache.set(normalizedId, entry); @@ -414,19 +571,40 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { if (entry.status === 'revoked') { return { allowed: false, + code: 'CLIENT_REVOKED', reason: 'CLIENT_REVOKED: Client access has been revoked by admin', }; } if (entry.token && presentedToken && entry.token !== presentedToken) { return { allowed: false, + code: 'TOKEN_ROTATED', reason: 'TOKEN_ROTATED: Client credentials have been rotated', }; } - return { allowed: true, rateLimit: entry.rateLimit }; + if (isClientTokenExpired(entry.expiresAt, now)) { + return { + allowed: false, + code: 'CLIENT_TOKEN_EXPIRED', + reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`, + }; + } + if (entry.allowedIps && entry.allowedIps.trim() && !isClientIpAllowed(clientIp, entry.allowedIps)) { + return { + allowed: false, + code: 'CLIENT_IP_FORBIDDEN', + reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`, + }; + } + return { + allowed: true, + rateLimit: entry.rateLimit, + storageQuotaMb: entry.storageQuotaMb, + }; } catch { return { allowed: false, + code: 'AUTH_UNAVAILABLE', reason: 'AUTH_UNAVAILABLE: Unable to verify client credentials against database', }; } @@ -888,13 +1066,42 @@ function applyRuntimeSettingsObject(settingsObj) { } } + if (settingsObj.enableS3Backup !== undefined) { + runtimeConfig.enableS3Backup = Boolean(settingsObj.enableS3Backup); + } + + if ( + settingsObj.diskPauseThresholdPct !== undefined && + settingsObj.diskPauseThresholdPct !== null + ) { + const pct = Number(settingsObj.diskPauseThresholdPct); + if (Number.isFinite(pct) && (pct === 0 || (pct >= 50 && pct <= 99))) { + runtimeConfig.diskPauseThresholdPct = Math.floor(pct); + } + } + + if ( + settingsObj.compressAfterDays !== undefined && + settingsObj.compressAfterDays !== null + ) { + const parsedCompress = Number(settingsObj.compressAfterDays); + if (Number.isFinite(parsedCompress) && parsedCompress >= 0) { + const nextCompressDays = Math.floor(parsedCompress); + const changed = nextCompressDays !== runtimeConfig.compressAfterDays; + runtimeConfig.compressAfterDays = nextCompressDays; + if (changed && nextCompressDays > 0) { + void writer.compressOldPartitionsAsync(nextCompressDays).catch(() => undefined); + } + } + } + if (settingsObj.autoPruneDays !== undefined && settingsObj.autoPruneDays !== null) { const parsedPrune = Number(settingsObj.autoPruneDays); if (Number.isFinite(parsedPrune) && parsedPrune >= 0) { const nextPruneDays = Math.floor(parsedPrune); const changed = nextPruneDays !== runtimeConfig.autoPruneDays; runtimeConfig.autoPruneDays = nextPruneDays; - if (changed && nextPruneDays > 0) { + if (changed && nextPruneDays > 0 && !runtimeConfig.enableS3Backup) { void writer.pruneExpiredPartitionsAsync(nextPruneDays).catch(() => undefined); } } @@ -948,6 +1155,9 @@ async function ensureDatabaseSchema() { secret TEXT NOT NULL, token TEXT NOT NULL, rate_limit INTEGER NOT NULL DEFAULT 100, + storage_quota_mb INTEGER NOT NULL DEFAULT 0, + allowed_ips TEXT NOT NULL DEFAULT '', + expires_at TIMESTAMPTZ, status TEXT NOT NULL DEFAULT 'active', total_events BIGINT NOT NULL DEFAULT 0, last_active TIMESTAMPTZ, @@ -968,6 +1178,21 @@ async function ensureDatabaseSchema() { jwt_secret TEXT NOT NULL, enable_idempotency BOOLEAN NOT NULL DEFAULT true, auto_prune_days INTEGER NOT NULL DEFAULT 30, + compress_after_days INTEGER NOT NULL DEFAULT 0, + enable_s3_backup BOOLEAN NOT NULL DEFAULT false, + s3_endpoint TEXT NOT NULL DEFAULT '', + s3_region TEXT NOT NULL DEFAULT 'auto', + s3_bucket TEXT NOT NULL DEFAULT '', + s3_prefix TEXT NOT NULL DEFAULT 'datastack/', + s3_access_key_id TEXT NOT NULL DEFAULT '', + s3_secret_access_key_enc TEXT NOT NULL DEFAULT '', + disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95, + enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false, + webhook_url TEXT NOT NULL DEFAULT '', + webhook_secret_enc TEXT NOT NULL DEFAULT '', + webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300, + enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false, + prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '', updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ) `); @@ -999,6 +1224,9 @@ async function ensureDatabaseSchema() { await pool.query(` ALTER TABLE client_sdks ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100, + ADD COLUMN IF NOT EXISTS storage_quota_mb INTEGER NOT NULL DEFAULT 0, + ADD COLUMN IF NOT EXISTS allowed_ips TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ, ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active', ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0, ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ, @@ -1016,6 +1244,21 @@ async function ensureDatabaseSchema() { ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true, ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30, + ADD COLUMN IF NOT EXISTS compress_after_days INTEGER NOT NULL DEFAULT 0, + ADD COLUMN IF NOT EXISTS enable_s3_backup BOOLEAN NOT NULL DEFAULT false, + ADD COLUMN IF NOT EXISTS s3_endpoint TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS s3_region TEXT NOT NULL DEFAULT 'auto', + ADD COLUMN IF NOT EXISTS s3_bucket TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS s3_prefix TEXT NOT NULL DEFAULT 'datastack/', + ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS s3_secret_access_key_enc TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95, + ADD COLUMN IF NOT EXISTS enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false, + ADD COLUMN IF NOT EXISTS webhook_url TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS webhook_secret_enc TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300, + ADD COLUMN IF NOT EXISTS enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false, + ADD COLUMN IF NOT EXISTS prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '', ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() `); await pool.query(` @@ -1043,7 +1286,7 @@ async function syncSettingsFromDatabase() { const res = await dbPool.query( `SELECT max_events_per_sec, max_payload_size_kb, batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb, jwt_secret, enable_idempotency, - auto_prune_days + auto_prune_days, compress_after_days, enable_s3_backup, disk_pause_threshold_pct FROM admin_settings WHERE id = 'default'`, ); @@ -1053,6 +1296,14 @@ async function syncSettingsFromDatabase() { row.auto_prune_days !== undefined && row.auto_prune_days !== null ? Number(row.auto_prune_days) : NaN; + const parsedCompress = + row.compress_after_days !== undefined && row.compress_after_days !== null + ? Number(row.compress_after_days) + : NaN; + const parsedDiskPct = + row.disk_pause_threshold_pct !== undefined && row.disk_pause_threshold_pct !== null + ? Number(row.disk_pause_threshold_pct) + : NaN; applyRuntimeSettingsObject({ maxEventsPerSec: Number(row.max_events_per_sec) || 100, @@ -1065,6 +1316,9 @@ async function syncSettingsFromDatabase() { fileRotationMb: Number(row.file_rotation_mb) || 100, enableIdempotency: Boolean(row.enable_idempotency), autoPruneDays: Number.isFinite(parsedPrune) ? parsedPrune : 30, + compressAfterDays: Number.isFinite(parsedCompress) ? Math.max(0, parsedCompress) : 0, + enableS3Backup: Boolean(row.enable_s3_backup), + diskPauseThresholdPct: Number.isFinite(parsedDiskPct) ? parsedDiskPct : 95, jwtSecret: row.jwt_secret ? String(row.jwt_secret).trim() : undefined, }); return true; @@ -1338,6 +1592,9 @@ app.post('/internal/client-sync', (req, res) => { status: nextStatus, token: nextToken, rateLimit: Math.max(1, Number(item.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)), + allowedIps: typeof item.allowedIps === 'string' ? item.allowedIps : '', + expiresAt: item.expiresAt ? String(item.expiresAt) : null, }); if (nextStatus === 'revoked' || (prev && nextToken && prev.token !== nextToken)) { io.to(`client:${cid}`).disconnectSockets(true); @@ -1380,6 +1637,9 @@ app.post('/internal/client-sync', (req, res) => { status: 'revoked', token: String(clientPayload.token || ''), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)), + allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '', + expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null, }); } else if (existingMem) { existingMem.status = 'revoked'; @@ -1390,6 +1650,9 @@ app.post('/internal/client-sync', (req, res) => { status: 'revoked', token: '', rateLimit: 100, + storageQuotaMb: 0, + allowedIps: '', + expiresAt: null, }); } io.to(`client:${normalizedId}`).disconnectSockets(true); @@ -1402,6 +1665,9 @@ app.post('/internal/client-sync', (req, res) => { status: clientPayload.status === 'revoked' ? 'revoked' : 'active', token: String(clientPayload.token || ''), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)), + allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '', + expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null, }); } } else if (action === 'updated') { @@ -1412,6 +1678,9 @@ app.post('/internal/client-sync', (req, res) => { status: clientPayload.status === 'revoked' ? 'revoked' : 'active', token: String(clientPayload.token || ''), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), + storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)), + allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '', + expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null, }); } io.to(`client:${normalizedId}`).disconnectSockets(true); @@ -1542,7 +1811,12 @@ async function startIngestionServer() { } pruneTimer = setInterval(() => { - if (runtimeConfig.autoPruneDays > 0) { + if (runtimeConfig.compressAfterDays > 0) { + void writer + .compressOldPartitionsAsync(runtimeConfig.compressAfterDays) + .catch(() => undefined); + } + if (runtimeConfig.autoPruneDays > 0 && !runtimeConfig.enableS3Backup) { void writer .pruneExpiredPartitionsAsync(runtimeConfig.autoPruneDays) .catch(() => undefined); @@ -1594,4 +1868,6 @@ module.exports = { parseAllowedCorsOrigins, isOriginAllowedByCors, corsOriginDelegate, + isClientIpAllowed, + isClientTokenExpired, }; diff --git a/server/socket/handlers.js b/server/socket/handlers.js index 5d493f5..b7779c9 100644 --- a/server/socket/handlers.js +++ b/server/socket/handlers.js @@ -26,6 +26,34 @@ function normalizeClientId(clientId) { .replace(/[^a-z0-9_-]/g, "_"); } +function extractSocketClientIp(socket) { + const headers = (socket && socket.handshake && socket.handshake.headers) || {}; + const realIp = + typeof headers["x-real-ip"] === "string" ? headers["x-real-ip"].trim() : ""; + if (realIp) { + return realIp.slice(0, 128); + } + const forwardedFor = + typeof headers["x-forwarded-for"] === "string" + ? headers["x-forwarded-for"] + : ""; + if (forwardedFor) { + const hops = forwardedFor + .split(",") + .map((part) => part.trim()) + .filter(Boolean); + const rightmost = hops[hops.length - 1]; + if (rightmost) { + return rightmost.slice(0, 128); + } + } + const addr = + (socket && socket.handshake && socket.handshake.address) || + (socket && socket.conn && socket.conn.remoteAddress) || + ""; + return String(addr).trim() || "127.0.0.1"; +} + function setupSocketHandlers(io, bufferQueue, worker, options = {}) { const getRuntimeConfig = typeof options.getRuntimeConfig === "function" @@ -340,17 +368,20 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { const decoded = jwt.verify(cleanToken, config.jwtSecret); const normalizedClientId = normalizeClientId(decoded.clientId); + const clientIp = extractSocketClientIp(socket); const dbCheck = await verifyClientInDb( normalizedClientId, cleanToken, true, + clientIp, ); if (!dbCheck.allowed) { - return next(new Error(dbCheck.reason || "CLIENT_REVOKED")); + return next(new Error(dbCheck.reason || dbCheck.code || "CLIENT_REVOKED")); } socket.data.clientId = normalizedClientId; socket.data.token = cleanToken; + socket.data.clientIp = clientIp; socket.data.rateLimit = dbCheck.rateLimit || decoded.rateLimit || config.maxEventsPerSec || 100; next(); @@ -401,16 +432,18 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { // Single event socket.on("sync_data", async (payload, callback) => { try { + const clientIp = socket.data.clientIp || extractSocketClientIp(socket); const dbCheck = await verifyClientInDb( clientId, socket.data.token, false, + clientIp, ); if (!dbCheck.allowed) { if (callback) { callback({ success: false, - code: "CLIENT_REVOKED", + code: dbCheck.code || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED", }); } @@ -418,6 +451,25 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { return; } + if (worker && typeof worker.checkStorageAdmission === "function") { + const config = getRuntimeConfig(); + const admission = await worker.checkStorageAdmission( + clientId, + dbCheck.storageQuotaMb || 0, + config.diskPauseThresholdPct ?? 95, + ); + if (!admission.allowed) { + if (callback) { + callback({ + success: false, + code: admission.code, + error: admission.reason, + }); + } + return; + } + } + const effectiveClientRate = dbCheck.rateLimit || socket.data.rateLimit || 100; const safePayload = @@ -499,10 +551,12 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { return; } + const clientIp = socket.data.clientIp || extractSocketClientIp(socket); const dbCheck = await verifyClientInDb( clientId, socket.data.token, false, + clientIp, ); if (!dbCheck.allowed) { if (callback) { @@ -510,7 +564,7 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { success: false, processed: 0, rejected: batch.length, - code: "CLIENT_REVOKED", + code: dbCheck.code || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED", }); } @@ -518,6 +572,27 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { return; } + if (worker && typeof worker.checkStorageAdmission === "function") { + const config = getRuntimeConfig(); + const admission = await worker.checkStorageAdmission( + clientId, + dbCheck.storageQuotaMb || 0, + config.diskPauseThresholdPct ?? 95, + ); + if (!admission.allowed) { + if (callback) { + callback({ + success: false, + processed: 0, + rejected: batch.length, + code: admission.code, + error: admission.reason, + }); + } + return; + } + } + const effectiveClientRate = dbCheck.rateLimit || socket.data.rateLimit || 100; let processed = 0; @@ -646,16 +721,19 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { // Heartbeat & Reconnection Verification Event socket.on("ping_verify", async (_payload, callback) => { try { + const clientIp = socket.data.clientIp || extractSocketClientIp(socket); const dbCheck = await verifyClientInDb( clientId, socket.data.token, true, + clientIp, ); if (!dbCheck.allowed) { if (callback) { callback({ success: false, status: "REVOKED", + code: dbCheck.code || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED", }); } @@ -691,4 +769,4 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) { }; } -module.exports = { setupSocketHandlers, normalizeClientId }; +module.exports = { setupSocketHandlers, normalizeClientId, extractSocketClientIp }; diff --git a/server/workers/csvWriter.js b/server/workers/csvWriter.js index 11edb66..958277f 100644 --- a/server/workers/csvWriter.js +++ b/server/workers/csvWriter.js @@ -1,12 +1,15 @@ const crypto = require('node:crypto'); const fs = require('fs'); const path = require('path'); +const { pipeline } = require('node:stream/promises'); +const zlib = require('node:zlib'); const { Parser } = require('json2csv'); const LOCK_RETRY_MIN_MS = 5; const LOCK_RETRY_MAX_MS = 18; const LOCK_TIMEOUT_MS = 10000; const STALE_LOCK_AGE_MS = 30000; +const QUOTA_CACHE_TTL_MS = 30000; function sleepMs(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); @@ -23,15 +26,42 @@ function resolveDefaultStorageDir() { /** * Background CSV Writer Worker * Flushes batches of real-time events to partitioned CSV files. - * Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv + * Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv(.gz) */ class CsvWriterWorker { constructor(baseDir = resolveDefaultStorageDir(), maxFileSize = 100 * 1024 * 1024) { this.baseDir = baseDir; this.maxFileSize = maxFileSize; + this.diskUsageCache = null; + this.clientBytesCache = new Map(); + this.statfsOverrideForTests = null; this.ensureBaseDir(); } + isPartitionFilename(filename) { + return typeof filename === 'string' && (filename.endsWith('.csv') || filename.endsWith('.csv.gz')); + } + + extractDateFromFilename(filename) { + return String(filename || '') + .replace(/\.csv(?:\.gz)?$/, '') + .split('-part')[0]; + } + + clearQuotaCaches(clientId) { + this.diskUsageCache = null; + if (clientId) { + this.clientBytesCache.delete(this.sanitizeClientId(clientId)); + } else { + this.clientBytesCache.clear(); + } + } + + setStatfsOverrideForTests(provider) { + this.statfsOverrideForTests = typeof provider === 'function' ? provider : null; + this.diskUsageCache = null; + } + setMaxFileSizeMb(megabytes) { const mb = Number(megabytes); if (Number.isFinite(mb) && mb >= 1) { @@ -282,6 +312,7 @@ class CsvWriterWorker { fs.writeFileSync(resolvedPath, fullCsv, 'utf8'); } + this.clearQuotaCaches(clientId); return resolvedPath; }); @@ -291,6 +322,101 @@ class CsvWriterWorker { return { written: events.length, files }; } + async compressOldPartitionsAsync(compressAfterDays) { + const days = Number(compressAfterDays); + if (!Number.isFinite(days) || days <= 0) { + return 0; + } + + const baseStat = await fs.promises.lstat(this.baseDir).catch(() => null); + if (!baseStat || !baseStat.isDirectory()) { + return 0; + } + + const cutoffDate = new Date(Date.now() - Math.floor(days) * 86400000) + .toISOString() + .split('T')[0]; + + let compressedCount = 0; + try { + const clientDirs = await fs.promises.readdir(this.baseDir); + for (const clientDir of clientDirs) { + const fullClientPath = path.join(this.baseDir, clientDir); + const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null); + if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) continue; + + const files = await fs.promises.readdir(fullClientPath).catch(() => []); + for (const file of files) { + if (!file.endsWith('.csv')) continue; + const datePart = this.extractDateFromFilename(file); + if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) continue; + const fullPath = path.join(fullClientPath, file); + const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); + if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue; + + const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`); + const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`; + let acquired = false; + const tryAcquireLock = async () => { + const handle = await fs.promises.open(lockPath, 'wx'); + try { + await handle.writeFile(lockValue, 'utf8'); + } finally { + await handle.close(); + } + return true; + }; + + try { + acquired = await tryAcquireLock(); + } catch (err) { + if (err && err.code === 'EEXIST') { + await this.removeStaleLockAtomicallyAsync(lockPath); + try { + acquired = await tryAcquireLock(); + } catch { + acquired = false; + } + } + } + + if (!acquired) continue; + + let targetGzPath = `${fullPath}.gz`; + if (fs.existsSync(targetGzPath)) { + const stem = fullPath.replace(/\.csv$/, ''); + let suffix = 2; + while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) { + suffix += 1; + } + targetGzPath = `${stem}-gz${suffix}.csv.gz`; + } + const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`; + + try { + await pipeline( + fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }), + zlib.createGzip({ level: 6 }), + fs.createWriteStream(tmpGzPath), + ); + await fs.promises.utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime).catch(() => undefined); + await fs.promises.rename(tmpGzPath, targetGzPath); + await fs.promises.unlink(fullPath); + this.clearQuotaCaches(clientDir); + compressedCount += 1; + } catch { + await fs.promises.unlink(tmpGzPath).catch(() => undefined); + } finally { + await this.releaseLockIfOwnedAsync(lockPath, lockValue); + } + } + } + } catch { + // Ignore directory scan failure + } + return compressedCount; + } + pruneExpiredPartitions(autoPruneDays) { const days = Number(autoPruneDays); if (!Number.isFinite(days) || days <= 0 || !fs.existsSync(this.baseDir)) { @@ -316,8 +442,8 @@ class CsvWriterWorker { const files = fs.readdirSync(fullClientPath); for (const file of files) { - if (!file.endsWith('.csv')) continue; - const datePart = file.replace('.csv', '').split('-part')[0]; + if (!this.isPartitionFilename(file)) continue; + const datePart = this.extractDateFromFilename(file); if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { const fullPath = path.join(fullClientPath, file); let fileLstat; @@ -361,6 +487,7 @@ class CsvWriterWorker { try { if (fs.existsSync(fullPath)) { fs.unlinkSync(fullPath); + this.clearQuotaCaches(clientDir); prunedCount += 1; } } catch { @@ -402,8 +529,8 @@ class CsvWriterWorker { const files = await fs.promises.readdir(fullClientPath).catch(() => []); for (const file of files) { - if (!file.endsWith('.csv')) continue; - const datePart = file.replace('.csv', '').split('-part')[0]; + if (!this.isPartitionFilename(file)) continue; + const datePart = this.extractDateFromFilename(file); if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { const fullPath = path.join(fullClientPath, file); const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); @@ -439,6 +566,7 @@ class CsvWriterWorker { try { await fs.promises.unlink(fullPath); + this.clearQuotaCaches(clientDir); prunedCount += 1; } catch { // Ignore individual file unlink failure @@ -453,6 +581,106 @@ class CsvWriterWorker { } return prunedCount; } + + async getDiskUsageAsync() { + const now = Date.now(); + if (this.diskUsageCache && now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS) { + return this.diskUsageCache; + } + + let probeDir = path.resolve(this.baseDir); + while (!fs.existsSync(probeDir)) { + const parent = path.dirname(probeDir); + if (parent === probeDir) break; + probeDir = parent; + } + + try { + const statfsRes = this.statfsOverrideForTests + ? await this.statfsOverrideForTests(probeDir) + : await fs.promises.statfs(probeDir); + const bsize = Math.max(1, Number(statfsRes.bsize) || 4096); + const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize); + const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize); + const usedBytes = Math.max(0, totalBytes - availableBytes); + const usedPct = + totalBytes > 0 + ? Number(Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2)) + : 0; + const snapshot = { usedPct, availableBytes, totalBytes, checkedAt: now }; + this.diskUsageCache = snapshot; + return snapshot; + } catch { + const fallback = { usedPct: 0, availableBytes: 0, totalBytes: 0, checkedAt: now }; + this.diskUsageCache = fallback; + return fallback; + } + } + + async getClientStorageBytesAsync(clientId) { + const safeClientId = this.sanitizeClientId(clientId); + const now = Date.now(); + const cached = this.clientBytesCache.get(safeClientId); + if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) { + return cached.bytes; + } + + const resolvedBase = path.resolve(this.baseDir); + const clientDir = path.resolve(resolvedBase, safeClientId); + if (!clientDir.startsWith(resolvedBase + path.sep)) { + return 0; + } + + let totalBytes = 0; + try { + const dirStat = await fs.promises.lstat(clientDir).catch(() => null); + if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) { + const entries = await fs.promises.readdir(clientDir).catch(() => []); + for (const entry of entries) { + if (!this.isPartitionFilename(entry)) continue; + const filePath = path.join(clientDir, entry); + const stats = await fs.promises.lstat(filePath).catch(() => null); + if (stats && !stats.isSymbolicLink() && stats.isFile()) { + totalBytes += stats.size; + } + } + } + } catch { + // ignore + } + + this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes }); + return totalBytes; + } + + async checkStorageAdmission(clientId, clientQuotaMb, diskPauseThresholdPct) { + const threshold = Number(diskPauseThresholdPct); + if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) { + const disk = await this.getDiskUsageAsync(); + if (disk.totalBytes > 0 && disk.usedPct >= threshold) { + return { + allowed: false, + code: 'DISK_SPACE_LOW', + reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`, + }; + } + } + + const quotaMb = Number(clientQuotaMb); + if (Number.isFinite(quotaMb) && quotaMb > 0) { + const currentBytes = await this.getClientStorageBytesAsync(clientId); + const quotaBytes = Math.floor(quotaMb) * 1024 * 1024; + if (currentBytes >= quotaBytes) { + return { + allowed: false, + code: 'STORAGE_QUOTA_EXCEEDED', + reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`, + }; + } + } + + return { allowed: true }; + } } module.exports = { CsvWriterWorker, resolveDefaultStorageDir }; diff --git a/tests/v2-phase01-storage-quotas.test.ts b/tests/v2-phase01-storage-quotas.test.ts new file mode 100644 index 0000000..6b98d79 --- /dev/null +++ b/tests/v2-phase01-storage-quotas.test.ts @@ -0,0 +1,727 @@ +import assert from "node:assert/strict"; +import fs from "node:fs/promises"; +import http from "node:http"; +import { createRequire } from "node:module"; +import type { AddressInfo } from "node:net"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import zlib from "node:zlib"; +import { NextRequest } from "next/server"; +import { io as ioClient, type Socket } from "socket.io-client"; +import { DELETE as deleteClientRoute, PATCH as patchClientRoute } from "../app/api/clients/[id]/route"; +import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route"; +import { GET as dataGet } from "../app/api/data/route"; +import { GET as eventsGet, POST as eventsPost } from "../app/api/events/route"; +import { GET as exportGet } from "../app/api/export/route"; +import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route"; +import { POST as archivePost } from "../app/api/storage/archive/route"; +import { GET as storageGet } from "../app/api/storage/route"; +import { POST as s3TestPost } from "../app/api/storage/s3-test/route"; +import { SettingsManager } from "../core/adminSettings/settingsManager"; +import { + createAdminSessionToken, + getAdminSessionSecret, +} from "../core/auth/adminSession"; +import { + createInitialAdmin, + resetAdminStateForTests, +} from "../core/auth/adminUsers"; +import { DataBufferEngine } from "../core/data/buffer"; +import { SDKManager } from "../core/sdk/sdkManager"; +import { + decryptSecret, + encryptSecret, + getMaskedHintFromEncrypted, + SecretVaultError, +} from "../core/Security/secretVault"; +import { CSVStorageManager } from "../core/Storage/csvStorage"; +import { + buildS3ObjectKey, + normalizeS3EndpointUrl, + testS3Connection, + uploadFileToS3, +} from "../core/Storage/s3Archive"; +import { + DEFAULT_EDITABLE_SETTINGS, + settingsToNumericDrafts, + validateEditableSettingsDraft, + validateSingleNumericSetting, +} from "../lib/vconUiHelpers"; + +const require = createRequire(import.meta.url); + +async function makeAdminCookie(): Promise { + resetAdminStateForTests(); + await createInitialAdmin("v2-p1@datastack.dev", "V2Phase01Pass!"); + const secret = getAdminSessionSecret()!; + const token = await createAdminSessionToken( + "v2-p1@datastack.dev", + secret, + Date.now() + 3600_000, + 1, + ); + return `datastack_admin_session=${token}`; +} + +test("V2-Phase-01 [Vault]: encryptSecret/decryptSecret AES-256-GCM roundtrip, tamper detection & masked hints", () => { + const secretA = "test-master-secret-key-at-least-32-characters"; + const secretB = "different-master-secret-key-at-least-32-chars"; + + const encrypted = encryptSecret("r2_secret_value_987654321", secretA); + assert.ok(encrypted.startsWith("v1:")); + assert.equal(decryptSecret(encrypted, secretA), "r2_secret_value_987654321"); + assert.equal(getMaskedHintFromEncrypted(encrypted, secretA), "••••4321"); + + // Tampered ciphertext throws SecretVaultError (V2-E01) + const parts = encrypted.split(":"); + const flippedChar = parts[3].startsWith("A") ? "B" : "A"; + const tampered = `${parts[0]}:${parts[1]}:${parts[2]}:${flippedChar}${parts[3].slice(1)}`; + assert.throws( + () => decryptSecret(tampered, secretA), + (err: unknown) => + err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED", + ); + + // Wrong master key throws SecretVaultError + assert.throws( + () => decryptSecret(encrypted, secretB), + (err: unknown) => + err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED", + ); + + // Too-short master secret rejects key derivation + assert.throws( + () => encryptSecret("hello", "short"), + (err: unknown) => + err instanceof SecretVaultError && err.code === "SECRET_VAULT_KEY_UNAVAILABLE", + ); +}); + +test("V2-Phase-01 [S3 SigV4]: buildS3ObjectKey, normalizeS3EndpointUrl, testS3Connection & uploadFileToS3 against mock S3 server", async () => { + assert.equal( + buildS3ObjectKey("/datastack/", "client_01", "2026-10-01.csv.gz"), + "datastack/client_01/2026-10-01.csv.gz", + ); + assert.equal( + normalizeS3EndpointUrl("", "us-east-1").url.origin, + "https://s3.us-east-1.amazonaws.com", + ); + + const requests: Array<{ + method: string; + url: string; + auth: string; + bodyLen: number; + }> = []; + let failUploads = false; + + const mockS3 = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const body = Buffer.concat(chunks); + requests.push({ + method: req.method || "", + url: req.url || "", + auth: String(req.headers.authorization || ""), + bodyLen: body.byteLength, + }); + if (req.method === "PUT" && failUploads) { + res.writeHead(500, { "content-type": "text/plain" }); + res.end("Simulated S3 outage"); + return; + } + if (req.method === "PUT") { + res.writeHead(200, { etag: '"mock-etag-1"' }); + res.end(""); + return; + } + res.writeHead(200, { "content-type": "application/xml" }); + res.end(""); + }); + }); + + await new Promise((resolve) => mockS3.listen(0, "127.0.0.1", resolve)); + const addr = mockS3.address() as AddressInfo; + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-s3-")); + + try { + const samplePath = path.join(tmpDir, "2026-10-01.csv.gz"); + const gzData = zlib.gzipSync( + Buffer.from('clientId,eventId,timestamp,payload\n"c1","e1","2026-10-01T00:00:00Z","{}"\n'), + ); + await fs.writeFile(samplePath, gzData); + + const cfg = { + endpoint: `http://127.0.0.1:${addr.port}`, + region: "auto", + bucket: "ds-archive", + prefix: "datastack/", + accessKeyId: "AKIA_TEST_123", + secretAccessKey: "secret_access_key_xyz_999", + }; + + const probe = await testS3Connection(cfg); + assert.equal(probe.ok, true); + assert.equal(probe.status, 200); + assert.match(requests[0].auth, /^AWS4-HMAC-SHA256 Credential=AKIA_TEST_123\//); + assert.match(requests[0].url, /^\/ds-archive\?list-type=2&max-keys=1$/); + + const uploadOk = await uploadFileToS3( + cfg, + samplePath, + "datastack/c1/2026-10-01.csv.gz", + ); + assert.equal(uploadOk.ok, true); + assert.equal(uploadOk.etag, '"mock-etag-1"'); + assert.equal(requests[1].method, "PUT"); + assert.equal(requests[1].url, "/ds-archive/datastack/c1/2026-10-01.csv.gz"); + assert.equal(requests[1].bodyLen, gzData.byteLength); + + failUploads = true; + const uploadErr = await uploadFileToS3( + cfg, + samplePath, + "datastack/c1/2026-10-01.csv.gz", + ); + assert.equal(uploadErr.ok, false); + assert.equal(uploadErr.code, "S3_ARCHIVE_FAILED"); + assert.match(uploadErr.error || "", /HTTP 500/); + } finally { + await new Promise((resolve) => mockS3.close(() => resolve())); + await fs.rm(tmpDir, { recursive: true, force: true }); + } +}); + +test("V2-Phase-01 [Compression, Export, S3 Safe Prune V2-E02, Corrupt Gzip V2-E03]", async () => { + const prevStorageDir = process.env.DATASTACK_STORAGE_DIR; + const prevDbUrl = process.env.DATABASE_URL; + const origBaseStorageDir = CSVStorageManager.baseStorageDir; + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-gz-")); + delete process.env.DATABASE_URL; + process.env.DATASTACK_STORAGE_DIR = tmpDir; + CSVStorageManager.baseStorageDir = tmpDir; + CSVStorageManager.clearRowCountCache(); + CSVStorageManager.clearQuotaCaches(); + + const uploadedKeys: string[] = []; + let failNextUpload = false; + const mockS3 = http.createServer((req, res) => { + req.on("data", () => undefined); + req.on("end", () => { + if (req.method === "PUT") { + if (failNextUpload) { + res.writeHead(502); + res.end("Upstream error"); + return; + } + uploadedKeys.push(req.url || ""); + res.writeHead(200, { etag: '"ok"' }); + res.end(""); + return; + } + res.writeHead(200); + res.end(""); + }); + }); + + await new Promise((resolve) => mockS3.listen(0, "127.0.0.1", resolve)); + const s3Addr = mockS3.address() as AddressInfo; + + try { + const cookie = await makeAdminCookie(); + const today = new Date().toISOString().slice(0, 10); + const oldDate = new Date(Date.now() - 45 * 86_400_000).toISOString().slice(0, 10); + await SettingsManager.updateSettings({ + autoPruneDays: 90, + compressAfterDays: 0, + enableS3Backup: false, + diskPauseThresholdPct: 0, + }); + + // 1. Write 2 old rows (45 days ago) and 1 today row + await CSVStorageManager.appendBatch([ + { + clientId: "v2_alpha", + eventId: "evt_old_1", + timestamp: `${oldDate}T10:00:00.000Z`, + payload: '{"stage":"old","seq":1}', + }, + { + clientId: "v2_alpha", + eventId: "evt_old_2", + timestamp: `${oldDate}T11:00:00.000Z`, + payload: '{"stage":"old","seq":2}', + }, + { + clientId: "v2_alpha", + eventId: "evt_today_1", + timestamp: `${today}T09:00:00.000Z`, + payload: '{"stage":"live","seq":3}', + }, + ]); + + // 2. Compress partitions older than 30 days -> ${oldDate}.csv becomes .csv.gz; today stays .csv + const compressed = await CSVStorageManager.compressOldPartitionsAsync(30); + assert.equal(compressed, 1); + + const files = await CSVStorageManager.listAllFilesAsync(); + assert.equal(files.length, 2); + const gzMeta = files.find((f) => f.filename === `${oldDate}.csv.gz`); + assert.ok(gzMeta, `Expected compressed ${oldDate}.csv.gz partition`); + assert.equal(gzMeta.linesCount, 2); + assert.equal(gzMeta.compressed, true); + + // Original uncompressed ${oldDate}.csv was removed after atomic rename + const legacyCsv = path.join(tmpDir, "v2_alpha", `${oldDate}.csv`); + await assert.rejects(() => fs.stat(legacyCsv)); + + // 3. Verify GET /api/data reads .csv.gz transparently + const dataRes = await dataGet( + new NextRequest( + `http://localhost:3000/api/data?clientId=v2_alpha&from=${oldDate}&to=${oldDate}`, + { headers: { cookie } }, + ), + ); + assert.equal(dataRes.status, 200); + const dataJson = await dataRes.json(); + assert.equal(dataJson.total, 2); + assert.equal(dataJson.events.length, 2); + assert.equal(dataJson.events[0].eventId, "evt_old_2"); + assert.equal(dataJson.events[1].eventId, "evt_old_1"); + + // 4. Verify GET /api/events reads compressed partition seamlessly + const evRes = await eventsGet( + new NextRequest("http://localhost:3000/api/events?clientId=v2_alpha&limit=10", { + headers: { cookie }, + }), + ); + assert.equal(evRes.status, 200); + const evJson = await evRes.json(); + const ids = (evJson.events as Array<{ eventId: string }>).map((e) => e.eventId); + assert.ok(ids.includes("evt_old_1")); + assert.ok(ids.includes("evt_today_1")); + + // 5. Verify GET /api/export CSV, JSON, and raw single .csv.gz download + const exportCsvRes = await exportGet( + new NextRequest( + `http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=csv`, + { headers: { cookie } }, + ), + ); + assert.equal(exportCsvRes.status, 200); + const exportedCsvText = await exportCsvRes.text(); + assert.match(exportedCsvText, /evt_old_1/); + assert.match(exportedCsvText, /evt_old_2/); + + const exportJsonRes = await exportGet( + new NextRequest( + `http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=json`, + { headers: { cookie } }, + ), + ); + assert.equal(exportJsonRes.status, 200); + const exportedJson = await exportJsonRes.json(); + assert.equal(exportedJson.length, 2); + + const exportSingleGzRes = await exportGet( + new NextRequest( + `http://localhost:3000/api/export?clientId=v2_alpha&file=${oldDate}.csv.gz`, + { headers: { cookie } }, + ), + ); + assert.equal(exportSingleGzRes.status, 200); + assert.equal(exportSingleGzRes.headers.get("content-type"), "application/gzip"); + const downloadedBuf = Buffer.from(await exportSingleGzRes.arrayBuffer()); + assert.match(zlib.gunzipSync(downloadedBuf).toString("utf8"), /evt_old_1/); + + // 6. Save S3 settings via POST /api/settings and verify GET /api/settings masks secret (zero plaintext leak) + const saveRes = await settingsPost( + new NextRequest("http://localhost:3000/api/settings", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ + compressAfterDays: 30, + autoPruneDays: 60, + diskPauseThresholdPct: 92, + enableS3Backup: true, + s3Endpoint: `http://127.0.0.1:${s3Addr.port}`, + s3Region: "auto", + s3Bucket: "prod-vault", + s3Prefix: "archive-test/", + s3AccessKeyId: "AKIA_V2_KEY", + s3SecretAccessKey: "super_secret_v2_token_7788", + }), + }), + ); + assert.equal(saveRes.status, 200); + + const getRes = await settingsGet( + new NextRequest("http://localhost:3000/api/settings", { headers: { cookie } }), + ); + assert.equal(getRes.status, 200); + const getText = await getRes.text(); + assert.doesNotMatch(getText, /super_secret_v2_token_7788/); + const parsedSettings = JSON.parse(getText).settings; + assert.equal(parsedSettings.hasS3SecretKey, true); + assert.equal(parsedSettings.s3SecretKeyMask, "••••7788"); + assert.equal(parsedSettings.compressAfterDays, 30); + assert.equal(parsedSettings.diskPauseThresholdPct, 92); + + // 7. POST /api/storage/s3-test succeeds with saved credentials + const s3TestRes = await s3TestPost( + new NextRequest("http://localhost:3000/api/storage/s3-test", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({}), + }), + ); + assert.equal(s3TestRes.status, 200); + assert.equal((await s3TestRes.json()).success, true); + + // 8. V2-E02: When S3 upload fails, POST /api/storage/archive returns 502 and NEVER deletes local partition + failNextUpload = true; + const failArchiveRes = await archivePost( + new NextRequest("http://localhost:3000/api/storage/archive", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ pruneAfterUpload: true, olderThanDays: 30 }), + }), + ); + assert.equal(failArchiveRes.status, 502); + const failArchiveJson = await failArchiveRes.json(); + assert.equal(failArchiveJson.code, "S3_ARCHIVE_FAILED"); + assert.equal(failArchiveJson.prunedCount, 0); + // File MUST still exist on disk! + await fs.stat(gzMeta.filePath); + + // 9. When S3 upload succeeds with pruneAfterUpload: false, local file stays + uploadedKeys receives .csv.gz + failNextUpload = false; + const okArchiveRes = await archivePost( + new NextRequest("http://localhost:3000/api/storage/archive", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ pruneAfterUpload: false, olderThanDays: 30 }), + }), + ); + assert.equal(okArchiveRes.status, 200); + const okArchiveJson = await okArchiveRes.json(); + assert.equal(okArchiveJson.archivedCount, 1); + assert.equal(okArchiveJson.prunedCount, 0); + assert.deepEqual(uploadedKeys, [`/prod-vault/archive-test/v2_alpha/${oldDate}.csv.gz`]); + await fs.stat(gzMeta.filePath); + + // 10. V2-E03: Corrupt .csv.gz cleanly surfaces error in /api/data instead of crashing + await fs.writeFile( + path.join(tmpDir, "v2_alpha", "2024-01-01.csv.gz"), + Buffer.from("not-valid-gzip-bytes"), + ); + const corruptDataRes = await dataGet( + new NextRequest( + "http://localhost:3000/api/data?clientId=v2_alpha&from=2024-01-01&to=2024-01-01", + { headers: { cookie } }, + ), + ); + assert.equal(corruptDataRes.status, 500); + const corruptJson = await corruptDataRes.json(); + assert.match(corruptJson.error || "", /Corrupt|\.gz/i); + } finally { + await new Promise((resolve) => mockS3.close(() => resolve())); + await DataBufferEngine.shutdown(); + CSVStorageManager.baseStorageDir = origBaseStorageDir; + CSVStorageManager.clearRowCountCache(); + CSVStorageManager.clearQuotaCaches(); + if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR; + else process.env.DATASTACK_STORAGE_DIR = prevStorageDir; + if (prevDbUrl === undefined) delete process.env.DATABASE_URL; + else process.env.DATABASE_URL = prevDbUrl; + await fs.rm(tmpDir, { recursive: true, force: true }); + resetAdminStateForTests(); + } +}); + +test("V2-Phase-01 [Quotas V2-E04 & V2-E05]: HTTP POST /api/events + Port 4000 Socket.IO enforce disk threshold & per-client storageQuotaMb while keeping Admin reads online", async () => { + const prevStorageDir = process.env.DATASTACK_STORAGE_DIR; + const prevDbUrl = process.env.DATABASE_URL; + const prevDisableListen = process.env.DATASTACK_DISABLE_LISTEN; + const origBaseStorageDir = CSVStorageManager.baseStorageDir; + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-quota-")); + + delete process.env.DATABASE_URL; + process.env.DATASTACK_STORAGE_DIR = tmpDir; + CSVStorageManager.baseStorageDir = tmpDir; + process.env.DATASTACK_DISABLE_LISTEN = "1"; + CSVStorageManager.clearQuotaCaches(); + await DataBufferEngine.shutdown(); + DataBufferEngine.setRedisClientForTests(null); + + const serverMod = require("../server/index.js") as { + server: http.Server; + io: { close: (cb: () => void) => void }; + writer: { + baseDir: string; + setStatfsOverrideForTests: ( + fn: ((dir: string) => Promise<{ blocks: number; bavail: number; bsize: number }>) | null, + ) => void; + clearQuotaCaches: () => void; + }; + runtimeConfig: { diskPauseThresholdPct: number }; + setDbPoolForTests: (pool: unknown) => void; + stopTimers: () => void; + }; + const origWriterBaseDir = serverMod.writer.baseDir; + serverMod.writer.baseDir = tmpDir; + serverMod.writer.clearQuotaCaches(); + + const openSockets: Socket[] = []; + let listeningUrl = ""; + + try { + const cookie = await makeAdminCookie(); + + // 1. Validate UI helper & API validation for diskPauseThresholdPct (0 or 50..99) + assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "0").valid, true); + assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "95").valid, true); + assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "25").valid, false); + const fullDraft = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS); + assert.equal(validateEditableSettingsDraft(fullDraft, true).valid, true); + + const badPctRes = await settingsPost( + new NextRequest("http://localhost:3000/api/settings", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ diskPauseThresholdPct: 25 }), + }), + ); + assert.equal(badPctRes.status, 400); + + await SettingsManager.updateSettingsAsync({ diskPauseThresholdPct: 90 }); + + // 2. Register client via /api/clients with 1 MB quota & edit quota via PATCH /api/clients/[id] + const createRes = await clientsPost( + new NextRequest("http://localhost:3000/api/clients", { + method: "POST", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ + clientId: "quota_client", + name: "Quota Client", + rateLimit: 200, + storageQuotaMb: 2, + }), + }), + ); + assert.equal(createRes.status, 201); + const createdClient = (await createRes.json()).client; + assert.equal(createdClient.storageQuotaMb, 2); + + const patchRes = await patchClientRoute( + new NextRequest("http://localhost:3000/api/clients/quota_client", { + method: "PATCH", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ storageQuotaMb: 1 }), + }), + { params: Promise.resolve({ id: "quota_client" }) }, + ); + assert.equal(patchRes.status, 200); + assert.equal((await patchRes.json()).client.storageQuotaMb, 1); + + const listRes = await clientsGet( + new NextRequest("http://localhost:3000/api/clients", { headers: { cookie } }), + ); + const listed = (await listRes.json()).clients.find( + (c: { clientId: string }) => c.clientId === "quota_client", + ); + assert.equal(listed.storageQuotaMb, 1); + + // Seed 1.1 MB into client dir to breach 1 MB quota (V2-E05) + const clientDir = path.join(tmpDir, "quota_client"); + await fs.mkdir(clientDir, { recursive: true }); + await fs.writeFile( + path.join(clientDir, "2026-10-10.csv"), + Buffer.alloc(Math.ceil(1.1 * 1024 * 1024), "x"), + ); + CSVStorageManager.clearQuotaCaches("quota_client"); + + const httpQuotaRes = await eventsPost( + new NextRequest("http://localhost:3000/api/events", { + method: "POST", + headers: { + "content-type": "application/json", + authorization: `Bearer ${createdClient.token}`, + }, + body: JSON.stringify({ eventId: "q_evt_1", data: { ok: true } }), + }), + ); + assert.equal(httpQuotaRes.status, 507); + assert.equal((await httpQuotaRes.json()).code, "STORAGE_QUOTA_EXCEEDED"); + + // Now test global VPS disk threshold >= 90% (V2-E04) across HTTP + Socket.IO + await patchClientRoute( + new NextRequest("http://localhost:3000/api/clients/quota_client", { + method: "PATCH", + headers: { "content-type": "application/json", cookie }, + body: JSON.stringify({ storageQuotaMb: 0 }), + }), + { params: Promise.resolve({ id: "quota_client" }) }, + ); + CSVStorageManager.setStatfsOverrideForTests(async () => ({ + blocks: 1000, + bavail: 50, // 95% used >= 90% threshold + bsize: 4096, + })); + + const httpDiskRes = await eventsPost( + new NextRequest("http://localhost:3000/api/events", { + method: "POST", + headers: { + "content-type": "application/json", + authorization: `Bearer ${createdClient.token}`, + }, + body: JSON.stringify({ eventId: "q_evt_2", data: { ok: true } }), + }), + ); + assert.equal(httpDiskRes.status, 507); + assert.equal((await httpDiskRes.json()).code, "DISK_SPACE_LOW"); + + // Admin read/storage/export MUST remain 100% operational while ingestion is paused! + const storageRes = await storageGet( + new NextRequest("http://localhost:3000/api/storage", { headers: { cookie } }), + ); + assert.equal(storageRes.status, 200); + const storageJson = await storageRes.json(); + assert.equal(storageJson.diskUsage.usedPct, 95); + + // Port 4000 Socket.IO enforcement for DISK_SPACE_LOW & STORAGE_QUOTA_EXCEEDED + let mockQuotaMb = 0; + serverMod.runtimeConfig.diskPauseThresholdPct = 90; + serverMod.setDbPoolForTests({ + query: async (sql: string) => { + if (sql.includes("FROM client_sdks")) { + return { + rowCount: 1, + rows: [ + { + client_id: "quota_client", + token: createdClient.token, + status: "active", + rate_limit: 200, + storage_quota_mb: mockQuotaMb, + }, + ], + }; + } + return { rowCount: 0, rows: [] }; + }, + }); + serverMod.writer.setStatfsOverrideForTests(async () => ({ + blocks: 1000, + bavail: 50, // 95% used + bsize: 4096, + })); + + await new Promise((resolve) => + serverMod.server.listen(0, "127.0.0.1", resolve), + ); + const addr = serverMod.server.address() as AddressInfo; + listeningUrl = `http://127.0.0.1:${addr.port}`; + + const socket = ioClient(listeningUrl, { + auth: { token: createdClient.token }, + transports: ["websocket"], + reconnection: false, + timeout: 3000, + }); + openSockets.push(socket); + await new Promise((resolve, reject) => { + socket.on("connect", () => resolve()); + socket.on("connect_error", reject); + }); + + const singleDiskAck = await new Promise<{ + success: boolean; + code?: string; + }>((resolve, reject) => { + socket.timeout(3000).emit("sync_data", { data: { k: 1 } }, (err: unknown, ack: { success: boolean; code?: string }) => { + if (err) reject(err); + else resolve(ack); + }); + }); + assert.equal(singleDiskAck.success, false); + assert.equal(singleDiskAck.code, "DISK_SPACE_LOW"); + + // Relax disk threshold & activate per-client 1 MB quota in Socket.IO worker + serverMod.writer.setStatfsOverrideForTests(async () => ({ + blocks: 1000, + bavail: 800, // 20% used + bsize: 4096, + })); + serverMod.writer.clearQuotaCaches(); + mockQuotaMb = 1; + // Wait out 2s verifyClientInDb cache or update query response cleanly + await new Promise((r) => setTimeout(r, 2100)); + + const batchQuotaAck = await new Promise<{ + success: boolean; + code?: string; + rejected?: number; + }>((resolve, reject) => { + socket.timeout(3000).emit( + "batch_sync", + [{ eventId: "b1", data: { k: 1 } }], + (err: unknown, ack: { success: boolean; code?: string; rejected?: number }) => { + if (err) reject(err); + else resolve(ack); + }, + ); + }); + assert.equal(batchQuotaAck.success, false); + assert.equal(batchQuotaAck.code, "STORAGE_QUOTA_EXCEEDED"); + assert.equal(batchQuotaAck.rejected, 1); + + // Clean delete route check + const delRes = await deleteClientRoute( + new NextRequest("http://localhost:3000/api/clients/quota_client?purgeFiles=true", { + method: "DELETE", + headers: { cookie }, + }), + { params: Promise.resolve({ id: "quota_client" }) }, + ); + assert.equal(delRes.status, 200); + } finally { + for (const s of openSockets) { + try { + s.disconnect(); + } catch { + // ignore + } + } + serverMod.writer.setStatfsOverrideForTests(null); + serverMod.writer.baseDir = origWriterBaseDir; + serverMod.writer.clearQuotaCaches(); + serverMod.setDbPoolForTests(null); + serverMod.stopTimers(); + await new Promise((resolve) => { + serverMod.io.close(() => { + if (serverMod.server.listening) { + serverMod.server.close(() => resolve()); + } else { + resolve(); + } + }); + }); + SDKManager.stopActivityFlushTimerForTests(); + CSVStorageManager.setStatfsOverrideForTests(null); + CSVStorageManager.clearQuotaCaches(); + await DataBufferEngine.shutdown(); + CSVStorageManager.baseStorageDir = origBaseStorageDir; + if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR; + else process.env.DATASTACK_STORAGE_DIR = prevStorageDir; + if (prevDisableListen === undefined) delete process.env.DATASTACK_DISABLE_LISTEN; + else process.env.DATASTACK_DISABLE_LISTEN = prevDisableListen; + if (prevDbUrl === undefined) delete process.env.DATABASE_URL; + else process.env.DATABASE_URL = prevDbUrl; + await fs.rm(tmpDir, { recursive: true, force: true }); + resetAdminStateForTests(); + } +}); diff --git a/tests/v2-phase02-alerts-ip-prometheus.test.ts b/tests/v2-phase02-alerts-ip-prometheus.test.ts new file mode 100644 index 0000000..4bf21c1 --- /dev/null +++ b/tests/v2-phase02-alerts-ip-prometheus.test.ts @@ -0,0 +1,631 @@ +import assert from "node:assert/strict"; +import fs from "node:fs/promises"; +import http from "node:http"; +import { createRequire } from "node:module"; +import type { AddressInfo } from "node:net"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { NextRequest } from "next/server"; +import { io as ioClient, type Socket } from "socket.io-client"; +import { PATCH as patchClientRoute } from "../app/api/clients/[id]/route"; +import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route"; +import { POST as eventsPost } from "../app/api/events/route"; +import { GET as exportGet } from "../app/api/export/route"; +import { GET as prometheusGet } from "../app/api/prometheus/route"; +import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route"; +import { POST as webhookTestPost } from "../app/api/settings/webhook-test/route"; +import { SettingsManager } from "../core/adminSettings/settingsManager"; +import { + buildWebhookSignature, + clearWebhookCooldownCacheForTests, + dispatchWebhookAlertAsync, + formatOutboundWebhookBody, +} from "../core/alerts/webhookNotifier"; +import { + createAdminSessionToken, + getAdminSessionSecret, +} from "../core/auth/adminSession"; +import { + createInitialAdmin, + resetAdminStateForTests, +} from "../core/auth/adminUsers"; +import { DataBufferEngine } from "../core/data/buffer"; +import { SDKManager } from "../core/sdk/sdkManager"; +import { + isClientIpAllowed, + isClientTokenExpired, + validateAllowedIpsInput, + validateClientExpiresAtInput, +} from "../core/Security/ipAllowlist"; +import { CSVStorageManager } from "../core/Storage/csvStorage"; +import { + DEFAULT_ALERTING_SETTINGS, + DEFAULT_EDITABLE_SETTINGS, + normalizeAlertingSettings, + settingsToNumericDrafts, + validateEditableSettingsDraft, + validateSingleNumericSetting, +} from "../lib/vconUiHelpers"; + +const require = createRequire(import.meta.url); + +async function makeAdminCookie(): Promise { + resetAdminStateForTests(); + await createInitialAdmin("v2-p2@datastack.dev", "V2Phase02Pass!"); + const secret = getAdminSessionSecret()!; + const token = await createAdminSessionToken( + "v2-p2@datastack.dev", + secret, + Date.now() + 3600_000, + 1, + ); + return `datastack_admin_session=${token}`; +} + +test("V2-Phase-02 [IP Allowlist & Token Expiry]: validateAllowedIpsInput, isClientIpAllowed (IPv4/IPv6/CIDR) & isClientTokenExpired", () => { + // Valid empty or wildcard + assert.deepEqual(validateAllowedIpsInput(""), { valid: true, normalized: "" }); + assert.deepEqual(validateAllowedIpsInput(" "), { valid: true, normalized: "" }); + assert.deepEqual(validateAllowedIpsInput("*"), { valid: true, normalized: "*" }); + + // Valid comma-separated IPv4, IPv4 CIDR, IPv6, IPv6 CIDR + const validList = validateAllowedIpsInput( + "203.0.113.10, 10.0.0.0/8, ::ffff:192.0.2.5, 2001:db8::/32, 203.0.113.10", + ); + assert.equal(validList.valid, true); + if (validList.valid) { + assert.equal( + validList.normalized, + "203.0.113.10, 10.0.0.0/8, 192.0.2.5, 2001:db8::/32", + ); + } + + // Invalid IP or CIDR prefix rejected + const badCidr = validateAllowedIpsInput("10.0.0.0/33"); + assert.equal(badCidr.valid, false); + const badIp = validateAllowedIpsInput("999.888.777.666"); + assert.equal(badIp.valid, false); + + // Matching behavior + assert.equal(isClientIpAllowed("198.51.100.77", ""), true); + assert.equal(isClientIpAllowed("198.51.100.77", "*"), true); + assert.equal( + isClientIpAllowed("203.0.113.10", "203.0.113.10, 10.0.0.0/8"), + true, + ); + assert.equal( + isClientIpAllowed("::ffff:203.0.113.10", "203.0.113.10, 10.0.0.0/8"), + true, + ); + assert.equal( + isClientIpAllowed("10.24.99.1", "203.0.113.10, 10.0.0.0/8"), + true, + ); + assert.equal( + isClientIpAllowed("192.168.1.50", "203.0.113.10, 10.0.0.0/8"), + false, + ); + // Loopback equivalence between 127.0.0.1 and ::1 + assert.equal(isClientIpAllowed("::1", "127.0.0.1"), true); + assert.equal(isClientIpAllowed("127.0.0.1", "::1"), true); + + // Expiration validation & evaluation + assert.deepEqual(validateClientExpiresAtInput(null), { + valid: true, + normalized: null, + }); + assert.deepEqual(validateClientExpiresAtInput(""), { + valid: true, + normalized: null, + }); + const validExp = validateClientExpiresAtInput("2030-06-15T12:00:00.000Z"); + assert.equal(validExp.valid, true); + if (validExp.valid) { + assert.equal(validExp.normalized, "2030-06-15T12:00:00.000Z"); + } + const badExp = validateClientExpiresAtInput("not-a-timestamp"); + assert.equal(badExp.valid, false); + + const nowMs = Date.parse("2026-11-19T12:00:00.000Z"); + assert.equal(isClientTokenExpired(null, nowMs), false); + assert.equal(isClientTokenExpired("2026-11-19T12:00:01.000Z", nowMs), false); + assert.equal(isClientTokenExpired("2026-11-19T12:00:00.000Z", nowMs), true); + assert.equal(isClientTokenExpired("2026-11-19T11:59:59.000Z", nowMs), true); +}); + +test("V2-Phase-02 [Webhook Notifier & /api/settings/webhook-test]: HMAC-SHA256 signing, cooldown suppression & V2-E06 error handling", async () => { + clearWebhookCooldownCacheForTests(); + const receivedRequests: Array<{ + headers: http.IncomingHttpHeaders; + body: string; + json: Record; + }> = []; + let responseStatus = 200; + + const mockWebhookServer = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk) => chunks.push(Buffer.from(chunk))); + req.on("end", () => { + const bodyStr = Buffer.concat(chunks).toString("utf8"); + let parsed: Record = {}; + try { + parsed = JSON.parse(bodyStr); + } catch { + // ignore + } + receivedRequests.push({ + headers: req.headers, + body: bodyStr, + json: parsed, + }); + res.writeHead(responseStatus, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ ok: responseStatus < 400 })); + }); + }); + + await new Promise((resolve) => mockWebhookServer.listen(0, "127.0.0.1", resolve)); + const port = (mockWebhookServer.address() as AddressInfo).port; + const webhookUrl = `http://127.0.0.1:${port}/hooks/datastack`; + + try { + // 1. Format & signature verification + const formatted = formatOutboundWebhookBody({ + event: "DISK_SPACE_LOW", + severity: "critical", + title: "Disk Threshold Reached", + message: "VPS disk is at 95.4%.", + clientId: "svc_alpha", + timestamp: "2026-11-19T12:00:00.000Z", + details: { usedPct: 95.4 }, + }); + assert.ok(formatted.text.includes("DISK_SPACE_LOW")); + assert.equal(formatted.content, formatted.text); + + const bodyJson = JSON.stringify(formatted); + const sig = buildWebhookSignature(bodyJson, "webhook_secret_xyz"); + assert.ok(sig.startsWith("sha256=")); + + // 2. Cooldown deduplication test + const first = await dispatchWebhookAlertAsync( + { + enabled: true, + url: webhookUrl, + secret: "webhook_secret_xyz", + cooldownSec: 120, + }, + { + event: "CLIENT_QUOTA_EXCEEDED", + severity: "warning", + title: "Client Quota Exceeded", + message: "Quota reached for svc_alpha", + clientId: "svc_alpha", + }, + ); + assert.equal(first.ok, true); + assert.equal(receivedRequests.length, 1); + assert.equal( + receivedRequests[0].headers["x-datastack-event"], + "CLIENT_QUOTA_EXCEEDED", + ); + assert.equal( + receivedRequests[0].headers["x-datastack-signature"], + buildWebhookSignature(receivedRequests[0].body, "webhook_secret_xyz"), + ); + + // Duplicate event for same client within cooldown window is suppressed + const second = await dispatchWebhookAlertAsync( + { + enabled: true, + url: webhookUrl, + secret: "webhook_secret_xyz", + cooldownSec: 120, + }, + { + event: "CLIENT_QUOTA_EXCEEDED", + severity: "warning", + title: "Client Quota Exceeded", + message: "Quota reached again for svc_alpha", + clientId: "svc_alpha", + }, + ); + assert.equal(second.ok, true); + assert.equal(second.suppressedByCooldown, true); + assert.equal(receivedRequests.length, 1); + + // 3. Test POST /api/settings/webhook-test route (200 & 502 WEBHOOK_DELIVERY_FAILED V2-E06) + const adminCookie = await makeAdminCookie(); + const okTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", { + method: "POST", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + webhookUrl, + webhookSecret: "test_route_hmac_secret", + }), + }); + const okTestRes = await webhookTestPost(okTestReq); + assert.equal(okTestRes.status, 200); + const okTestData = await okTestRes.json(); + assert.equal(okTestData.success, true); + assert.equal(receivedRequests.length, 2); + assert.equal(receivedRequests[1].headers["x-datastack-event"], "TEST_ALERT"); + + // Simulate upstream 500 error -> 502 WEBHOOK_DELIVERY_FAILED (V2-E06) + responseStatus = 500; + const failTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", { + method: "POST", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + webhookUrl, + }), + }); + const failTestRes = await webhookTestPost(failTestReq); + assert.equal(failTestRes.status, 502); + const failTestData = await failTestRes.json(); + assert.equal(failTestData.code, "WEBHOOK_DELIVERY_FAILED"); + } finally { + await new Promise((resolve) => mockWebhookServer.close(() => resolve())); + } +}); + +test("V2-Phase-02 [HTTP & Socket Ingestion Security]: Per-client IP allowlist (403 V2-E07) & Token expiration (401 V2-E08)", async () => { + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-sec-")); + const prevStorageDir = process.env.DATASTACK_STORAGE_DIR; + process.env.DATASTACK_STORAGE_DIR = tmpDir; + CSVStorageManager.resetForTests(tmpDir); + await CSVStorageManager.initializeAsync(); + clearWebhookCooldownCacheForTests(); + + try { + const adminCookie = await makeAdminCookie(); + + // 1. Create client with strict IP allowlist (only 10.10.0.0/16) and future expiration + const futureIso = new Date(Date.now() + 3600_000).toISOString(); + const createRes = await clientsPost( + new NextRequest("http://localhost:3000/api/clients", { + method: "POST", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + name: "IP Restricted Client", + rateLimit: 100, + storageQuotaMb: 0, + allowedIps: "10.10.0.0/16", + expiresAt: futureIso, + }), + }), + ); + assert.equal(createRes.status, 200); + const createData = await createRes.json(); + const client = createData.client; + assert.equal(client.allowedIps, "10.10.0.0/16"); + assert.equal(client.expiresAt, futureIso); + + // 2. Request from forbidden IP 192.168.1.25 -> 403 CLIENT_IP_FORBIDDEN (V2-E07) + const forbiddenIpRes = await eventsPost( + new NextRequest("http://localhost:3000/api/events", { + method: "POST", + headers: { + authorization: `Bearer ${client.token}`, + "content-type": "application/json", + "x-forwarded-for": "192.168.1.25", + }, + body: JSON.stringify({ + eventId: "evt_ip_denied", + payload: { step: 1 }, + }), + }), + ); + assert.equal(forbiddenIpRes.status, 403); + const forbiddenIpJson = await forbiddenIpRes.json(); + assert.equal(forbiddenIpJson.code, "CLIENT_IP_FORBIDDEN"); + + // 3. Request from allowed CIDR IP 10.10.42.9 -> 200 OK + const allowedIpRes = await eventsPost( + new NextRequest("http://localhost:3000/api/events", { + method: "POST", + headers: { + authorization: `Bearer ${client.token}`, + "content-type": "application/json", + "x-forwarded-for": "10.10.42.9", + }, + body: JSON.stringify({ + eventId: "evt_ip_allowed", + payload: { step: 2 }, + }), + }), + ); + assert.equal(allowedIpRes.status, 200); + + // 4. Patch client with past expiresAt -> 401 CLIENT_TOKEN_EXPIRED (V2-E08) + const pastIso = new Date(Date.now() - 60_000).toISOString(); + const patchRes = await patchClientRoute( + new NextRequest( + `http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`, + { + method: "PATCH", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + expiresAt: pastIso, + }), + }, + ), + { params: Promise.resolve({ id: client.clientId }) }, + ); + assert.equal(patchRes.status, 200); + const patchData = await patchRes.json(); + assert.equal(patchData.client.expiresAt, pastIso); + + const expiredRes = await eventsPost( + new NextRequest("http://localhost:3000/api/events", { + method: "POST", + headers: { + authorization: `Bearer ${client.token}`, + "content-type": "application/json", + "x-forwarded-for": "10.10.42.9", + }, + body: JSON.stringify({ + eventId: "evt_expired", + payload: { step: 3 }, + }), + }), + ); + assert.equal(expiredRes.status, 401); + const expiredJson = await expiredRes.json(); + assert.equal(expiredJson.code, "CLIENT_TOKEN_EXPIRED"); + + // 5. Verify Port 4000 Socket.IO server enforces IP allowlist & token expiry + process.env.DATASTACK_DISABLE_LISTEN = "1"; + const ingestionApp = require("../server/index.js"); + + // Sync expired client to Port 4000 in-memory cache and verify verifyClientInDb returns CLIENT_TOKEN_EXPIRED + await new Promise((resolve) => + ingestionApp.server.listen(0, "127.0.0.1", () => resolve()), + ); + const port4000 = (ingestionApp.server.address() as AddressInfo).port; + + try { + const expiredCheck = await ingestionApp.verifyClientInDb( + client.clientId, + client.token, + true, + "10.10.42.9", + ); + assert.equal(expiredCheck.ok, false); + assert.equal(expiredCheck.reason, "CLIENT_TOKEN_EXPIRED"); + + // Clear expiration via PATCH, keep IP allowlist 10.10.0.0/16, and check Socket.IO handshake from 127.0.0.1 + await patchClientRoute( + new NextRequest( + `http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`, + { + method: "PATCH", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + expiresAt: null, + }), + }, + ), + { params: Promise.resolve({ id: client.clientId }) }, + ); + + const ipForbiddenCheck = await ingestionApp.verifyClientInDb( + client.clientId, + client.token, + true, + "127.0.0.1", + ); + assert.equal(ipForbiddenCheck.ok, false); + assert.equal(ipForbiddenCheck.reason, "CLIENT_IP_FORBIDDEN"); + + // Socket.IO connection from 127.0.0.1 should fail handshake with CLIENT_IP_FORBIDDEN + const socketError = await new Promise((resolve) => { + const sock: Socket = ioClient(`http://127.0.0.1:${port4000}`, { + auth: { clientId: client.clientId, token: client.token }, + transports: ["websocket"], + reconnection: false, + }); + sock.on("connect_error", (err) => { + sock.close(); + resolve(err.message); + }); + sock.on("connect", () => { + sock.close(); + resolve("CONNECTED_UNEXPECTEDLY"); + }); + }); + assert.ok(socketError.includes("CLIENT_IP_FORBIDDEN")); + } finally { + await new Promise((resolve) => + ingestionApp.server.close(() => resolve()), + ); + } + } finally { + await DataBufferEngine.flushAllAsync(); + SDKManager.stopActivityFlushTimerForTests(); + if (prevStorageDir === undefined) { + delete process.env.DATASTACK_STORAGE_DIR; + } else { + process.env.DATASTACK_STORAGE_DIR = prevStorageDir; + } + CSVStorageManager.resetForTests(); + await fs.rm(tmpDir, { recursive: true, force: true }); + } +}); + +test("V2-Phase-02 [Prometheus Metrics & NDJSON Streaming Export]: GET /api/prometheus (V2-FEAT-04 / V2-E09) & GET /api/export?format=ndjson (V2-FEAT-06)", async () => { + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-prom-")); + const prevStorageDir = process.env.DATASTACK_STORAGE_DIR; + process.env.DATASTACK_STORAGE_DIR = tmpDir; + CSVStorageManager.resetForTests(tmpDir); + await CSVStorageManager.initializeAsync(); + + try { + const adminCookie = await makeAdminCookie(); + + // 1. Unauthenticated scrape when Prometheus is disabled -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09) + const unauthPromRes = await prometheusGet( + new NextRequest("http://localhost:3000/api/prometheus"), + ); + assert.equal(unauthPromRes.status, 401); + const unauthPromJson = await unauthPromRes.json(); + assert.equal(unauthPromJson.code, "PROMETHEUS_UNAUTHORIZED"); + + // 2. Configure Webhook & Prometheus settings via POST /api/settings + const saveSettingsRes = await settingsPost( + new NextRequest("http://localhost:3000/api/settings", { + method: "POST", + headers: { + cookie: adminCookie, + "content-type": "application/json", + }, + body: JSON.stringify({ + enableWebhookAlerts: true, + webhookUrl: "https://hooks.example.com/datastack", + webhookSecret: "super_secret_webhook_hmac_9988", + webhookCooldownSec: 180, + enablePrometheusMetrics: true, + prometheusMetricsToken: "prom_scrape_bearer_token_7766", + }), + }), + ); + assert.equal(saveSettingsRes.status, 200); + const savedData = await saveSettingsRes.json(); + assert.equal(savedData.settings.enableWebhookAlerts, true); + assert.equal(savedData.settings.webhookCooldownSec, 180); + assert.equal(savedData.settings.hasWebhookSecret, true); + assert.equal(savedData.settings.webhookSecretMask, "••••9988"); + assert.equal("webhookSecretEnc" in savedData.settings, false); + assert.equal(savedData.settings.enablePrometheusMetrics, true); + assert.equal(savedData.settings.hasPrometheusToken, true); + assert.equal(savedData.settings.prometheusTokenMask, "••••7766"); + assert.equal("prometheusMetricsTokenEnc" in savedData.settings, false); + + // 3. Wrong Bearer token -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09) + const wrongTokenRes = await prometheusGet( + new NextRequest("http://localhost:3000/api/prometheus", { + headers: { + authorization: "Bearer wrong_token_value", + }, + }), + ); + assert.equal(wrongTokenRes.status, 401); + + // 4. Valid Bearer token -> 200 Prometheus text exposition format + const validTokenRes = await prometheusGet( + new NextRequest("http://localhost:3000/api/prometheus", { + headers: { + authorization: "Bearer prom_scrape_bearer_token_7766", + }, + }), + ); + assert.equal(validTokenRes.status, 200); + assert.ok( + validTokenRes.headers + .get("content-type") + ?.includes("text/plain; version=0.0.4"), + ); + const metricsBody = await validTokenRes.text(); + assert.ok(metricsBody.includes("datastack_up 1")); + assert.ok(metricsBody.includes("datastack_events_total")); + assert.ok(metricsBody.includes("datastack_storage_bytes_total")); + assert.ok(metricsBody.includes("datastack_disk_used_percent")); + + // 5. Seed events and export as streaming NDJSON (GET /api/export?format=ndjson) (V2-FEAT-06) + await CSVStorageManager.appendBatchAsync("client_ndjson", [ + { + clientId: "client_ndjson", + eventId: "evt_nd_1", + timestamp: "2026-11-19T10:00:00.000Z", + payload: JSON.stringify({ action: "purchase", amount: 49 }), + }, + { + clientId: "client_ndjson", + eventId: "evt_nd_2", + timestamp: "2026-11-19T10:01:00.000Z", + payload: JSON.stringify({ action: "refund", amount: 12 }), + }, + ]); + + const ndjsonRes = await exportGet( + new NextRequest( + "http://localhost:3000/api/export?format=ndjson&clientId=client_ndjson", + { + headers: { cookie: adminCookie }, + }, + ), + ); + assert.equal(ndjsonRes.status, 200); + assert.ok( + ndjsonRes.headers + .get("content-type") + ?.includes("application/x-ndjson"), + ); + assert.ok( + ndjsonRes.headers + .get("content-disposition") + ?.includes("datastack-client_ndjson.ndjson"), + ); + const ndjsonText = await ndjsonRes.text(); + const lines = ndjsonText + .trim() + .split("\n") + .map((line) => JSON.parse(line)); + assert.equal(lines.length, 2); + assert.equal(lines[0].clientId, "client_ndjson"); + assert.equal(lines[0].eventId, "evt_nd_1"); + assert.deepEqual(lines[0].payload, { action: "purchase", amount: 49 }); + assert.equal(lines[1].eventId, "evt_nd_2"); + assert.deepEqual(lines[1].payload, { action: "refund", amount: 12 }); + + // 6. Verify UI helper validation for webhookCooldownSec & normalizeAlertingSettings + assert.equal( + validateSingleNumericSetting("webhookCooldownSec", "5").valid, + false, + ); + assert.equal( + validateSingleNumericSetting("webhookCooldownSec", "300").valid, + true, + ); + const drafts = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS); + const validatedDraft = validateEditableSettingsDraft(drafts, true); + assert.equal(validatedDraft.valid, true); + assert.deepEqual(validatedDraft.values, DEFAULT_EDITABLE_SETTINGS); + assert.deepEqual( + normalizeAlertingSettings(undefined), + DEFAULT_ALERTING_SETTINGS, + ); + } finally { + await DataBufferEngine.flushAllAsync(); + SDKManager.stopActivityFlushTimerForTests(); + await SettingsManager.updateSettingsAsync({ + enableWebhookAlerts: false, + webhookUrl: "", + clearWebhookSecret: true, + webhookCooldownSec: 300, + enablePrometheusMetrics: false, + clearPrometheusToken: true, + }); + if (prevStorageDir === undefined) { + delete process.env.DATASTACK_STORAGE_DIR; + } else { + process.env.DATASTACK_STORAGE_DIR = prevStorageDir; + } + CSVStorageManager.resetForTests(); + await fs.rm(tmpDir, { recursive: true, force: true }); + } +});