feat(client): implement per-client storage and security controls

- Added storage quota, IP allowlisting, and expiration logic to client management APIs.
- Updated CSV storage manager to support auto-compression and improved partition file parsing.
- Integrated security checks into event ingestion and data retrieval routes.
This commit is contained in:
Team Dark Devil committed 2026-11-19 23:02:27 +06:00
1 parent 89e4d68690
commit 2873b9904a
32 files changed
+6001 -167

No files matched your search

+16 -1
View File
@@ -1,7 +1,7 @@
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory # MEMORY.md — DataStack Cross-IDE Persistent Project Memory
> **Last Synchronized**: 2026-11-19 > **Last Synchronized**: 2026-11-19
> **Status**: **Production & Deployment Hardening Complete (`BASELINE-2026-11-19-R2` + Deployment Hardening) — `4 / 4` R2 Phases (`18/18` bugs) + `5/5` Docker/Coolify Redeploy Fixes (`DEPLOY-BUG-01`–`05`) Verified (`85/85` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).** > **Status**: **V2-Phase-01 Complete & Verified (`BASELINE-2026-11-19-V2`) — `1 / 4` V2 Phases Done (Storage `.csv.gz` Auto-Compression, S3/R2 Off-Site Archive with AES-256-GCM Secret Vault, Per-Client `storageQuotaMb` & VPS Disk Pause Guard `HTTP 507`) — Verified (`89/89` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`). > **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
--- ---
@@ -300,6 +300,21 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **
- **`DEPLOY-BUG-05`**: Clarified Coolify `Base Directory: /` and `Docker Compose Location: /docker/docker-compose.yml` in `docs/users/installation.md` and `README.md`. - **`DEPLOY-BUG-05`**: Clarified Coolify `Base Directory: /` and `Docker Compose Location: /docker/docker-compose.yml` in `docs/users/installation.md` and `README.md`.
- Verified: `85/85` full TS suite pass (`11/11` in `tests/phase07-deployment.test.ts`), `10/10` Python SDK tests pass, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings). - Verified: `85/85` full TS suite pass (`11/11` in `tests/phase07-deployment.test.ts`), `10/10` Python SDK tests pass, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings).
- **V2 Baseline Freeze & Governance Planning Complete (`BASELINE-2026-11-19-V2`)**:
- Froze the clean, Deployment-Hardened runtime state (`85/85` TS tests, `10/10` Python tests, clean `npm run build` & `eslint .`, zero runtime code changes) as **`BASELINE-2026-11-19-V2`**.
- Created raw-source verified V2 planning & contract documents in `notes/`:
- `notes/V2_FEATURE_GAP_AUDIT.md` (`V2-FEAT-01`–`V2-FEAT-07` gap analysis + full `/vcon/*` control mapping).
- `notes/V2_DATABASE_AND_SETTINGS_CONTRACT.md` (zero-loss additive PostgreSQL DDL rules preserving `admin_users` `id = 1 CHECK (id = 1)`, AES-256-GCM DB secret encryption + masked `GET /api/settings`, Port `3000`↔`4000` live sync).
- `notes/BASELINE_FREEZE.md`, `notes/PRODUCTION_READINESS_SCOPE_LOCK.md` (Doc `001` Scope Locks for `V2-Phase-01`–`04`), `notes/PRODUCTION_ROADMAP.md` (Doc `002`), `notes/PHASE_EXECUTION_LOG.md` (Doc `003`, `0 / 4` V2 phases done), `notes/ERROR_HANDLING_MATRIX.md` (Doc `004`, including `V2-E01`–`V2-E12`), `notes/ACTUAL_IMPLEMENTATION_STATUS.md`, and `notes/README.md`.
- Roadmap grouped into 4 Scope-Locked V2 Phases (`V2-Phase-01` Archival/Gzip + Quota/Disk Guard; `V2-Phase-02` Webhooks/Alerts + IP Allowlist/Expiry + Prometheus/NDJSON; `V2-Phase-03` Multi-Admin RBAC + Audit Log; `V2-Phase-04` Node.js/TypeScript SDK + `/vcon/sdk` guide).
- **`V2-Phase-01` Complete & Verified (`2026-11-19`)** (`V2-FEAT-01`, `V2-FEAT-03`, `V2-E01`–`V2-E05`):
- **AES-256-GCM Secret Vault (`core/Security/secretVault.ts`)**: Deterministic HKDF-SHA256 key derivation (`encryptSecret`, `decryptSecret`, `getMaskedHintFromEncrypted`, `SecretVaultError`) so `s3_secret_access_key_enc` is encrypted at rest in PostgreSQL and never leaked in plaintext by `GET /api/settings` (`hasS3SecretKey` + `s3SecretKeyMask: "••••xxxx"`).
- **S3 / Cloudflare R2 / MinIO SigV4 Client (`core/Storage/s3Archive.ts`)**: Pure Node `node:crypto` + `fetch` AWS Signature v4 client (`testS3Connection`, `uploadFileToS3`, `buildS3ObjectKey`, `normalizeS3EndpointUrl`) with `POST /api/storage/s3-test` and `POST /api/storage/archive`.
- **Transparent `.csv.gz` Auto-Compression & Safe Archive-Before-Prune (`core/Storage/csvStorage.ts`, `server/workers/csvWriter.js`)**: Cooperatively locked atomic `.csv` → `.csv.gz` compression (`compressOldPartitionsAsync` / `compressOldPartitions`) excluding today's active UTC partition; transparent `.csv` + `.csv.gz` reading/streaming across `GET /api/data` (surfacing `500 Corrupt .csv.gz partition` on corrupt gzip — `V2-E03`), `GET /api/events`, and `GET /api/export` (`application/gzip` for raw single `.csv.gz` download); fail-safe `beforeDeleteFile` S3 upload hook (`V2-E02`) that never deletes local partitions if S3 upload fails (`502 S3_ARCHIVE_FAILED`).
- **Per-Client Storage Quota (`storageQuotaMb`) & VPS Disk Pause Guard (`diskPauseThresholdPct`)**: Additive PostgreSQL columns (`client_sdks.storage_quota_mb` + 9 `admin_settings` columns in `core/Database/database.ts`, synced in `server/index.js` while preserving `ddlCount === 8`), `30s` TTL cached `statfs` + directory byte counters, and HTTP `507 Insufficient Storage` (`DISK_SPACE_LOW` / `STORAGE_QUOTA_EXCEEDED` — `V2-E04`, `V2-E05`) on `POST /api/events` and Port `4000` `sync_data`/`batch_sync` while keeping all Admin read/export/settings routes online.
- **Admin UI Controls (`SettingsWorkspace.tsx`, `NewClientModal.tsx`, `ClientsTable.tsx`, `lib/vconUiHelpers.ts`)**: Added live VPS disk usage bar, `Auto-compress (.csv → .csv.gz)`, `Disk pause threshold (0 or 50–99%)`, full `S3 / R2 off-site archive` configuration with `Test S3 Connection` & `Archive Now`, and per-client `Storage quota (MB, 0 = unlimited)` in modal and table.
- **Verified**: `89/89` TS tests (`4/4` in `tests/v2-phase01-storage-quotas.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/settings` (Storage tab) and `/vcon/clients` (`0` console errors).
--- ---
## 5. Known Constraints & Operational Notes for Future Contributors ## 5. Known Constraints & Operational Notes for Future Contributors
+61 -2
View File
@@ -1,9 +1,21 @@
import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
import { SDKManager } from "@/core/sdk/sdkManager"; import { SDKManager } from "@/core/sdk/sdkManager";
import {
validateAllowedIpsInput,
validateClientExpiresAtInput,
} from "@/core/Security/ipAllowlist";
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
const SESSION_COOKIE = "datastack_admin_session"; const SESSION_COOKIE = "datastack_admin_session";
const ALLOWED_PATCH_KEYS = new Set(["action", "name", "rateLimit", "status"]); const ALLOWED_PATCH_KEYS = new Set([
"action",
"name",
"rateLimit",
"storageQuotaMb",
"allowedIps",
"expiresAt",
"status",
]);
export async function GET( export async function GET(
req: NextRequest, req: NextRequest,
@@ -125,7 +137,14 @@ export async function PATCH(
} }
} }
const updates: { name?: string; rateLimit?: number; status?: "active" | "revoked" } = {}; const updates: {
name?: string;
rateLimit?: number;
storageQuotaMb?: number;
allowedIps?: string;
expiresAt?: string | null;
status?: "active" | "revoked";
} = {};
if (body.name !== undefined) { if (body.name !== undefined) {
if ( if (
typeof body.name !== "string" || typeof body.name !== "string" ||
@@ -153,6 +172,46 @@ export async function PATCH(
} }
updates.rateLimit = body.rateLimit; updates.rateLimit = body.rateLimit;
} }
if (body.storageQuotaMb !== undefined) {
if (
typeof body.storageQuotaMb !== "number" ||
!Number.isInteger(body.storageQuotaMb) ||
body.storageQuotaMb < 0 ||
body.storageQuotaMb > 1048576
) {
return NextResponse.json(
{ error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
updates.storageQuotaMb = body.storageQuotaMb;
}
if (body.allowedIps !== undefined) {
if (typeof body.allowedIps !== "string") {
return NextResponse.json(
{ error: "allowedIps must be a string.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
const ipValidation = validateAllowedIpsInput(body.allowedIps);
if (!ipValidation.valid) {
return NextResponse.json(
{ error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
updates.allowedIps = ipValidation.normalized;
}
if (body.expiresAt !== undefined) {
const expValidation = validateClientExpiresAtInput(body.expiresAt);
if (!expValidation.valid) {
return NextResponse.json(
{ error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
updates.expiresAt = expValidation.normalized;
}
if (body.status !== undefined) { if (body.status !== undefined) {
if (body.status !== "active" && body.status !== "revoked") { if (body.status !== "active" && body.status !== "revoked") {
return NextResponse.json( return NextResponse.json(
+53 -1
View File
@@ -1,5 +1,9 @@
import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
import { SDKManager } from "@/core/sdk/sdkManager"; import { SDKManager } from "@/core/sdk/sdkManager";
import {
validateAllowedIpsInput,
validateClientExpiresAtInput,
} from "@/core/Security/ipAllowlist";
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
const SESSION_COOKIE = "datastack_admin_session"; const SESSION_COOKIE = "datastack_admin_session";
@@ -58,7 +62,14 @@ export async function POST(req: NextRequest) {
} }
try { try {
const { clientId, name, rateLimit } = body; const {
clientId,
name,
rateLimit,
storageQuotaMb,
allowedIps,
expiresAt,
} = body;
if (typeof clientId !== "string" || typeof name !== "string" || !clientId.trim() || !name.trim()) { if (typeof clientId !== "string" || typeof name !== "string" || !clientId.trim() || !name.trim()) {
return NextResponse.json( return NextResponse.json(
@@ -83,6 +94,44 @@ export async function POST(req: NextRequest) {
); );
} }
const parsedQuotaMb = storageQuotaMb === undefined ? 0 : Number(storageQuotaMb);
if (!Number.isInteger(parsedQuotaMb) || parsedQuotaMb < 0 || parsedQuotaMb > 1048576) {
return NextResponse.json(
{ error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
let normalizedAllowedIps = "";
if (allowedIps !== undefined && allowedIps !== null) {
if (typeof allowedIps !== "string") {
return NextResponse.json(
{ error: "allowedIps must be a string.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
const ipValidation = validateAllowedIpsInput(allowedIps);
if (!ipValidation.valid) {
return NextResponse.json(
{ error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
normalizedAllowedIps = ipValidation.normalized;
}
let normalizedExpiresAt: string | null = null;
if (expiresAt !== undefined) {
const expValidation = validateClientExpiresAtInput(expiresAt);
if (!expValidation.valid) {
return NextResponse.json(
{ error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" },
{ status: 400 },
);
}
normalizedExpiresAt = expValidation.normalized;
}
const existing = await SDKManager.getClientAsync(safeClientId); const existing = await SDKManager.getClientAsync(safeClientId);
if (existing) { if (existing) {
return NextResponse.json( return NextResponse.json(
@@ -95,6 +144,9 @@ export async function POST(req: NextRequest) {
clientId: safeClientId, clientId: safeClientId,
name: name.trim().slice(0, 120), name: name.trim().slice(0, 120),
rateLimit: parsedRateLimit, rateLimit: parsedRateLimit,
storageQuotaMb: parsedQuotaMb,
allowedIps: normalizedAllowedIps,
expiresAt: normalizedExpiresAt,
}); });
return NextResponse.json({ client: newClient }, { status: 201 }); return NextResponse.json({ client: newClient }, { status: 201 });
+12 -6
View File
@@ -6,7 +6,6 @@ import { CSVStorageManager } from "@/core/Storage/csvStorage";
import { requireAdminApiSession } from "@/lib/adminAuth"; import { requireAdminApiSession } from "@/lib/adminAuth";
import { parse } from "csv-parse"; import { parse } from "csv-parse";
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
import { createReadStream } from "node:fs";
import { lstat, readdir } from "node:fs/promises"; import { lstat, readdir } from "node:fs/promises";
import path from "node:path"; import path from "node:path";
@@ -135,10 +134,10 @@ export async function GET(request: NextRequest) {
try { try {
filenames = (await readdir(clientDir)) filenames = (await readdir(clientDir))
.filter((filename) => { .filter((filename) => {
if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?\.csv$/.test(filename)) { if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?(?:-gz\d+)?\.csv(?:\.gz)?$/.test(filename)) {
return false; return false;
} }
const fileDate = filename.slice(0, 10); const fileDate = CSVStorageManager.extractDateFromFilename(filename);
if ((from && fileDate < from) || (to && fileDate > to)) { if ((from && fileDate < from) || (to && fileDate > to)) {
return false; return false;
} }
@@ -150,7 +149,7 @@ export async function GET(request: NextRequest) {
} }
for (const filename of filenames) { for (const filename of filenames) {
const fileDate = filename.slice(0, 10); const fileDate = CSVStorageManager.extractDateFromFilename(filename);
const filePath = path.join(clientDir, filename); const filePath = path.join(clientDir, filename);
let fileLstat; let fileLstat;
@@ -161,10 +160,13 @@ export async function GET(request: NextRequest) {
} }
if (fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue; if (fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue;
const fileStream = createReadStream(filePath); const fileStream = CSVStorageManager.openPartitionTextStream(filePath);
const parser = fileStream.pipe( const parser = fileStream.pipe(
parse({ columns: true, skip_empty_lines: true, bom: true }), parse({ columns: true, skip_empty_lines: true, bom: true }),
); );
fileStream.on("error", (err) => {
if (!parser.destroyed) parser.destroy(err);
});
try { try {
for await (const record of parser as AsyncIterable< for await (const record of parser as AsyncIterable<
@@ -180,7 +182,11 @@ export async function GET(request: NextRequest) {
if (event.eventId) addIfMatching(event); if (event.eventId) addIfMatching(event);
} }
} catch (err) { } catch (err) {
console.warn(`[API /data] Failed reading CSV partition ${filePath}:`, err); console.warn(`[API /data] Failed reading partition ${filePath}:`, err);
if (filename.endsWith(".csv.gz")) {
const detail = err instanceof Error ? err.message : String(err);
throw new Error(`Corrupt .csv.gz partition (${filename}): ${detail}`);
}
} finally { } finally {
if (!parser.destroyed) parser.destroy(); if (!parser.destroyed) parser.destroy();
if (!fileStream.destroyed) fileStream.destroy(); if (!fileStream.destroyed) fileStream.destroy();
+81 -4
View File
@@ -3,6 +3,10 @@ import { verifyRequestAdminSession } from '@/core/auth/adminUsers';
import { AuthService } from '@/core/auth/jwt'; import { AuthService } from '@/core/auth/jwt';
import { DataBufferEngine, REDIS_BUFFER_KEY } from '@/core/data/buffer'; import { DataBufferEngine, REDIS_BUFFER_KEY } from '@/core/data/buffer';
import { SDKManager } from '@/core/sdk/sdkManager'; import { SDKManager } from '@/core/sdk/sdkManager';
import {
isClientIpAllowed,
isClientTokenExpired,
} from '@/core/Security/ipAllowlist';
import { import {
SecurityEngine, SecurityEngine,
getInternalIngestionHeaders, getInternalIngestionHeaders,
@@ -15,8 +19,7 @@ import {
import { parse } from 'csv-parse'; import { parse } from 'csv-parse';
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import crypto from 'node:crypto'; import crypto from 'node:crypto';
import { createReadStream } from 'node:fs'; import { extractClientIp, requireAdminApiSession } from '@/lib/adminAuth';
import { requireAdminApiSession } from '@/lib/adminAuth';
import type { BufferedBucket, BufferedSummary } from '@/lib/vconUiHelpers'; import type { BufferedBucket, BufferedSummary } from '@/lib/vconUiHelpers';
const SESSION_COOKIE = 'datastack_admin_session'; const SESSION_COOKIE = 'datastack_admin_session';
@@ -233,10 +236,13 @@ async function getRecentStoredEvents(
for (const file of files) { for (const file of files) {
const fileTail: DataEvent[] = []; const fileTail: DataEvent[] = [];
const fileStream = createReadStream(file.filePath); const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath);
const parser = fileStream.pipe( const parser = fileStream.pipe(
parse({ columns: true, skip_empty_lines: true, bom: true }), parse({ columns: true, skip_empty_lines: true, bom: true }),
); );
fileStream.on('error', (err) => {
if (!parser.destroyed) parser.destroy(err);
});
try { try {
for await (const record of parser as AsyncIterable< for await (const record of parser as AsyncIterable<
@@ -254,7 +260,7 @@ async function getRecentStoredEvents(
}); });
} }
} catch (err) { } catch (err) {
console.warn(`[API /events] Failed reading CSV partition ${file.filePath}:`, err); console.warn(`[API /events] Failed reading partition ${file.filePath}:`, err);
} finally { } finally {
if (!parser.destroyed) parser.destroy(); if (!parser.destroyed) parser.destroy();
if (!fileStream.destroyed) fileStream.destroy(); if (!fileStream.destroyed) fileStream.destroy();
@@ -476,6 +482,77 @@ export async function POST(req: NextRequest) {
{ status: 401 }, { status: 401 },
); );
} }
if (isClientTokenExpired(registeredClient.expiresAt)) {
SettingsManager.triggerWebhookAlert({
event: 'CLIENT_TOKEN_EXPIRED',
severity: 'warning',
title: 'Expired Client Token Rejected',
message: `Client '${clientId}' attempted HTTP ingestion with an expired token (expiresAt=${registeredClient.expiresAt}).`,
clientId,
details: { expiresAt: registeredClient.expiresAt },
});
return NextResponse.json(
{
error: `CLIENT_TOKEN_EXPIRED: Client '${clientId}' token expired at ${registeredClient.expiresAt}. Rotate or extend client expiration.`,
code: 'CLIENT_TOKEN_EXPIRED',
},
{ status: 401 },
);
}
if (registeredClient.allowedIps && registeredClient.allowedIps.trim()) {
const clientIp = extractClientIp(req);
if (!isClientIpAllowed(clientIp, registeredClient.allowedIps)) {
SettingsManager.triggerWebhookAlert({
event: 'CLIENT_IP_FORBIDDEN',
severity: 'warning',
title: 'Unauthorized Client IP Blocked',
message: `Client '${clientId}' request from IP '${clientIp}' was blocked by allowedIps policy.`,
clientId,
details: {
clientIp,
allowedIps: registeredClient.allowedIps,
},
});
return NextResponse.json(
{
error: `CLIENT_IP_FORBIDDEN: Request IP '${clientIp}' is not authorized for client '${clientId}'.`,
code: 'CLIENT_IP_FORBIDDEN',
},
{ status: 403 },
);
}
}
}
const storageAdmission = await CSVStorageManager.checkStorageAdmissionAsync(
clientId,
registeredClient.storageQuotaMb,
settings.diskPauseThresholdPct,
);
if (!storageAdmission.allowed) {
if (storageAdmission.code) {
SettingsManager.triggerWebhookAlert({
event: storageAdmission.code,
severity:
storageAdmission.code === 'DISK_SPACE_LOW' ? 'critical' : 'warning',
title:
storageAdmission.code === 'DISK_SPACE_LOW'
? 'VPS Disk Pause Threshold Reached'
: 'Client Storage Quota Exceeded',
message:
storageAdmission.reason ||
`Storage admission paused (${storageAdmission.code}) for client '${clientId}'.`,
clientId,
});
}
return NextResponse.json(
{
success: false,
error: storageAdmission.reason,
code: storageAdmission.code,
},
{ status: 507 },
);
} }
const effectiveClientRate = const effectiveClientRate =
+42 -14
View File
@@ -12,7 +12,7 @@ import { NextRequest, NextResponse } from "next/server";
import { createReadStream } from "node:fs"; import { createReadStream } from "node:fs";
import path from "node:path"; import path from "node:path";
type ExportFormat = "csv" | "json"; type ExportFormat = "csv" | "json" | "ndjson";
type CsvRecord = Record<string, string>; type CsvRecord = Record<string, string>;
type StreamChunk = string | Uint8Array; type StreamChunk = string | Uint8Array;
@@ -46,10 +46,13 @@ function toByteStream(chunks: AsyncIterable<StreamChunk>) {
async function* readFileRecords( async function* readFileRecords(
file: StorageFileMeta, file: StorageFileMeta,
): AsyncGenerator<CsvRecord> { ): AsyncGenerator<CsvRecord> {
const fileStream = createReadStream(file.filePath); const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath);
const parser = fileStream.pipe( const parser = fileStream.pipe(
parse({ columns: true, skip_empty_lines: true, bom: true }), parse({ columns: true, skip_empty_lines: true, bom: true }),
); );
fileStream.on("error", (err) => {
if (!parser.destroyed) parser.destroy(err);
});
try { try {
for await (const record of parser as AsyncIterable<CsvRecord>) { for await (const record of parser as AsyncIterable<CsvRecord>) {
@@ -72,13 +75,13 @@ function makeExportStream(
if (format === "json") { if (format === "json") {
yield "["; yield "[";
} else if (includeHeader) { } else if (format === "csv" && includeHeader) {
yield `${fields.join(",")}\n`; yield `${fields.join(",")}\n`;
} }
for (const file of files) { for (const file of files) {
for await (const record of readFileRecords(file)) { for await (const record of readFileRecords(file)) {
if (format === "json") { if (format === "json" || format === "ndjson") {
const payload = record.payload ?? ""; const payload = record.payload ?? "";
let parsedPayload: unknown = payload; let parsedPayload: unknown = payload;
try { try {
@@ -86,7 +89,15 @@ function makeExportStream(
} catch { } catch {
// Keep non-JSON payloads as strings. // Keep non-JSON payloads as strings.
} }
yield `${isFirst ? "" : ","}${JSON.stringify({ ...record, payload: parsedPayload })}`; const serialized = JSON.stringify({
...record,
payload: parsedPayload,
});
if (format === "ndjson") {
yield `${serialized}\n`;
} else {
yield `${isFirst ? "" : ","}${serialized}`;
}
} else { } else {
const line = fields const line = fields
.map((field) => `"${String(record[field] ?? "").replaceAll('"', '""')}"`) .map((field) => `"${String(record[field] ?? "").replaceAll('"', '""')}"`)
@@ -224,9 +235,14 @@ export async function GET(request: NextRequest) {
: SDKManager.normalizeClientId(rawClientId) || "all"; : SDKManager.normalizeClientId(rawClientId) || "all";
const sanitizedClientId = const sanitizedClientId =
clientId === "all" ? "all" : CSVStorageManager.sanitizeClientId(clientId); clientId === "all" ? "all" : CSVStorageManager.sanitizeClientId(clientId);
const filename = searchParams.get("filename"); const filename = searchParams.get("filename") || searchParams.get("file");
const rawFormat = searchParams.get("format"); const rawFormat = (searchParams.get("format") || "csv").toLowerCase();
const format: ExportFormat = rawFormat === "json" ? "json" : "csv"; const format: ExportFormat =
rawFormat === "json"
? "json"
: rawFormat === "ndjson"
? "ndjson"
: "csv";
const includeHeader = searchParams.get("includeHeader") !== "false"; const includeHeader = searchParams.get("includeHeader") !== "false";
let files = await CSVStorageManager.listAllFilesAsync(); let files = await CSVStorageManager.listAllFilesAsync();
@@ -259,21 +275,31 @@ export async function GET(request: NextRequest) {
); );
} }
const file = matches[0]; const file = matches[0];
const baseStem = file.filename.replace(/\.csv(?:\.gz)?$/i, "");
const downloadName = const downloadName =
format === "json" format === "json"
? `${file.filename.replace(/\.csv$/i, "")}.json` ? `${baseStem}.json`
: format === "ndjson"
? `${baseStem}.ndjson`
: !includeHeader && file.compressed
? `${baseStem}.csv`
: file.filename; : file.filename;
const bodyStream = const bodyStream =
format === "json" || !includeHeader format === "json" || format === "ndjson" || !includeHeader
? makeExportStream([file], format, includeHeader) ? makeExportStream([file], format, includeHeader)
: streamFile(file); : streamFile(file);
const contentType =
format === "json"
? "application/json; charset=utf-8"
: format === "ndjson"
? "application/x-ndjson; charset=utf-8"
: file.compressed && includeHeader
? "application/gzip"
: "text/csv; charset=utf-8";
return new NextResponse(bodyStream, { return new NextResponse(bodyStream, {
status: 200, status: 200,
headers: { headers: {
"Content-Type": "Content-Type": contentType,
format === "json"
? "application/json; charset=utf-8"
: "text/csv; charset=utf-8",
"Content-Disposition": `attachment; filename="${downloadName.replace(/"/g, "")}"`, "Content-Disposition": `attachment; filename="${downloadName.replace(/"/g, "")}"`,
"Cache-Control": "no-store", "Cache-Control": "no-store",
}, },
@@ -294,6 +320,8 @@ export async function GET(request: NextRequest) {
"Content-Type": "Content-Type":
format === "json" format === "json"
? "application/json; charset=utf-8" ? "application/json; charset=utf-8"
: format === "ndjson"
? "application/x-ndjson; charset=utf-8"
: "text/csv; charset=utf-8", : "text/csv; charset=utf-8",
"Content-Disposition": `attachment; filename="${baseName}_${datePart}.${format}"`, "Content-Disposition": `attachment; filename="${baseName}_${datePart}.${format}"`,
"Cache-Control": "no-store", "Cache-Control": "no-store",
+207
View File
@@ -0,0 +1,207 @@
import { SettingsManager } from "@/core/adminSettings/settingsManager";
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
import { DataBufferEngine } from "@/core/data/buffer";
import { SDKManager } from "@/core/sdk/sdkManager";
import { getInternalIngestionHeaders } from "@/core/Security/security";
import { CSVStorageManager } from "@/core/Storage/csvStorage";
import { ADMIN_SESSION_COOKIE } from "@/lib/adminAuth";
import { NextRequest, NextResponse } from "next/server";
import { createHash, timingSafeEqual } from "node:crypto";
function verifyBearerTokenConstantTime(
expectedToken: string,
providedToken: string,
): boolean {
const expected = expectedToken.trim();
const provided = providedToken.trim();
if (!expected || !provided) return false;
const expectedDigest = createHash("sha256")
.update(expected, "utf8")
.digest();
const providedDigest = createHash("sha256")
.update(provided, "utf8")
.digest();
return timingSafeEqual(expectedDigest, providedDigest);
}
function escapePrometheusLabel(raw: string): string {
return String(raw || "")
.replace(/\\/g, "\\\\")
.replace(/\n/g, "\\n")
.replace(/"/g, '\\"');
}
export async function GET(req: NextRequest) {
try {
let authorized = false;
// 1. Check Bearer token (or ?token= query param) against encrypted Prometheus token
const authHeader = req.headers.get("authorization") || "";
const bearerCandidate = authHeader.toLowerCase().startsWith("bearer ")
? authHeader.slice(7).trim()
: req.nextUrl.searchParams.get("token")?.trim() || "";
const promConfig = await SettingsManager.getDecryptedPrometheusConfigAsync();
if (
promConfig.enabled &&
promConfig.token.trim().length > 0 &&
bearerCandidate.length > 0 &&
verifyBearerTokenConstantTime(promConfig.token, bearerCandidate)
) {
authorized = true;
}
// 2. Fallback: valid Admin Console session cookie
if (!authorized) {
const sessionCookie = req.cookies.get(ADMIN_SESSION_COOKIE)?.value;
if (sessionCookie) {
const adminAuth = await verifyRequestAdminSession(sessionCookie);
if (adminAuth.valid) {
authorized = true;
}
}
}
if (!authorized) {
return NextResponse.json(
{
error:
"PROMETHEUS_UNAUTHORIZED: Valid bearer scrape token or admin session required.",
code: "PROMETHEUS_UNAUTHORIZED",
},
{ status: 401 },
);
}
const [clients, storageStats, diskHealth, bufferStats] = await Promise.all([
SDKManager.listClientsAsync(),
CSVStorageManager.getStatsAsync(),
CSVStorageManager.getDiskSpaceHealthAsync(),
DataBufferEngine.getBufferStatsAsync(),
]);
let activeSockets = 0;
let bridgeUp = 0;
let remoteBufferSize = 0;
const headers = getInternalIngestionHeaders();
if (headers["x-datastack-internal-secret"]) {
const ingestionBaseUrl = (
process.env.INGESTION_INTERNAL_URL || "http://127.0.0.1:4000"
).replace(/\/$/, "");
try {
const res = await fetch(`${ingestionBaseUrl}/internal/metrics`, {
headers,
cache: "no-store",
signal: AbortSignal.timeout(1000),
});
if (res.ok) {
bridgeUp = 1;
const data = (await res.json()) as {
activeSockets?: number;
bufferSize?: number;
};
activeSockets = Math.max(0, Number(data.activeSockets) || 0);
remoteBufferSize = Math.max(0, Number(data.bufferSize) || 0);
}
} catch {
bridgeUp = 0;
}
}
const totalBufferedEvents =
bufferStats.localBufferSize +
(bufferStats.redisEnabled && bufferStats.redisConnected
? bufferStats.redisBufferSize
: remoteBufferSize);
let activeClientsCount = 0;
let revokedClientsCount = 0;
for (const c of clients) {
if (c.status === "revoked") {
revokedClientsCount += 1;
} else {
activeClientsCount += 1;
}
}
const lines: string[] = [
"# HELP datastack_up Whether the DataStack control plane is up (1 = healthy).",
"# TYPE datastack_up gauge",
"datastack_up 1",
"# HELP datastack_ingestion_bridge_up Whether the Port 4000 Socket.IO ingestion engine is reachable (1 = reachable, 0 = unreachable).",
"# TYPE datastack_ingestion_bridge_up gauge",
`datastack_ingestion_bridge_up ${bridgeUp}`,
"# HELP datastack_active_sockets Currently connected Socket.IO SDK clients on Port 4000.",
"# TYPE datastack_active_sockets gauge",
`datastack_active_sockets ${activeSockets}`,
"# HELP datastack_buffer_events_total Total events currently queued in memory or Redis awaiting CSV flush.",
"# TYPE datastack_buffer_events_total gauge",
`datastack_buffer_events_total ${totalBufferedEvents}`,
"# HELP datastack_storage_files_total Total CSV and gzipped CSV partition files on disk.",
"# TYPE datastack_storage_files_total gauge",
`datastack_storage_files_total ${storageStats.totalFiles}`,
"# HELP datastack_storage_bytes_total Total bytes occupied by CSV and gzipped CSV partition files on disk.",
"# TYPE datastack_storage_bytes_total gauge",
`datastack_storage_bytes_total ${storageStats.totalSizeBytes}`,
"# HELP datastack_storage_rows_total Total telemetry rows persisted across all partition files.",
"# TYPE datastack_storage_rows_total gauge",
`datastack_storage_rows_total ${storageStats.totalRows}`,
"# HELP datastack_disk_used_percent Percentage of underlying filesystem capacity currently used.",
"# TYPE datastack_disk_used_percent gauge",
`datastack_disk_used_percent ${diskHealth.usedPct}`,
"# HELP datastack_disk_free_bytes Free bytes available on the underlying storage volume.",
"# TYPE datastack_disk_free_bytes gauge",
`datastack_disk_free_bytes ${diskHealth.freeBytes}`,
"# HELP datastack_disk_total_bytes Total capacity in bytes of the underlying storage volume.",
"# TYPE datastack_disk_total_bytes gauge",
`datastack_disk_total_bytes ${diskHealth.totalBytes}`,
"# HELP datastack_clients_total Number of registered SDK clients partitioned by status.",
"# TYPE datastack_clients_total gauge",
`datastack_clients_total{status="active"} ${activeClientsCount}`,
`datastack_clients_total{status="revoked"} ${revokedClientsCount}`,
"# HELP datastack_client_events_total Cumulative ingested events per registered client.",
"# TYPE datastack_client_events_total counter",
];
for (const c of clients) {
const safeId = escapePrometheusLabel(c.clientId);
lines.push(
`datastack_client_events_total{client_id="${safeId}"} ${Math.max(0, Number(c.totalEvents) || 0)}`,
);
}
lines.push(
"# HELP datastack_client_storage_bytes Disk usage in bytes per registered client.",
"# TYPE datastack_client_storage_bytes gauge",
);
const perClientBytes = new Map<string, number>();
for (const f of storageStats.files) {
perClientBytes.set(
f.clientId,
(perClientBytes.get(f.clientId) || 0) + f.sizeBytes,
);
}
for (const c of clients) {
const safeId = escapePrometheusLabel(c.clientId);
const usedBytes = perClientBytes.get(c.clientId) || 0;
lines.push(
`datastack_client_storage_bytes{client_id="${safeId}"} ${usedBytes}`,
);
}
const body = `${lines.join("\n")}\n`;
return new NextResponse(body, {
status: 200,
headers: {
"Content-Type": "text/plain; version=0.0.4; charset=utf-8",
"Cache-Control": "no-store, no-cache, must-revalidate",
},
});
} catch (err: unknown) {
const message =
err instanceof Error ? err.message : "Failed to generate Prometheus metrics";
return NextResponse.json({ error: message }, { status: 500 });
}
}
+190 -3
View File
@@ -1,8 +1,9 @@
import { import {
SettingsManager, SettingsManager,
type AdminSettings, type AdminSettingsUpdateInput,
} from "@/core/adminSettings/settingsManager"; } from "@/core/adminSettings/settingsManager";
import { verifyRequestAdminSession } from "@/core/auth/adminUsers"; import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
import { SecretVaultError } from "@/core/Security/secretVault";
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
const SESSION_COOKIE = "datastack_admin_session"; const SESSION_COOKIE = "datastack_admin_session";
@@ -16,6 +17,24 @@ const ALLOWED_SETTING_KEYS = new Set([
"enableIdempotency", "enableIdempotency",
"autoPruneDays", "autoPruneDays",
"jwtSecret", "jwtSecret",
"compressAfterDays",
"enableS3Backup",
"s3Endpoint",
"s3Region",
"s3Bucket",
"s3Prefix",
"s3AccessKeyId",
"s3SecretAccessKey",
"clearS3SecretKey",
"diskPauseThresholdPct",
"enableWebhookAlerts",
"webhookUrl",
"webhookSecret",
"clearWebhookSecret",
"webhookCooldownSec",
"enablePrometheusMetrics",
"prometheusMetricsToken",
"clearPrometheusToken",
]); ]);
function parseBoundedInt( function parseBoundedInt(
@@ -33,6 +52,21 @@ function parseBoundedInt(
return raw; return raw;
} }
function parseOptionalString(
raw: unknown,
field: string,
maxLength: number,
): string {
if (typeof raw !== "string") {
throw new Error(`${field} must be a string.`);
}
const trimmed = raw.trim();
if (trimmed.length > maxLength) {
throw new Error(`${field} must be at most ${maxLength} characters.`);
}
return trimmed;
}
export async function GET(req: NextRequest) { export async function GET(req: NextRequest) {
try { try {
const auth = await verifyRequestAdminSession( const auth = await verifyRequestAdminSession(
@@ -109,7 +143,7 @@ export async function POST(req: NextRequest) {
); );
} }
const validated: Partial<AdminSettings> = {}; const validated: AdminSettingsUpdateInput = {};
try { try {
if (body.maxEventsPerSec !== undefined) { if (body.maxEventsPerSec !== undefined) {
validated.maxEventsPerSec = parseBoundedInt( validated.maxEventsPerSec = parseBoundedInt(
@@ -159,14 +193,161 @@ export async function POST(req: NextRequest) {
3650, 3650,
); );
} }
if (body.compressAfterDays !== undefined) {
validated.compressAfterDays = parseBoundedInt(
body.compressAfterDays,
"compressAfterDays",
0,
3650,
);
}
if (body.diskPauseThresholdPct !== undefined) {
const pct = parseBoundedInt(
body.diskPauseThresholdPct,
"diskPauseThresholdPct",
0,
99,
);
if (pct !== 0 && pct < 50) {
throw new Error(
"diskPauseThresholdPct must be 0 (disabled) or between 50 and 99.",
);
}
validated.diskPauseThresholdPct = pct;
}
if (body.enableIdempotency !== undefined) { if (body.enableIdempotency !== undefined) {
if (typeof body.enableIdempotency !== "boolean") { if (typeof body.enableIdempotency !== "boolean") {
throw new Error("enableIdempotency must be a boolean."); throw new Error("enableIdempotency must be a boolean.");
} }
validated.enableIdempotency = body.enableIdempotency; validated.enableIdempotency = body.enableIdempotency;
} }
if (body.enableS3Backup !== undefined) {
if (typeof body.enableS3Backup !== "boolean") {
throw new Error("enableS3Backup must be a boolean.");
}
validated.enableS3Backup = body.enableS3Backup;
}
if (body.s3Endpoint !== undefined) {
const ep = parseOptionalString(body.s3Endpoint, "s3Endpoint", 512);
if (ep) {
let parsedEp: URL;
try {
parsedEp = new URL(
/^https?:\/\//i.test(ep) ? ep : `https://${ep}`,
);
} catch {
throw new Error("s3Endpoint must be a valid http(s) URL.");
}
if (parsedEp.protocol !== "http:" && parsedEp.protocol !== "https:") {
throw new Error("s3Endpoint must use http:// or https://.");
}
}
validated.s3Endpoint = ep;
}
if (body.s3Region !== undefined) {
validated.s3Region =
parseOptionalString(body.s3Region, "s3Region", 64) || "auto";
}
if (body.s3Bucket !== undefined) {
validated.s3Bucket = parseOptionalString(
body.s3Bucket,
"s3Bucket",
128,
);
}
if (body.s3Prefix !== undefined) {
validated.s3Prefix = parseOptionalString(
body.s3Prefix,
"s3Prefix",
256,
);
}
if (body.s3AccessKeyId !== undefined) {
validated.s3AccessKeyId = parseOptionalString(
body.s3AccessKeyId,
"s3AccessKeyId",
256,
);
}
if (body.s3SecretAccessKey !== undefined) {
validated.s3SecretAccessKey = parseOptionalString(
body.s3SecretAccessKey,
"s3SecretAccessKey",
512,
);
}
if (body.clearS3SecretKey !== undefined) {
if (typeof body.clearS3SecretKey !== "boolean") {
throw new Error("clearS3SecretKey must be a boolean.");
}
validated.clearS3SecretKey = body.clearS3SecretKey;
}
if (body.enableWebhookAlerts !== undefined) {
if (typeof body.enableWebhookAlerts !== "boolean") {
throw new Error("enableWebhookAlerts must be a boolean.");
}
validated.enableWebhookAlerts = body.enableWebhookAlerts;
}
if (body.webhookUrl !== undefined) {
const wh = parseOptionalString(body.webhookUrl, "webhookUrl", 1024);
if (wh) {
let parsedWh: URL;
try {
parsedWh = new URL(wh);
} catch {
throw new Error("webhookUrl must be a valid http(s) URL.");
}
if (parsedWh.protocol !== "http:" && parsedWh.protocol !== "https:") {
throw new Error("webhookUrl must use http:// or https://.");
}
}
validated.webhookUrl = wh;
}
if (body.webhookSecret !== undefined) {
validated.webhookSecret = parseOptionalString(
body.webhookSecret,
"webhookSecret",
512,
);
}
if (body.clearWebhookSecret !== undefined) {
if (typeof body.clearWebhookSecret !== "boolean") {
throw new Error("clearWebhookSecret must be a boolean.");
}
validated.clearWebhookSecret = body.clearWebhookSecret;
}
if (body.webhookCooldownSec !== undefined) {
validated.webhookCooldownSec = parseBoundedInt(
body.webhookCooldownSec,
"webhookCooldownSec",
10,
86400,
);
}
if (body.enablePrometheusMetrics !== undefined) {
if (typeof body.enablePrometheusMetrics !== "boolean") {
throw new Error("enablePrometheusMetrics must be a boolean.");
}
validated.enablePrometheusMetrics = body.enablePrometheusMetrics;
}
if (body.prometheusMetricsToken !== undefined) {
validated.prometheusMetricsToken = parseOptionalString(
body.prometheusMetricsToken,
"prometheusMetricsToken",
512,
);
}
if (body.clearPrometheusToken !== undefined) {
if (typeof body.clearPrometheusToken !== "boolean") {
throw new Error("clearPrometheusToken must be a boolean.");
}
validated.clearPrometheusToken = body.clearPrometheusToken;
}
if (body.jwtSecret !== undefined) { if (body.jwtSecret !== undefined) {
if (typeof body.jwtSecret !== "string" || body.jwtSecret.trim().length < 32) { if (
typeof body.jwtSecret !== "string" ||
body.jwtSecret.trim().length < 32
) {
throw new Error( throw new Error(
"jwtSecret must be a string with at least 32 characters.", "jwtSecret must be a string with at least 32 characters.",
); );
@@ -208,6 +389,12 @@ export async function POST(req: NextRequest) {
: {}), : {}),
}); });
} catch (err: unknown) { } catch (err: unknown) {
if (err instanceof SecretVaultError) {
return NextResponse.json(
{ error: err.message, code: err.code },
{ status: 500 },
);
}
const message = const message =
err instanceof Error ? err.message : "Settings update failed"; err instanceof Error ? err.message : "Settings update failed";
return NextResponse.json({ error: message }, { status: 500 }); return NextResponse.json({ error: message }, { status: 500 });
+119
View File
@@ -0,0 +1,119 @@
import { SettingsManager } from "@/core/adminSettings/settingsManager";
import { dispatchWebhookAlertAsync } from "@/core/alerts/webhookNotifier";
import { SecretVaultError } from "@/core/Security/secretVault";
import { requireAdminApiSession } from "@/lib/adminAuth";
import { NextRequest, NextResponse } from "next/server";
export async function POST(req: NextRequest) {
const auth = await requireAdminApiSession(req);
if (!auth.valid) {
return auth.response;
}
let body: Record<string, unknown> = {};
try {
const text = await req.text();
if (text.trim()) {
const parsed = JSON.parse(text);
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
body = parsed as Record<string, unknown>;
}
}
} catch {
return NextResponse.json(
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
{ status: 400 },
);
}
try {
const savedConfig = await SettingsManager.getDecryptedWebhookConfigAsync();
const webhookUrl =
typeof body.webhookUrl === "string"
? body.webhookUrl.trim()
: savedConfig.webhookUrl.trim();
const clearWebhookSecret = body.clearWebhookSecret === true;
const webhookSecret = clearWebhookSecret
? ""
: typeof body.webhookSecret === "string" && body.webhookSecret.trim()
? body.webhookSecret.trim()
: savedConfig.webhookSecret;
if (!webhookUrl) {
return NextResponse.json(
{
success: false,
code: "VALIDATION_ERROR",
error: "Webhook URL is required to send a test alert.",
},
{ status: 400 },
);
}
try {
const parsedUrl = new URL(webhookUrl);
if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") {
throw new Error("Webhook URL must use http:// or https://.");
}
} catch {
return NextResponse.json(
{
success: false,
code: "VALIDATION_ERROR",
error: "Webhook URL must be a valid http:// or https:// endpoint.",
},
{ status: 400 },
);
}
const result = await dispatchWebhookAlertAsync(
{
event: "TEST_ALERT",
severity: "info",
title: "Webhook Verification",
message: "Test alert dispatched from DataStack Admin Console.",
details: {
triggeredBy: auth.auth.user?.email || "admin",
},
},
{
enabled: true,
webhookUrl,
webhookSecret,
cooldownSec: savedConfig.cooldownSec,
},
{ bypassCooldown: true },
);
if (!result.ok) {
return NextResponse.json(
{
success: false,
code: "WEBHOOK_DELIVERY_FAILED",
error: result.error || "Webhook target failed to accept test alert.",
status: result.status,
},
{ status: 502 },
);
}
return NextResponse.json({
success: true,
status: result.status,
message: `Webhook test alert delivered (HTTP ${result.status ?? 200}).`,
});
} catch (err: unknown) {
if (err instanceof SecretVaultError) {
return NextResponse.json(
{ success: false, code: err.code, error: err.message },
{ status: 400 },
);
}
const message =
err instanceof Error ? err.message : "Webhook test alert failed";
return NextResponse.json(
{ success: false, code: "WEBHOOK_DELIVERY_FAILED", error: message },
{ status: 502 },
);
}
}
+163
View File
@@ -0,0 +1,163 @@
import { SettingsManager } from "@/core/adminSettings/settingsManager";
import { SecretVaultError } from "@/core/Security/secretVault";
import { CSVStorageManager } from "@/core/Storage/csvStorage";
import {
buildS3ObjectKey,
normalizeS3EndpointUrl,
uploadFileToS3,
} from "@/core/Storage/s3Archive";
import { requireAdminApiSession } from "@/lib/adminAuth";
import { NextRequest, NextResponse } from "next/server";
export async function POST(req: NextRequest) {
const auth = await requireAdminApiSession(req);
if (!auth.valid) {
return auth.response;
}
let body: Record<string, unknown> = {};
try {
const text = await req.text();
if (text.trim()) {
const parsed = JSON.parse(text);
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
body = parsed as Record<string, unknown>;
}
}
} catch {
return NextResponse.json(
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
{ status: 400 },
);
}
try {
const settings = await SettingsManager.getSettingsAsync();
const s3Config = await SettingsManager.getDecryptedS3ConfigAsync();
if (!s3Config.bucket.trim() || !s3Config.accessKeyId.trim() || !s3Config.secretAccessKey.trim()) {
return NextResponse.json(
{
success: false,
code: "S3_INVALID_CONFIG",
error: "Configure S3 Bucket, Access Key ID, and Secret Access Key before archiving.",
},
{ status: 400 },
);
}
try {
normalizeS3EndpointUrl(s3Config.endpoint, s3Config.region);
} catch (urlErr: unknown) {
return NextResponse.json(
{
success: false,
code: "S3_INVALID_CONFIG",
error: urlErr instanceof Error ? urlErr.message : "Invalid S3 endpoint URL.",
},
{ status: 400 },
);
}
const pruneAfterUpload = body.pruneAfterUpload === true;
const rawOlderThanDays =
typeof body.olderThanDays === "number" && Number.isFinite(body.olderThanDays)
? Math.max(0, Math.floor(body.olderThanDays))
: settings.autoPruneDays > 0
? settings.autoPruneDays
: 0;
const shouldCompressFirst =
body.compressFirst !== false && settings.compressAfterDays > 0;
let compressedCount = 0;
if (shouldCompressFirst) {
compressedCount = await CSVStorageManager.compressOldPartitionsAsync(
settings.compressAfterDays,
);
}
const archiveResult =
await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync(
settings.autoPruneDays,
{
archiveOnlyWithoutDelete: !pruneAfterUpload,
maxCutoffDays: rawOlderThanDays,
beforeDeleteFile: async (file) => {
const key = buildS3ObjectKey(
s3Config.prefix,
file.clientId,
file.filename,
);
const uploadRes = await uploadFileToS3(
s3Config,
file.filePath,
key,
);
if (!uploadRes.ok) {
throw new Error(
uploadRes.error || `Failed to upload ${file.filename}`,
);
}
},
},
);
if (
archiveResult.errors.length > 0 &&
archiveResult.archivedCount === 0
) {
SettingsManager.triggerWebhookAlert({
event: "S3_ARCHIVE_FAILED",
severity: "critical",
title: "S3 Cold Archive Upload Failed",
message: `Manual S3 archive failed: ${archiveResult.errors[0]}`,
details: { errorCount: archiveResult.errors.length },
});
return NextResponse.json(
{
success: false,
code: "S3_ARCHIVE_FAILED",
error: archiveResult.errors[0],
compressedCount,
...archiveResult,
},
{ status: 502 },
);
}
if (archiveResult.errors.length > 0) {
SettingsManager.triggerWebhookAlert({
event: "S3_ARCHIVE_FAILED",
severity: "warning",
title: "Partial S3 Cold Archive Errors",
message: `S3 archive encountered ${archiveResult.errors.length} error(s): ${archiveResult.errors[0]}`,
details: { errorCount: archiveResult.errors.length },
});
}
return NextResponse.json({
success: archiveResult.errors.length === 0,
compressedCount,
...archiveResult,
});
} catch (err: unknown) {
if (err instanceof SecretVaultError) {
return NextResponse.json(
{ success: false, code: err.code, error: err.message },
{ status: 400 },
);
}
const message =
err instanceof Error ? err.message : "Manual S3 archive failed";
SettingsManager.triggerWebhookAlert({
event: "S3_ARCHIVE_FAILED",
severity: "critical",
title: "S3 Cold Archive Upload Failed",
message,
});
return NextResponse.json(
{ success: false, code: "S3_ARCHIVE_FAILED", error: message },
{ status: 500 },
);
}
}
+5 -2
View File
@@ -23,7 +23,10 @@ export async function GET(req: NextRequest) {
? CSVStorageManager.sanitizeClientId(clientId) ? CSVStorageManager.sanitizeClientId(clientId)
: undefined; : undefined;
const allFiles = await CSVStorageManager.listAllFilesAsync(); const [allFiles, diskUsage] = await Promise.all([
CSVStorageManager.listAllFilesAsync(),
CSVStorageManager.getDiskUsageAsync(),
]);
let files = allFiles; let files = allFiles;
if (clientId) { if (clientId) {
@@ -39,7 +42,7 @@ export async function GET(req: NextRequest) {
const stats = CSVStorageManager.formatStorageStats(files); const stats = CSVStorageManager.formatStorageStats(files);
return NextResponse.json({ stats, files }); return NextResponse.json({ stats, files, diskUsage });
} catch (err: unknown) { } catch (err: unknown) {
const message = err instanceof Error ? err.message : 'Storage stats failed'; const message = err instanceof Error ? err.message : 'Storage stats failed';
return NextResponse.json({ error: message }, { status: 500 }); return NextResponse.json({ error: message }, { status: 500 });
+105
View File
@@ -0,0 +1,105 @@
import { SettingsManager } from "@/core/adminSettings/settingsManager";
import {
decryptSecret,
SecretVaultError,
} from "@/core/Security/secretVault";
import {
testS3Connection,
type S3ArchiveConfig,
} from "@/core/Storage/s3Archive";
import { requireAdminApiSession } from "@/lib/adminAuth";
import { NextRequest, NextResponse } from "next/server";
export async function POST(req: NextRequest) {
const auth = await requireAdminApiSession(req);
if (!auth.valid) {
return auth.response;
}
let body: Record<string, unknown> = {};
try {
const text = await req.text();
if (text.trim()) {
const parsed = JSON.parse(text);
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
body = parsed as Record<string, unknown>;
}
}
} catch {
return NextResponse.json(
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
{ status: 400 },
);
}
try {
const saved = await SettingsManager.getSettingsAsync();
const draftSecret =
typeof body.s3SecretAccessKey === "string" &&
body.s3SecretAccessKey.trim().length > 0
? body.s3SecretAccessKey.trim()
: "";
let effectiveSecret = draftSecret;
if (!effectiveSecret && !body.clearS3SecretKey) {
effectiveSecret = decryptSecret(saved.s3SecretAccessKeyEnc);
}
const config: S3ArchiveConfig = {
endpoint:
typeof body.s3Endpoint === "string"
? body.s3Endpoint.trim()
: saved.s3Endpoint,
region:
typeof body.s3Region === "string" && body.s3Region.trim()
? body.s3Region.trim()
: saved.s3Region || "auto",
bucket:
typeof body.s3Bucket === "string"
? body.s3Bucket.trim()
: saved.s3Bucket,
prefix:
typeof body.s3Prefix === "string"
? body.s3Prefix.trim()
: saved.s3Prefix,
accessKeyId:
typeof body.s3AccessKeyId === "string"
? body.s3AccessKeyId.trim()
: saved.s3AccessKeyId,
secretAccessKey: effectiveSecret,
};
const result = await testS3Connection(config);
if (!result.ok) {
return NextResponse.json(
{
success: false,
code: result.code || "S3_CONNECTION_FAILED",
error: result.error || "S3 connectivity check failed.",
...(result.status !== undefined ? { s3Status: result.status } : {}),
},
{ status: result.code === "S3_INVALID_CONFIG" ? 400 : 502 },
);
}
return NextResponse.json({
success: true,
endpoint: result.endpoint,
bucket: result.bucket,
status: result.status,
});
} catch (err: unknown) {
if (err instanceof SecretVaultError) {
return NextResponse.json(
{ success: false, code: err.code, error: err.message },
{ status: 400 },
);
}
const message =
err instanceof Error ? err.message : "S3 connectivity check failed";
return NextResponse.json(
{ success: false, code: "S3_CONNECTION_FAILED", error: message },
{ status: 500 },
);
}
}
+103 -5
View File
@@ -47,6 +47,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
const [editClient, setEditClient] = useState<ClientItem | null>(null); const [editClient, setEditClient] = useState<ClientItem | null>(null);
const [editName, setEditName] = useState(''); const [editName, setEditName] = useState('');
const [editRateLimit, setEditRateLimit] = useState('100'); const [editRateLimit, setEditRateLimit] = useState('100');
const [editStorageQuotaMb, setEditStorageQuotaMb] = useState('0');
const [editAllowedIps, setEditAllowedIps] = useState('');
const [editExpiresAt, setEditExpiresAt] = useState('');
const [editError, setEditError] = useState(''); const [editError, setEditError] = useState('');
const [processing, setProcessing] = useState(false); const [processing, setProcessing] = useState(false);
@@ -59,6 +62,18 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
setEditClient(client); setEditClient(client);
setEditName(client.name); setEditName(client.name);
setEditRateLimit(String(client.rateLimit || 100)); setEditRateLimit(String(client.rateLimit || 100));
setEditStorageQuotaMb(String(client.storageQuotaMb ?? 0));
setEditAllowedIps(client.allowedIps || '');
if (client.expiresAt) {
const parsedDate = new Date(client.expiresAt);
setEditExpiresAt(
!Number.isNaN(parsedDate.getTime())
? parsedDate.toISOString().slice(0, 16)
: '',
);
} else {
setEditExpiresAt('');
}
setEditError(''); setEditError('');
}; };
@@ -68,6 +83,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
const trimmedName = editName.trim(); const trimmedName = editName.trim();
const numericRateLimit = Number(editRateLimit); const numericRateLimit = Number(editRateLimit);
const numericQuotaMb = Number(
editStorageQuotaMb.trim() === '' ? '0' : editStorageQuotaMb,
);
if (!trimmedName) { if (!trimmedName) {
setEditError('Display name is required.'); setEditError('Display name is required.');
return; return;
@@ -76,6 +94,19 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
setEditError('Rate limit must be an integer between 1 and 100,000 events/s.'); setEditError('Rate limit must be an integer between 1 and 100,000 events/s.');
return; return;
} }
if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) {
setEditError('Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).');
return;
}
let normalizedExpiresAt: string | null = null;
if (editExpiresAt.trim()) {
const parsedDate = new Date(editExpiresAt.trim());
if (Number.isNaN(parsedDate.getTime())) {
setEditError('Expiration date/time is invalid.');
return;
}
normalizedExpiresAt = parsedDate.toISOString();
}
setEditError(''); setEditError('');
setProcessing(true); setProcessing(true);
@@ -86,6 +117,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
body: JSON.stringify({ body: JSON.stringify({
name: trimmedName, name: trimmedName,
rateLimit: Math.floor(numericRateLimit), rateLimit: Math.floor(numericRateLimit),
storageQuotaMb: Math.floor(numericQuotaMb),
allowedIps: editAllowedIps.trim(),
expiresAt: normalizedExpiresAt,
}), }),
}); });
if (response.status === 401) { if (response.status === 401) {
@@ -224,11 +258,26 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
) : ( ) : (
clients.map((client) => { clients.map((client) => {
const revoked = client.status === 'revoked'; const revoked = client.status === 'revoked';
const online = client.status === 'active' && Boolean(client.lastActive) && const expired =
Boolean(client.expiresAt) &&
now >= new Date(client.expiresAt!).getTime();
const online =
client.status === 'active' &&
!expired &&
Boolean(client.lastActive) &&
now - new Date(client.lastActive!).getTime() < 60_000; now - new Date(client.lastActive!).getTime() < 60_000;
const pending = client.status === 'active' && !client.lastActive; const pending = client.status === 'active' && !expired && !client.lastActive;
const label = revoked ? 'revoked' : pending ? 'pending' : online ? 'online' : 'offline'; const label = revoked
const dotClass = revoked ? 'revoked'
: expired
? 'expired'
: pending
? 'pending'
: online
? 'online'
: 'offline';
const dotClass =
revoked || expired
? 'bg-[var(--danger)]' ? 'bg-[var(--danger)]'
: online : online
? 'bg-[var(--accent)]' ? 'bg-[var(--accent)]'
@@ -249,6 +298,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
revoked revoked
</span> </span>
)} )}
{expired && (
<span className="border border-[var(--danger)] bg-[var(--danger-soft)] px-1 py-0 text-[9px] uppercase tracking-wider text-[var(--danger)]">
expired
</span>
)}
</div> </div>
<div className="mt-0.5 truncate font-mono text-[10.5px] text-[var(--text-faint)]">{client.clientId}</div> <div className="mt-0.5 truncate font-mono text-[10.5px] text-[var(--text-faint)]">{client.clientId}</div>
</div> </div>
@@ -259,7 +313,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
{client.tokenPreview || 'JWT Active'} {client.tokenPreview || 'JWT Active'}
</div> </div>
<div className="mt-0.5 text-[10px] text-[var(--text-faint)]"> <div className="mt-0.5 text-[10px] text-[var(--text-faint)]">
{client.rateLimit}/s limit {client.rateLimit}/s · {(client.storageQuotaMb ?? 0) > 0 ? `${client.storageQuotaMb}MB quota` : 'Unlimited'}
{client.allowedIps ? ` · IP: ${client.allowedIps}` : ''}
{client.expiresAt
? ` · Exp: ${new Date(client.expiresAt).toISOString().slice(0, 10)}`
: ''}
</div> </div>
</TableCell> </TableCell>
<TableCell className="text-right font-mono tabular-nums"> <TableCell className="text-right font-mono tabular-nums">
@@ -375,6 +433,46 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
onChange={(event) => setEditRateLimit(event.target.value)} onChange={(event) => setEditRateLimit(event.target.value)}
required required
/> />
<Input
label="Storage quota (MB, 0 = unlimited)"
compact
type="number"
min="0"
max="1048576"
value={editStorageQuotaMb}
onChange={(event) => setEditStorageQuotaMb(event.target.value)}
required
/>
<Input
label="Allowed IPs / CIDR (blank = allow all)"
compact
value={editAllowedIps}
onChange={(event) => setEditAllowedIps(event.target.value)}
placeholder="203.0.113.10, 10.0.0.0/8"
className="font-mono text-[11px]"
/>
<div className="flex items-end gap-1.5">
<div className="min-w-0 flex-1">
<Input
label="Token expiration (optional)"
compact
type="datetime-local"
value={editExpiresAt}
onChange={(event) => setEditExpiresAt(event.target.value)}
className="font-mono text-[11px]"
/>
</div>
{editExpiresAt && (
<Button
type="button"
size="xs"
variant="ghost"
onClick={() => setEditExpiresAt('')}
>
Clear
</Button>
)}
</div>
{editError && ( {editError && (
<p role="alert" className="text-[11px] text-[var(--danger)]"> <p role="alert" className="text-[11px] text-[var(--danger)]">
{editError} {editError}
+8 -2
View File
@@ -33,7 +33,7 @@ interface StorageFile {
modifiedAt: string; modifiedAt: string;
} }
type ExportFormat = 'csv' | 'json'; type ExportFormat = 'csv' | 'json' | 'ndjson';
type RangePreset = 'today' | '7d' | '30d' | 'all' | 'custom'; type RangePreset = 'today' | '7d' | '30d' | 'all' | 'custom';
function formatBytes(bytes: number) { function formatBytes(bytes: number) {
@@ -368,7 +368,7 @@ export function ExportWorkspace() {
Format Format
</legend> </legend>
<div <div
className="grid grid-cols-2 gap-2" className="grid grid-cols-3 gap-2"
role="radiogroup" role="radiogroup"
aria-label="Export format" aria-label="Export format"
> >
@@ -384,6 +384,12 @@ export function ExportWorkspace() {
label="JSON" label="JSON"
onSelect={setFormat} onSelect={setFormat}
/> />
<FormatOption
value="ndjson"
selected={format === 'ndjson'}
label="NDJSON"
onSelect={setFormat}
/>
</div> </div>
</fieldset> </fieldset>
+53 -1
View File
@@ -19,6 +19,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
const [clientId, setClientId] = useState(''); const [clientId, setClientId] = useState('');
const [name, setName] = useState(''); const [name, setName] = useState('');
const [rateLimit, setRateLimit] = useState('100'); const [rateLimit, setRateLimit] = useState('100');
const [storageQuotaMb, setStorageQuotaMb] = useState('0');
const [allowedIps, setAllowedIps] = useState('');
const [expiresAt, setExpiresAt] = useState('');
const [submitting, setSubmitting] = useState(false); const [submitting, setSubmitting] = useState(false);
const [credentials, setCredentials] = useState<ClientCredentials | null>(null); const [credentials, setCredentials] = useState<ClientCredentials | null>(null);
@@ -35,12 +38,33 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
error('Validation error', 'Rate limit must be an integer between 1 and 100,000 events/s.'); error('Validation error', 'Rate limit must be an integer between 1 and 100,000 events/s.');
return; return;
} }
const numericQuotaMb = Number(storageQuotaMb.trim() === '' ? '0' : storageQuotaMb);
if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) {
error('Validation error', 'Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).');
return;
}
let normalizedExpiresAt: string | null = null;
if (expiresAt.trim()) {
const parsedDate = new Date(expiresAt.trim());
if (Number.isNaN(parsedDate.getTime())) {
error('Validation error', 'Token expiration must be a valid date/time.');
return;
}
normalizedExpiresAt = parsedDate.toISOString();
}
setSubmitting(true); setSubmitting(true);
try { try {
const response = await fetch('/api/clients', { const response = await fetch('/api/clients', {
method: 'POST', method: 'POST',
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ clientId: trimmedId, name: trimmedName, rateLimit: numericRate }), body: JSON.stringify({
clientId: trimmedId,
name: trimmedName,
rateLimit: numericRate,
storageQuotaMb: numericQuotaMb,
allowedIps: allowedIps.trim(),
expiresAt: normalizedExpiresAt,
}),
}); });
if (response.status === 401) { if (response.status === 401) {
window.location.assign('/vcon/login'); window.location.assign('/vcon/login');
@@ -53,6 +77,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
setClientId(''); setClientId('');
setName(''); setName('');
setRateLimit('100'); setRateLimit('100');
setStorageQuotaMb('0');
setAllowedIps('');
setExpiresAt('');
onClose(); onClose();
success('Client registered', `${data.client.clientId} is ready to connect.`); success('Client registered', `${data.client.clientId} is ready to connect.`);
await onCreated(); await onCreated();
@@ -103,6 +130,31 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
value={rateLimit} value={rateLimit}
onChange={(event) => setRateLimit(event.target.value)} onChange={(event) => setRateLimit(event.target.value)}
/> />
<Input
compact
label="Storage quota (MB, 0 = unlimited)"
type="number"
min="0"
max="1048576"
value={storageQuotaMb}
onChange={(event) => setStorageQuotaMb(event.target.value)}
/>
<Input
compact
label="Allowed IPs / CIDR (blank = allow all)"
placeholder="203.0.113.10, 10.0.0.0/8"
value={allowedIps}
onChange={(event) => setAllowedIps(event.target.value)}
className="font-mono text-[11px]"
/>
<Input
compact
label="Token expiration (optional)"
type="datetime-local"
value={expiresAt}
onChange={(event) => setExpiresAt(event.target.value)}
className="font-mono text-[11px]"
/>
</form> </form>
</Modal> </Modal>
+792 -15
View File
@@ -16,15 +16,21 @@ import { StatStrip } from '@/components/ui/StatStrip';
import { useToast } from '@/components/ui/Toast'; import { useToast } from '@/components/ui/Toast';
import { getCached, setCached } from '@/lib/vconCache'; import { getCached, setCached } from '@/lib/vconCache';
import { import {
type AlertingAndMetricsSettingsFields,
DEFAULT_ALERTING_SETTINGS,
DEFAULT_EDITABLE_SETTINGS, DEFAULT_EDITABLE_SETTINGS,
DEFAULT_S3_SETTINGS,
describeSettingsBridgeSync, describeSettingsBridgeSync,
type EditableSettings, type EditableSettings,
mapBridgeStatusToSocketUiState, mapBridgeStatusToSocketUiState,
normalizeAlertingSettings,
normalizeEditableSettings, normalizeEditableSettings,
normalizeS3Settings,
NUMERIC_SETTING_KEYS, NUMERIC_SETTING_KEYS,
type NumericSettingDrafts, type NumericSettingDrafts,
type NumericSettingKey, type NumericSettingKey,
resolvePublicEndpoints, resolvePublicEndpoints,
type S3SettingsFields,
SETTINGS_LIMITS, SETTINGS_LIMITS,
settingsToNumericDrafts, settingsToNumericDrafts,
type SettingsValidationErrors, type SettingsValidationErrors,
@@ -34,12 +40,29 @@ import {
} from '@/lib/vconUiHelpers'; } from '@/lib/vconUiHelpers';
import { useCallback, useEffect, useMemo, useState } from 'react'; import { useCallback, useEffect, useMemo, useState } from 'react';
interface PublicSettings extends EditableSettings { interface PublicSettings
extends EditableSettings,
S3SettingsFields,
AlertingAndMetricsSettingsFields {
serverPort: number; serverPort: number;
webPort: number; webPort: number;
hasS3SecretKey?: boolean;
s3SecretKeyMask?: string;
hasWebhookSecret?: boolean;
webhookSecretMask?: string;
hasPrometheusToken?: boolean;
prometheusTokenMask?: string;
updatedAt: string; updatedAt: string;
} }
interface DiskUsageSummary {
available: boolean;
usedPct: number;
freeBytes: number;
totalBytes: number;
checkedAt?: string;
}
const TABS = [ const TABS = [
{ id: 'general', label: 'General' }, { id: 'general', label: 'General' },
{ id: 'connections', label: 'Connections' }, { id: 'connections', label: 'Connections' },
@@ -55,24 +78,89 @@ const FIELD_TO_TAB: Record<NumericSettingKey, TabId> = {
batchFlushIntervalMs: 'connections', batchFlushIntervalMs: 'connections',
fileRotationMb: 'storage', fileRotationMb: 'storage',
autoPruneDays: 'storage', autoPruneDays: 'storage',
compressAfterDays: 'storage',
diskPauseThresholdPct: 'storage',
maxPayloadSizeKb: 'security', maxPayloadSizeKb: 'security',
webhookCooldownSec: 'security',
}; };
function formatGiB(bytes: number): string {
if (!Number.isFinite(bytes) || bytes <= 0) return '0.0 GB';
return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)} GB`;
}
export function SettingsWorkspace() { export function SettingsWorkspace() {
const { success, warning, error } = useToast(); const { success, warning, error } = useToast();
const [savedSettings, setSavedSettings] = useState<EditableSettings>(() => const [savedSettings, setSavedSettings] = useState<EditableSettings>(() =>
getCached<EditableSettings>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) normalizeEditableSettings(
getCached<Partial<EditableSettings>>('vcon:settings', DEFAULT_EDITABLE_SETTINGS)
)
); );
const [numericDrafts, setNumericDrafts] = useState<NumericSettingDrafts>(() => const [numericDrafts, setNumericDrafts] = useState<NumericSettingDrafts>(() =>
settingsToNumericDrafts( settingsToNumericDrafts(
getCached<EditableSettings>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) normalizeEditableSettings(
getCached<Partial<EditableSettings>>('vcon:settings', DEFAULT_EDITABLE_SETTINGS)
)
) )
); );
const [enableIdempotency, setEnableIdempotency] = useState<boolean>( const [enableIdempotency, setEnableIdempotency] = useState<boolean>(
() => () =>
getCached<EditableSettings>('vcon:settings', DEFAULT_EDITABLE_SETTINGS) normalizeEditableSettings(
.enableIdempotency getCached<Partial<EditableSettings>>('vcon:settings', DEFAULT_EDITABLE_SETTINGS)
).enableIdempotency
); );
const [savedS3Settings, setSavedS3Settings] = useState<S3SettingsFields>(() =>
normalizeS3Settings(
getCached<Partial<S3SettingsFields>>('vcon:s3-settings', DEFAULT_S3_SETTINGS)
)
);
const [s3Draft, setS3Draft] = useState<S3SettingsFields>(() =>
normalizeS3Settings(
getCached<Partial<S3SettingsFields>>('vcon:s3-settings', DEFAULT_S3_SETTINGS)
)
);
const [hasS3SecretKey, setHasS3SecretKey] = useState(false);
const [s3SecretKeyMask, setS3SecretKeyMask] = useState('');
const [s3SecretAccessKey, setS3SecretAccessKey] = useState('');
const [clearS3SecretKey, setClearS3SecretKey] = useState(false);
const [savedAlertingSettings, setSavedAlertingSettings] =
useState<AlertingAndMetricsSettingsFields>(() =>
normalizeAlertingSettings(
getCached<Partial<AlertingAndMetricsSettingsFields>>(
'vcon:alerting-settings',
DEFAULT_ALERTING_SETTINGS
)
)
);
const [alertingDraft, setAlertingDraft] =
useState<AlertingAndMetricsSettingsFields>(() =>
normalizeAlertingSettings(
getCached<Partial<AlertingAndMetricsSettingsFields>>(
'vcon:alerting-settings',
DEFAULT_ALERTING_SETTINGS
)
)
);
const [hasWebhookSecret, setHasWebhookSecret] = useState(false);
const [webhookSecretMask, setWebhookSecretMask] = useState('');
const [webhookSecret, setWebhookSecret] = useState('');
const [clearWebhookSecret, setClearWebhookSecret] = useState(false);
const [testingWebhook, setTestingWebhook] = useState(false);
const [webhookTestStatus, setWebhookTestStatus] = useState<{
ok: boolean;
text: string;
} | null>(null);
const [hasPrometheusToken, setHasPrometheusToken] = useState(false);
const [prometheusTokenMask, setPrometheusTokenMask] = useState('');
const [prometheusMetricsToken, setPrometheusMetricsToken] = useState('');
const [clearPrometheusToken, setClearPrometheusToken] = useState(false);
const [diskUsage, setDiskUsage] = useState<DiskUsageSummary | null>(null);
const [testingS3, setTestingS3] = useState(false);
const [archivingNow, setArchivingNow] = useState(false);
const [s3ActionStatus, setS3ActionStatus] = useState<{
ok: boolean;
text: string;
} | null>(null);
const [fieldErrors, setFieldErrors] = useState<SettingsValidationErrors>({}); const [fieldErrors, setFieldErrors] = useState<SettingsValidationErrors>({});
const [activeTab, setActiveTab] = useState<TabId>('general'); const [activeTab, setActiveTab] = useState<TabId>('general');
const [jwtSecret, setJwtSecret] = useState(''); const [jwtSecret, setJwtSecret] = useState('');
@@ -169,40 +257,73 @@ export function SettingsWorkspace() {
); );
const loadedPort = const loadedPort =
Number.isFinite(runtimePort) && runtimePort > 0 ? runtimePort : settingsPort; Number.isFinite(runtimePort) && runtimePort > 0 ? runtimePort : settingsPort;
const rawSettings = settingsData.settings as PublicSettings;
const loaded: EditableSettings = { const loaded: EditableSettings = {
maxEventsPerSec: Number( maxEventsPerSec: Number(
settingsData.settings.maxEventsPerSec ?? rawSettings.maxEventsPerSec ??
DEFAULT_EDITABLE_SETTINGS.maxEventsPerSec DEFAULT_EDITABLE_SETTINGS.maxEventsPerSec
), ),
maxPayloadSizeKb: Number( maxPayloadSizeKb: Number(
settingsData.settings.maxPayloadSizeKb ?? rawSettings.maxPayloadSizeKb ??
DEFAULT_EDITABLE_SETTINGS.maxPayloadSizeKb DEFAULT_EDITABLE_SETTINGS.maxPayloadSizeKb
), ),
batchFlushThreshold: Number( batchFlushThreshold: Number(
settingsData.settings.batchFlushThreshold ?? rawSettings.batchFlushThreshold ??
DEFAULT_EDITABLE_SETTINGS.batchFlushThreshold DEFAULT_EDITABLE_SETTINGS.batchFlushThreshold
), ),
batchFlushIntervalMs: Number( batchFlushIntervalMs: Number(
settingsData.settings.batchFlushIntervalMs ?? rawSettings.batchFlushIntervalMs ??
DEFAULT_EDITABLE_SETTINGS.batchFlushIntervalMs DEFAULT_EDITABLE_SETTINGS.batchFlushIntervalMs
), ),
fileRotationMb: Number( fileRotationMb: Number(
settingsData.settings.fileRotationMb ?? rawSettings.fileRotationMb ??
DEFAULT_EDITABLE_SETTINGS.fileRotationMb DEFAULT_EDITABLE_SETTINGS.fileRotationMb
), ),
enableIdempotency: Boolean( enableIdempotency: Boolean(
settingsData.settings.enableIdempotency ?? rawSettings.enableIdempotency ??
DEFAULT_EDITABLE_SETTINGS.enableIdempotency DEFAULT_EDITABLE_SETTINGS.enableIdempotency
), ),
autoPruneDays: Number( autoPruneDays: Number(
settingsData.settings.autoPruneDays ?? rawSettings.autoPruneDays ??
DEFAULT_EDITABLE_SETTINGS.autoPruneDays DEFAULT_EDITABLE_SETTINGS.autoPruneDays
), ),
compressAfterDays: Number(
rawSettings.compressAfterDays ??
DEFAULT_EDITABLE_SETTINGS.compressAfterDays
),
diskPauseThresholdPct: Number(
rawSettings.diskPauseThresholdPct ??
DEFAULT_EDITABLE_SETTINGS.diskPauseThresholdPct
),
webhookCooldownSec: Number(
rawSettings.webhookCooldownSec ??
DEFAULT_EDITABLE_SETTINGS.webhookCooldownSec
),
}; };
const cachedLoaded = setCached('vcon:settings', loaded); const loadedS3 = normalizeS3Settings(rawSettings);
const loadedAlerting = normalizeAlertingSettings(rawSettings);
const cachedLoaded = setCached('vcon:settings', normalizeEditableSettings(loaded));
const cachedS3 = setCached('vcon:s3-settings', loadedS3);
const cachedAlerting = setCached('vcon:alerting-settings', loadedAlerting);
setSavedSettings(cachedLoaded); setSavedSettings(cachedLoaded);
setNumericDrafts(settingsToNumericDrafts(cachedLoaded)); setNumericDrafts(settingsToNumericDrafts(cachedLoaded));
setEnableIdempotency(cachedLoaded.enableIdempotency); setEnableIdempotency(cachedLoaded.enableIdempotency);
setSavedS3Settings(cachedS3);
setS3Draft(cachedS3);
setHasS3SecretKey(Boolean(rawSettings.hasS3SecretKey));
setS3SecretKeyMask(String(rawSettings.s3SecretKeyMask || ''));
setS3SecretAccessKey('');
setClearS3SecretKey(false);
setSavedAlertingSettings(cachedAlerting);
setAlertingDraft(cachedAlerting);
setHasWebhookSecret(Boolean(rawSettings.hasWebhookSecret));
setWebhookSecretMask(String(rawSettings.webhookSecretMask || ''));
setWebhookSecret('');
setClearWebhookSecret(false);
setHasPrometheusToken(Boolean(rawSettings.hasPrometheusToken));
setPrometheusTokenMask(String(rawSettings.prometheusTokenMask || ''));
setPrometheusMetricsToken('');
setClearPrometheusToken(false);
setFieldErrors({}); setFieldErrors({});
setServerPort(setCached('vcon:server-port', loadedPort)); setServerPort(setCached('vcon:server-port', loadedPort));
setStoragePath( setStoragePath(
@@ -211,6 +332,9 @@ export function SettingsWorkspace() {
String(storageData.stats?.storageDirectory ?? '') String(storageData.stats?.storageDirectory ?? '')
) )
); );
if (storageData.diskUsage && typeof storageData.diskUsage === 'object') {
setDiskUsage(storageData.diskUsage as DiskUsageSummary);
}
if (metricsData) { if (metricsData) {
setActiveSockets(Number(metricsData.activeSockets) || 0); setActiveSockets(Number(metricsData.activeSockets) || 0);
setSocketStatus( setSocketStatus(
@@ -228,14 +352,57 @@ export function SettingsWorkspace() {
.finally(() => setLoading(false)); .finally(() => setLoading(false));
}, [error]); }, [error]);
const s3ConfigDirty = useMemo(
() =>
s3Draft.enableS3Backup !== savedS3Settings.enableS3Backup ||
s3Draft.s3Endpoint.trim() !== savedS3Settings.s3Endpoint ||
(s3Draft.s3Region.trim() || 'auto') !== (savedS3Settings.s3Region || 'auto') ||
s3Draft.s3Bucket.trim() !== savedS3Settings.s3Bucket ||
s3Draft.s3Prefix.trim() !== savedS3Settings.s3Prefix ||
s3Draft.s3AccessKeyId.trim() !== savedS3Settings.s3AccessKeyId ||
s3SecretAccessKey.trim().length > 0 ||
clearS3SecretKey,
[clearS3SecretKey, s3Draft, s3SecretAccessKey, savedS3Settings]
);
const alertingConfigDirty = useMemo(
() =>
alertingDraft.enableWebhookAlerts !==
savedAlertingSettings.enableWebhookAlerts ||
alertingDraft.webhookUrl.trim() !== savedAlertingSettings.webhookUrl ||
webhookSecret.trim().length > 0 ||
clearWebhookSecret ||
alertingDraft.enablePrometheusMetrics !==
savedAlertingSettings.enablePrometheusMetrics ||
prometheusMetricsToken.trim().length > 0 ||
clearPrometheusToken,
[
alertingDraft,
clearPrometheusToken,
clearWebhookSecret,
prometheusMetricsToken,
savedAlertingSettings,
webhookSecret,
]
);
const isDirty = useMemo( const isDirty = useMemo(
() => () =>
jwtSecret.trim().length > 0 || jwtSecret.trim().length > 0 ||
enableIdempotency !== savedSettings.enableIdempotency || enableIdempotency !== savedSettings.enableIdempotency ||
s3ConfigDirty ||
alertingConfigDirty ||
NUMERIC_SETTING_KEYS.some( NUMERIC_SETTING_KEYS.some(
(key) => numericDrafts[key].trim() !== String(savedSettings[key]) (key) => numericDrafts[key].trim() !== String(savedSettings[key])
), ),
[enableIdempotency, jwtSecret, numericDrafts, savedSettings] [
alertingConfigDirty,
enableIdempotency,
jwtSecret,
numericDrafts,
s3ConfigDirty,
savedSettings,
]
); );
const updateNumberDraft = (key: NumericSettingKey, value: string) => { const updateNumberDraft = (key: NumericSettingKey, value: string) => {
@@ -257,6 +424,11 @@ export function SettingsWorkspace() {
const executeSave = async (validated: EditableSettings) => { const executeSave = async (validated: EditableSettings) => {
const trimmedSecret = jwtSecret.trim(); const trimmedSecret = jwtSecret.trim();
const normalizedS3 = normalizeS3Settings(s3Draft);
const trimmedS3Secret = s3SecretAccessKey.trim();
const normalizedAlerting = normalizeAlertingSettings(alertingDraft);
const trimmedWebhookSecret = webhookSecret.trim();
const trimmedPrometheusToken = prometheusMetricsToken.trim();
setConfirmRotateOpen(false); setConfirmRotateOpen(false);
setSaving(true); setSaving(true);
try { try {
@@ -265,6 +437,30 @@ export function SettingsWorkspace() {
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ body: JSON.stringify({
...validated, ...validated,
enableS3Backup: normalizedS3.enableS3Backup,
s3Endpoint: normalizedS3.s3Endpoint,
s3Region: normalizedS3.s3Region,
s3Bucket: normalizedS3.s3Bucket,
s3Prefix: normalizedS3.s3Prefix,
s3AccessKeyId: normalizedS3.s3AccessKeyId,
...(trimmedS3Secret
? { s3SecretAccessKey: trimmedS3Secret }
: clearS3SecretKey
? { clearS3SecretKey: true }
: {}),
enableWebhookAlerts: normalizedAlerting.enableWebhookAlerts,
webhookUrl: normalizedAlerting.webhookUrl,
...(trimmedWebhookSecret
? { webhookSecret: trimmedWebhookSecret }
: clearWebhookSecret
? { clearWebhookSecret: true }
: {}),
enablePrometheusMetrics: normalizedAlerting.enablePrometheusMetrics,
...(trimmedPrometheusToken
? { prometheusMetricsToken: trimmedPrometheusToken }
: clearPrometheusToken
? { clearPrometheusToken: true }
: {}),
...(trimmedSecret ? { jwtSecret: trimmedSecret } : {}), ...(trimmedSecret ? { jwtSecret: trimmedSecret } : {}),
}), }),
}); });
@@ -275,10 +471,32 @@ export function SettingsWorkspace() {
const data = await response.json(); const data = await response.json();
if (!response.ok) throw new Error(data.error || 'Settings update failed'); if (!response.ok) throw new Error(data.error || 'Settings update failed');
const synced = normalizeEditableSettings(data.settings ?? validated); const synced = normalizeEditableSettings(data.settings ?? validated);
const syncedS3 = normalizeS3Settings(data.settings ?? normalizedS3);
const syncedAlerting = normalizeAlertingSettings(
data.settings ?? normalizedAlerting
);
setCached('vcon:settings', synced); setCached('vcon:settings', synced);
setCached('vcon:s3-settings', syncedS3);
setCached('vcon:alerting-settings', syncedAlerting);
setSavedSettings(synced); setSavedSettings(synced);
setNumericDrafts(settingsToNumericDrafts(synced)); setNumericDrafts(settingsToNumericDrafts(synced));
setEnableIdempotency(synced.enableIdempotency); setEnableIdempotency(synced.enableIdempotency);
setSavedS3Settings(syncedS3);
setS3Draft(syncedS3);
setHasS3SecretKey(Boolean(data.settings?.hasS3SecretKey));
setS3SecretKeyMask(String(data.settings?.s3SecretKeyMask || ''));
setS3SecretAccessKey('');
setClearS3SecretKey(false);
setSavedAlertingSettings(syncedAlerting);
setAlertingDraft(syncedAlerting);
setHasWebhookSecret(Boolean(data.settings?.hasWebhookSecret));
setWebhookSecretMask(String(data.settings?.webhookSecretMask || ''));
setWebhookSecret('');
setClearWebhookSecret(false);
setHasPrometheusToken(Boolean(data.settings?.hasPrometheusToken));
setPrometheusTokenMask(String(data.settings?.prometheusTokenMask || ''));
setPrometheusMetricsToken('');
setClearPrometheusToken(false);
setFieldErrors({}); setFieldErrors({});
setJwtSecret(''); setJwtSecret('');
setJwtError(''); setJwtError('');
@@ -316,6 +534,127 @@ export function SettingsWorkspace() {
} }
}; };
const handleTestWebhookAlert = async () => {
setTestingWebhook(true);
setWebhookTestStatus(null);
try {
const normalizedAlerting = normalizeAlertingSettings(alertingDraft);
const trimmedWebhookSecret = webhookSecret.trim();
const response = await fetch('/api/settings/webhook-test', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
webhookUrl: normalizedAlerting.webhookUrl,
...(trimmedWebhookSecret ? { webhookSecret: trimmedWebhookSecret } : {}),
...(clearWebhookSecret ? { clearWebhookSecret: true } : {}),
}),
});
if (response.status === 401) {
window.location.assign('/vcon/login');
return;
}
const data = await response.json().catch(() => ({}));
if (!response.ok || !data.success) {
const msg = String(data.error || 'Webhook test alert failed.');
setWebhookTestStatus({ ok: false, text: msg });
error('Webhook test failed', msg);
return;
}
const msg = `Test alert delivered (HTTP ${data.status}).`;
setWebhookTestStatus({ ok: true, text: msg });
success('Webhook alert delivered', msg);
} catch (testErr) {
const msg =
testErr instanceof Error ? testErr.message : 'Webhook test alert failed.';
setWebhookTestStatus({ ok: false, text: msg });
error('Webhook test error', msg);
} finally {
setTestingWebhook(false);
}
};
const handleTestS3Connection = async () => {
setTestingS3(true);
setS3ActionStatus(null);
try {
const normalizedS3 = normalizeS3Settings(s3Draft);
const trimmedS3Secret = s3SecretAccessKey.trim();
const response = await fetch('/api/storage/s3-test', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
s3Endpoint: normalizedS3.s3Endpoint,
s3Region: normalizedS3.s3Region,
s3Bucket: normalizedS3.s3Bucket,
s3Prefix: normalizedS3.s3Prefix,
s3AccessKeyId: normalizedS3.s3AccessKeyId,
...(trimmedS3Secret ? { s3SecretAccessKey: trimmedS3Secret } : {}),
...(clearS3SecretKey ? { clearS3SecretKey: true } : {}),
}),
});
if (response.status === 401) {
window.location.assign('/vcon/login');
return;
}
const data = await response.json().catch(() => ({}));
if (!response.ok || !data.success) {
const msg = String(data.error || 'S3 connection check failed.');
setS3ActionStatus({ ok: false, text: msg });
error('S3 test failed', msg);
return;
}
const msg = `Connected to bucket "${data.bucket}" (HTTP ${data.status}).`;
setS3ActionStatus({ ok: true, text: msg });
success('S3 connection verified', msg);
} catch (testErr) {
const msg =
testErr instanceof Error ? testErr.message : 'S3 connection check failed.';
setS3ActionStatus({ ok: false, text: msg });
error('S3 test error', msg);
} finally {
setTestingS3(false);
}
};
const handleArchiveNow = async () => {
if (s3ConfigDirty) {
warning('Unsaved S3 settings', 'Save your S3 credentials before archiving.');
return;
}
setArchivingNow(true);
setS3ActionStatus(null);
try {
const response = await fetch('/api/storage/archive', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ pruneAfterUpload: false }),
});
if (response.status === 401) {
window.location.assign('/vcon/login');
return;
}
const data = await response.json().catch(() => ({}));
if (!response.ok || !data.success) {
const msg = String(data.error || 'Off-site S3 archive failed.');
setS3ActionStatus({ ok: false, text: msg });
error('Archive failed', msg);
return;
}
const compressed = Number(data.compressedCount) || 0;
const archived = Number(data.archivedCount) || 0;
const msg = `Compressed ${compressed} file(s) and archived ${archived} partition(s) to S3 (local copies retained).`;
setS3ActionStatus({ ok: true, text: msg });
success('Archive completed', msg);
} catch (archiveErr) {
const msg =
archiveErr instanceof Error ? archiveErr.message : 'Off-site S3 archive failed.';
setS3ActionStatus({ ok: false, text: msg });
error('Archive error', msg);
} finally {
setArchivingNow(false);
}
};
const handleFormSubmit = async (event: React.FormEvent<HTMLFormElement>) => { const handleFormSubmit = async (event: React.FormEvent<HTMLFormElement>) => {
event.preventDefault(); event.preventDefault();
const validation = validateEditableSettingsDraft( const validation = validateEditableSettingsDraft(
@@ -335,6 +674,38 @@ export function SettingsWorkspace() {
return; return;
} }
const trimmedEndpoint = s3Draft.s3Endpoint.trim();
if (trimmedEndpoint) {
try {
const parsed = new URL(
/^https?:\/\//i.test(trimmedEndpoint)
? trimmedEndpoint
: `https://${trimmedEndpoint}`
);
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
throw new Error('Invalid protocol');
}
} catch {
setActiveTab('storage');
error('Invalid S3 Endpoint', 'S3 Endpoint must be a valid http(s) URL.');
return;
}
}
const trimmedWebhookUrl = alertingDraft.webhookUrl.trim();
if (trimmedWebhookUrl) {
try {
const parsed = new URL(trimmedWebhookUrl);
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
throw new Error('Invalid protocol');
}
} catch {
setActiveTab('security');
error('Invalid Webhook URL', 'Webhook URL must be a valid http(s) URL.');
return;
}
}
setFieldErrors({}); setFieldErrors({});
const trimmedSecret = jwtSecret.trim(); const trimmedSecret = jwtSecret.trim();
if (trimmedSecret.length > 0 && trimmedSecret.length < 32) { if (trimmedSecret.length > 0 && trimmedSecret.length < 32) {
@@ -373,6 +744,16 @@ export function SettingsWorkspace() {
const discardChanges = () => { const discardChanges = () => {
setNumericDrafts(settingsToNumericDrafts(savedSettings)); setNumericDrafts(settingsToNumericDrafts(savedSettings));
setEnableIdempotency(savedSettings.enableIdempotency); setEnableIdempotency(savedSettings.enableIdempotency);
setS3Draft(savedS3Settings);
setS3SecretAccessKey('');
setClearS3SecretKey(false);
setS3ActionStatus(null);
setAlertingDraft(savedAlertingSettings);
setWebhookSecret('');
setClearWebhookSecret(false);
setWebhookTestStatus(null);
setPrometheusMetricsToken('');
setClearPrometheusToken(false);
setFieldErrors({}); setFieldErrors({});
setJwtSecret(''); setJwtSecret('');
setJwtError(''); setJwtError('');
@@ -563,7 +944,7 @@ export function SettingsWorkspace() {
{activeTab === 'storage' && ( {activeTab === 'storage' && (
<> <>
<SettingsGroup title="CSV files"> <SettingsGroup title="CSV files & compression">
<SettingsField label="Root path"> <SettingsField label="Root path">
<Input <Input
compact compact
@@ -572,6 +953,41 @@ export function SettingsWorkspace() {
className="border-[var(--border)] bg-[var(--surface-2)] font-mono text-[11px] text-[var(--text-muted)]" className="border-[var(--border)] bg-[var(--surface-2)] font-mono text-[11px] text-[var(--text-muted)]"
/> />
</SettingsField> </SettingsField>
<SettingsField label="VPS disk usage">
<div className="flex flex-col gap-1 border border-[var(--border)] bg-[var(--surface-2)] px-2.5 py-1.5">
<div className="flex items-center justify-between gap-2 font-mono text-[11px] text-[var(--text)]">
<span>
{diskUsage?.available
? `${diskUsage.usedPct.toFixed(1)}% used`
: 'Statfs unavailable'}
</span>
{diskUsage?.available && (
<span className="text-[10.5px] text-[var(--text-muted)]">
{formatGiB(diskUsage.freeBytes)} free /{' '}
{formatGiB(diskUsage.totalBytes)}
</span>
)}
</div>
{diskUsage?.available && (
<div className="h-1.5 w-full border border-[var(--border)] bg-[var(--bg-main)]">
<div
className={`h-full ${
Number(numericDrafts.diskPauseThresholdPct) > 0 &&
diskUsage.usedPct >=
Number(numericDrafts.diskPauseThresholdPct)
? 'bg-[var(--danger)]'
: diskUsage.usedPct >= 80
? 'bg-[var(--warning)]'
: 'bg-[var(--accent)]'
}`}
style={{
width: `${Math.min(100, Math.max(0, diskUsage.usedPct))}%`,
}}
/>
</div>
)}
</div>
</SettingsField>
<SettingsField label="File rotation"> <SettingsField label="File rotation">
<Input <Input
compact compact
@@ -602,7 +1018,200 @@ export function SettingsWorkspace() {
className="font-mono" className="font-mono"
/> />
</SettingsField> </SettingsField>
<SettingsField label="Auto-compress (.csv → .csv.gz)">
<Input
compact
type="number"
min={SETTINGS_LIMITS.compressAfterDays.min}
max={SETTINGS_LIMITS.compressAfterDays.max}
suffix="days"
value={numericDrafts.compressAfterDays}
error={fieldErrors.compressAfterDays}
onChange={(event) =>
updateNumberDraft('compressAfterDays', event.target.value)
}
className="font-mono"
/>
</SettingsField>
<SettingsField label="Disk pause threshold (0 or 50–99%)">
<Input
compact
type="number"
min={SETTINGS_LIMITS.diskPauseThresholdPct.min}
max={SETTINGS_LIMITS.diskPauseThresholdPct.max}
suffix="%"
value={numericDrafts.diskPauseThresholdPct}
error={fieldErrors.diskPauseThresholdPct}
onChange={(event) =>
updateNumberDraft('diskPauseThresholdPct', event.target.value)
}
className="font-mono"
/>
</SettingsField>
</SettingsGroup> </SettingsGroup>
<SettingsGroup title="S3 / R2 off-site archive">
<SettingsField label="Automatic archive before prune">
<FormSwitch
label="Enable off-site S3 / R2 backup"
checked={s3Draft.enableS3Backup}
onChange={(checked) =>
setS3Draft((prev) => ({ ...prev, enableS3Backup: checked }))
}
/>
</SettingsField>
<SettingsField label="S3 Endpoint URL">
<Input
compact
placeholder="https://<account>.r2.cloudflarestorage.com"
value={s3Draft.s3Endpoint}
onChange={(event) =>
setS3Draft((prev) => ({
...prev,
s3Endpoint: event.target.value,
}))
}
className="font-mono text-[11px]"
/>
</SettingsField>
<SettingsField label="Region">
<Input
compact
placeholder="auto"
value={s3Draft.s3Region}
onChange={(event) =>
setS3Draft((prev) => ({
...prev,
s3Region: event.target.value,
}))
}
className="font-mono"
/>
</SettingsField>
<SettingsField label="Bucket name">
<Input
compact
placeholder="my-datastack-archive"
value={s3Draft.s3Bucket}
onChange={(event) =>
setS3Draft((prev) => ({
...prev,
s3Bucket: event.target.value,
}))
}
className="font-mono"
/>
</SettingsField>
<SettingsField label="Key prefix">
<Input
compact
placeholder="datastack/"
value={s3Draft.s3Prefix}
onChange={(event) =>
setS3Draft((prev) => ({
...prev,
s3Prefix: event.target.value,
}))
}
className="font-mono"
/>
</SettingsField>
<SettingsField label="Access Key ID">
<Input
compact
placeholder="AKIA... / R2 Access Key ID"
value={s3Draft.s3AccessKeyId}
onChange={(event) =>
setS3Draft((prev) => ({
...prev,
s3AccessKeyId: event.target.value,
}))
}
className="font-mono text-[11px]"
/>
</SettingsField>
<SettingsField label="Secret Access Key (write-only)">
<div className="flex items-center gap-1.5">
<Input
compact
type="password"
autoComplete="new-password"
value={s3SecretAccessKey}
onChange={(event) => {
setS3SecretAccessKey(event.target.value);
if (clearS3SecretKey) setClearS3SecretKey(false);
}}
placeholder={
clearS3SecretKey
? 'Saved key will be cleared on Save'
: hasS3SecretKey
? `Saved (${s3SecretKeyMask || 'encrypted'}) — blank keeps current`
: 'Enter Secret Access Key...'
}
className="font-mono text-[11px]"
/>
{hasS3SecretKey && (
<Button
type="button"
size="xs"
variant={clearS3SecretKey ? 'danger' : 'ghost'}
onClick={() => {
setClearS3SecretKey((prev) => !prev);
setS3SecretAccessKey('');
}}
>
{clearS3SecretKey ? 'Undo clear' : 'Clear key'}
</Button>
)}
</div>
</SettingsField>
<SettingsField label="Archive diagnostics">
<div className="flex flex-col gap-1.5">
<div className="flex flex-wrap items-center gap-1.5">
<Button
type="button"
size="xs"
variant="secondary"
icon="Plug"
loading={testingS3}
disabled={testingS3 || archivingNow}
onClick={() => void handleTestS3Connection()}
>
Test S3 Connection
</Button>
<Button
type="button"
size="xs"
variant="secondary"
icon="UploadCloud"
loading={archivingNow}
disabled={testingS3 || archivingNow || s3ConfigDirty}
onClick={() => void handleArchiveNow()}
>
Archive Now
</Button>
</div>
{s3ConfigDirty && (
<span className="text-[10.5px] text-[var(--text-faint)]">
Save S3 changes before running Archive Now.
</span>
)}
{s3ActionStatus && (
<div
role="status"
className={`border px-2 py-1 text-[11px] ${
s3ActionStatus.ok
? 'border-[var(--accent)] bg-[var(--surface-2)] text-[var(--text)]'
: 'border-[var(--danger)] bg-[var(--danger-soft)] text-[var(--danger)]'
}`}
>
{s3ActionStatus.text}
</div>
)}
</div>
</SettingsField>
</SettingsGroup>
<SettingsGroup title="Schema"> <SettingsGroup title="Schema">
<SettingsField label="Row format"> <SettingsField label="Row format">
<Input <Input
@@ -617,6 +1226,7 @@ export function SettingsWorkspace() {
)} )}
{activeTab === 'security' && ( {activeTab === 'security' && (
<>
<SettingsGroup title="Authentication"> <SettingsGroup title="Authentication">
<SettingsField label="JWT secret (min 32 chars)"> <SettingsField label="JWT secret (min 32 chars)">
<Input <Input
@@ -651,6 +1261,173 @@ export function SettingsWorkspace() {
/> />
</SettingsField> </SettingsField>
</SettingsGroup> </SettingsGroup>
<SettingsGroup title="Outbound webhook alerts">
<SettingsField label="Alert notifications">
<FormSwitch
label="Enable webhook alerts (Slack / Discord / JSON)"
checked={alertingDraft.enableWebhookAlerts}
onChange={(checked) =>
setAlertingDraft((prev) => ({
...prev,
enableWebhookAlerts: checked,
}))
}
/>
</SettingsField>
<SettingsField label="Webhook URL">
<Input
compact
placeholder="https://hooks.slack.com/services/..."
value={alertingDraft.webhookUrl}
onChange={(event) =>
setAlertingDraft((prev) => ({
...prev,
webhookUrl: event.target.value,
}))
}
className="font-mono text-[11px]"
/>
</SettingsField>
<SettingsField label="Webhook HMAC secret (write-only)">
<div className="flex items-center gap-1.5">
<Input
compact
type="password"
autoComplete="new-password"
value={webhookSecret}
onChange={(event) => {
setWebhookSecret(event.target.value);
if (clearWebhookSecret) setClearWebhookSecret(false);
}}
placeholder={
clearWebhookSecret
? 'Saved secret will be cleared on Save'
: hasWebhookSecret
? `Saved (${webhookSecretMask || 'encrypted'}) — blank keeps current`
: 'Optional HMAC-SHA256 signing secret...'
}
className="font-mono text-[11px]"
/>
{hasWebhookSecret && (
<Button
type="button"
size="xs"
variant={clearWebhookSecret ? 'danger' : 'ghost'}
onClick={() => {
setClearWebhookSecret((prev) => !prev);
setWebhookSecret('');
}}
>
{clearWebhookSecret ? 'Undo clear' : 'Clear secret'}
</Button>
)}
</div>
</SettingsField>
<SettingsField label="Alert cooldown">
<Input
compact
type="number"
min={SETTINGS_LIMITS.webhookCooldownSec.min}
max={SETTINGS_LIMITS.webhookCooldownSec.max}
suffix="sec"
value={numericDrafts.webhookCooldownSec}
error={fieldErrors.webhookCooldownSec}
onChange={(event) =>
updateNumberDraft('webhookCooldownSec', event.target.value)
}
className="font-mono"
/>
</SettingsField>
<SettingsField label="Webhook diagnostics">
<div className="flex flex-col gap-1.5">
<div className="flex flex-wrap items-center gap-1.5">
<Button
type="button"
size="xs"
variant="secondary"
icon="Bell"
loading={testingWebhook}
disabled={testingWebhook}
onClick={() => void handleTestWebhookAlert()}
>
Send Test Alert
</Button>
</div>
{webhookTestStatus && (
<div
role="status"
className={`border px-2 py-1 text-[11px] ${
webhookTestStatus.ok
? 'border-[var(--accent)] bg-[var(--surface-2)] text-[var(--text)]'
: 'border-[var(--danger)] bg-[var(--danger-soft)] text-[var(--danger)]'
}`}
>
{webhookTestStatus.text}
</div>
)}
</div>
</SettingsField>
</SettingsGroup>
<SettingsGroup title="Prometheus metrics (/api/prometheus)">
<SettingsField label="Scrape endpoint">
<FormSwitch
label="Enable /api/prometheus scrape endpoint"
checked={alertingDraft.enablePrometheusMetrics}
onChange={(checked) =>
setAlertingDraft((prev) => ({
...prev,
enablePrometheusMetrics: checked,
}))
}
/>
</SettingsField>
<SettingsField label="Scrape URL">
<Input
compact
value={baseUrl ? `${baseUrl}/api/prometheus` : '/api/prometheus'}
readOnly
className="border-[var(--border)] bg-[var(--surface-2)] font-mono text-[11px] text-[var(--text-muted)]"
/>
</SettingsField>
<SettingsField label="Bearer scrape token (write-only)">
<div className="flex items-center gap-1.5">
<Input
compact
type="password"
autoComplete="new-password"
value={prometheusMetricsToken}
onChange={(event) => {
setPrometheusMetricsToken(event.target.value);
if (clearPrometheusToken) setClearPrometheusToken(false);
}}
placeholder={
clearPrometheusToken
? 'Saved token will be cleared on Save'
: hasPrometheusToken
? `Saved (${prometheusTokenMask || 'encrypted'}) — blank keeps current`
: 'Enter Bearer token for external Prometheus scrape...'
}
className="font-mono text-[11px]"
/>
{hasPrometheusToken && (
<Button
type="button"
size="xs"
variant={clearPrometheusToken ? 'danger' : 'ghost'}
onClick={() => {
setClearPrometheusToken((prev) => !prev);
setPrometheusMetricsToken('');
}}
>
{clearPrometheusToken ? 'Undo clear' : 'Clear token'}
</Button>
)}
</div>
</SettingsField>
</SettingsGroup>
</>
)} )}
<SettingsFooter <SettingsFooter
+3
View File
@@ -2,6 +2,9 @@ export interface ClientItem {
clientId: string; clientId: string;
name: string; name: string;
rateLimit: number; rateLimit: number;
storageQuotaMb?: number;
allowedIps?: string;
expiresAt?: string | null;
status: "active" | "revoked"; status: "active" | "revoked";
totalEvents: number; totalEvents: number;
lastActive?: string; lastActive?: string;
+36
View File
@@ -17,6 +17,9 @@ export const CORE_SCHEMA_TABLE_DDLS = [
secret TEXT NOT NULL, secret TEXT NOT NULL,
token TEXT NOT NULL, token TEXT NOT NULL,
rate_limit INTEGER NOT NULL DEFAULT 100, rate_limit INTEGER NOT NULL DEFAULT 100,
storage_quota_mb INTEGER NOT NULL DEFAULT 0,
allowed_ips TEXT NOT NULL DEFAULT '',
expires_at TIMESTAMPTZ,
status TEXT NOT NULL DEFAULT 'active', status TEXT NOT NULL DEFAULT 'active',
total_events BIGINT NOT NULL DEFAULT 0, total_events BIGINT NOT NULL DEFAULT 0,
last_active TIMESTAMPTZ, last_active TIMESTAMPTZ,
@@ -37,6 +40,21 @@ export const CORE_SCHEMA_TABLE_DDLS = [
jwt_secret TEXT NOT NULL, jwt_secret TEXT NOT NULL,
enable_idempotency BOOLEAN NOT NULL DEFAULT true, enable_idempotency BOOLEAN NOT NULL DEFAULT true,
auto_prune_days INTEGER NOT NULL DEFAULT 30, auto_prune_days INTEGER NOT NULL DEFAULT 30,
compress_after_days INTEGER NOT NULL DEFAULT 0,
enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
s3_endpoint TEXT NOT NULL DEFAULT '',
s3_region TEXT NOT NULL DEFAULT 'auto',
s3_bucket TEXT NOT NULL DEFAULT '',
s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
s3_access_key_id TEXT NOT NULL DEFAULT '',
s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
webhook_url TEXT NOT NULL DEFAULT '',
webhook_secret_enc TEXT NOT NULL DEFAULT '',
webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
) )
`, `,
@@ -71,6 +89,9 @@ export const CORE_SCHEMA_ALTER_DDLS = [
` `
ALTER TABLE client_sdks ALTER TABLE client_sdks
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
ADD COLUMN IF NOT EXISTS storage_quota_mb INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS allowed_ips TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active', ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0, ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ, ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
@@ -88,6 +109,21 @@ export const CORE_SCHEMA_ALTER_DDLS = [
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true, ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30, ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
ADD COLUMN IF NOT EXISTS compress_after_days INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS s3_endpoint TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_region TEXT NOT NULL DEFAULT 'auto',
ADD COLUMN IF NOT EXISTS s3_bucket TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
ADD COLUMN IF NOT EXISTS enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS webhook_url TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS webhook_secret_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
ADD COLUMN IF NOT EXISTS enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
`, `,
` `
+219
View File
@@ -0,0 +1,219 @@
import net from "node:net";
/**
* Per-Client IP / CIDR Allowlist & Token Expiration Validator (V2-Phase-02)
*/
export function stripIpv6MappedPrefix(ip: string): string {
const trimmed = String(ip || "").trim();
if (trimmed.toLowerCase().startsWith("::ffff:")) {
const candidate = trimmed.slice(7);
if (net.isIPv4(candidate)) {
return candidate;
}
}
return trimmed;
}
export function ipv4ToUint32(ip: string): number | null {
const clean = stripIpv6MappedPrefix(ip);
if (!net.isIPv4(clean)) return null;
const parts = clean.split(".").map((octet) => Number(octet));
if (
parts.length !== 4 ||
parts.some((n) => !Number.isInteger(n) || n < 0 || n > 255)
) {
return null;
}
return (
(((parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3]) >>> 0)
);
}
export function parseAllowedIpsList(raw?: string | null): string[] {
if (!raw || typeof raw !== "string") return [];
return raw
.split(/[\s,;]+/)
.map((entry) => entry.trim())
.filter(Boolean);
}
export function validateAllowedIpsInput(
raw: unknown,
): { valid: true; normalized: string } | { valid: false; error: string } {
if (raw === undefined || raw === null || raw === "") {
return { valid: true, normalized: "" };
}
if (typeof raw !== "string") {
return { valid: false, error: "allowedIps must be a string." };
}
if (raw.length > 2048) {
return {
valid: false,
error: "allowedIps cannot exceed 2048 characters.",
};
}
const entries = parseAllowedIpsList(raw);
if (entries.length === 0) {
return { valid: true, normalized: "" };
}
if (entries.length > 64) {
return {
valid: false,
error: "allowedIps cannot contain more than 64 IP/CIDR entries.",
};
}
const normalizedEntries: string[] = [];
for (const entry of entries) {
if (entry === "*") {
normalizedEntries.push("*");
continue;
}
if (entry.includes("/")) {
const [ipPart, prefixPart, ...extra] = entry.split("/");
if (extra.length > 0 || !ipPart || !prefixPart || !/^\d+$/.test(prefixPart)) {
return {
valid: false,
error: `Invalid CIDR entry '${entry}'. Expected format like 203.0.113.0/24.`,
};
}
const prefix = Number(prefixPart);
const cleanIp = stripIpv6MappedPrefix(ipPart);
if (net.isIPv4(cleanIp)) {
if (prefix < 0 || prefix > 32) {
return {
valid: false,
error: `Invalid IPv4 CIDR prefix in '${entry}' (must be 0..32).`,
};
}
normalizedEntries.push(`${cleanIp}/${prefix}`);
continue;
}
if (net.isIPv6(cleanIp)) {
if (prefix < 0 || prefix > 128) {
return {
valid: false,
error: `Invalid IPv6 CIDR prefix in '${entry}' (must be 0..128).`,
};
}
normalizedEntries.push(`${cleanIp.toLowerCase()}/${prefix}`);
continue;
}
return {
valid: false,
error: `Invalid IP address in CIDR entry '${entry}'.`,
};
}
const cleanIp = stripIpv6MappedPrefix(entry);
const ipVersion = net.isIP(cleanIp);
if (ipVersion === 4) {
normalizedEntries.push(cleanIp);
} else if (ipVersion === 6) {
normalizedEntries.push(cleanIp.toLowerCase());
} else {
return {
valid: false,
error: `Invalid IP or CIDR entry '${entry}'.`,
};
}
}
return {
valid: true,
normalized: Array.from(new Set(normalizedEntries)).join(", "),
};
}
export function isClientIpAllowed(
clientIp: string,
allowedIpsRaw?: string | null,
): boolean {
const entries = parseAllowedIpsList(allowedIpsRaw);
if (entries.length === 0) {
return true;
}
const cleanClientIp = stripIpv6MappedPrefix(clientIp);
if (!cleanClientIp || cleanClientIp === "unknown") {
return entries.includes("*");
}
const clientIpv4Int = ipv4ToUint32(cleanClientIp);
const isLoopbackClient =
cleanClientIp === "127.0.0.1" || cleanClientIp.toLowerCase() === "::1";
for (const rule of entries) {
if (rule === "*") return true;
if (rule.includes("/")) {
const [ruleIpRaw, prefixRaw] = rule.split("/");
const ruleIp = stripIpv6MappedPrefix(ruleIpRaw);
const prefix = Number(prefixRaw);
const ruleIpv4Int = ipv4ToUint32(ruleIp);
if (clientIpv4Int !== null && ruleIpv4Int !== null && prefix >= 0 && prefix <= 32) {
if (prefix === 0) return true;
const mask = prefix === 32 ? 0xffffffff : (~((1 << (32 - prefix)) - 1)) >>> 0;
if ((clientIpv4Int & mask) >>> 0 === (ruleIpv4Int & mask) >>> 0) {
return true;
}
}
continue;
}
const cleanRule = stripIpv6MappedPrefix(rule);
if (cleanRule.toLowerCase() === cleanClientIp.toLowerCase()) {
return true;
}
if (
isLoopbackClient &&
(cleanRule === "127.0.0.1" || cleanRule.toLowerCase() === "::1")
) {
return true;
}
}
return false;
}
export function validateClientExpiresAtInput(
raw: unknown,
): { valid: true; iso: string | null } | { valid: false; error: string } {
if (raw === undefined || raw === null || raw === "") {
return { valid: true, iso: null };
}
if (typeof raw !== "string") {
return {
valid: false,
error: "expiresAt must be an ISO timestamp string or null.",
};
}
const trimmed = raw.trim();
if (!trimmed) {
return { valid: true, iso: null };
}
const parsedMs = Date.parse(trimmed);
if (!Number.isFinite(parsedMs)) {
return {
valid: false,
error: "expiresAt must be a valid ISO-8601 date or timestamp.",
};
}
return { valid: true, iso: new Date(parsedMs).toISOString() };
}
export function isClientTokenExpired(
expiresAt?: string | null,
nowMs = Date.now(),
): boolean {
if (!expiresAt || typeof expiresAt !== "string") return false;
const trimmed = expiresAt.trim();
if (!trimmed) return false;
const parsedMs = Date.parse(trimmed);
if (!Number.isFinite(parsedMs)) return false;
return nowMs >= parsedMs;
}
+152
View File
@@ -0,0 +1,152 @@
import {
createCipheriv,
createDecipheriv,
hkdfSync,
randomBytes,
} from "node:crypto";
const DEV_DEFAULT_SESSION_SECRET =
"datastack-admin-session-default-secret-32-chars";
const VAULT_PREFIX = "v1";
const HKDF_SALT = Buffer.from("datastack-vault-salt-v1", "utf8");
const HKDF_INFO = Buffer.from("datastack-aes-256-gcm-v1", "utf8");
export type SecretVaultErrorCode =
| "SECRET_VAULT_KEY_UNAVAILABLE"
| "SECRET_VAULT_DECRYPT_FAILED"
| "SECRET_VAULT_INVALID_INPUT";
export class SecretVaultError extends Error {
public readonly code: SecretVaultErrorCode;
constructor(code: SecretVaultErrorCode, message: string) {
super(`${code}: ${message}`);
this.name = "SecretVaultError";
this.code = code;
}
}
export function resolveVaultMasterSecret(secretOverride?: string): string | null {
const candidate = (
secretOverride ??
process.env.DATASTACK_ADMIN_SESSION_SECRET ??
""
).trim();
if (process.env.NODE_ENV === "production") {
return candidate.length >= 32 ? candidate : null;
}
if (candidate.length > 0) {
return candidate.length >= 32 ? candidate : null;
}
return DEV_DEFAULT_SESSION_SECRET;
}
function deriveAesKey(secretOverride?: string): Buffer {
const master = resolveVaultMasterSecret(secretOverride);
if (!master) {
throw new SecretVaultError(
"SECRET_VAULT_KEY_UNAVAILABLE",
"DATASTACK_ADMIN_SESSION_SECRET (>= 32 characters) is required to encrypt or decrypt stored secrets.",
);
}
const derived = hkdfSync(
"sha256",
Buffer.from(master, "utf8"),
HKDF_SALT,
HKDF_INFO,
32,
);
return Buffer.from(derived);
}
export function encryptSecret(
plaintext: string,
secretOverride?: string,
): string {
if (typeof plaintext !== "string") {
throw new SecretVaultError(
"SECRET_VAULT_INVALID_INPUT",
"Secret value must be a string.",
);
}
if (!plaintext) {
return "";
}
const key = deriveAesKey(secretOverride);
const iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", key, iv);
const encrypted = Buffer.concat([
cipher.update(plaintext, "utf8"),
cipher.final(),
]);
const authTag = cipher.getAuthTag();
return [
VAULT_PREFIX,
iv.toString("base64url"),
authTag.toString("base64url"),
encrypted.toString("base64url"),
].join(":");
}
export function decryptSecret(
ciphertext: string,
secretOverride?: string,
): string {
if (!ciphertext || typeof ciphertext !== "string") {
return "";
}
const parts = ciphertext.trim().split(":");
if (parts.length !== 4 || parts[0] !== VAULT_PREFIX) {
throw new SecretVaultError(
"SECRET_VAULT_DECRYPT_FAILED",
"Stored secret ciphertext has an unrecognized format.",
);
}
const key = deriveAesKey(secretOverride);
try {
const iv = Buffer.from(parts[1], "base64url");
const authTag = Buffer.from(parts[2], "base64url");
const payload = Buffer.from(parts[3], "base64url");
if (iv.length !== 12 || authTag.length !== 16) {
throw new Error("Invalid IV or auth tag length");
}
const decipher = createDecipheriv("aes-256-gcm", key, iv);
decipher.setAuthTag(authTag);
const decrypted = Buffer.concat([
decipher.update(payload),
decipher.final(),
]);
return decrypted.toString("utf8");
} catch (err: unknown) {
if (err instanceof SecretVaultError) {
throw err;
}
throw new SecretVaultError(
"SECRET_VAULT_DECRYPT_FAILED",
"Unable to decrypt stored secret. Verify DATASTACK_ADMIN_SESSION_SECRET matches the key used to encrypt it.",
);
}
}
export function maskSecretHint(plaintext: string): string {
const trimmed = String(plaintext || "").trim();
if (!trimmed) return "";
if (trimmed.length >= 8) {
return `••••${trimmed.slice(-4)}`;
}
return "••••••••";
}
export function getMaskedHintFromEncrypted(
ciphertext: string,
secretOverride?: string,
): string {
if (!ciphertext || !ciphertext.trim()) return "";
try {
const plain = decryptSecret(ciphertext, secretOverride);
return maskSecretHint(plain);
} catch {
return "••••(locked)";
}
}
+414 -29
View File
@@ -1,7 +1,10 @@
import crypto from 'node:crypto'; import crypto from 'node:crypto';
import fs from 'fs'; import fs from 'fs';
import path from 'path'; import path from 'path';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { StringDecoder } from 'node:string_decoder'; import { StringDecoder } from 'node:string_decoder';
import zlib from 'node:zlib';
import { Parser } from 'json2csv'; import { Parser } from 'json2csv';
export interface DataEvent { export interface DataEvent {
@@ -19,8 +22,28 @@ export interface StorageFileMeta {
sizeBytes: number; sizeBytes: number;
linesCount: number; linesCount: number;
modifiedAt: Date; modifiedAt: Date;
compressed?: boolean;
} }
export interface DiskUsageSnapshot {
usedPct: number;
availableBytes: number;
totalBytes: number;
checkedAt: number;
}
export interface StorageAdmissionResult {
allowed: boolean;
code?: 'DISK_SPACE_LOW' | 'STORAGE_QUOTA_EXCEEDED';
reason?: string;
}
export type StatfsProvider = (dir: string) => Promise<{
blocks: number;
bavail: number;
bsize: number;
}>;
interface RowCountCacheEntry { interface RowCountCacheEntry {
sizeBytes: number; sizeBytes: number;
mtimeMs: number; mtimeMs: number;
@@ -31,6 +54,7 @@ const LOCK_RETRY_MIN_MS = 5;
const LOCK_RETRY_MAX_MS = 18; const LOCK_RETRY_MAX_MS = 18;
const LOCK_TIMEOUT_MS = 10_000; const LOCK_TIMEOUT_MS = 10_000;
const STALE_LOCK_AGE_MS = 30_000; const STALE_LOCK_AGE_MS = 30_000;
const QUOTA_CACHE_TTL_MS = 30_000;
function sleepMs(ms: number): Promise<void> { function sleepMs(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms)); return new Promise((resolve) => setTimeout(resolve, ms));
@@ -49,11 +73,39 @@ export class CSVStorageManager {
public static baseStorageDir = resolveDefaultStorageDir(); public static baseStorageDir = resolveDefaultStorageDir();
public static maxFileSizeBytes = 100 * 1024 * 1024; // 100MB rotation limit public static maxFileSizeBytes = 100 * 1024 * 1024; // 100MB rotation limit
private static rowCountCache = new Map<string, RowCountCacheEntry>(); private static rowCountCache = new Map<string, RowCountCacheEntry>();
private static diskUsageCache: DiskUsageSnapshot | null = null;
private static clientBytesCache = new Map<
string,
{ checkedAt: number; bytes: number }
>();
private static statfsOverrideForTests: StatfsProvider | null = null;
public static isPartitionFilename(filename: string): boolean {
return filename.endsWith('.csv') || filename.endsWith('.csv.gz');
}
public static extractDateFromFilename(filename: string): string {
return filename.replace(/\.csv(?:\.gz)?$/, '').split('-part')[0];
}
public static clearRowCountCache(): void { public static clearRowCountCache(): void {
this.rowCountCache.clear(); this.rowCountCache.clear();
} }
public static clearQuotaCaches(clientId?: string): void {
this.diskUsageCache = null;
if (clientId) {
this.clientBytesCache.delete(this.sanitizeClientId(clientId));
} else {
this.clientBytesCache.clear();
}
}
public static setStatfsOverrideForTests(provider: StatfsProvider | null): void {
this.statfsOverrideForTests = provider;
this.diskUsageCache = null;
}
public static sanitizeClientId(clientId: string): string { public static sanitizeClientId(clientId: string): string {
const raw = String(clientId || 'unknown').trim().replace(/[^a-zA-Z0-9_-]/g, '_'); const raw = String(clientId || 'unknown').trim().replace(/[^a-zA-Z0-9_-]/g, '_');
return raw || 'unknown'; return raw || 'unknown';
@@ -299,6 +351,12 @@ export class CSVStorageManager {
private static countCsvDataRowsChunkedSync(filePath: string): number { private static countCsvDataRowsChunkedSync(filePath: string): number {
const counter = this.createChunkRowCounter(); const counter = this.createChunkRowCounter();
if (filePath.endsWith('.gz')) {
const compressed = fs.readFileSync(filePath);
const decompressed = zlib.gunzipSync(compressed);
counter.pushChunk(decompressed);
return counter.finish();
}
const buf = Buffer.allocUnsafe(64 * 1024); const buf = Buffer.allocUnsafe(64 * 1024);
const fd = fs.openSync(filePath, 'r'); const fd = fs.openSync(filePath, 'r');
try { try {
@@ -312,14 +370,34 @@ export class CSVStorageManager {
} }
} }
public static openPartitionTextStream(filePath: string): Readable {
const fileStream = fs.createReadStream(filePath, {
highWaterMark: 64 * 1024,
});
if (!filePath.endsWith('.gz')) {
return fileStream;
}
const gunzip = zlib.createGunzip();
fileStream.on('error', (err) => {
if (!gunzip.destroyed) {
gunzip.destroy(err);
}
});
gunzip.on('close', () => {
if (!fileStream.destroyed) {
fileStream.destroy();
}
});
fileStream.pipe(gunzip);
return gunzip;
}
private static async countCsvDataRowsStreamAsync( private static async countCsvDataRowsStreamAsync(
filePath: string, filePath: string,
): Promise<number> { ): Promise<number> {
const counter = this.createChunkRowCounter(); const counter = this.createChunkRowCounter();
return new Promise<number>((resolve, reject) => { return new Promise<number>((resolve, reject) => {
const stream = fs.createReadStream(filePath, { const stream = this.openPartitionTextStream(filePath);
highWaterMark: 64 * 1024,
});
stream.on('data', (chunk) => counter.pushChunk(chunk)); stream.on('data', (chunk) => counter.pushChunk(chunk));
stream.on('end', () => resolve(counter.finish())); stream.on('end', () => resolve(counter.finish()));
stream.on('error', (err) => reject(err)); stream.on('error', (err) => reject(err));
@@ -437,8 +515,9 @@ export class CSVStorageManager {
fs.writeFileSync(resolvedPath, fullCsv, 'utf8'); fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
} }
// Invalidate row cache for modified file immediately // Invalidate row & quota caches for modified file immediately
this.rowCountCache.delete(resolvedPath); this.rowCountCache.delete(resolvedPath);
this.clearQuotaCaches(clientId);
return resolvedPath; return resolvedPath;
}); });
@@ -449,7 +528,7 @@ export class CSVStorageManager {
} }
/** /**
* List all stored CSV files with cached row counts (invalidated on mtime/size changes) * List all stored CSV and gzipped CSV (.csv / .csv.gz) files with cached row counts
*/ */
public static listAllFiles(): StorageFileMeta[] { public static listAllFiles(): StorageFileMeta[] {
const results: StorageFileMeta[] = []; const results: StorageFileMeta[] = [];
@@ -473,7 +552,7 @@ export class CSVStorageManager {
const files = fs.readdirSync(fullClientPath); const files = fs.readdirSync(fullClientPath);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const fullFilePath = path.join(fullClientPath, file); const fullFilePath = path.join(fullClientPath, file);
let stats: fs.Stats; let stats: fs.Stats;
try { try {
@@ -505,7 +584,7 @@ export class CSVStorageManager {
} }
} }
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
results.push({ results.push({
clientId: clientDir, clientId: clientDir,
@@ -515,6 +594,7 @@ export class CSVStorageManager {
sizeBytes: stats.size, sizeBytes: stats.size,
linesCount, linesCount,
modifiedAt: stats.mtime, modifiedAt: stats.mtime,
compressed: file.endsWith('.csv.gz'),
}); });
} }
} }
@@ -551,7 +631,7 @@ export class CSVStorageManager {
const files = await fs.promises.readdir(fullClientPath); const files = await fs.promises.readdir(fullClientPath);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const fullFilePath = path.join(fullClientPath, file); const fullFilePath = path.join(fullClientPath, file);
const stats = await fs.promises.lstat(fullFilePath).catch(() => null); const stats = await fs.promises.lstat(fullFilePath).catch(() => null);
if (!stats || stats.isSymbolicLink() || !stats.isFile()) continue; if (!stats || stats.isSymbolicLink() || !stats.isFile()) continue;
@@ -578,7 +658,7 @@ export class CSVStorageManager {
} }
} }
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
results.push({ results.push({
clientId: clientDir, clientId: clientDir,
@@ -588,6 +668,7 @@ export class CSVStorageManager {
sizeBytes: stats.size, sizeBytes: stats.size,
linesCount, linesCount,
modifiedAt: stats.mtime, modifiedAt: stats.mtime,
compressed: file.endsWith('.csv.gz'),
}); });
} }
} }
@@ -605,7 +686,121 @@ export class CSVStorageManager {
} }
/** /**
* Prune CSV partitions older than autoPruneDays under cooperative partition locks * Compress uncompressed .csv partitions older than compressAfterDays into .csv.gz under cooperative partition locks
*/
public static async compressOldPartitionsAsync(
compressAfterDays: number,
): Promise<number> {
if (!Number.isFinite(compressAfterDays) || compressAfterDays <= 0) {
return 0;
}
const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null);
if (!baseStat || !baseStat.isDirectory()) {
return 0;
}
const cutoffDate = new Date(
Date.now() - Math.floor(compressAfterDays) * 86_400_000,
)
.toISOString()
.split('T')[0];
let compressedCount = 0;
try {
const clientDirs = await fs.promises.readdir(this.baseStorageDir);
for (const clientDir of clientDirs) {
const fullClientPath = path.join(this.baseStorageDir, clientDir);
const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null);
if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) {
continue;
}
const files = await fs.promises
.readdir(fullClientPath)
.catch(() => [] as string[]);
for (const file of files) {
if (!file.endsWith('.csv')) continue;
const datePart = this.extractDateFromFilename(file);
if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) {
continue;
}
const fullPath = path.join(fullClientPath, file);
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) {
continue;
}
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
let acquired = false;
const tryAcquireLock = async (): Promise<boolean> => {
const handle = await fs.promises.open(lockPath, 'wx');
try {
await handle.writeFile(lockValue, 'utf8');
} finally {
await handle.close();
}
return true;
};
try {
acquired = await tryAcquireLock();
} catch (err: unknown) {
if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') {
await this.removeStaleLockAtomicallyAsync(lockPath);
try {
acquired = await tryAcquireLock();
} catch {
acquired = false;
}
}
}
if (!acquired) continue;
let targetGzPath = `${fullPath}.gz`;
if (fs.existsSync(targetGzPath)) {
const stem = fullPath.replace(/\.csv$/, '');
let suffix = 2;
while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) {
suffix += 1;
}
targetGzPath = `${stem}-gz${suffix}.csv.gz`;
}
const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`;
try {
await pipeline(
fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }),
zlib.createGzip({ level: 6 }),
fs.createWriteStream(tmpGzPath),
);
await fs.promises
.utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime)
.catch(() => undefined);
await fs.promises.rename(tmpGzPath, targetGzPath);
await fs.promises.unlink(fullPath);
this.rowCountCache.delete(fullPath);
this.rowCountCache.delete(targetGzPath);
this.clearQuotaCaches(clientDir);
compressedCount += 1;
} catch {
await fs.promises.unlink(tmpGzPath).catch(() => undefined);
} finally {
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
}
}
}
} catch {
// Ignore directory scan failure
}
return compressedCount;
}
/**
* Prune CSV / CSV.GZ partitions older than autoPruneDays under cooperative partition locks
*/ */
public static pruneExpiredPartitions(autoPruneDays: number): number { public static pruneExpiredPartitions(autoPruneDays: number): number {
if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) { if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) {
@@ -636,8 +831,8 @@ export class CSVStorageManager {
const files = fs.readdirSync(fullClientPath); const files = fs.readdirSync(fullClientPath);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
const fullPath = path.join(fullClientPath, file); const fullPath = path.join(fullClientPath, file);
let fileLstat: fs.Stats; let fileLstat: fs.Stats;
@@ -683,6 +878,7 @@ export class CSVStorageManager {
if (fs.existsSync(fullPath)) { if (fs.existsSync(fullPath)) {
fs.unlinkSync(fullPath); fs.unlinkSync(fullPath);
this.rowCountCache.delete(fullPath); this.rowCountCache.delete(fullPath);
this.clearQuotaCaches(clientDir);
prunedCount += 1; prunedCount += 1;
} }
} catch { } catch {
@@ -701,27 +897,54 @@ export class CSVStorageManager {
} }
/** /**
* Non-blocking async partition pruner using fs.promises under cooperative partition locks * Archive and/or prune expired partitions under cooperative partition locks.
* Never deletes a local partition if options.beforeDeleteFile throws or rejects (V2-E02).
*/ */
public static async pruneExpiredPartitionsAsync( public static async pruneOrArchiveExpiredPartitionsAsync(
autoPruneDays: number, autoPruneDays: number,
): Promise<number> { options?: {
if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) { beforeDeleteFile?: (file: {
return 0; clientId: string;
filename: string;
filePath: string;
date: string;
}) => Promise<void>;
archiveOnlyWithoutDelete?: boolean;
maxCutoffDays?: number;
},
): Promise<{
scannedCount: number;
archivedCount: number;
prunedCount: number;
errors: string[];
}> {
const targetDays =
options?.maxCutoffDays !== undefined
? options.maxCutoffDays
: autoPruneDays;
if (!Number.isFinite(targetDays) || targetDays < 0) {
return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
}
if (options?.maxCutoffDays === undefined && targetDays <= 0) {
return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
} }
const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null); const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null);
if (!baseStat || !baseStat.isDirectory()) { if (!baseStat || !baseStat.isDirectory()) {
return 0; return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
} }
const cutoffDate = new Date( const cutoffDate = new Date(
Date.now() - Math.floor(autoPruneDays) * 86_400_000, Date.now() - Math.floor(targetDays) * 86_400_000,
) )
.toISOString() .toISOString()
.split('T')[0]; .split('T')[0];
let scannedCount = 0;
let archivedCount = 0;
let prunedCount = 0; let prunedCount = 0;
const errors: string[] = [];
try { try {
const clientDirs = await fs.promises.readdir(this.baseStorageDir); const clientDirs = await fs.promises.readdir(this.baseStorageDir);
for (const clientDir of clientDirs) { for (const clientDir of clientDirs) {
@@ -731,16 +954,21 @@ export class CSVStorageManager {
continue; continue;
} }
const files = await fs.promises.readdir(fullClientPath).catch(() => [] as string[]); const files = await fs.promises
.readdir(fullClientPath)
.catch(() => [] as string[]);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) {
continue;
}
const fullPath = path.join(fullClientPath, file); const fullPath = path.join(fullClientPath, file);
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) { if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) {
continue; continue;
} }
scannedCount += 1;
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`); const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`; const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
@@ -771,22 +999,176 @@ export class CSVStorageManager {
if (!acquired) continue; if (!acquired) continue;
try { try {
if (options?.beforeDeleteFile) {
await options.beforeDeleteFile({
clientId: clientDir,
filename: file,
filePath: fullPath,
date: datePart,
});
archivedCount += 1;
}
if (!options?.archiveOnlyWithoutDelete) {
await fs.promises.unlink(fullPath); await fs.promises.unlink(fullPath);
this.rowCountCache.delete(fullPath); this.rowCountCache.delete(fullPath);
this.clearQuotaCaches(clientDir);
prunedCount += 1; prunedCount += 1;
} catch { }
// Ignore individual unlink error or concurrent removal } catch (err: unknown) {
const msg = err instanceof Error ? err.message : 'Archive/prune error';
errors.push(`${clientDir}/${file}: ${msg}`);
} finally { } finally {
await this.releaseLockIfOwnedAsync(lockPath, lockValue); await this.releaseLockIfOwnedAsync(lockPath, lockValue);
} }
} }
} }
}
} catch { } catch {
// Ignore directory scan failure // Ignore directory scan failure
} }
return prunedCount; return { scannedCount, archivedCount, prunedCount, errors };
}
/**
* Non-blocking async partition pruner using fs.promises under cooperative partition locks
*/
public static async pruneExpiredPartitionsAsync(
autoPruneDays: number,
): Promise<number> {
const res = await this.pruneOrArchiveExpiredPartitionsAsync(autoPruneDays);
return res.prunedCount;
}
/**
* Cached VPS filesystem disk usage snapshot (30s TTL) for disk-full protection
*/
public static async getDiskUsageAsync(): Promise<DiskUsageSnapshot> {
const now = Date.now();
if (
this.diskUsageCache &&
now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS
) {
return this.diskUsageCache;
}
let probeDir = path.resolve(this.baseStorageDir);
while (!fs.existsSync(probeDir)) {
const parent = path.dirname(probeDir);
if (parent === probeDir) break;
probeDir = parent;
}
try {
const statfsRes = this.statfsOverrideForTests
? await this.statfsOverrideForTests(probeDir)
: await fs.promises.statfs(probeDir);
const bsize = Math.max(1, Number(statfsRes.bsize) || 4096);
const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize);
const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize);
const usedBytes = Math.max(0, totalBytes - availableBytes);
const usedPct =
totalBytes > 0
? Number(
Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2),
)
: 0;
const snapshot: DiskUsageSnapshot = {
usedPct,
availableBytes,
totalBytes,
checkedAt: now,
};
this.diskUsageCache = snapshot;
return snapshot;
} catch {
const fallback: DiskUsageSnapshot = {
usedPct: 0,
availableBytes: 0,
totalBytes: 0,
checkedAt: now,
};
this.diskUsageCache = fallback;
return fallback;
}
}
/**
* Cached per-client directory byte count (30s TTL)
*/
public static async getClientStorageBytesAsync(
clientId: string,
): Promise<number> {
const safeClientId = this.sanitizeClientId(clientId);
const now = Date.now();
const cached = this.clientBytesCache.get(safeClientId);
if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) {
return cached.bytes;
}
const resolvedBase = path.resolve(this.baseStorageDir);
const clientDir = path.resolve(resolvedBase, safeClientId);
if (!clientDir.startsWith(resolvedBase + path.sep)) {
return 0;
}
let totalBytes = 0;
try {
const dirStat = await fs.promises.lstat(clientDir).catch(() => null);
if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) {
const entries = await fs.promises
.readdir(clientDir)
.catch(() => [] as string[]);
for (const entry of entries) {
if (!this.isPartitionFilename(entry)) continue;
const filePath = path.join(clientDir, entry);
const stats = await fs.promises.lstat(filePath).catch(() => null);
if (stats && !stats.isSymbolicLink() && stats.isFile()) {
totalBytes += stats.size;
}
}
}
} catch {
// Fall back to 0 on unreadable directory
}
this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes });
return totalBytes;
}
/**
* Checks global VPS disk threshold and per-client MB quota before admitting new events
*/
public static async checkStorageAdmissionAsync(
clientId: string,
clientQuotaMb: number,
diskPauseThresholdPct: number,
): Promise<StorageAdmissionResult> {
const threshold = Number(diskPauseThresholdPct);
if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) {
const disk = await this.getDiskUsageAsync();
if (disk.totalBytes > 0 && disk.usedPct >= threshold) {
return {
allowed: false,
code: 'DISK_SPACE_LOW',
reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`,
};
}
}
const quotaMb = Number(clientQuotaMb);
if (Number.isFinite(quotaMb) && quotaMb > 0) {
const currentBytes = await this.getClientStorageBytesAsync(clientId);
const quotaBytes = Math.floor(quotaMb) * 1024 * 1024;
if (currentBytes >= quotaBytes) {
return {
allowed: false,
code: 'STORAGE_QUOTA_EXCEEDED',
reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`,
};
}
}
return { allowed: true };
} }
public static formatStorageStats(files: StorageFileMeta[]) { public static formatStorageStats(files: StorageFileMeta[]) {
@@ -826,12 +1208,12 @@ export class CSVStorageManager {
} }
/** /**
* Reads file content for export / download with path traversal protection * Reads file content for export / download with path traversal protection (supports .csv & .csv.gz)
*/ */
public static getFileContent(clientId: string, filename: string): string | null { public static getFileContent(clientId: string, filename: string): string | null {
const safeClientId = this.sanitizeClientId(clientId); const safeClientId = this.sanitizeClientId(clientId);
const safeFilename = path.basename(String(filename || '')); const safeFilename = path.basename(String(filename || ''));
if (!safeClientId || !safeFilename || !safeFilename.endsWith('.csv')) { if (!safeClientId || !safeFilename || !this.isPartitionFilename(safeFilename)) {
return null; return null;
} }
const resolvedBase = path.resolve(this.baseStorageDir); const resolvedBase = path.resolve(this.baseStorageDir);
@@ -844,6 +1226,9 @@ export class CSVStorageManager {
if (stats.isSymbolicLink() || !stats.isFile()) { if (stats.isSymbolicLink() || !stats.isFile()) {
return null; return null;
} }
if (safeFilename.endsWith('.gz')) {
return zlib.gunzipSync(fs.readFileSync(fullPath)).toString('utf8');
}
return fs.readFileSync(fullPath, 'utf8'); return fs.readFileSync(fullPath, 'utf8');
} }
return null; return null;
+335
View File
@@ -0,0 +1,335 @@
import crypto from "node:crypto";
import fs from "node:fs";
export interface S3ArchiveConfig {
endpoint: string;
region: string;
bucket: string;
prefix: string;
accessKeyId: string;
secretAccessKey: string;
}
export interface S3OperationResult {
ok: boolean;
status?: number;
endpoint?: string;
bucket?: string;
key?: string;
etag?: string;
code?: "S3_INVALID_CONFIG" | "S3_CONNECTION_FAILED" | "S3_ARCHIVE_FAILED";
error?: string;
}
function sha256Hex(data: Buffer | string): string {
return crypto.createHash("sha256").update(data).digest("hex");
}
function hmacSha256(key: Buffer | string, data: string): Buffer {
return crypto.createHmac("sha256", key).update(data, "utf8").digest();
}
function encodeRfc3986(value: string): string {
return encodeURIComponent(value).replace(
/[!'()*]/g,
(ch) => `%${ch.charCodeAt(0).toString(16).toUpperCase()}`,
);
}
function encodeS3Path(rawPath: string): string {
return rawPath
.split("/")
.map((segment) => encodeRfc3986(segment))
.join("/");
}
export function buildS3ObjectKey(
prefix: string,
clientId: string,
filename: string,
): string {
const cleanPrefix = String(prefix || "")
.trim()
.replace(/^\/+|\/+$/g, "");
const cleanClient = String(clientId || "unknown")
.trim()
.replace(/^\/+|\/+$/g, "");
const cleanFile = String(filename || "")
.trim()
.replace(/^\/+/, "");
return [cleanPrefix, cleanClient, cleanFile].filter(Boolean).join("/");
}
export function normalizeS3EndpointUrl(
endpoint: string,
region: string,
): { url: URL; region: string } {
const trimmedRegion = String(region || "").trim() || "auto";
const trimmedEndpoint = String(endpoint || "").trim().replace(/\/+$/, "");
if (trimmedEndpoint) {
const withProtocol = /^https?:\/\//i.test(trimmedEndpoint)
? trimmedEndpoint
: `https://${trimmedEndpoint}`;
const parsed = new URL(withProtocol);
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
throw new Error("S3 endpoint must use http:// or https://");
}
return { url: parsed, region: trimmedRegion };
}
if (trimmedRegion === "auto") {
throw new Error(
"S3 Endpoint URL is required when Region is set to 'auto' (e.g. Cloudflare R2 or MinIO).",
);
}
return {
url: new URL(`https://s3.${trimmedRegion}.amazonaws.com`),
region: trimmedRegion,
};
}
export function signS3Request(params: {
method: "GET" | "HEAD" | "PUT";
endpointUrl: URL;
bucket: string;
objectKey?: string;
queryParams?: Record<string, string>;
region: string;
accessKeyId: string;
secretAccessKey: string;
payloadHashHex: string;
contentType?: string;
now?: Date;
}): { targetUrl: string; headers: Record<string, string> } {
const now = params.now ?? new Date();
const amzDate = now.toISOString().replace(/[:-]|\.\d{3}/g, "");
const dateStamp = amzDate.slice(0, 8);
const basePath = params.endpointUrl.pathname.replace(/\/+$/, "");
const bucketPart = encodeRfc3986(params.bucket.trim());
const keyPart = params.objectKey ? `/${encodeS3Path(params.objectKey)}` : "";
const canonicalUri = `${basePath}/${bucketPart}${keyPart}`.replace(
/\/{2,}/g,
"/",
);
const queryEntries = Object.entries(params.queryParams ?? {}).sort(([a], [b]) =>
a < b ? -1 : a > b ? 1 : 0,
);
const canonicalQueryString = queryEntries
.map(([k, v]) => `${encodeRfc3986(k)}=${encodeRfc3986(v)}`)
.join("&");
const hostHeader = params.endpointUrl.host;
const headerMap: Record<string, string> = {
host: hostHeader,
"x-amz-content-sha256": params.payloadHashHex,
"x-amz-date": amzDate,
};
if (params.contentType) {
headerMap["content-type"] = params.contentType;
}
const sortedHeaderKeys = Object.keys(headerMap).sort();
const canonicalHeaders = sortedHeaderKeys
.map((key) => `${key}:${headerMap[key].trim()}\n`)
.join("");
const signedHeaders = sortedHeaderKeys.join(";");
const canonicalRequest = [
params.method,
canonicalUri,
canonicalQueryString,
canonicalHeaders,
signedHeaders,
params.payloadHashHex,
].join("\n");
const credentialScope = `${dateStamp}/${params.region}/s3/aws4_request`;
const stringToSign = [
"AWS4-HMAC-SHA256",
amzDate,
credentialScope,
sha256Hex(canonicalRequest),
].join("\n");
const kDate = hmacSha256(`AWS4${params.secretAccessKey}`, dateStamp);
const kRegion = hmacSha256(kDate, params.region);
const kService = hmacSha256(kRegion, "s3");
const kSigning = hmacSha256(kService, "aws4_request");
const signature = crypto
.createHmac("sha256", kSigning)
.update(stringToSign, "utf8")
.digest("hex");
const authorization = `AWS4-HMAC-SHA256 Credential=${params.accessKeyId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}`;
const targetUrl = `${params.endpointUrl.origin}${canonicalUri}${
canonicalQueryString ? `?${canonicalQueryString}` : ""
}`;
return {
targetUrl,
headers: {
"x-amz-date": amzDate,
"x-amz-content-sha256": params.payloadHashHex,
...(params.contentType ? { "content-type": params.contentType } : {}),
Authorization: authorization,
},
};
}
function validateS3Config(config: S3ArchiveConfig): string | null {
if (!config.bucket?.trim()) return "S3 Bucket name is required.";
if (!config.accessKeyId?.trim()) return "S3 Access Key ID is required.";
if (!config.secretAccessKey?.trim()) return "S3 Secret Access Key is required.";
return null;
}
export async function testS3Connection(
config: S3ArchiveConfig,
): Promise<S3OperationResult> {
const validationError = validateS3Config(config);
if (validationError) {
return {
ok: false,
code: "S3_INVALID_CONFIG",
error: validationError,
};
}
try {
const { url, region } = normalizeS3EndpointUrl(
config.endpoint,
config.region,
);
const signed = signS3Request({
method: "GET",
endpointUrl: url,
bucket: config.bucket.trim(),
queryParams: { "list-type": "2", "max-keys": "1" },
region,
accessKeyId: config.accessKeyId.trim(),
secretAccessKey: config.secretAccessKey,
payloadHashHex: sha256Hex(""),
});
const response = await fetch(signed.targetUrl, {
method: "GET",
headers: signed.headers,
cache: "no-store",
signal: AbortSignal.timeout(8000),
});
if (!response.ok) {
const bodyPreview = await response.text().catch(() => "");
return {
ok: false,
status: response.status,
endpoint: url.origin,
bucket: config.bucket.trim(),
code: "S3_CONNECTION_FAILED",
error: `S3 endpoint returned HTTP ${response.status}${
bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : ""
}`,
};
}
return {
ok: true,
status: response.status,
endpoint: url.origin,
bucket: config.bucket.trim(),
};
} catch (err: unknown) {
return {
ok: false,
code: "S3_CONNECTION_FAILED",
error:
err instanceof Error ? err.message : "Unable to reach S3 endpoint",
};
}
}
export async function uploadFileToS3(
config: S3ArchiveConfig,
localFilePath: string,
objectKey: string,
): Promise<S3OperationResult> {
const validationError = validateS3Config(config);
if (validationError) {
return {
ok: false,
code: "S3_INVALID_CONFIG",
error: validationError,
key: objectKey,
};
}
try {
const { url, region } = normalizeS3EndpointUrl(
config.endpoint,
config.region,
);
const bodyBuffer = await fs.promises.readFile(localFilePath);
const payloadHashHex = sha256Hex(bodyBuffer);
const contentType = localFilePath.endsWith(".gz")
? "application/gzip"
: "text/csv; charset=utf-8";
const signed = signS3Request({
method: "PUT",
endpointUrl: url,
bucket: config.bucket.trim(),
objectKey,
region,
accessKeyId: config.accessKeyId.trim(),
secretAccessKey: config.secretAccessKey,
payloadHashHex,
contentType,
});
const response = await fetch(signed.targetUrl, {
method: "PUT",
headers: {
...signed.headers,
"content-length": String(bodyBuffer.byteLength),
},
body: new Uint8Array(bodyBuffer),
cache: "no-store",
signal: AbortSignal.timeout(30000),
});
if (!response.ok) {
const bodyPreview = await response.text().catch(() => "");
return {
ok: false,
status: response.status,
endpoint: url.origin,
bucket: config.bucket.trim(),
key: objectKey,
code: "S3_ARCHIVE_FAILED",
error: `S3 upload failed with HTTP ${response.status}${
bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : ""
}`,
};
}
return {
ok: true,
status: response.status,
endpoint: url.origin,
bucket: config.bucket.trim(),
key: objectKey,
etag: response.headers.get("etag") ?? undefined,
};
} catch (err: unknown) {
return {
ok: false,
key: objectKey,
code: "S3_ARCHIVE_FAILED",
error: err instanceof Error ? err.message : "S3 upload request failed",
};
}
}
+443 -55
View File
@@ -1,12 +1,27 @@
import {
dispatchWebhookAlertAsync,
type WebhookAlertPayload,
type WebhookRuntimeConfig,
} from "../alerts/webhookNotifier";
import { AuthService, resolveDefaultJwtSecret } from "../auth/jwt"; import { AuthService, resolveDefaultJwtSecret } from "../auth/jwt";
import { DataBufferEngine } from "../data/buffer"; import { DataBufferEngine } from "../data/buffer";
import { DatabaseEngine } from "../Database/database"; import { DatabaseEngine } from "../Database/database";
import { SDKManager } from "../sdk/sdkManager"; import { SDKManager } from "../sdk/sdkManager";
import {
decryptSecret,
encryptSecret,
getMaskedHintFromEncrypted,
} from "../Security/secretVault";
import { import {
SecurityEngine, SecurityEngine,
getInternalIngestionHeaders, getInternalIngestionHeaders,
} from "../Security/security"; } from "../Security/security";
import { CSVStorageManager } from "../Storage/csvStorage"; import { CSVStorageManager } from "../Storage/csvStorage";
import {
buildS3ObjectKey,
type S3ArchiveConfig,
uploadFileToS3,
} from "../Storage/s3Archive";
export interface AdminSettings { export interface AdminSettings {
serverPort: number; serverPort: number;
@@ -19,9 +34,49 @@ export interface AdminSettings {
jwtSecret: string; jwtSecret: string;
enableIdempotency: boolean; enableIdempotency: boolean;
autoPruneDays: number; autoPruneDays: number;
compressAfterDays: number;
enableS3Backup: boolean;
s3Endpoint: string;
s3Region: string;
s3Bucket: string;
s3Prefix: string;
s3AccessKeyId: string;
s3SecretAccessKeyEnc: string;
diskPauseThresholdPct: number;
enableWebhookAlerts: boolean;
webhookUrl: string;
webhookSecretEnc: string;
webhookCooldownSec: number;
enablePrometheusMetrics: boolean;
prometheusMetricsTokenEnc: string;
updatedAt: string; updatedAt: string;
} }
export interface PublicAdminSettings
extends Omit<
AdminSettings,
| "jwtSecret"
| "s3SecretAccessKeyEnc"
| "webhookSecretEnc"
| "prometheusMetricsTokenEnc"
> {
hasS3SecretKey: boolean;
s3SecretKeyMask: string;
hasWebhookSecret: boolean;
webhookSecretMask: string;
hasPrometheusToken: boolean;
prometheusTokenMask: string;
}
export type AdminSettingsUpdateInput = Partial<AdminSettings> & {
s3SecretAccessKey?: string;
clearS3SecretKey?: boolean;
webhookSecret?: string;
clearWebhookSecret?: boolean;
prometheusMetricsToken?: string;
clearPrometheusToken?: boolean;
};
export interface SettingsBridgeSyncStatus { export interface SettingsBridgeSyncStatus {
attempted: boolean; attempted: boolean;
ok: boolean; ok: boolean;
@@ -36,9 +91,11 @@ type SettingsGlobals = typeof globalThis & {
__dataStackSettingsHydrated?: boolean; __dataStackSettingsHydrated?: boolean;
__dataStackLastSettingsBridgeSync?: SettingsBridgeSyncStatus; __dataStackLastSettingsBridgeSync?: SettingsBridgeSyncStatus;
__dataStackLastPruneMeta?: { days: number; ranAtMs: number }; __dataStackLastPruneMeta?: { days: number; ranAtMs: number };
__dataStackLastCompressMeta?: { days: number; ranAtMs: number };
}; };
const PRUNE_COOLDOWN_MS = 60 * 60 * 1000; const PRUNE_COOLDOWN_MS = 60 * 60 * 1000;
const COMPRESS_COOLDOWN_MS = 60 * 60 * 1000;
const globals = globalThis as SettingsGlobals; const globals = globalThis as SettingsGlobals;
function getDefaultSettings(): AdminSettings { function getDefaultSettings(): AdminSettings {
@@ -53,6 +110,21 @@ function getDefaultSettings(): AdminSettings {
jwtSecret: resolveDefaultJwtSecret(), jwtSecret: resolveDefaultJwtSecret(),
enableIdempotency: true, enableIdempotency: true,
autoPruneDays: 30, autoPruneDays: 30,
compressAfterDays: 0,
enableS3Backup: false,
s3Endpoint: "",
s3Region: "auto",
s3Bucket: "",
s3Prefix: "datastack/",
s3AccessKeyId: "",
s3SecretAccessKeyEnc: "",
diskPauseThresholdPct: 95,
enableWebhookAlerts: false,
webhookUrl: "",
webhookSecretEnc: "",
webhookCooldownSec: 300,
enablePrometheusMetrics: false,
prometheusMetricsTokenEnc: "",
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}; };
} }
@@ -64,8 +136,132 @@ function getSettingsState(): AdminSettings {
return globals.__dataStackAdminSettings; return globals.__dataStackAdminSettings;
} }
function toPublicSettings(settings: AdminSettings): PublicAdminSettings {
const s3SecretKeyMask = getMaskedHintFromEncrypted(
settings.s3SecretAccessKeyEnc,
);
const webhookSecretMask = getMaskedHintFromEncrypted(
settings.webhookSecretEnc,
);
const prometheusTokenMask = getMaskedHintFromEncrypted(
settings.prometheusMetricsTokenEnc,
);
return {
serverPort: settings.serverPort,
webPort: settings.webPort,
maxEventsPerSec: settings.maxEventsPerSec,
maxPayloadSizeKb: settings.maxPayloadSizeKb,
batchFlushThreshold: settings.batchFlushThreshold,
batchFlushIntervalMs: settings.batchFlushIntervalMs,
fileRotationMb: settings.fileRotationMb,
enableIdempotency: settings.enableIdempotency,
autoPruneDays: settings.autoPruneDays,
compressAfterDays: settings.compressAfterDays,
enableS3Backup: settings.enableS3Backup,
s3Endpoint: settings.s3Endpoint,
s3Region: settings.s3Region,
s3Bucket: settings.s3Bucket,
s3Prefix: settings.s3Prefix,
s3AccessKeyId: settings.s3AccessKeyId,
hasS3SecretKey: Boolean(settings.s3SecretAccessKeyEnc),
s3SecretKeyMask,
diskPauseThresholdPct: settings.diskPauseThresholdPct,
enableWebhookAlerts: settings.enableWebhookAlerts,
webhookUrl: settings.webhookUrl,
hasWebhookSecret: Boolean(settings.webhookSecretEnc),
webhookSecretMask,
webhookCooldownSec: settings.webhookCooldownSec,
enablePrometheusMetrics: settings.enablePrometheusMetrics,
hasPrometheusToken: Boolean(settings.prometheusMetricsTokenEnc),
prometheusTokenMask,
updatedAt: settings.updatedAt,
};
}
export class SettingsManager { export class SettingsManager {
private static applyRuntimeSettings(settings: AdminSettings, forcePrune = false) { public static triggerWebhookAlert(payload: WebhookAlertPayload): void {
const current = getSettingsState();
if (!current.enableWebhookAlerts || !current.webhookUrl.trim()) {
return;
}
void (async () => {
try {
let secret = "";
if (current.webhookSecretEnc) {
try {
secret = decryptSecret(current.webhookSecretEnc);
} catch {
secret = "";
}
}
await dispatchWebhookAlertAsync(payload, {
enabled: current.enableWebhookAlerts,
webhookUrl: current.webhookUrl,
webhookSecret: secret,
cooldownSec: current.webhookCooldownSec,
});
} catch {
// Never allow background webhook dispatch to throw into callers
}
})();
}
private static async executeArchiveAndPrune(
settings: AdminSettings,
): Promise<void> {
if (!settings.enableS3Backup) {
await CSVStorageManager.pruneExpiredPartitionsAsync(
settings.autoPruneDays,
);
return;
}
const s3Config: S3ArchiveConfig = {
endpoint: settings.s3Endpoint,
region: settings.s3Region,
bucket: settings.s3Bucket,
prefix: settings.s3Prefix,
accessKeyId: settings.s3AccessKeyId,
secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc),
};
await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync(
settings.autoPruneDays,
{
beforeDeleteFile: async (file) => {
const objectKey = buildS3ObjectKey(
s3Config.prefix,
file.clientId,
file.filename,
);
const result = await uploadFileToS3(
s3Config,
file.filePath,
objectKey,
);
if (!result.ok) {
const errMsg =
result.error || `Failed to upload ${file.filename} to S3`;
this.triggerWebhookAlert({
event: "S3_ARCHIVE_FAILED",
severity: "critical",
title: "Off-Site S3 Archive Upload Failed",
message: errMsg,
clientId: file.clientId,
details: { filename: file.filename, bucket: s3Config.bucket },
});
throw new Error(errMsg);
}
},
},
);
}
private static applyRuntimeSettings(
settings: AdminSettings,
forcePrune = false,
forceCompress = false,
) {
SecurityEngine.maxEventsPerSec = settings.maxEventsPerSec; SecurityEngine.maxEventsPerSec = settings.maxEventsPerSec;
SecurityEngine.enableIdempotency = settings.enableIdempotency; SecurityEngine.enableIdempotency = settings.enableIdempotency;
SecurityEngine.maxPayloadSizeBytes = settings.maxPayloadSizeKb * 1024; SecurityEngine.maxPayloadSizeBytes = settings.maxPayloadSizeKb * 1024;
@@ -76,10 +272,34 @@ export class SettingsManager {
} }
CSVStorageManager.maxFileSizeBytes = settings.fileRotationMb * 1024 * 1024; CSVStorageManager.maxFileSizeBytes = settings.fileRotationMb * 1024 * 1024;
const hasConfiguredDb = Boolean(process.env.DATABASE_URL?.trim()); const hasConfiguredDb = Boolean(process.env.DATABASE_URL?.trim());
if ( const dbReadyForMaintenance =
settings.autoPruneDays > 0 && !hasConfiguredDb || globals.__dataStackSettingsHydrated === true;
(!hasConfiguredDb || globals.__dataStackSettingsHydrated === true)
) { if (settings.compressAfterDays > 0 && dbReadyForMaintenance) {
const prevComp = globals.__dataStackLastCompressMeta;
const now = Date.now();
const shouldCompress =
forceCompress ||
!prevComp ||
prevComp.days !== settings.compressAfterDays ||
now - prevComp.ranAtMs >= COMPRESS_COOLDOWN_MS;
if (shouldCompress) {
globals.__dataStackLastCompressMeta = {
days: settings.compressAfterDays,
ranAtMs: now,
};
void CSVStorageManager.compressOldPartitionsAsync(
settings.compressAfterDays,
).catch((err) => {
console.warn(
"[DataStack Storage] Background partition compression failed:",
err,
);
});
}
}
if (settings.autoPruneDays > 0 && dbReadyForMaintenance) {
const prev = globals.__dataStackLastPruneMeta; const prev = globals.__dataStackLastPruneMeta;
const now = Date.now(); const now = Date.now();
const shouldPrune = const shouldPrune =
@@ -92,11 +312,9 @@ export class SettingsManager {
days: settings.autoPruneDays, days: settings.autoPruneDays,
ranAtMs: now, ranAtMs: now,
}; };
void CSVStorageManager.pruneExpiredPartitionsAsync( void this.executeArchiveAndPrune(settings).catch((err) => {
settings.autoPruneDays,
).catch((err) => {
console.warn( console.warn(
"[DataStack Storage] Background async partition prune failed:", "[DataStack Storage] Background async partition prune/archive failed:",
err, err,
); );
}); });
@@ -115,7 +333,12 @@ export class SettingsManager {
const result = await pool.query( const result = await pool.query(
`SELECT server_port, web_port, max_events_per_sec, max_payload_size_kb, `SELECT server_port, web_port, max_events_per_sec, max_payload_size_kb,
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb, batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
jwt_secret, enable_idempotency, auto_prune_days, updated_at jwt_secret, enable_idempotency, auto_prune_days,
compress_after_days, enable_s3_backup, s3_endpoint, s3_region,
s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc,
disk_pause_threshold_pct, enable_webhook_alerts, webhook_url,
webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics,
prometheus_metrics_token_enc, updated_at
FROM admin_settings FROM admin_settings
WHERE id = 'default'`, WHERE id = 'default'`,
); );
@@ -126,6 +349,21 @@ export class SettingsManager {
row.auto_prune_days !== undefined && row.auto_prune_days !== null row.auto_prune_days !== undefined && row.auto_prune_days !== null
? Number(row.auto_prune_days) ? Number(row.auto_prune_days)
: NaN; : NaN;
const parsedCompressDays =
row.compress_after_days !== undefined &&
row.compress_after_days !== null
? Number(row.compress_after_days)
: NaN;
const parsedDiskThreshold =
row.disk_pause_threshold_pct !== undefined &&
row.disk_pause_threshold_pct !== null
? Number(row.disk_pause_threshold_pct)
: NaN;
const parsedWebhookCooldown =
row.webhook_cooldown_sec !== undefined &&
row.webhook_cooldown_sec !== null
? Number(row.webhook_cooldown_sec)
: NaN;
const envSocketPort = Number(process.env.PORT_SOCKET); const envSocketPort = Number(process.env.PORT_SOCKET);
const envWebPort = Number(process.env.PORT); const envWebPort = Number(process.env.PORT);
const loaded: AdminSettings = { const loaded: AdminSettings = {
@@ -142,9 +380,34 @@ export class SettingsManager {
batchFlushThreshold: Number(row.batch_flush_threshold) || 500, batchFlushThreshold: Number(row.batch_flush_threshold) || 500,
batchFlushIntervalMs: Number(row.batch_flush_interval_ms) || 2000, batchFlushIntervalMs: Number(row.batch_flush_interval_ms) || 2000,
fileRotationMb: Number(row.file_rotation_mb) || 100, fileRotationMb: Number(row.file_rotation_mb) || 100,
jwtSecret: String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(), jwtSecret:
String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(),
enableIdempotency: Boolean(row.enable_idempotency), enableIdempotency: Boolean(row.enable_idempotency),
autoPruneDays: Number.isFinite(parsedPruneDays) ? parsedPruneDays : 30, autoPruneDays: Number.isFinite(parsedPruneDays) ? parsedPruneDays : 30,
compressAfterDays: Number.isFinite(parsedCompressDays)
? Math.max(0, parsedCompressDays)
: 0,
enableS3Backup: Boolean(row.enable_s3_backup),
s3Endpoint: String(row.s3_endpoint ?? ""),
s3Region: String(row.s3_region || "auto"),
s3Bucket: String(row.s3_bucket ?? ""),
s3Prefix: String(row.s3_prefix ?? "datastack/"),
s3AccessKeyId: String(row.s3_access_key_id ?? ""),
s3SecretAccessKeyEnc: String(row.s3_secret_access_key_enc ?? ""),
diskPauseThresholdPct: Number.isFinite(parsedDiskThreshold)
? parsedDiskThreshold
: 95,
enableWebhookAlerts: Boolean(row.enable_webhook_alerts),
webhookUrl: String(row.webhook_url ?? ""),
webhookSecretEnc: String(row.webhook_secret_enc ?? ""),
webhookCooldownSec:
Number.isFinite(parsedWebhookCooldown) && parsedWebhookCooldown >= 10
? Math.floor(parsedWebhookCooldown)
: 300,
enablePrometheusMetrics: Boolean(row.enable_prometheus_metrics),
prometheusMetricsTokenEnc: String(
row.prometheus_metrics_token_enc ?? "",
),
updatedAt: row.updated_at updatedAt: row.updated_at
? new Date(row.updated_at as string | Date).toISOString() ? new Date(row.updated_at as string | Date).toISOString()
: new Date().toISOString(), : new Date().toISOString(),
@@ -198,8 +461,17 @@ export class SettingsManager {
`INSERT INTO admin_settings ( `INSERT INTO admin_settings (
id, server_port, web_port, max_events_per_sec, max_payload_size_kb, id, server_port, web_port, max_events_per_sec, max_payload_size_kb,
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb, batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
jwt_secret, enable_idempotency, auto_prune_days, updated_at jwt_secret, enable_idempotency, auto_prune_days,
) VALUES ('default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, NOW()) compress_after_days, enable_s3_backup, s3_endpoint, s3_region,
s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc,
disk_pause_threshold_pct, enable_webhook_alerts, webhook_url,
webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics,
prometheus_metrics_token_enc, updated_at
) VALUES (
'default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
$11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21,
$22, $23, $24, $25, NOW()
)
ON CONFLICT (id) DO UPDATE SET ON CONFLICT (id) DO UPDATE SET
server_port = EXCLUDED.server_port, server_port = EXCLUDED.server_port,
web_port = EXCLUDED.web_port, web_port = EXCLUDED.web_port,
@@ -211,6 +483,21 @@ export class SettingsManager {
jwt_secret = EXCLUDED.jwt_secret, jwt_secret = EXCLUDED.jwt_secret,
enable_idempotency = EXCLUDED.enable_idempotency, enable_idempotency = EXCLUDED.enable_idempotency,
auto_prune_days = EXCLUDED.auto_prune_days, auto_prune_days = EXCLUDED.auto_prune_days,
compress_after_days = EXCLUDED.compress_after_days,
enable_s3_backup = EXCLUDED.enable_s3_backup,
s3_endpoint = EXCLUDED.s3_endpoint,
s3_region = EXCLUDED.s3_region,
s3_bucket = EXCLUDED.s3_bucket,
s3_prefix = EXCLUDED.s3_prefix,
s3_access_key_id = EXCLUDED.s3_access_key_id,
s3_secret_access_key_enc = EXCLUDED.s3_secret_access_key_enc,
disk_pause_threshold_pct = EXCLUDED.disk_pause_threshold_pct,
enable_webhook_alerts = EXCLUDED.enable_webhook_alerts,
webhook_url = EXCLUDED.webhook_url,
webhook_secret_enc = EXCLUDED.webhook_secret_enc,
webhook_cooldown_sec = EXCLUDED.webhook_cooldown_sec,
enable_prometheus_metrics = EXCLUDED.enable_prometheus_metrics,
prometheus_metrics_token_enc = EXCLUDED.prometheus_metrics_token_enc,
updated_at = NOW()`, updated_at = NOW()`,
[ [
settings.serverPort, settings.serverPort,
@@ -223,6 +510,21 @@ export class SettingsManager {
settings.jwtSecret, settings.jwtSecret,
settings.enableIdempotency, settings.enableIdempotency,
settings.autoPruneDays, settings.autoPruneDays,
settings.compressAfterDays,
settings.enableS3Backup,
settings.s3Endpoint,
settings.s3Region,
settings.s3Bucket,
settings.s3Prefix,
settings.s3AccessKeyId,
settings.s3SecretAccessKeyEnc,
settings.diskPauseThresholdPct,
settings.enableWebhookAlerts,
settings.webhookUrl,
settings.webhookSecretEnc,
settings.webhookCooldownSec,
settings.enablePrometheusMetrics,
settings.prometheusMetricsTokenEnc,
], ],
); );
} catch (err: unknown) { } catch (err: unknown) {
@@ -250,37 +552,49 @@ export class SettingsManager {
return { ...current }; return { ...current };
} }
public static getPublicSettings(): Omit<AdminSettings, "jwtSecret"> { public static getPublicSettings(): PublicAdminSettings {
const settings = this.getSettings(); return toPublicSettings(this.getSettings());
}
public static async getPublicSettingsAsync(): Promise<PublicAdminSettings> {
const settings = await this.getSettingsAsync();
return toPublicSettings(settings);
}
public static async getDecryptedS3ConfigAsync(): Promise<S3ArchiveConfig> {
const settings = await this.getSettingsAsync();
return { return {
serverPort: settings.serverPort, endpoint: settings.s3Endpoint,
webPort: settings.webPort, region: settings.s3Region,
maxEventsPerSec: settings.maxEventsPerSec, bucket: settings.s3Bucket,
maxPayloadSizeKb: settings.maxPayloadSizeKb, prefix: settings.s3Prefix,
batchFlushThreshold: settings.batchFlushThreshold, accessKeyId: settings.s3AccessKeyId,
batchFlushIntervalMs: settings.batchFlushIntervalMs, secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc),
fileRotationMb: settings.fileRotationMb,
enableIdempotency: settings.enableIdempotency,
autoPruneDays: settings.autoPruneDays,
updatedAt: settings.updatedAt,
}; };
} }
public static async getPublicSettingsAsync(): Promise< public static async getDecryptedWebhookConfigAsync(): Promise<WebhookRuntimeConfig> {
Omit<AdminSettings, "jwtSecret">
> {
const settings = await this.getSettingsAsync(); const settings = await this.getSettingsAsync();
return { return {
serverPort: settings.serverPort, enabled: settings.enableWebhookAlerts,
webPort: settings.webPort, webhookUrl: settings.webhookUrl,
maxEventsPerSec: settings.maxEventsPerSec, webhookSecret: settings.webhookSecretEnc
maxPayloadSizeKb: settings.maxPayloadSizeKb, ? decryptSecret(settings.webhookSecretEnc)
batchFlushThreshold: settings.batchFlushThreshold, : "",
batchFlushIntervalMs: settings.batchFlushIntervalMs, cooldownSec: settings.webhookCooldownSec,
fileRotationMb: settings.fileRotationMb, };
enableIdempotency: settings.enableIdempotency, }
autoPruneDays: settings.autoPruneDays,
updatedAt: settings.updatedAt, public static async getDecryptedPrometheusConfigAsync(): Promise<{
enabled: boolean;
token: string;
}> {
const settings = await this.getSettingsAsync();
return {
enabled: settings.enablePrometheusMetrics,
token: settings.prometheusMetricsTokenEnc
? decryptSecret(settings.prometheusMetricsTokenEnc)
: "",
}; };
} }
@@ -326,6 +640,12 @@ export class SettingsManager {
fileRotationMb: settings.fileRotationMb, fileRotationMb: settings.fileRotationMb,
enableIdempotency: settings.enableIdempotency, enableIdempotency: settings.enableIdempotency,
autoPruneDays: settings.autoPruneDays, autoPruneDays: settings.autoPruneDays,
compressAfterDays: settings.compressAfterDays,
enableS3Backup: settings.enableS3Backup,
diskPauseThresholdPct: settings.diskPauseThresholdPct,
enableWebhookAlerts: settings.enableWebhookAlerts,
webhookUrl: settings.webhookUrl,
webhookCooldownSec: settings.webhookCooldownSec,
...(jwtSecretChanged ? { jwtSecret: settings.jwtSecret } : {}), ...(jwtSecretChanged ? { jwtSecret: settings.jwtSecret } : {}),
}), }),
cache: "no-store", cache: "no-store",
@@ -355,7 +675,10 @@ export class SettingsManager {
globals.__dataStackLastSettingsBridgeSync = result; globals.__dataStackLastSettingsBridgeSync = result;
return result; return result;
} catch (err: unknown) { } catch (err: unknown) {
const message = err instanceof Error ? err.message : "Port 4000 settings-sync unreachable"; const message =
err instanceof Error
? err.message
: "Port 4000 settings-sync unreachable";
if ( if (
process.env.NODE_ENV === "production" || process.env.NODE_ENV === "production" ||
Boolean(process.env.INGESTION_INTERNAL_URL) Boolean(process.env.INGESTION_INTERNAL_URL)
@@ -374,18 +697,79 @@ export class SettingsManager {
} }
} }
public static updateSettings(updates: Partial<AdminSettings>): AdminSettings { private static mergeSettingsUpdate(
current: AdminSettings,
updates: AdminSettingsUpdateInput,
): AdminSettings {
const {
s3SecretAccessKey,
clearS3SecretKey,
webhookSecret,
clearWebhookSecret,
prometheusMetricsToken,
clearPrometheusToken,
...rest
} = updates;
let nextEncryptedSecret = current.s3SecretAccessKeyEnc;
if (clearS3SecretKey) {
nextEncryptedSecret = "";
} else if (
typeof s3SecretAccessKey === "string" &&
s3SecretAccessKey.trim().length > 0
) {
nextEncryptedSecret = encryptSecret(s3SecretAccessKey.trim());
} else if (rest.s3SecretAccessKeyEnc !== undefined) {
nextEncryptedSecret = rest.s3SecretAccessKeyEnc;
}
let nextWebhookSecretEnc = current.webhookSecretEnc;
if (clearWebhookSecret) {
nextWebhookSecretEnc = "";
} else if (
typeof webhookSecret === "string" &&
webhookSecret.trim().length > 0
) {
nextWebhookSecretEnc = encryptSecret(webhookSecret.trim());
} else if (rest.webhookSecretEnc !== undefined) {
nextWebhookSecretEnc = rest.webhookSecretEnc;
}
let nextPrometheusTokenEnc = current.prometheusMetricsTokenEnc;
if (clearPrometheusToken) {
nextPrometheusTokenEnc = "";
} else if (
typeof prometheusMetricsToken === "string" &&
prometheusMetricsToken.trim().length > 0
) {
nextPrometheusTokenEnc = encryptSecret(prometheusMetricsToken.trim());
} else if (rest.prometheusMetricsTokenEnc !== undefined) {
nextPrometheusTokenEnc = rest.prometheusMetricsTokenEnc;
}
return {
...current,
...rest,
s3SecretAccessKeyEnc: nextEncryptedSecret,
webhookSecretEnc: nextWebhookSecretEnc,
prometheusMetricsTokenEnc: nextPrometheusTokenEnc,
updatedAt: new Date().toISOString(),
};
}
public static updateSettings(
updates: AdminSettingsUpdateInput,
): AdminSettings {
const current = getSettingsState(); const current = getSettingsState();
const previousJwtSecret = current.jwtSecret; const previousJwtSecret = current.jwtSecret;
const jwtSecretChanged = const jwtSecretChanged =
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret; Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
const nextSettings: AdminSettings = { const nextSettings = this.mergeSettingsUpdate(current, updates);
...current,
...updates,
updatedAt: new Date().toISOString(),
};
globals.__dataStackAdminSettings = nextSettings; globals.__dataStackAdminSettings = nextSettings;
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); this.applyRuntimeSettings(
nextSettings,
updates.autoPruneDays !== undefined,
updates.compressAfterDays !== undefined,
);
if (jwtSecretChanged) { if (jwtSecretChanged) {
SDKManager.resignAllClientTokens(); SDKManager.resignAllClientTokens();
} }
@@ -403,17 +787,13 @@ export class SettingsManager {
} }
public static async updateSettingsAsync( public static async updateSettingsAsync(
updates: Partial<AdminSettings>, updates: AdminSettingsUpdateInput,
): Promise<AdminSettings> { ): Promise<AdminSettings> {
const current = await this.getSettingsAsync(); const current = await this.getSettingsAsync();
const previousJwtSecret = current.jwtSecret; const previousJwtSecret = current.jwtSecret;
const jwtSecretChanged = const jwtSecretChanged =
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret; Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
const nextSettings: AdminSettings = { const nextSettings = this.mergeSettingsUpdate(current, updates);
...current,
...updates,
updatedAt: new Date().toISOString(),
};
if ( if (
jwtSecretChanged && jwtSecretChanged &&
@@ -450,11 +830,15 @@ export class SettingsManager {
try { try {
await this.persistSettingsToDb(nextSettings); await this.persistSettingsToDb(nextSettings);
globals.__dataStackAdminSettings = nextSettings; globals.__dataStackAdminSettings = nextSettings;
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); this.applyRuntimeSettings(
nextSettings,
updates.autoPruneDays !== undefined,
updates.compressAfterDays !== undefined,
);
await SDKManager.resignAllClientTokensAsync(); await SDKManager.resignAllClientTokensAsync();
} catch (err) { } catch (err) {
globals.__dataStackAdminSettings = current; globals.__dataStackAdminSettings = current;
this.applyRuntimeSettings(current, false); this.applyRuntimeSettings(current, false, false);
await this.persistSettingsToDb(current).catch(() => undefined); await this.persistSettingsToDb(current).catch(() => undefined);
if (preSyncStatus.synced) { if (preSyncStatus.synced) {
await this.notifyIngestionSettingsSync( await this.notifyIngestionSettingsSync(
@@ -482,7 +866,11 @@ export class SettingsManager {
await this.persistSettingsToDb(nextSettings); await this.persistSettingsToDb(nextSettings);
globals.__dataStackAdminSettings = nextSettings; globals.__dataStackAdminSettings = nextSettings;
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined); this.applyRuntimeSettings(
nextSettings,
updates.autoPruneDays !== undefined,
updates.compressAfterDays !== undefined,
);
await this.notifyIngestionSettingsSync( await this.notifyIngestionSettingsSync(
nextSettings, nextSettings,
previousJwtSecret, previousJwtSecret,
+183
View File
@@ -0,0 +1,183 @@
import crypto from "node:crypto";
export type WebhookAlertEventType =
| "DISK_SPACE_LOW"
| "STORAGE_QUOTA_EXCEEDED"
| "S3_ARCHIVE_FAILED"
| "CLIENT_IP_FORBIDDEN"
| "CLIENT_TOKEN_EXPIRED"
| "ADMIN_LOGIN_LOCKOUT"
| "TEST_ALERT";
export interface WebhookAlertPayload {
event: WebhookAlertEventType;
severity: "info" | "warning" | "critical";
title: string;
message: string;
clientId?: string;
details?: Record<string, unknown>;
timestamp?: string;
}
export interface WebhookRuntimeConfig {
enabled: boolean;
webhookUrl: string;
webhookSecret: string;
cooldownSec: number;
}
export interface WebhookDispatchResult {
ok: boolean;
skipped?: boolean;
reason?: string;
status?: number;
error?: string;
}
type WebhookGlobals = typeof globalThis & {
__dataStackWebhookCooldownMap?: Map<string, number>;
};
const globals = globalThis as WebhookGlobals;
const DEFAULT_WEBHOOK_TIMEOUT_MS = 5000;
function getCooldownMap(): Map<string, number> {
if (!globals.__dataStackWebhookCooldownMap) {
globals.__dataStackWebhookCooldownMap = new Map<string, number>();
}
return globals.__dataStackWebhookCooldownMap;
}
export function clearWebhookCooldownCacheForTests(): void {
getCooldownMap().clear();
}
export function buildWebhookSignature(
rawBody: string,
secret: string,
): string {
const hmac = crypto
.createHmac("sha256", secret)
.update(rawBody, "utf8")
.digest("hex");
return `sha256=${hmac}`;
}
export function formatOutboundWebhookBody(payload: WebhookAlertPayload): string {
const timestamp = payload.timestamp || new Date().toISOString();
const prefix =
payload.severity === "critical"
? "[CRITICAL]"
: payload.severity === "warning"
? "[WARNING]"
: "[INFO]";
const clientTag = payload.clientId ? ` (client: ${payload.clientId})` : "";
const summaryLine = `${prefix} DataStack ${payload.event}${clientTag} — ${payload.title}: ${payload.message}`;
return JSON.stringify({
instance: "DataStack",
event: payload.event,
severity: payload.severity,
title: payload.title,
message: payload.message,
...(payload.clientId ? { clientId: payload.clientId } : {}),
...(payload.details ? { details: payload.details } : {}),
timestamp,
// Slack & Discord compatible top-level fields
text: summaryLine,
content: summaryLine,
});
}
export async function dispatchWebhookAlertAsync(
payload: WebhookAlertPayload,
config: WebhookRuntimeConfig,
options?: { bypassCooldown?: boolean; timeoutMs?: number },
): Promise<WebhookDispatchResult> {
const webhookUrl = String(config.webhookUrl || "").trim();
if (!options?.bypassCooldown && !config.enabled) {
return { ok: false, skipped: true, reason: "WEBHOOK_DISABLED" };
}
if (!webhookUrl) {
return {
ok: false,
skipped: !options?.bypassCooldown,
error: "Webhook URL is not configured.",
};
}
let parsedUrl: URL;
try {
parsedUrl = new URL(webhookUrl);
if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") {
return {
ok: false,
error: "Webhook URL must use http:// or https://.",
};
}
} catch {
return {
ok: false,
error: "Webhook URL is not a valid URL.",
};
}
const dedupeKey = `${payload.event}:${payload.clientId || "global"}`;
const cooldownMs = Math.max(0, (Number(config.cooldownSec) || 300) * 1000);
const now = Date.now();
const cooldownMap = getCooldownMap();
if (!options?.bypassCooldown && cooldownMs > 0) {
const lastSentAt = cooldownMap.get(dedupeKey);
if (lastSentAt !== undefined && now - lastSentAt < cooldownMs) {
return { ok: true, skipped: true, reason: "COOLDOWN_ACTIVE" };
}
cooldownMap.set(dedupeKey, now);
}
const timestamp = payload.timestamp || new Date(now).toISOString();
const bodyStr = formatOutboundWebhookBody({ ...payload, timestamp });
const headers: Record<string, string> = {
"Content-Type": "application/json",
"User-Agent": "DataStack-Webhook-Notifier/2.0",
"X-DataStack-Event": payload.event,
"X-DataStack-Timestamp": timestamp,
};
const secret = String(config.webhookSecret || "").trim();
if (secret) {
headers["X-DataStack-Signature"] = buildWebhookSignature(bodyStr, secret);
}
const timeoutMs = Math.max(500, options?.timeoutMs || DEFAULT_WEBHOOK_TIMEOUT_MS);
try {
const res = await fetch(parsedUrl.toString(), {
method: "POST",
headers,
body: bodyStr,
cache: "no-store",
signal: AbortSignal.timeout(timeoutMs),
});
if (!res.ok) {
if (!options?.bypassCooldown) {
// Keep cooldown entry so failing webhooks don't DDOS a broken upstream endpoint
}
const errText = await res.text().catch(() => "");
return {
ok: false,
status: res.status,
error: `Webhook endpoint returned HTTP ${res.status}${errText ? `: ${errText.slice(0, 160)}` : ""}`,
};
}
return { ok: true, status: res.status };
} catch (err: unknown) {
const msg =
err instanceof Error ? err.message : "Webhook delivery failed";
return {
ok: false,
error: `Webhook delivery failed: ${msg}`,
};
}
}
+4 -1
View File
@@ -13,8 +13,10 @@ export class ApiEngine {
*/ */
public static async getDashboardMetrics() { public static async getDashboardMetrics() {
const settings = await SettingsManager.getSettingsAsync(); const settings = await SettingsManager.getSettingsAsync();
const [storageStats, localBufferStats, clients] = await Promise.all([ const [storageStats, diskUsage, localBufferStats, clients] =
await Promise.all([
CSVStorageManager.getStorageStatsAsync(), CSVStorageManager.getStorageStatsAsync(),
CSVStorageManager.getDiskUsageAsync(),
DataBufferEngine.getBufferStatsAsync(), DataBufferEngine.getBufferStatsAsync(),
SDKManager.listClientsAsync(), SDKManager.listClientsAsync(),
]); ]);
@@ -129,6 +131,7 @@ export class ApiEngine {
activeClients, activeClients,
bufferStats, bufferStats,
storageStats, storageStats,
diskUsage,
securityStats, securityStats,
runtime: { runtime: {
serverPort: liveServerPort, serverPort: liveServerPort,
+122 -9
View File
@@ -23,6 +23,9 @@ export interface ClientSDK {
secret: string; secret: string;
token: string; token: string;
rateLimit: number; // max events per sec rateLimit: number; // max events per sec
storageQuotaMb: number; // max storage in MB per client (0 = unlimited)
allowedIps: string; // comma-separated IPv4/IPv6/CIDR list ("" = allow all)
expiresAt?: string | null; // ISO timestamp or null/undefined if never expires
status: "active" | "revoked"; status: "active" | "revoked";
totalEvents: number; totalEvents: number;
lastActive?: string; lastActive?: string;
@@ -103,6 +106,9 @@ interface RemoteInMemoryClient {
status?: string; status?: string;
token?: string; token?: string;
rateLimit?: number; rateLimit?: number;
storageQuotaMb?: number;
allowedIps?: string;
expiresAt?: string | null;
} }
function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): number { function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): number {
@@ -119,6 +125,9 @@ function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): numbe
secret: "", secret: "",
token: String(item.token || ""), token: String(item.token || ""),
rateLimit: Math.max(1, Number(item.rateLimit) || 100), rateLimit: Math.max(1, Number(item.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)),
allowedIps: typeof item.allowedIps === "string" ? item.allowedIps : "",
expiresAt: item.expiresAt ? String(item.expiresAt) : null,
status: item.status === "revoked" ? "revoked" : "active", status: item.status === "revoked" ? "revoked" : "active",
totalEvents: 0, totalEvents: 0,
createdAt: new Date().toISOString(), createdAt: new Date().toISOString(),
@@ -135,6 +144,9 @@ function serializeSyncClient(client: ClientSDK) {
status: client.status, status: client.status,
token: client.token, token: client.token,
rateLimit: client.rateLimit, rateLimit: client.rateLimit,
storageQuotaMb: client.storageQuotaMb,
allowedIps: client.allowedIps || "",
expiresAt: client.expiresAt ?? null,
}; };
} }
@@ -224,12 +236,26 @@ function notifyIngestionClientSync(
} }
function mapRowToClient(row: Record<string, unknown>): ClientSDK { function mapRowToClient(row: Record<string, unknown>): ClientSDK {
const parsedQuota = Number(row.storage_quota_mb);
let expiresAt: string | null = null;
if (row.expires_at) {
const d = new Date(row.expires_at as string | Date);
if (!Number.isNaN(d.getTime())) {
expiresAt = d.toISOString();
}
}
return { return {
clientId: String(row.client_id), clientId: String(row.client_id),
name: String(row.name), name: String(row.name),
secret: String(row.secret), secret: String(row.secret),
token: String(row.token), token: String(row.token),
rateLimit: Number(row.rate_limit) || 100, rateLimit: Number(row.rate_limit) || 100,
storageQuotaMb:
Number.isFinite(parsedQuota) && parsedQuota >= 0
? Math.floor(parsedQuota)
: 0,
allowedIps: typeof row.allowed_ips === "string" ? row.allowed_ips : "",
expiresAt,
status: row.status === "revoked" ? "revoked" : "active", status: row.status === "revoked" ? "revoked" : "active",
totalEvents: Number(row.total_events) || 0, totalEvents: Number(row.total_events) || 0,
lastActive: row.last_active lastActive: row.last_active
@@ -256,7 +282,7 @@ export class SDKManager {
const [result, deletedResult] = await Promise.all([ const [result, deletedResult] = await Promise.all([
pool.query( pool.query(
`SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at `SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at
FROM client_sdks FROM client_sdks
ORDER BY created_at DESC`, ORDER BY created_at DESC`,
), ),
@@ -464,7 +490,7 @@ export class SDKManager {
await this.flushPendingClientActivityAsync(); await this.flushPendingClientActivityAsync();
} }
const result = await pool.query( const result = await pool.query(
`SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at `SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at
FROM client_sdks FROM client_sdks
WHERE client_id = $1`, WHERE client_id = $1`,
[normalizedId], [normalizedId],
@@ -500,10 +526,19 @@ export class SDKManager {
clientId: string; clientId: string;
name: string; name: string;
rateLimit?: number; rateLimit?: number;
storageQuotaMb?: number;
allowedIps?: string;
expiresAt?: string | null;
}): ClientSDK { }): ClientSDK {
const safeClientId = this.normalizeClientId(data.clientId); const safeClientId = this.normalizeClientId(data.clientId);
const secret = this.generateClientSecret(safeClientId); const secret = this.generateClientSecret(safeClientId);
const rateLimit = Math.max(1, Number(data.rateLimit) || 100); const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
const storageQuotaMb = Math.max(
0,
Math.floor(Number(data.storageQuotaMb) || 0),
);
const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : "";
const expiresAt = data.expiresAt ? String(data.expiresAt) : null;
const token = AuthService.generateClientToken({ const token = AuthService.generateClientToken({
clientId: safeClientId, clientId: safeClientId,
@@ -517,6 +552,9 @@ export class SDKManager {
secret, secret,
token, token,
rateLimit, rateLimit,
storageQuotaMb,
allowedIps,
expiresAt,
status: "active", status: "active",
totalEvents: 0, totalEvents: 0,
createdAt: new Date().toISOString(), createdAt: new Date().toISOString(),
@@ -548,13 +586,16 @@ export class SDKManager {
[client.clientId], [client.clientId],
); );
await pool.query( await pool.query(
`INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, status, total_events, created_at, updated_at) `INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, created_at, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW()) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW())
ON CONFLICT (client_id) DO UPDATE SET ON CONFLICT (client_id) DO UPDATE SET
name = EXCLUDED.name, name = EXCLUDED.name,
secret = EXCLUDED.secret, secret = EXCLUDED.secret,
token = EXCLUDED.token, token = EXCLUDED.token,
rate_limit = EXCLUDED.rate_limit, rate_limit = EXCLUDED.rate_limit,
storage_quota_mb = EXCLUDED.storage_quota_mb,
allowed_ips = EXCLUDED.allowed_ips,
expires_at = EXCLUDED.expires_at,
status = EXCLUDED.status, status = EXCLUDED.status,
updated_at = NOW()`, updated_at = NOW()`,
[ [
@@ -563,6 +604,9 @@ export class SDKManager {
client.secret, client.secret,
client.token, client.token,
client.rateLimit, client.rateLimit,
client.storageQuotaMb,
client.allowedIps || "",
client.expiresAt ?? null,
client.status, client.status,
client.totalEvents, client.totalEvents,
client.createdAt, client.createdAt,
@@ -578,11 +622,20 @@ export class SDKManager {
clientId: string; clientId: string;
name: string; name: string;
rateLimit?: number; rateLimit?: number;
storageQuotaMb?: number;
allowedIps?: string;
expiresAt?: string | null;
}): Promise<ClientSDK> { }): Promise<ClientSDK> {
await ensureDbSettingsHydrated(); await ensureDbSettingsHydrated();
const safeClientId = this.normalizeClientId(data.clientId); const safeClientId = this.normalizeClientId(data.clientId);
const secret = this.generateClientSecret(safeClientId); const secret = this.generateClientSecret(safeClientId);
const rateLimit = Math.max(1, Number(data.rateLimit) || 100); const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
const storageQuotaMb = Math.max(
0,
Math.floor(Number(data.storageQuotaMb) || 0),
);
const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : "";
const expiresAt = data.expiresAt ? String(data.expiresAt) : null;
const token = AuthService.generateClientToken({ const token = AuthService.generateClientToken({
clientId: safeClientId, clientId: safeClientId,
@@ -596,6 +649,9 @@ export class SDKManager {
secret, secret,
token, token,
rateLimit, rateLimit,
storageQuotaMb,
allowedIps,
expiresAt,
status: "active", status: "active",
totalEvents: 0, totalEvents: 0,
createdAt: new Date().toISOString(), createdAt: new Date().toISOString(),
@@ -683,7 +739,17 @@ export class SDKManager {
public static updateClient( public static updateClient(
clientId: string, clientId: string,
updates: Partial<Pick<ClientSDK, "name" | "rateLimit" | "status">>, updates: Partial<
Pick<
ClientSDK,
| "name"
| "rateLimit"
| "storageQuotaMb"
| "allowedIps"
| "expiresAt"
| "status"
>
>,
): ClientSDK | null { ): ClientSDK | null {
const normalizedId = this.normalizeClientId(clientId); const normalizedId = this.normalizeClientId(clientId);
const client = getClientsMap().get(normalizedId); const client = getClientsMap().get(normalizedId);
@@ -695,6 +761,20 @@ export class SDKManager {
...(updates.rateLimit !== undefined ...(updates.rateLimit !== undefined
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) } ? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
: {}), : {}),
...(updates.storageQuotaMb !== undefined
? {
storageQuotaMb: Math.max(
0,
Math.floor(Number(updates.storageQuotaMb) || 0),
),
}
: {}),
...(updates.allowedIps !== undefined
? { allowedIps: String(updates.allowedIps).trim() }
: {}),
...(updates.expiresAt !== undefined
? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null }
: {}),
...(updates.status !== undefined ? { status: updates.status } : {}), ...(updates.status !== undefined ? { status: updates.status } : {}),
}; };
@@ -722,9 +802,17 @@ export class SDKManager {
try { try {
await pool.query( await pool.query(
`UPDATE client_sdks `UPDATE client_sdks
SET name = $2, rate_limit = $3, status = $4, updated_at = NOW() SET name = $2, rate_limit = $3, storage_quota_mb = $4, allowed_ips = $5, expires_at = $6, status = $7, updated_at = NOW()
WHERE client_id = $1`, WHERE client_id = $1`,
[client.clientId, client.name, client.rateLimit, client.status], [
client.clientId,
client.name,
client.rateLimit,
client.storageQuotaMb,
client.allowedIps || "",
client.expiresAt ?? null,
client.status,
],
); );
} catch (err: unknown) { } catch (err: unknown) {
const detail = err instanceof Error ? err.message : "Unknown DB error"; const detail = err instanceof Error ? err.message : "Unknown DB error";
@@ -734,7 +822,17 @@ export class SDKManager {
public static async updateClientAsync( public static async updateClientAsync(
clientId: string, clientId: string,
updates: Partial<Pick<ClientSDK, "name" | "rateLimit" | "status">>, updates: Partial<
Pick<
ClientSDK,
| "name"
| "rateLimit"
| "storageQuotaMb"
| "allowedIps"
| "expiresAt"
| "status"
>
>,
): Promise<ClientSDK | null> { ): Promise<ClientSDK | null> {
await ensureDbSettingsHydrated(); await ensureDbSettingsHydrated();
const normalizedId = this.normalizeClientId(clientId); const normalizedId = this.normalizeClientId(clientId);
@@ -747,6 +845,20 @@ export class SDKManager {
...(updates.rateLimit !== undefined ...(updates.rateLimit !== undefined
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) } ? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
: {}), : {}),
...(updates.storageQuotaMb !== undefined
? {
storageQuotaMb: Math.max(
0,
Math.floor(Number(updates.storageQuotaMb) || 0),
),
}
: {}),
...(updates.allowedIps !== undefined
? { allowedIps: String(updates.allowedIps).trim() }
: {}),
...(updates.expiresAt !== undefined
? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null }
: {}),
...(updates.status !== undefined ? { status: updates.status } : {}), ...(updates.status !== undefined ? { status: updates.status } : {}),
}; };
@@ -853,7 +965,7 @@ export class SDKManager {
const dirStat = lstatSync(targetDir); const dirStat = lstatSync(targetDir);
if (!dirStat.isSymbolicLink() && dirStat.isDirectory()) { if (!dirStat.isSymbolicLink() && dirStat.isDirectory()) {
for (const entry of readdirSync(targetDir)) { for (const entry of readdirSync(targetDir)) {
if (!entry.endsWith(".csv")) continue; if (!CSVStorageManager.isPartitionFilename(entry)) continue;
const entryPath = resolve(targetDir, entry); const entryPath = resolve(targetDir, entry);
try { try {
const entryStat = lstatSync(entryPath); const entryStat = lstatSync(entryPath);
@@ -871,6 +983,7 @@ export class SDKManager {
rmSync(targetDir, { recursive: true, force: true }); rmSync(targetDir, { recursive: true, force: true });
CSVStorageManager.clearRowCountCache(); CSVStorageManager.clearRowCountCache();
CSVStorageManager.clearQuotaCaches(safeId);
return purgedFiles; return purgedFiles;
} }
+123
View File
@@ -6,8 +6,41 @@ export interface EditableSettings {
fileRotationMb: number; fileRotationMb: number;
enableIdempotency: boolean; enableIdempotency: boolean;
autoPruneDays: number; autoPruneDays: number;
compressAfterDays: number;
diskPauseThresholdPct: number;
webhookCooldownSec: number;
} }
export interface S3SettingsFields {
enableS3Backup: boolean;
s3Endpoint: string;
s3Region: string;
s3Bucket: string;
s3Prefix: string;
s3AccessKeyId: string;
}
export interface AlertingAndMetricsSettingsFields {
enableWebhookAlerts: boolean;
webhookUrl: string;
enablePrometheusMetrics: boolean;
}
export const DEFAULT_S3_SETTINGS: S3SettingsFields = {
enableS3Backup: false,
s3Endpoint: '',
s3Region: 'auto',
s3Bucket: '',
s3Prefix: 'datastack/',
s3AccessKeyId: '',
};
export const DEFAULT_ALERTING_SETTINGS: AlertingAndMetricsSettingsFields = {
enableWebhookAlerts: false,
webhookUrl: '',
enablePrometheusMetrics: false,
};
export const SETTINGS_LIMITS = { export const SETTINGS_LIMITS = {
maxEventsPerSec: { min: 1, max: 100000, defaultValue: 100 }, maxEventsPerSec: { min: 1, max: 100000, defaultValue: 100 },
maxPayloadSizeKb: { min: 1, max: 10240, defaultValue: 100 }, maxPayloadSizeKb: { min: 1, max: 10240, defaultValue: 100 },
@@ -15,6 +48,9 @@ export const SETTINGS_LIMITS = {
batchFlushIntervalMs: { min: 100, max: 60000, defaultValue: 2000 }, batchFlushIntervalMs: { min: 100, max: 60000, defaultValue: 2000 },
fileRotationMb: { min: 1, max: 2048, defaultValue: 100 }, fileRotationMb: { min: 1, max: 2048, defaultValue: 100 },
autoPruneDays: { min: 0, max: 3650, defaultValue: 30 }, autoPruneDays: { min: 0, max: 3650, defaultValue: 30 },
compressAfterDays: { min: 0, max: 3650, defaultValue: 0 },
diskPauseThresholdPct: { min: 0, max: 99, defaultValue: 95 },
webhookCooldownSec: { min: 10, max: 86400, defaultValue: 300 },
} as const; } as const;
export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = { export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = {
@@ -25,6 +61,9 @@ export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = {
fileRotationMb: SETTINGS_LIMITS.fileRotationMb.defaultValue, fileRotationMb: SETTINGS_LIMITS.fileRotationMb.defaultValue,
enableIdempotency: true, enableIdempotency: true,
autoPruneDays: SETTINGS_LIMITS.autoPruneDays.defaultValue, autoPruneDays: SETTINGS_LIMITS.autoPruneDays.defaultValue,
compressAfterDays: SETTINGS_LIMITS.compressAfterDays.defaultValue,
diskPauseThresholdPct: SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue,
webhookCooldownSec: SETTINGS_LIMITS.webhookCooldownSec.defaultValue,
}; };
function clampInt(raw: unknown, min: number, max: number, fallback: number): number { function clampInt(raw: unknown, min: number, max: number, fallback: number): number {
@@ -34,6 +73,57 @@ function clampInt(raw: unknown, min: number, max: number, fallback: number): num
return Math.max(min, Math.min(max, rounded)); return Math.max(min, Math.min(max, rounded));
} }
function normalizeDiskPauseThresholdPct(raw: unknown): number {
const clamped = clampInt(
raw,
SETTINGS_LIMITS.diskPauseThresholdPct.min,
SETTINGS_LIMITS.diskPauseThresholdPct.max,
SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue,
);
if (clamped !== 0 && clamped < 50) {
return 50;
}
return clamped;
}
export function normalizeS3Settings(
input: Partial<S3SettingsFields> | null | undefined,
): S3SettingsFields {
return {
enableS3Backup:
typeof input?.enableS3Backup === 'boolean'
? input.enableS3Backup
: DEFAULT_S3_SETTINGS.enableS3Backup,
s3Endpoint: typeof input?.s3Endpoint === 'string' ? input.s3Endpoint.trim() : '',
s3Region:
typeof input?.s3Region === 'string' && input.s3Region.trim()
? input.s3Region.trim()
: DEFAULT_S3_SETTINGS.s3Region,
s3Bucket: typeof input?.s3Bucket === 'string' ? input.s3Bucket.trim() : '',
s3Prefix:
typeof input?.s3Prefix === 'string' ? input.s3Prefix.trim() : DEFAULT_S3_SETTINGS.s3Prefix,
s3AccessKeyId:
typeof input?.s3AccessKeyId === 'string' ? input.s3AccessKeyId.trim() : '',
};
}
export function normalizeAlertingSettings(
input: Partial<AlertingAndMetricsSettingsFields> | null | undefined,
): AlertingAndMetricsSettingsFields {
return {
enableWebhookAlerts:
typeof input?.enableWebhookAlerts === 'boolean'
? input.enableWebhookAlerts
: DEFAULT_ALERTING_SETTINGS.enableWebhookAlerts,
webhookUrl:
typeof input?.webhookUrl === 'string' ? input.webhookUrl.trim() : '',
enablePrometheusMetrics:
typeof input?.enablePrometheusMetrics === 'boolean'
? input.enablePrometheusMetrics
: DEFAULT_ALERTING_SETTINGS.enablePrometheusMetrics,
};
}
export function normalizeEditableSettings( export function normalizeEditableSettings(
input: Partial<EditableSettings>, input: Partial<EditableSettings>,
): EditableSettings { ): EditableSettings {
@@ -78,6 +168,21 @@ export function normalizeEditableSettings(
SETTINGS_LIMITS.autoPruneDays.max, SETTINGS_LIMITS.autoPruneDays.max,
SETTINGS_LIMITS.autoPruneDays.defaultValue, SETTINGS_LIMITS.autoPruneDays.defaultValue,
), ),
compressAfterDays: clampInt(
input.compressAfterDays,
SETTINGS_LIMITS.compressAfterDays.min,
SETTINGS_LIMITS.compressAfterDays.max,
SETTINGS_LIMITS.compressAfterDays.defaultValue,
),
diskPauseThresholdPct: normalizeDiskPauseThresholdPct(
input.diskPauseThresholdPct,
),
webhookCooldownSec: clampInt(
input.webhookCooldownSec,
SETTINGS_LIMITS.webhookCooldownSec.min,
SETTINGS_LIMITS.webhookCooldownSec.max,
SETTINGS_LIMITS.webhookCooldownSec.defaultValue,
),
}; };
} }
@@ -398,6 +503,9 @@ export const NUMERIC_SETTING_KEYS: readonly NumericSettingKey[] = [
'batchFlushIntervalMs', 'batchFlushIntervalMs',
'fileRotationMb', 'fileRotationMb',
'autoPruneDays', 'autoPruneDays',
'compressAfterDays',
'diskPauseThresholdPct',
'webhookCooldownSec',
] as const; ] as const;
export function settingsToNumericDrafts( export function settingsToNumericDrafts(
@@ -410,6 +518,9 @@ export function settingsToNumericDrafts(
batchFlushIntervalMs: String(settings.batchFlushIntervalMs), batchFlushIntervalMs: String(settings.batchFlushIntervalMs),
fileRotationMb: String(settings.fileRotationMb), fileRotationMb: String(settings.fileRotationMb),
autoPruneDays: String(settings.autoPruneDays), autoPruneDays: String(settings.autoPruneDays),
compressAfterDays: String(settings.compressAfterDays),
diskPauseThresholdPct: String(settings.diskPauseThresholdPct),
webhookCooldownSec: String(settings.webhookCooldownSec),
}; };
} }
@@ -429,6 +540,15 @@ export function validateSingleNumericSetting(
if (!Number.isSafeInteger(parsed)) { if (!Number.isSafeInteger(parsed)) {
return { valid: false, error: 'Enter a valid integer.' }; return { valid: false, error: 'Enter a valid integer.' };
} }
if (key === 'diskPauseThresholdPct') {
if (parsed !== 0 && (parsed < 50 || parsed > max)) {
return {
valid: false,
error: 'Must be 0 (disabled) or between 50 and 99.',
};
}
return { valid: true, value: parsed };
}
if (parsed < min || parsed > max) { if (parsed < min || parsed > max) {
return { return {
valid: false, valid: false,
@@ -475,6 +595,9 @@ export function validateEditableSettingsDraft(
batchFlushIntervalMs: parsedValues.batchFlushIntervalMs!, batchFlushIntervalMs: parsedValues.batchFlushIntervalMs!,
fileRotationMb: parsedValues.fileRotationMb!, fileRotationMb: parsedValues.fileRotationMb!,
autoPruneDays: parsedValues.autoPruneDays!, autoPruneDays: parsedValues.autoPruneDays!,
compressAfterDays: parsedValues.compressAfterDays!,
diskPauseThresholdPct: parsedValues.diskPauseThresholdPct!,
webhookCooldownSec: parsedValues.webhookCooldownSec!,
enableIdempotency: Boolean(enableIdempotency), enableIdempotency: Boolean(enableIdempotency),
}, },
}; };
+284 -8
View File
@@ -199,6 +199,9 @@ const runtimeConfig = {
fileRotationMb: 100, fileRotationMb: 100,
enableIdempotency: true, enableIdempotency: true,
autoPruneDays: 30, autoPruneDays: 30,
compressAfterDays: 0,
enableS3Backup: false,
diskPauseThresholdPct: 95,
}; };
let previousJwtSecret = ''; let previousJwtSecret = '';
@@ -287,17 +290,113 @@ const inMemoryClientActivity = new Map();
let hasSyncedInMemoryClients = false; let hasSyncedInMemoryClients = false;
const CLIENT_CACHE_TTL_MS = 2000; const CLIENT_CACHE_TTL_MS = 2000;
async function verifyClientInDb(clientId, presentedToken, bypassCache = false) { function stripIpv6MappedPrefix(rawIp) {
let ip = String(rawIp || '').trim();
if (!ip) return '';
if (ip.startsWith('[') && ip.includes(']')) {
ip = ip.slice(1, ip.indexOf(']'));
} else if (/^\d{1,3}(?:\.\d{1,3}){3}:\d+$/.test(ip)) {
ip = ip.slice(0, ip.lastIndexOf(':'));
}
const pctIndex = ip.indexOf('%');
if (pctIndex !== -1) {
ip = ip.slice(0, pctIndex);
}
if (/^::ffff:\d{1,3}(?:\.\d{1,3}){3}$/i.test(ip)) {
return ip.slice(7);
}
return ip.toLowerCase();
}
function ipv4ToUint32(ip) {
const parts = ip.split('.');
if (parts.length !== 4) return null;
let acc = 0;
for (const part of parts) {
if (!/^\d{1,3}$/.test(part)) return null;
if (part.length > 1 && part.startsWith('0')) return null;
const num = Number(part);
if (num < 0 || num > 255) return null;
acc = ((acc << 8) | num) >>> 0;
}
return acc >>> 0;
}
function matchesIpv4Cidr(clientIp, cidrPattern) {
const [networkStr, prefixStr] = cidrPattern.split('/');
if (!networkStr || prefixStr === undefined) return false;
const prefix = Number(prefixStr);
if (!Number.isInteger(prefix) || prefix < 0 || prefix > 32) return false;
const clientNum = ipv4ToUint32(clientIp);
const networkNum = ipv4ToUint32(networkStr);
if (clientNum === null || networkNum === null) return false;
if (prefix === 0) return true;
const mask = (0xffffffff << (32 - prefix)) >>> 0;
return (clientNum & mask) >>> 0 === (networkNum & mask) >>> 0;
}
function isClientIpAllowed(clientIp, rawAllowedIps) {
if (!rawAllowedIps || !String(rawAllowedIps).trim()) return true;
const entries = String(rawAllowedIps)
.split(/[\s,]+/)
.map((entry) => entry.trim())
.filter(Boolean);
if (entries.length === 0) return true;
if (entries.includes('*') || entries.includes('0.0.0.0/0') || entries.includes('::/0')) {
return true;
}
const normalizedClientIp = stripIpv6MappedPrefix(clientIp);
if (!normalizedClientIp || normalizedClientIp === 'unknown') {
return false;
}
for (const entry of entries) {
const normalizedEntry = stripIpv6MappedPrefix(entry);
if (normalizedEntry.includes('/')) {
if (matchesIpv4Cidr(normalizedClientIp, normalizedEntry)) {
return true;
}
const [netAddr, prefixStr] = normalizedEntry.split('/');
if (Number(prefixStr) === 128 && normalizedClientIp === stripIpv6MappedPrefix(netAddr)) {
return true;
}
} else {
if (normalizedClientIp === normalizedEntry) {
return true;
}
if (
(normalizedClientIp === '127.0.0.1' && normalizedEntry === '::1') ||
(normalizedClientIp === '::1' && normalizedEntry === '127.0.0.1')
) {
return true;
}
}
}
return false;
}
function isClientTokenExpired(expiresAt, nowMs = Date.now()) {
if (!expiresAt) return false;
const ts =
expiresAt instanceof Date
? expiresAt.getTime()
: new Date(String(expiresAt)).getTime();
if (Number.isNaN(ts)) return false;
return ts <= nowMs;
}
async function verifyClientInDb(clientId, presentedToken, bypassCache = false, clientIp = '') {
const normalizedId = normalizeClientId(clientId); const normalizedId = normalizeClientId(clientId);
if (!normalizedId) { if (!normalizedId) {
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Missing client identifier', reason: 'CLIENT_REVOKED: Missing client identifier',
}; };
} }
if (deletedClientsSet.has(normalizedId)) { if (deletedClientsSet.has(normalizedId)) {
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been deleted or revoked by admin', reason: 'CLIENT_REVOKED: Client access has been deleted or revoked by admin',
}; };
} }
@@ -313,6 +412,7 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
deletedClientsSet.add(normalizedId); deletedClientsSet.add(normalizedId);
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been deleted by admin', reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
}; };
} }
@@ -326,6 +426,9 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
status: parsed.status === 'revoked' ? 'revoked' : 'active', status: parsed.status === 'revoked' ? 'revoked' : 'active',
token: String(parsed.token || ''), token: String(parsed.token || ''),
rateLimit: Math.max(1, Number(parsed.rateLimit) || 100), rateLimit: Math.max(1, Number(parsed.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(parsed.storageQuotaMb) || 0)),
allowedIps: typeof parsed.allowedIps === 'string' ? parsed.allowedIps : '',
expiresAt: parsed.expiresAt ? String(parsed.expiresAt) : null,
}; };
inMemoryClientRegistry.set(normalizedId, memClient); inMemoryClientRegistry.set(normalizedId, memClient);
hasSyncedInMemoryClients = true; hasSyncedInMemoryClients = true;
@@ -341,20 +444,41 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
if (memClient.status === 'revoked') { if (memClient.status === 'revoked') {
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been revoked by admin', reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
}; };
} }
if (memClient.token && presentedToken && memClient.token !== presentedToken) { if (memClient.token && presentedToken && memClient.token !== presentedToken) {
return { return {
allowed: false, allowed: false,
code: 'TOKEN_ROTATED',
reason: 'TOKEN_ROTATED: Client credentials have been rotated', reason: 'TOKEN_ROTATED: Client credentials have been rotated',
}; };
} }
return { allowed: true, rateLimit: memClient.rateLimit || 100 }; if (isClientTokenExpired(memClient.expiresAt)) {
return {
allowed: false,
code: 'CLIENT_TOKEN_EXPIRED',
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
};
}
if (memClient.allowedIps && memClient.allowedIps.trim() && !isClientIpAllowed(clientIp, memClient.allowedIps)) {
return {
allowed: false,
code: 'CLIENT_IP_FORBIDDEN',
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
};
}
return {
allowed: true,
rateLimit: memClient.rateLimit || 100,
storageQuotaMb: memClient.storageQuotaMb || 0,
};
} }
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted', reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
}; };
} }
@@ -365,16 +489,36 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
if (prevCached.status === 'revoked') { if (prevCached.status === 'revoked') {
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been revoked by admin', reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
}; };
} }
if (prevCached.token && presentedToken && prevCached.token !== presentedToken) { if (prevCached.token && presentedToken && prevCached.token !== presentedToken) {
return { return {
allowed: false, allowed: false,
code: 'TOKEN_ROTATED',
reason: 'TOKEN_ROTATED: Client credentials have been rotated', reason: 'TOKEN_ROTATED: Client credentials have been rotated',
}; };
} }
return { allowed: true, rateLimit: prevCached.rateLimit }; if (isClientTokenExpired(prevCached.expiresAt, now)) {
return {
allowed: false,
code: 'CLIENT_TOKEN_EXPIRED',
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
};
}
if (prevCached.allowedIps && prevCached.allowedIps.trim() && !isClientIpAllowed(clientIp, prevCached.allowedIps)) {
return {
allowed: false,
code: 'CLIENT_IP_FORBIDDEN',
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
};
}
return {
allowed: true,
rateLimit: prevCached.rateLimit,
storageQuotaMb: prevCached.storageQuotaMb || 0,
};
} }
try { try {
@@ -387,26 +531,39 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
deletedClientsSet.add(normalizedId); deletedClientsSet.add(normalizedId);
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been deleted by admin', reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
}; };
} }
const res = await dbPool.query( const res = await dbPool.query(
`SELECT status, token, rate_limit FROM client_sdks WHERE client_id = $1`, `SELECT status, token, rate_limit, storage_quota_mb, allowed_ips, expires_at FROM client_sdks WHERE client_id = $1`,
[normalizedId], [normalizedId],
); );
if (!res.rowCount || res.rowCount === 0) { if (!res.rowCount || res.rowCount === 0) {
clientAuthCache.delete(normalizedId); clientAuthCache.delete(normalizedId);
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted', reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
}; };
} }
const row = res.rows[0]; const row = res.rows[0];
const parsedQuota = Number(row.storage_quota_mb);
let expiresAtIso = null;
if (row.expires_at) {
const d = new Date(row.expires_at);
if (!Number.isNaN(d.getTime())) {
expiresAtIso = d.toISOString();
}
}
const entry = { const entry = {
status: row.status, status: row.status,
token: row.token, token: row.token,
rateLimit: Number(row.rate_limit) || 100, rateLimit: Number(row.rate_limit) || 100,
storageQuotaMb: Number.isFinite(parsedQuota) && parsedQuota >= 0 ? Math.floor(parsedQuota) : 0,
allowedIps: typeof row.allowed_ips === 'string' ? row.allowed_ips : '',
expiresAt: expiresAtIso,
checkedAt: now, checkedAt: now,
}; };
clientAuthCache.set(normalizedId, entry); clientAuthCache.set(normalizedId, entry);
@@ -414,19 +571,40 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
if (entry.status === 'revoked') { if (entry.status === 'revoked') {
return { return {
allowed: false, allowed: false,
code: 'CLIENT_REVOKED',
reason: 'CLIENT_REVOKED: Client access has been revoked by admin', reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
}; };
} }
if (entry.token && presentedToken && entry.token !== presentedToken) { if (entry.token && presentedToken && entry.token !== presentedToken) {
return { return {
allowed: false, allowed: false,
code: 'TOKEN_ROTATED',
reason: 'TOKEN_ROTATED: Client credentials have been rotated', reason: 'TOKEN_ROTATED: Client credentials have been rotated',
}; };
} }
return { allowed: true, rateLimit: entry.rateLimit }; if (isClientTokenExpired(entry.expiresAt, now)) {
return {
allowed: false,
code: 'CLIENT_TOKEN_EXPIRED',
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
};
}
if (entry.allowedIps && entry.allowedIps.trim() && !isClientIpAllowed(clientIp, entry.allowedIps)) {
return {
allowed: false,
code: 'CLIENT_IP_FORBIDDEN',
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
};
}
return {
allowed: true,
rateLimit: entry.rateLimit,
storageQuotaMb: entry.storageQuotaMb,
};
} catch { } catch {
return { return {
allowed: false, allowed: false,
code: 'AUTH_UNAVAILABLE',
reason: 'AUTH_UNAVAILABLE: Unable to verify client credentials against database', reason: 'AUTH_UNAVAILABLE: Unable to verify client credentials against database',
}; };
} }
@@ -888,13 +1066,42 @@ function applyRuntimeSettingsObject(settingsObj) {
} }
} }
if (settingsObj.enableS3Backup !== undefined) {
runtimeConfig.enableS3Backup = Boolean(settingsObj.enableS3Backup);
}
if (
settingsObj.diskPauseThresholdPct !== undefined &&
settingsObj.diskPauseThresholdPct !== null
) {
const pct = Number(settingsObj.diskPauseThresholdPct);
if (Number.isFinite(pct) && (pct === 0 || (pct >= 50 && pct <= 99))) {
runtimeConfig.diskPauseThresholdPct = Math.floor(pct);
}
}
if (
settingsObj.compressAfterDays !== undefined &&
settingsObj.compressAfterDays !== null
) {
const parsedCompress = Number(settingsObj.compressAfterDays);
if (Number.isFinite(parsedCompress) && parsedCompress >= 0) {
const nextCompressDays = Math.floor(parsedCompress);
const changed = nextCompressDays !== runtimeConfig.compressAfterDays;
runtimeConfig.compressAfterDays = nextCompressDays;
if (changed && nextCompressDays > 0) {
void writer.compressOldPartitionsAsync(nextCompressDays).catch(() => undefined);
}
}
}
if (settingsObj.autoPruneDays !== undefined && settingsObj.autoPruneDays !== null) { if (settingsObj.autoPruneDays !== undefined && settingsObj.autoPruneDays !== null) {
const parsedPrune = Number(settingsObj.autoPruneDays); const parsedPrune = Number(settingsObj.autoPruneDays);
if (Number.isFinite(parsedPrune) && parsedPrune >= 0) { if (Number.isFinite(parsedPrune) && parsedPrune >= 0) {
const nextPruneDays = Math.floor(parsedPrune); const nextPruneDays = Math.floor(parsedPrune);
const changed = nextPruneDays !== runtimeConfig.autoPruneDays; const changed = nextPruneDays !== runtimeConfig.autoPruneDays;
runtimeConfig.autoPruneDays = nextPruneDays; runtimeConfig.autoPruneDays = nextPruneDays;
if (changed && nextPruneDays > 0) { if (changed && nextPruneDays > 0 && !runtimeConfig.enableS3Backup) {
void writer.pruneExpiredPartitionsAsync(nextPruneDays).catch(() => undefined); void writer.pruneExpiredPartitionsAsync(nextPruneDays).catch(() => undefined);
} }
} }
@@ -948,6 +1155,9 @@ async function ensureDatabaseSchema() {
secret TEXT NOT NULL, secret TEXT NOT NULL,
token TEXT NOT NULL, token TEXT NOT NULL,
rate_limit INTEGER NOT NULL DEFAULT 100, rate_limit INTEGER NOT NULL DEFAULT 100,
storage_quota_mb INTEGER NOT NULL DEFAULT 0,
allowed_ips TEXT NOT NULL DEFAULT '',
expires_at TIMESTAMPTZ,
status TEXT NOT NULL DEFAULT 'active', status TEXT NOT NULL DEFAULT 'active',
total_events BIGINT NOT NULL DEFAULT 0, total_events BIGINT NOT NULL DEFAULT 0,
last_active TIMESTAMPTZ, last_active TIMESTAMPTZ,
@@ -968,6 +1178,21 @@ async function ensureDatabaseSchema() {
jwt_secret TEXT NOT NULL, jwt_secret TEXT NOT NULL,
enable_idempotency BOOLEAN NOT NULL DEFAULT true, enable_idempotency BOOLEAN NOT NULL DEFAULT true,
auto_prune_days INTEGER NOT NULL DEFAULT 30, auto_prune_days INTEGER NOT NULL DEFAULT 30,
compress_after_days INTEGER NOT NULL DEFAULT 0,
enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
s3_endpoint TEXT NOT NULL DEFAULT '',
s3_region TEXT NOT NULL DEFAULT 'auto',
s3_bucket TEXT NOT NULL DEFAULT '',
s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
s3_access_key_id TEXT NOT NULL DEFAULT '',
s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
webhook_url TEXT NOT NULL DEFAULT '',
webhook_secret_enc TEXT NOT NULL DEFAULT '',
webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
) )
`); `);
@@ -999,6 +1224,9 @@ async function ensureDatabaseSchema() {
await pool.query(` await pool.query(`
ALTER TABLE client_sdks ALTER TABLE client_sdks
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
ADD COLUMN IF NOT EXISTS storage_quota_mb INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS allowed_ips TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active', ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0, ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ, ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
@@ -1016,6 +1244,21 @@ async function ensureDatabaseSchema() {
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true, ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30, ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
ADD COLUMN IF NOT EXISTS compress_after_days INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS s3_endpoint TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_region TEXT NOT NULL DEFAULT 'auto',
ADD COLUMN IF NOT EXISTS s3_bucket TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
ADD COLUMN IF NOT EXISTS enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS webhook_url TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS webhook_secret_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
ADD COLUMN IF NOT EXISTS enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
`); `);
await pool.query(` await pool.query(`
@@ -1043,7 +1286,7 @@ async function syncSettingsFromDatabase() {
const res = await dbPool.query( const res = await dbPool.query(
`SELECT max_events_per_sec, max_payload_size_kb, batch_flush_threshold, `SELECT max_events_per_sec, max_payload_size_kb, batch_flush_threshold,
batch_flush_interval_ms, file_rotation_mb, jwt_secret, enable_idempotency, batch_flush_interval_ms, file_rotation_mb, jwt_secret, enable_idempotency,
auto_prune_days auto_prune_days, compress_after_days, enable_s3_backup, disk_pause_threshold_pct
FROM admin_settings FROM admin_settings
WHERE id = 'default'`, WHERE id = 'default'`,
); );
@@ -1053,6 +1296,14 @@ async function syncSettingsFromDatabase() {
row.auto_prune_days !== undefined && row.auto_prune_days !== null row.auto_prune_days !== undefined && row.auto_prune_days !== null
? Number(row.auto_prune_days) ? Number(row.auto_prune_days)
: NaN; : NaN;
const parsedCompress =
row.compress_after_days !== undefined && row.compress_after_days !== null
? Number(row.compress_after_days)
: NaN;
const parsedDiskPct =
row.disk_pause_threshold_pct !== undefined && row.disk_pause_threshold_pct !== null
? Number(row.disk_pause_threshold_pct)
: NaN;
applyRuntimeSettingsObject({ applyRuntimeSettingsObject({
maxEventsPerSec: Number(row.max_events_per_sec) || 100, maxEventsPerSec: Number(row.max_events_per_sec) || 100,
@@ -1065,6 +1316,9 @@ async function syncSettingsFromDatabase() {
fileRotationMb: Number(row.file_rotation_mb) || 100, fileRotationMb: Number(row.file_rotation_mb) || 100,
enableIdempotency: Boolean(row.enable_idempotency), enableIdempotency: Boolean(row.enable_idempotency),
autoPruneDays: Number.isFinite(parsedPrune) ? parsedPrune : 30, autoPruneDays: Number.isFinite(parsedPrune) ? parsedPrune : 30,
compressAfterDays: Number.isFinite(parsedCompress) ? Math.max(0, parsedCompress) : 0,
enableS3Backup: Boolean(row.enable_s3_backup),
diskPauseThresholdPct: Number.isFinite(parsedDiskPct) ? parsedDiskPct : 95,
jwtSecret: row.jwt_secret ? String(row.jwt_secret).trim() : undefined, jwtSecret: row.jwt_secret ? String(row.jwt_secret).trim() : undefined,
}); });
return true; return true;
@@ -1338,6 +1592,9 @@ app.post('/internal/client-sync', (req, res) => {
status: nextStatus, status: nextStatus,
token: nextToken, token: nextToken,
rateLimit: Math.max(1, Number(item.rateLimit) || 100), rateLimit: Math.max(1, Number(item.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)),
allowedIps: typeof item.allowedIps === 'string' ? item.allowedIps : '',
expiresAt: item.expiresAt ? String(item.expiresAt) : null,
}); });
if (nextStatus === 'revoked' || (prev && nextToken && prev.token !== nextToken)) { if (nextStatus === 'revoked' || (prev && nextToken && prev.token !== nextToken)) {
io.to(`client:${cid}`).disconnectSockets(true); io.to(`client:${cid}`).disconnectSockets(true);
@@ -1380,6 +1637,9 @@ app.post('/internal/client-sync', (req, res) => {
status: 'revoked', status: 'revoked',
token: String(clientPayload.token || ''), token: String(clientPayload.token || ''),
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
}); });
} else if (existingMem) { } else if (existingMem) {
existingMem.status = 'revoked'; existingMem.status = 'revoked';
@@ -1390,6 +1650,9 @@ app.post('/internal/client-sync', (req, res) => {
status: 'revoked', status: 'revoked',
token: '', token: '',
rateLimit: 100, rateLimit: 100,
storageQuotaMb: 0,
allowedIps: '',
expiresAt: null,
}); });
} }
io.to(`client:${normalizedId}`).disconnectSockets(true); io.to(`client:${normalizedId}`).disconnectSockets(true);
@@ -1402,6 +1665,9 @@ app.post('/internal/client-sync', (req, res) => {
status: clientPayload.status === 'revoked' ? 'revoked' : 'active', status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
token: String(clientPayload.token || ''), token: String(clientPayload.token || ''),
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
}); });
} }
} else if (action === 'updated') { } else if (action === 'updated') {
@@ -1412,6 +1678,9 @@ app.post('/internal/client-sync', (req, res) => {
status: clientPayload.status === 'revoked' ? 'revoked' : 'active', status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
token: String(clientPayload.token || ''), token: String(clientPayload.token || ''),
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100), rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
}); });
} }
io.to(`client:${normalizedId}`).disconnectSockets(true); io.to(`client:${normalizedId}`).disconnectSockets(true);
@@ -1542,7 +1811,12 @@ async function startIngestionServer() {
} }
pruneTimer = setInterval(() => { pruneTimer = setInterval(() => {
if (runtimeConfig.autoPruneDays > 0) { if (runtimeConfig.compressAfterDays > 0) {
void writer
.compressOldPartitionsAsync(runtimeConfig.compressAfterDays)
.catch(() => undefined);
}
if (runtimeConfig.autoPruneDays > 0 && !runtimeConfig.enableS3Backup) {
void writer void writer
.pruneExpiredPartitionsAsync(runtimeConfig.autoPruneDays) .pruneExpiredPartitionsAsync(runtimeConfig.autoPruneDays)
.catch(() => undefined); .catch(() => undefined);
@@ -1594,4 +1868,6 @@ module.exports = {
parseAllowedCorsOrigins, parseAllowedCorsOrigins,
isOriginAllowedByCors, isOriginAllowedByCors,
corsOriginDelegate, corsOriginDelegate,
isClientIpAllowed,
isClientTokenExpired,
}; };
+82 -4
View File
@@ -26,6 +26,34 @@ function normalizeClientId(clientId) {
.replace(/[^a-z0-9_-]/g, "_"); .replace(/[^a-z0-9_-]/g, "_");
} }
function extractSocketClientIp(socket) {
const headers = (socket && socket.handshake && socket.handshake.headers) || {};
const realIp =
typeof headers["x-real-ip"] === "string" ? headers["x-real-ip"].trim() : "";
if (realIp) {
return realIp.slice(0, 128);
}
const forwardedFor =
typeof headers["x-forwarded-for"] === "string"
? headers["x-forwarded-for"]
: "";
if (forwardedFor) {
const hops = forwardedFor
.split(",")
.map((part) => part.trim())
.filter(Boolean);
const rightmost = hops[hops.length - 1];
if (rightmost) {
return rightmost.slice(0, 128);
}
}
const addr =
(socket && socket.handshake && socket.handshake.address) ||
(socket && socket.conn && socket.conn.remoteAddress) ||
"";
return String(addr).trim() || "127.0.0.1";
}
function setupSocketHandlers(io, bufferQueue, worker, options = {}) { function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
const getRuntimeConfig = const getRuntimeConfig =
typeof options.getRuntimeConfig === "function" typeof options.getRuntimeConfig === "function"
@@ -340,17 +368,20 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
const decoded = jwt.verify(cleanToken, config.jwtSecret); const decoded = jwt.verify(cleanToken, config.jwtSecret);
const normalizedClientId = normalizeClientId(decoded.clientId); const normalizedClientId = normalizeClientId(decoded.clientId);
const clientIp = extractSocketClientIp(socket);
const dbCheck = await verifyClientInDb( const dbCheck = await verifyClientInDb(
normalizedClientId, normalizedClientId,
cleanToken, cleanToken,
true, true,
clientIp,
); );
if (!dbCheck.allowed) { if (!dbCheck.allowed) {
return next(new Error(dbCheck.reason || "CLIENT_REVOKED")); return next(new Error(dbCheck.reason || dbCheck.code || "CLIENT_REVOKED"));
} }
socket.data.clientId = normalizedClientId; socket.data.clientId = normalizedClientId;
socket.data.token = cleanToken; socket.data.token = cleanToken;
socket.data.clientIp = clientIp;
socket.data.rateLimit = socket.data.rateLimit =
dbCheck.rateLimit || decoded.rateLimit || config.maxEventsPerSec || 100; dbCheck.rateLimit || decoded.rateLimit || config.maxEventsPerSec || 100;
next(); next();
@@ -401,16 +432,18 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
// Single event // Single event
socket.on("sync_data", async (payload, callback) => { socket.on("sync_data", async (payload, callback) => {
try { try {
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
const dbCheck = await verifyClientInDb( const dbCheck = await verifyClientInDb(
clientId, clientId,
socket.data.token, socket.data.token,
false, false,
clientIp,
); );
if (!dbCheck.allowed) { if (!dbCheck.allowed) {
if (callback) { if (callback) {
callback({ callback({
success: false, success: false,
code: "CLIENT_REVOKED", code: dbCheck.code || "CLIENT_REVOKED",
error: dbCheck.reason || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED",
}); });
} }
@@ -418,6 +451,25 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
return; return;
} }
if (worker && typeof worker.checkStorageAdmission === "function") {
const config = getRuntimeConfig();
const admission = await worker.checkStorageAdmission(
clientId,
dbCheck.storageQuotaMb || 0,
config.diskPauseThresholdPct ?? 95,
);
if (!admission.allowed) {
if (callback) {
callback({
success: false,
code: admission.code,
error: admission.reason,
});
}
return;
}
}
const effectiveClientRate = const effectiveClientRate =
dbCheck.rateLimit || socket.data.rateLimit || 100; dbCheck.rateLimit || socket.data.rateLimit || 100;
const safePayload = const safePayload =
@@ -499,10 +551,12 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
return; return;
} }
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
const dbCheck = await verifyClientInDb( const dbCheck = await verifyClientInDb(
clientId, clientId,
socket.data.token, socket.data.token,
false, false,
clientIp,
); );
if (!dbCheck.allowed) { if (!dbCheck.allowed) {
if (callback) { if (callback) {
@@ -510,7 +564,7 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
success: false, success: false,
processed: 0, processed: 0,
rejected: batch.length, rejected: batch.length,
code: "CLIENT_REVOKED", code: dbCheck.code || "CLIENT_REVOKED",
error: dbCheck.reason || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED",
}); });
} }
@@ -518,6 +572,27 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
return; return;
} }
if (worker && typeof worker.checkStorageAdmission === "function") {
const config = getRuntimeConfig();
const admission = await worker.checkStorageAdmission(
clientId,
dbCheck.storageQuotaMb || 0,
config.diskPauseThresholdPct ?? 95,
);
if (!admission.allowed) {
if (callback) {
callback({
success: false,
processed: 0,
rejected: batch.length,
code: admission.code,
error: admission.reason,
});
}
return;
}
}
const effectiveClientRate = const effectiveClientRate =
dbCheck.rateLimit || socket.data.rateLimit || 100; dbCheck.rateLimit || socket.data.rateLimit || 100;
let processed = 0; let processed = 0;
@@ -646,16 +721,19 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
// Heartbeat & Reconnection Verification Event // Heartbeat & Reconnection Verification Event
socket.on("ping_verify", async (_payload, callback) => { socket.on("ping_verify", async (_payload, callback) => {
try { try {
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
const dbCheck = await verifyClientInDb( const dbCheck = await verifyClientInDb(
clientId, clientId,
socket.data.token, socket.data.token,
true, true,
clientIp,
); );
if (!dbCheck.allowed) { if (!dbCheck.allowed) {
if (callback) { if (callback) {
callback({ callback({
success: false, success: false,
status: "REVOKED", status: "REVOKED",
code: dbCheck.code || "CLIENT_REVOKED",
error: dbCheck.reason || "CLIENT_REVOKED", error: dbCheck.reason || "CLIENT_REVOKED",
}); });
} }
@@ -691,4 +769,4 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
}; };
} }
module.exports = { setupSocketHandlers, normalizeClientId }; module.exports = { setupSocketHandlers, normalizeClientId, extractSocketClientIp };
+233 -5
View File
@@ -1,12 +1,15 @@
const crypto = require('node:crypto'); const crypto = require('node:crypto');
const fs = require('fs'); const fs = require('fs');
const path = require('path'); const path = require('path');
const { pipeline } = require('node:stream/promises');
const zlib = require('node:zlib');
const { Parser } = require('json2csv'); const { Parser } = require('json2csv');
const LOCK_RETRY_MIN_MS = 5; const LOCK_RETRY_MIN_MS = 5;
const LOCK_RETRY_MAX_MS = 18; const LOCK_RETRY_MAX_MS = 18;
const LOCK_TIMEOUT_MS = 10000; const LOCK_TIMEOUT_MS = 10000;
const STALE_LOCK_AGE_MS = 30000; const STALE_LOCK_AGE_MS = 30000;
const QUOTA_CACHE_TTL_MS = 30000;
function sleepMs(ms) { function sleepMs(ms) {
return new Promise((resolve) => setTimeout(resolve, ms)); return new Promise((resolve) => setTimeout(resolve, ms));
@@ -23,15 +26,42 @@ function resolveDefaultStorageDir() {
/** /**
* Background CSV Writer Worker * Background CSV Writer Worker
* Flushes batches of real-time events to partitioned CSV files. * Flushes batches of real-time events to partitioned CSV files.
* Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv * Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv(.gz)
*/ */
class CsvWriterWorker { class CsvWriterWorker {
constructor(baseDir = resolveDefaultStorageDir(), maxFileSize = 100 * 1024 * 1024) { constructor(baseDir = resolveDefaultStorageDir(), maxFileSize = 100 * 1024 * 1024) {
this.baseDir = baseDir; this.baseDir = baseDir;
this.maxFileSize = maxFileSize; this.maxFileSize = maxFileSize;
this.diskUsageCache = null;
this.clientBytesCache = new Map();
this.statfsOverrideForTests = null;
this.ensureBaseDir(); this.ensureBaseDir();
} }
isPartitionFilename(filename) {
return typeof filename === 'string' && (filename.endsWith('.csv') || filename.endsWith('.csv.gz'));
}
extractDateFromFilename(filename) {
return String(filename || '')
.replace(/\.csv(?:\.gz)?$/, '')
.split('-part')[0];
}
clearQuotaCaches(clientId) {
this.diskUsageCache = null;
if (clientId) {
this.clientBytesCache.delete(this.sanitizeClientId(clientId));
} else {
this.clientBytesCache.clear();
}
}
setStatfsOverrideForTests(provider) {
this.statfsOverrideForTests = typeof provider === 'function' ? provider : null;
this.diskUsageCache = null;
}
setMaxFileSizeMb(megabytes) { setMaxFileSizeMb(megabytes) {
const mb = Number(megabytes); const mb = Number(megabytes);
if (Number.isFinite(mb) && mb >= 1) { if (Number.isFinite(mb) && mb >= 1) {
@@ -282,6 +312,7 @@ class CsvWriterWorker {
fs.writeFileSync(resolvedPath, fullCsv, 'utf8'); fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
} }
this.clearQuotaCaches(clientId);
return resolvedPath; return resolvedPath;
}); });
@@ -291,6 +322,101 @@ class CsvWriterWorker {
return { written: events.length, files }; return { written: events.length, files };
} }
async compressOldPartitionsAsync(compressAfterDays) {
const days = Number(compressAfterDays);
if (!Number.isFinite(days) || days <= 0) {
return 0;
}
const baseStat = await fs.promises.lstat(this.baseDir).catch(() => null);
if (!baseStat || !baseStat.isDirectory()) {
return 0;
}
const cutoffDate = new Date(Date.now() - Math.floor(days) * 86400000)
.toISOString()
.split('T')[0];
let compressedCount = 0;
try {
const clientDirs = await fs.promises.readdir(this.baseDir);
for (const clientDir of clientDirs) {
const fullClientPath = path.join(this.baseDir, clientDir);
const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null);
if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) continue;
const files = await fs.promises.readdir(fullClientPath).catch(() => []);
for (const file of files) {
if (!file.endsWith('.csv')) continue;
const datePart = this.extractDateFromFilename(file);
if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) continue;
const fullPath = path.join(fullClientPath, file);
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue;
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
let acquired = false;
const tryAcquireLock = async () => {
const handle = await fs.promises.open(lockPath, 'wx');
try {
await handle.writeFile(lockValue, 'utf8');
} finally {
await handle.close();
}
return true;
};
try {
acquired = await tryAcquireLock();
} catch (err) {
if (err && err.code === 'EEXIST') {
await this.removeStaleLockAtomicallyAsync(lockPath);
try {
acquired = await tryAcquireLock();
} catch {
acquired = false;
}
}
}
if (!acquired) continue;
let targetGzPath = `${fullPath}.gz`;
if (fs.existsSync(targetGzPath)) {
const stem = fullPath.replace(/\.csv$/, '');
let suffix = 2;
while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) {
suffix += 1;
}
targetGzPath = `${stem}-gz${suffix}.csv.gz`;
}
const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`;
try {
await pipeline(
fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }),
zlib.createGzip({ level: 6 }),
fs.createWriteStream(tmpGzPath),
);
await fs.promises.utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime).catch(() => undefined);
await fs.promises.rename(tmpGzPath, targetGzPath);
await fs.promises.unlink(fullPath);
this.clearQuotaCaches(clientDir);
compressedCount += 1;
} catch {
await fs.promises.unlink(tmpGzPath).catch(() => undefined);
} finally {
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
}
}
}
} catch {
// Ignore directory scan failure
}
return compressedCount;
}
pruneExpiredPartitions(autoPruneDays) { pruneExpiredPartitions(autoPruneDays) {
const days = Number(autoPruneDays); const days = Number(autoPruneDays);
if (!Number.isFinite(days) || days <= 0 || !fs.existsSync(this.baseDir)) { if (!Number.isFinite(days) || days <= 0 || !fs.existsSync(this.baseDir)) {
@@ -316,8 +442,8 @@ class CsvWriterWorker {
const files = fs.readdirSync(fullClientPath); const files = fs.readdirSync(fullClientPath);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
const fullPath = path.join(fullClientPath, file); const fullPath = path.join(fullClientPath, file);
let fileLstat; let fileLstat;
@@ -361,6 +487,7 @@ class CsvWriterWorker {
try { try {
if (fs.existsSync(fullPath)) { if (fs.existsSync(fullPath)) {
fs.unlinkSync(fullPath); fs.unlinkSync(fullPath);
this.clearQuotaCaches(clientDir);
prunedCount += 1; prunedCount += 1;
} }
} catch { } catch {
@@ -402,8 +529,8 @@ class CsvWriterWorker {
const files = await fs.promises.readdir(fullClientPath).catch(() => []); const files = await fs.promises.readdir(fullClientPath).catch(() => []);
for (const file of files) { for (const file of files) {
if (!file.endsWith('.csv')) continue; if (!this.isPartitionFilename(file)) continue;
const datePart = file.replace('.csv', '').split('-part')[0]; const datePart = this.extractDateFromFilename(file);
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) { if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
const fullPath = path.join(fullClientPath, file); const fullPath = path.join(fullClientPath, file);
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null); const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
@@ -439,6 +566,7 @@ class CsvWriterWorker {
try { try {
await fs.promises.unlink(fullPath); await fs.promises.unlink(fullPath);
this.clearQuotaCaches(clientDir);
prunedCount += 1; prunedCount += 1;
} catch { } catch {
// Ignore individual file unlink failure // Ignore individual file unlink failure
@@ -453,6 +581,106 @@ class CsvWriterWorker {
} }
return prunedCount; return prunedCount;
} }
async getDiskUsageAsync() {
const now = Date.now();
if (this.diskUsageCache && now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS) {
return this.diskUsageCache;
}
let probeDir = path.resolve(this.baseDir);
while (!fs.existsSync(probeDir)) {
const parent = path.dirname(probeDir);
if (parent === probeDir) break;
probeDir = parent;
}
try {
const statfsRes = this.statfsOverrideForTests
? await this.statfsOverrideForTests(probeDir)
: await fs.promises.statfs(probeDir);
const bsize = Math.max(1, Number(statfsRes.bsize) || 4096);
const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize);
const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize);
const usedBytes = Math.max(0, totalBytes - availableBytes);
const usedPct =
totalBytes > 0
? Number(Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2))
: 0;
const snapshot = { usedPct, availableBytes, totalBytes, checkedAt: now };
this.diskUsageCache = snapshot;
return snapshot;
} catch {
const fallback = { usedPct: 0, availableBytes: 0, totalBytes: 0, checkedAt: now };
this.diskUsageCache = fallback;
return fallback;
}
}
async getClientStorageBytesAsync(clientId) {
const safeClientId = this.sanitizeClientId(clientId);
const now = Date.now();
const cached = this.clientBytesCache.get(safeClientId);
if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) {
return cached.bytes;
}
const resolvedBase = path.resolve(this.baseDir);
const clientDir = path.resolve(resolvedBase, safeClientId);
if (!clientDir.startsWith(resolvedBase + path.sep)) {
return 0;
}
let totalBytes = 0;
try {
const dirStat = await fs.promises.lstat(clientDir).catch(() => null);
if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) {
const entries = await fs.promises.readdir(clientDir).catch(() => []);
for (const entry of entries) {
if (!this.isPartitionFilename(entry)) continue;
const filePath = path.join(clientDir, entry);
const stats = await fs.promises.lstat(filePath).catch(() => null);
if (stats && !stats.isSymbolicLink() && stats.isFile()) {
totalBytes += stats.size;
}
}
}
} catch {
// ignore
}
this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes });
return totalBytes;
}
async checkStorageAdmission(clientId, clientQuotaMb, diskPauseThresholdPct) {
const threshold = Number(diskPauseThresholdPct);
if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) {
const disk = await this.getDiskUsageAsync();
if (disk.totalBytes > 0 && disk.usedPct >= threshold) {
return {
allowed: false,
code: 'DISK_SPACE_LOW',
reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`,
};
}
}
const quotaMb = Number(clientQuotaMb);
if (Number.isFinite(quotaMb) && quotaMb > 0) {
const currentBytes = await this.getClientStorageBytesAsync(clientId);
const quotaBytes = Math.floor(quotaMb) * 1024 * 1024;
if (currentBytes >= quotaBytes) {
return {
allowed: false,
code: 'STORAGE_QUOTA_EXCEEDED',
reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`,
};
}
}
return { allowed: true };
}
} }
module.exports = { CsvWriterWorker, resolveDefaultStorageDir }; module.exports = { CsvWriterWorker, resolveDefaultStorageDir };
+727
View File
@@ -0,0 +1,727 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import http from "node:http";
import { createRequire } from "node:module";
import type { AddressInfo } from "node:net";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import zlib from "node:zlib";
import { NextRequest } from "next/server";
import { io as ioClient, type Socket } from "socket.io-client";
import { DELETE as deleteClientRoute, PATCH as patchClientRoute } from "../app/api/clients/[id]/route";
import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route";
import { GET as dataGet } from "../app/api/data/route";
import { GET as eventsGet, POST as eventsPost } from "../app/api/events/route";
import { GET as exportGet } from "../app/api/export/route";
import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route";
import { POST as archivePost } from "../app/api/storage/archive/route";
import { GET as storageGet } from "../app/api/storage/route";
import { POST as s3TestPost } from "../app/api/storage/s3-test/route";
import { SettingsManager } from "../core/adminSettings/settingsManager";
import {
createAdminSessionToken,
getAdminSessionSecret,
} from "../core/auth/adminSession";
import {
createInitialAdmin,
resetAdminStateForTests,
} from "../core/auth/adminUsers";
import { DataBufferEngine } from "../core/data/buffer";
import { SDKManager } from "../core/sdk/sdkManager";
import {
decryptSecret,
encryptSecret,
getMaskedHintFromEncrypted,
SecretVaultError,
} from "../core/Security/secretVault";
import { CSVStorageManager } from "../core/Storage/csvStorage";
import {
buildS3ObjectKey,
normalizeS3EndpointUrl,
testS3Connection,
uploadFileToS3,
} from "../core/Storage/s3Archive";
import {
DEFAULT_EDITABLE_SETTINGS,
settingsToNumericDrafts,
validateEditableSettingsDraft,
validateSingleNumericSetting,
} from "../lib/vconUiHelpers";
const require = createRequire(import.meta.url);
async function makeAdminCookie(): Promise<string> {
resetAdminStateForTests();
await createInitialAdmin("v2-p1@datastack.dev", "V2Phase01Pass!");
const secret = getAdminSessionSecret()!;
const token = await createAdminSessionToken(
"v2-p1@datastack.dev",
secret,
Date.now() + 3600_000,
1,
);
return `datastack_admin_session=${token}`;
}
test("V2-Phase-01 [Vault]: encryptSecret/decryptSecret AES-256-GCM roundtrip, tamper detection & masked hints", () => {
const secretA = "test-master-secret-key-at-least-32-characters";
const secretB = "different-master-secret-key-at-least-32-chars";
const encrypted = encryptSecret("r2_secret_value_987654321", secretA);
assert.ok(encrypted.startsWith("v1:"));
assert.equal(decryptSecret(encrypted, secretA), "r2_secret_value_987654321");
assert.equal(getMaskedHintFromEncrypted(encrypted, secretA), "••••4321");
// Tampered ciphertext throws SecretVaultError (V2-E01)
const parts = encrypted.split(":");
const flippedChar = parts[3].startsWith("A") ? "B" : "A";
const tampered = `${parts[0]}:${parts[1]}:${parts[2]}:${flippedChar}${parts[3].slice(1)}`;
assert.throws(
() => decryptSecret(tampered, secretA),
(err: unknown) =>
err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED",
);
// Wrong master key throws SecretVaultError
assert.throws(
() => decryptSecret(encrypted, secretB),
(err: unknown) =>
err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED",
);
// Too-short master secret rejects key derivation
assert.throws(
() => encryptSecret("hello", "short"),
(err: unknown) =>
err instanceof SecretVaultError && err.code === "SECRET_VAULT_KEY_UNAVAILABLE",
);
});
test("V2-Phase-01 [S3 SigV4]: buildS3ObjectKey, normalizeS3EndpointUrl, testS3Connection & uploadFileToS3 against mock S3 server", async () => {
assert.equal(
buildS3ObjectKey("/datastack/", "client_01", "2026-10-01.csv.gz"),
"datastack/client_01/2026-10-01.csv.gz",
);
assert.equal(
normalizeS3EndpointUrl("", "us-east-1").url.origin,
"https://s3.us-east-1.amazonaws.com",
);
const requests: Array<{
method: string;
url: string;
auth: string;
bodyLen: number;
}> = [];
let failUploads = false;
const mockS3 = http.createServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", (chunk: Buffer) => chunks.push(chunk));
req.on("end", () => {
const body = Buffer.concat(chunks);
requests.push({
method: req.method || "",
url: req.url || "",
auth: String(req.headers.authorization || ""),
bodyLen: body.byteLength,
});
if (req.method === "PUT" && failUploads) {
res.writeHead(500, { "content-type": "text/plain" });
res.end("Simulated S3 outage");
return;
}
if (req.method === "PUT") {
res.writeHead(200, { etag: '"mock-etag-1"' });
res.end("");
return;
}
res.writeHead(200, { "content-type": "application/xml" });
res.end("<ListBucketResult></ListBucketResult>");
});
});
await new Promise<void>((resolve) => mockS3.listen(0, "127.0.0.1", resolve));
const addr = mockS3.address() as AddressInfo;
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-s3-"));
try {
const samplePath = path.join(tmpDir, "2026-10-01.csv.gz");
const gzData = zlib.gzipSync(
Buffer.from('clientId,eventId,timestamp,payload\n"c1","e1","2026-10-01T00:00:00Z","{}"\n'),
);
await fs.writeFile(samplePath, gzData);
const cfg = {
endpoint: `http://127.0.0.1:${addr.port}`,
region: "auto",
bucket: "ds-archive",
prefix: "datastack/",
accessKeyId: "AKIA_TEST_123",
secretAccessKey: "secret_access_key_xyz_999",
};
const probe = await testS3Connection(cfg);
assert.equal(probe.ok, true);
assert.equal(probe.status, 200);
assert.match(requests[0].auth, /^AWS4-HMAC-SHA256 Credential=AKIA_TEST_123\//);
assert.match(requests[0].url, /^\/ds-archive\?list-type=2&max-keys=1$/);
const uploadOk = await uploadFileToS3(
cfg,
samplePath,
"datastack/c1/2026-10-01.csv.gz",
);
assert.equal(uploadOk.ok, true);
assert.equal(uploadOk.etag, '"mock-etag-1"');
assert.equal(requests[1].method, "PUT");
assert.equal(requests[1].url, "/ds-archive/datastack/c1/2026-10-01.csv.gz");
assert.equal(requests[1].bodyLen, gzData.byteLength);
failUploads = true;
const uploadErr = await uploadFileToS3(
cfg,
samplePath,
"datastack/c1/2026-10-01.csv.gz",
);
assert.equal(uploadErr.ok, false);
assert.equal(uploadErr.code, "S3_ARCHIVE_FAILED");
assert.match(uploadErr.error || "", /HTTP 500/);
} finally {
await new Promise<void>((resolve) => mockS3.close(() => resolve()));
await fs.rm(tmpDir, { recursive: true, force: true });
}
});
test("V2-Phase-01 [Compression, Export, S3 Safe Prune V2-E02, Corrupt Gzip V2-E03]", async () => {
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
const prevDbUrl = process.env.DATABASE_URL;
const origBaseStorageDir = CSVStorageManager.baseStorageDir;
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-gz-"));
delete process.env.DATABASE_URL;
process.env.DATASTACK_STORAGE_DIR = tmpDir;
CSVStorageManager.baseStorageDir = tmpDir;
CSVStorageManager.clearRowCountCache();
CSVStorageManager.clearQuotaCaches();
const uploadedKeys: string[] = [];
let failNextUpload = false;
const mockS3 = http.createServer((req, res) => {
req.on("data", () => undefined);
req.on("end", () => {
if (req.method === "PUT") {
if (failNextUpload) {
res.writeHead(502);
res.end("Upstream error");
return;
}
uploadedKeys.push(req.url || "");
res.writeHead(200, { etag: '"ok"' });
res.end("");
return;
}
res.writeHead(200);
res.end("<OK/>");
});
});
await new Promise<void>((resolve) => mockS3.listen(0, "127.0.0.1", resolve));
const s3Addr = mockS3.address() as AddressInfo;
try {
const cookie = await makeAdminCookie();
const today = new Date().toISOString().slice(0, 10);
const oldDate = new Date(Date.now() - 45 * 86_400_000).toISOString().slice(0, 10);
await SettingsManager.updateSettings({
autoPruneDays: 90,
compressAfterDays: 0,
enableS3Backup: false,
diskPauseThresholdPct: 0,
});
// 1. Write 2 old rows (45 days ago) and 1 today row
await CSVStorageManager.appendBatch([
{
clientId: "v2_alpha",
eventId: "evt_old_1",
timestamp: `${oldDate}T10:00:00.000Z`,
payload: '{"stage":"old","seq":1}',
},
{
clientId: "v2_alpha",
eventId: "evt_old_2",
timestamp: `${oldDate}T11:00:00.000Z`,
payload: '{"stage":"old","seq":2}',
},
{
clientId: "v2_alpha",
eventId: "evt_today_1",
timestamp: `${today}T09:00:00.000Z`,
payload: '{"stage":"live","seq":3}',
},
]);
// 2. Compress partitions older than 30 days -> ${oldDate}.csv becomes .csv.gz; today stays .csv
const compressed = await CSVStorageManager.compressOldPartitionsAsync(30);
assert.equal(compressed, 1);
const files = await CSVStorageManager.listAllFilesAsync();
assert.equal(files.length, 2);
const gzMeta = files.find((f) => f.filename === `${oldDate}.csv.gz`);
assert.ok(gzMeta, `Expected compressed ${oldDate}.csv.gz partition`);
assert.equal(gzMeta.linesCount, 2);
assert.equal(gzMeta.compressed, true);
// Original uncompressed ${oldDate}.csv was removed after atomic rename
const legacyCsv = path.join(tmpDir, "v2_alpha", `${oldDate}.csv`);
await assert.rejects(() => fs.stat(legacyCsv));
// 3. Verify GET /api/data reads .csv.gz transparently
const dataRes = await dataGet(
new NextRequest(
`http://localhost:3000/api/data?clientId=v2_alpha&from=${oldDate}&to=${oldDate}`,
{ headers: { cookie } },
),
);
assert.equal(dataRes.status, 200);
const dataJson = await dataRes.json();
assert.equal(dataJson.total, 2);
assert.equal(dataJson.events.length, 2);
assert.equal(dataJson.events[0].eventId, "evt_old_2");
assert.equal(dataJson.events[1].eventId, "evt_old_1");
// 4. Verify GET /api/events reads compressed partition seamlessly
const evRes = await eventsGet(
new NextRequest("http://localhost:3000/api/events?clientId=v2_alpha&limit=10", {
headers: { cookie },
}),
);
assert.equal(evRes.status, 200);
const evJson = await evRes.json();
const ids = (evJson.events as Array<{ eventId: string }>).map((e) => e.eventId);
assert.ok(ids.includes("evt_old_1"));
assert.ok(ids.includes("evt_today_1"));
// 5. Verify GET /api/export CSV, JSON, and raw single .csv.gz download
const exportCsvRes = await exportGet(
new NextRequest(
`http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=csv`,
{ headers: { cookie } },
),
);
assert.equal(exportCsvRes.status, 200);
const exportedCsvText = await exportCsvRes.text();
assert.match(exportedCsvText, /evt_old_1/);
assert.match(exportedCsvText, /evt_old_2/);
const exportJsonRes = await exportGet(
new NextRequest(
`http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=json`,
{ headers: { cookie } },
),
);
assert.equal(exportJsonRes.status, 200);
const exportedJson = await exportJsonRes.json();
assert.equal(exportedJson.length, 2);
const exportSingleGzRes = await exportGet(
new NextRequest(
`http://localhost:3000/api/export?clientId=v2_alpha&file=${oldDate}.csv.gz`,
{ headers: { cookie } },
),
);
assert.equal(exportSingleGzRes.status, 200);
assert.equal(exportSingleGzRes.headers.get("content-type"), "application/gzip");
const downloadedBuf = Buffer.from(await exportSingleGzRes.arrayBuffer());
assert.match(zlib.gunzipSync(downloadedBuf).toString("utf8"), /evt_old_1/);
// 6. Save S3 settings via POST /api/settings and verify GET /api/settings masks secret (zero plaintext leak)
const saveRes = await settingsPost(
new NextRequest("http://localhost:3000/api/settings", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({
compressAfterDays: 30,
autoPruneDays: 60,
diskPauseThresholdPct: 92,
enableS3Backup: true,
s3Endpoint: `http://127.0.0.1:${s3Addr.port}`,
s3Region: "auto",
s3Bucket: "prod-vault",
s3Prefix: "archive-test/",
s3AccessKeyId: "AKIA_V2_KEY",
s3SecretAccessKey: "super_secret_v2_token_7788",
}),
}),
);
assert.equal(saveRes.status, 200);
const getRes = await settingsGet(
new NextRequest("http://localhost:3000/api/settings", { headers: { cookie } }),
);
assert.equal(getRes.status, 200);
const getText = await getRes.text();
assert.doesNotMatch(getText, /super_secret_v2_token_7788/);
const parsedSettings = JSON.parse(getText).settings;
assert.equal(parsedSettings.hasS3SecretKey, true);
assert.equal(parsedSettings.s3SecretKeyMask, "••••7788");
assert.equal(parsedSettings.compressAfterDays, 30);
assert.equal(parsedSettings.diskPauseThresholdPct, 92);
// 7. POST /api/storage/s3-test succeeds with saved credentials
const s3TestRes = await s3TestPost(
new NextRequest("http://localhost:3000/api/storage/s3-test", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({}),
}),
);
assert.equal(s3TestRes.status, 200);
assert.equal((await s3TestRes.json()).success, true);
// 8. V2-E02: When S3 upload fails, POST /api/storage/archive returns 502 and NEVER deletes local partition
failNextUpload = true;
const failArchiveRes = await archivePost(
new NextRequest("http://localhost:3000/api/storage/archive", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ pruneAfterUpload: true, olderThanDays: 30 }),
}),
);
assert.equal(failArchiveRes.status, 502);
const failArchiveJson = await failArchiveRes.json();
assert.equal(failArchiveJson.code, "S3_ARCHIVE_FAILED");
assert.equal(failArchiveJson.prunedCount, 0);
// File MUST still exist on disk!
await fs.stat(gzMeta.filePath);
// 9. When S3 upload succeeds with pruneAfterUpload: false, local file stays + uploadedKeys receives .csv.gz
failNextUpload = false;
const okArchiveRes = await archivePost(
new NextRequest("http://localhost:3000/api/storage/archive", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ pruneAfterUpload: false, olderThanDays: 30 }),
}),
);
assert.equal(okArchiveRes.status, 200);
const okArchiveJson = await okArchiveRes.json();
assert.equal(okArchiveJson.archivedCount, 1);
assert.equal(okArchiveJson.prunedCount, 0);
assert.deepEqual(uploadedKeys, [`/prod-vault/archive-test/v2_alpha/${oldDate}.csv.gz`]);
await fs.stat(gzMeta.filePath);
// 10. V2-E03: Corrupt .csv.gz cleanly surfaces error in /api/data instead of crashing
await fs.writeFile(
path.join(tmpDir, "v2_alpha", "2024-01-01.csv.gz"),
Buffer.from("not-valid-gzip-bytes"),
);
const corruptDataRes = await dataGet(
new NextRequest(
"http://localhost:3000/api/data?clientId=v2_alpha&from=2024-01-01&to=2024-01-01",
{ headers: { cookie } },
),
);
assert.equal(corruptDataRes.status, 500);
const corruptJson = await corruptDataRes.json();
assert.match(corruptJson.error || "", /Corrupt|\.gz/i);
} finally {
await new Promise<void>((resolve) => mockS3.close(() => resolve()));
await DataBufferEngine.shutdown();
CSVStorageManager.baseStorageDir = origBaseStorageDir;
CSVStorageManager.clearRowCountCache();
CSVStorageManager.clearQuotaCaches();
if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR;
else process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
if (prevDbUrl === undefined) delete process.env.DATABASE_URL;
else process.env.DATABASE_URL = prevDbUrl;
await fs.rm(tmpDir, { recursive: true, force: true });
resetAdminStateForTests();
}
});
test("V2-Phase-01 [Quotas V2-E04 & V2-E05]: HTTP POST /api/events + Port 4000 Socket.IO enforce disk threshold & per-client storageQuotaMb while keeping Admin reads online", async () => {
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
const prevDbUrl = process.env.DATABASE_URL;
const prevDisableListen = process.env.DATASTACK_DISABLE_LISTEN;
const origBaseStorageDir = CSVStorageManager.baseStorageDir;
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-quota-"));
delete process.env.DATABASE_URL;
process.env.DATASTACK_STORAGE_DIR = tmpDir;
CSVStorageManager.baseStorageDir = tmpDir;
process.env.DATASTACK_DISABLE_LISTEN = "1";
CSVStorageManager.clearQuotaCaches();
await DataBufferEngine.shutdown();
DataBufferEngine.setRedisClientForTests(null);
const serverMod = require("../server/index.js") as {
server: http.Server;
io: { close: (cb: () => void) => void };
writer: {
baseDir: string;
setStatfsOverrideForTests: (
fn: ((dir: string) => Promise<{ blocks: number; bavail: number; bsize: number }>) | null,
) => void;
clearQuotaCaches: () => void;
};
runtimeConfig: { diskPauseThresholdPct: number };
setDbPoolForTests: (pool: unknown) => void;
stopTimers: () => void;
};
const origWriterBaseDir = serverMod.writer.baseDir;
serverMod.writer.baseDir = tmpDir;
serverMod.writer.clearQuotaCaches();
const openSockets: Socket[] = [];
let listeningUrl = "";
try {
const cookie = await makeAdminCookie();
// 1. Validate UI helper & API validation for diskPauseThresholdPct (0 or 50..99)
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "0").valid, true);
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "95").valid, true);
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "25").valid, false);
const fullDraft = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS);
assert.equal(validateEditableSettingsDraft(fullDraft, true).valid, true);
const badPctRes = await settingsPost(
new NextRequest("http://localhost:3000/api/settings", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ diskPauseThresholdPct: 25 }),
}),
);
assert.equal(badPctRes.status, 400);
await SettingsManager.updateSettingsAsync({ diskPauseThresholdPct: 90 });
// 2. Register client via /api/clients with 1 MB quota & edit quota via PATCH /api/clients/[id]
const createRes = await clientsPost(
new NextRequest("http://localhost:3000/api/clients", {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({
clientId: "quota_client",
name: "Quota Client",
rateLimit: 200,
storageQuotaMb: 2,
}),
}),
);
assert.equal(createRes.status, 201);
const createdClient = (await createRes.json()).client;
assert.equal(createdClient.storageQuotaMb, 2);
const patchRes = await patchClientRoute(
new NextRequest("http://localhost:3000/api/clients/quota_client", {
method: "PATCH",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ storageQuotaMb: 1 }),
}),
{ params: Promise.resolve({ id: "quota_client" }) },
);
assert.equal(patchRes.status, 200);
assert.equal((await patchRes.json()).client.storageQuotaMb, 1);
const listRes = await clientsGet(
new NextRequest("http://localhost:3000/api/clients", { headers: { cookie } }),
);
const listed = (await listRes.json()).clients.find(
(c: { clientId: string }) => c.clientId === "quota_client",
);
assert.equal(listed.storageQuotaMb, 1);
// Seed 1.1 MB into client dir to breach 1 MB quota (V2-E05)
const clientDir = path.join(tmpDir, "quota_client");
await fs.mkdir(clientDir, { recursive: true });
await fs.writeFile(
path.join(clientDir, "2026-10-10.csv"),
Buffer.alloc(Math.ceil(1.1 * 1024 * 1024), "x"),
);
CSVStorageManager.clearQuotaCaches("quota_client");
const httpQuotaRes = await eventsPost(
new NextRequest("http://localhost:3000/api/events", {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${createdClient.token}`,
},
body: JSON.stringify({ eventId: "q_evt_1", data: { ok: true } }),
}),
);
assert.equal(httpQuotaRes.status, 507);
assert.equal((await httpQuotaRes.json()).code, "STORAGE_QUOTA_EXCEEDED");
// Now test global VPS disk threshold >= 90% (V2-E04) across HTTP + Socket.IO
await patchClientRoute(
new NextRequest("http://localhost:3000/api/clients/quota_client", {
method: "PATCH",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ storageQuotaMb: 0 }),
}),
{ params: Promise.resolve({ id: "quota_client" }) },
);
CSVStorageManager.setStatfsOverrideForTests(async () => ({
blocks: 1000,
bavail: 50, // 95% used >= 90% threshold
bsize: 4096,
}));
const httpDiskRes = await eventsPost(
new NextRequest("http://localhost:3000/api/events", {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${createdClient.token}`,
},
body: JSON.stringify({ eventId: "q_evt_2", data: { ok: true } }),
}),
);
assert.equal(httpDiskRes.status, 507);
assert.equal((await httpDiskRes.json()).code, "DISK_SPACE_LOW");
// Admin read/storage/export MUST remain 100% operational while ingestion is paused!
const storageRes = await storageGet(
new NextRequest("http://localhost:3000/api/storage", { headers: { cookie } }),
);
assert.equal(storageRes.status, 200);
const storageJson = await storageRes.json();
assert.equal(storageJson.diskUsage.usedPct, 95);
// Port 4000 Socket.IO enforcement for DISK_SPACE_LOW & STORAGE_QUOTA_EXCEEDED
let mockQuotaMb = 0;
serverMod.runtimeConfig.diskPauseThresholdPct = 90;
serverMod.setDbPoolForTests({
query: async (sql: string) => {
if (sql.includes("FROM client_sdks")) {
return {
rowCount: 1,
rows: [
{
client_id: "quota_client",
token: createdClient.token,
status: "active",
rate_limit: 200,
storage_quota_mb: mockQuotaMb,
},
],
};
}
return { rowCount: 0, rows: [] };
},
});
serverMod.writer.setStatfsOverrideForTests(async () => ({
blocks: 1000,
bavail: 50, // 95% used
bsize: 4096,
}));
await new Promise<void>((resolve) =>
serverMod.server.listen(0, "127.0.0.1", resolve),
);
const addr = serverMod.server.address() as AddressInfo;
listeningUrl = `http://127.0.0.1:${addr.port}`;
const socket = ioClient(listeningUrl, {
auth: { token: createdClient.token },
transports: ["websocket"],
reconnection: false,
timeout: 3000,
});
openSockets.push(socket);
await new Promise<void>((resolve, reject) => {
socket.on("connect", () => resolve());
socket.on("connect_error", reject);
});
const singleDiskAck = await new Promise<{
success: boolean;
code?: string;
}>((resolve, reject) => {
socket.timeout(3000).emit("sync_data", { data: { k: 1 } }, (err: unknown, ack: { success: boolean; code?: string }) => {
if (err) reject(err);
else resolve(ack);
});
});
assert.equal(singleDiskAck.success, false);
assert.equal(singleDiskAck.code, "DISK_SPACE_LOW");
// Relax disk threshold & activate per-client 1 MB quota in Socket.IO worker
serverMod.writer.setStatfsOverrideForTests(async () => ({
blocks: 1000,
bavail: 800, // 20% used
bsize: 4096,
}));
serverMod.writer.clearQuotaCaches();
mockQuotaMb = 1;
// Wait out 2s verifyClientInDb cache or update query response cleanly
await new Promise((r) => setTimeout(r, 2100));
const batchQuotaAck = await new Promise<{
success: boolean;
code?: string;
rejected?: number;
}>((resolve, reject) => {
socket.timeout(3000).emit(
"batch_sync",
[{ eventId: "b1", data: { k: 1 } }],
(err: unknown, ack: { success: boolean; code?: string; rejected?: number }) => {
if (err) reject(err);
else resolve(ack);
},
);
});
assert.equal(batchQuotaAck.success, false);
assert.equal(batchQuotaAck.code, "STORAGE_QUOTA_EXCEEDED");
assert.equal(batchQuotaAck.rejected, 1);
// Clean delete route check
const delRes = await deleteClientRoute(
new NextRequest("http://localhost:3000/api/clients/quota_client?purgeFiles=true", {
method: "DELETE",
headers: { cookie },
}),
{ params: Promise.resolve({ id: "quota_client" }) },
);
assert.equal(delRes.status, 200);
} finally {
for (const s of openSockets) {
try {
s.disconnect();
} catch {
// ignore
}
}
serverMod.writer.setStatfsOverrideForTests(null);
serverMod.writer.baseDir = origWriterBaseDir;
serverMod.writer.clearQuotaCaches();
serverMod.setDbPoolForTests(null);
serverMod.stopTimers();
await new Promise<void>((resolve) => {
serverMod.io.close(() => {
if (serverMod.server.listening) {
serverMod.server.close(() => resolve());
} else {
resolve();
}
});
});
SDKManager.stopActivityFlushTimerForTests();
CSVStorageManager.setStatfsOverrideForTests(null);
CSVStorageManager.clearQuotaCaches();
await DataBufferEngine.shutdown();
CSVStorageManager.baseStorageDir = origBaseStorageDir;
if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR;
else process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
if (prevDisableListen === undefined) delete process.env.DATASTACK_DISABLE_LISTEN;
else process.env.DATASTACK_DISABLE_LISTEN = prevDisableListen;
if (prevDbUrl === undefined) delete process.env.DATABASE_URL;
else process.env.DATABASE_URL = prevDbUrl;
await fs.rm(tmpDir, { recursive: true, force: true });
resetAdminStateForTests();
}
});
@@ -0,0 +1,631 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import http from "node:http";
import { createRequire } from "node:module";
import type { AddressInfo } from "node:net";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import { NextRequest } from "next/server";
import { io as ioClient, type Socket } from "socket.io-client";
import { PATCH as patchClientRoute } from "../app/api/clients/[id]/route";
import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route";
import { POST as eventsPost } from "../app/api/events/route";
import { GET as exportGet } from "../app/api/export/route";
import { GET as prometheusGet } from "../app/api/prometheus/route";
import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route";
import { POST as webhookTestPost } from "../app/api/settings/webhook-test/route";
import { SettingsManager } from "../core/adminSettings/settingsManager";
import {
buildWebhookSignature,
clearWebhookCooldownCacheForTests,
dispatchWebhookAlertAsync,
formatOutboundWebhookBody,
} from "../core/alerts/webhookNotifier";
import {
createAdminSessionToken,
getAdminSessionSecret,
} from "../core/auth/adminSession";
import {
createInitialAdmin,
resetAdminStateForTests,
} from "../core/auth/adminUsers";
import { DataBufferEngine } from "../core/data/buffer";
import { SDKManager } from "../core/sdk/sdkManager";
import {
isClientIpAllowed,
isClientTokenExpired,
validateAllowedIpsInput,
validateClientExpiresAtInput,
} from "../core/Security/ipAllowlist";
import { CSVStorageManager } from "../core/Storage/csvStorage";
import {
DEFAULT_ALERTING_SETTINGS,
DEFAULT_EDITABLE_SETTINGS,
normalizeAlertingSettings,
settingsToNumericDrafts,
validateEditableSettingsDraft,
validateSingleNumericSetting,
} from "../lib/vconUiHelpers";
const require = createRequire(import.meta.url);
async function makeAdminCookie(): Promise<string> {
resetAdminStateForTests();
await createInitialAdmin("v2-p2@datastack.dev", "V2Phase02Pass!");
const secret = getAdminSessionSecret()!;
const token = await createAdminSessionToken(
"v2-p2@datastack.dev",
secret,
Date.now() + 3600_000,
1,
);
return `datastack_admin_session=${token}`;
}
test("V2-Phase-02 [IP Allowlist & Token Expiry]: validateAllowedIpsInput, isClientIpAllowed (IPv4/IPv6/CIDR) & isClientTokenExpired", () => {
// Valid empty or wildcard
assert.deepEqual(validateAllowedIpsInput(""), { valid: true, normalized: "" });
assert.deepEqual(validateAllowedIpsInput(" "), { valid: true, normalized: "" });
assert.deepEqual(validateAllowedIpsInput("*"), { valid: true, normalized: "*" });
// Valid comma-separated IPv4, IPv4 CIDR, IPv6, IPv6 CIDR
const validList = validateAllowedIpsInput(
"203.0.113.10, 10.0.0.0/8, ::ffff:192.0.2.5, 2001:db8::/32, 203.0.113.10",
);
assert.equal(validList.valid, true);
if (validList.valid) {
assert.equal(
validList.normalized,
"203.0.113.10, 10.0.0.0/8, 192.0.2.5, 2001:db8::/32",
);
}
// Invalid IP or CIDR prefix rejected
const badCidr = validateAllowedIpsInput("10.0.0.0/33");
assert.equal(badCidr.valid, false);
const badIp = validateAllowedIpsInput("999.888.777.666");
assert.equal(badIp.valid, false);
// Matching behavior
assert.equal(isClientIpAllowed("198.51.100.77", ""), true);
assert.equal(isClientIpAllowed("198.51.100.77", "*"), true);
assert.equal(
isClientIpAllowed("203.0.113.10", "203.0.113.10, 10.0.0.0/8"),
true,
);
assert.equal(
isClientIpAllowed("::ffff:203.0.113.10", "203.0.113.10, 10.0.0.0/8"),
true,
);
assert.equal(
isClientIpAllowed("10.24.99.1", "203.0.113.10, 10.0.0.0/8"),
true,
);
assert.equal(
isClientIpAllowed("192.168.1.50", "203.0.113.10, 10.0.0.0/8"),
false,
);
// Loopback equivalence between 127.0.0.1 and ::1
assert.equal(isClientIpAllowed("::1", "127.0.0.1"), true);
assert.equal(isClientIpAllowed("127.0.0.1", "::1"), true);
// Expiration validation & evaluation
assert.deepEqual(validateClientExpiresAtInput(null), {
valid: true,
normalized: null,
});
assert.deepEqual(validateClientExpiresAtInput(""), {
valid: true,
normalized: null,
});
const validExp = validateClientExpiresAtInput("2030-06-15T12:00:00.000Z");
assert.equal(validExp.valid, true);
if (validExp.valid) {
assert.equal(validExp.normalized, "2030-06-15T12:00:00.000Z");
}
const badExp = validateClientExpiresAtInput("not-a-timestamp");
assert.equal(badExp.valid, false);
const nowMs = Date.parse("2026-11-19T12:00:00.000Z");
assert.equal(isClientTokenExpired(null, nowMs), false);
assert.equal(isClientTokenExpired("2026-11-19T12:00:01.000Z", nowMs), false);
assert.equal(isClientTokenExpired("2026-11-19T12:00:00.000Z", nowMs), true);
assert.equal(isClientTokenExpired("2026-11-19T11:59:59.000Z", nowMs), true);
});
test("V2-Phase-02 [Webhook Notifier & /api/settings/webhook-test]: HMAC-SHA256 signing, cooldown suppression & V2-E06 error handling", async () => {
clearWebhookCooldownCacheForTests();
const receivedRequests: Array<{
headers: http.IncomingHttpHeaders;
body: string;
json: Record<string, unknown>;
}> = [];
let responseStatus = 200;
const mockWebhookServer = http.createServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
req.on("end", () => {
const bodyStr = Buffer.concat(chunks).toString("utf8");
let parsed: Record<string, unknown> = {};
try {
parsed = JSON.parse(bodyStr);
} catch {
// ignore
}
receivedRequests.push({
headers: req.headers,
body: bodyStr,
json: parsed,
});
res.writeHead(responseStatus, { "Content-Type": "application/json" });
res.end(JSON.stringify({ ok: responseStatus < 400 }));
});
});
await new Promise<void>((resolve) => mockWebhookServer.listen(0, "127.0.0.1", resolve));
const port = (mockWebhookServer.address() as AddressInfo).port;
const webhookUrl = `http://127.0.0.1:${port}/hooks/datastack`;
try {
// 1. Format & signature verification
const formatted = formatOutboundWebhookBody({
event: "DISK_SPACE_LOW",
severity: "critical",
title: "Disk Threshold Reached",
message: "VPS disk is at 95.4%.",
clientId: "svc_alpha",
timestamp: "2026-11-19T12:00:00.000Z",
details: { usedPct: 95.4 },
});
assert.ok(formatted.text.includes("DISK_SPACE_LOW"));
assert.equal(formatted.content, formatted.text);
const bodyJson = JSON.stringify(formatted);
const sig = buildWebhookSignature(bodyJson, "webhook_secret_xyz");
assert.ok(sig.startsWith("sha256="));
// 2. Cooldown deduplication test
const first = await dispatchWebhookAlertAsync(
{
enabled: true,
url: webhookUrl,
secret: "webhook_secret_xyz",
cooldownSec: 120,
},
{
event: "CLIENT_QUOTA_EXCEEDED",
severity: "warning",
title: "Client Quota Exceeded",
message: "Quota reached for svc_alpha",
clientId: "svc_alpha",
},
);
assert.equal(first.ok, true);
assert.equal(receivedRequests.length, 1);
assert.equal(
receivedRequests[0].headers["x-datastack-event"],
"CLIENT_QUOTA_EXCEEDED",
);
assert.equal(
receivedRequests[0].headers["x-datastack-signature"],
buildWebhookSignature(receivedRequests[0].body, "webhook_secret_xyz"),
);
// Duplicate event for same client within cooldown window is suppressed
const second = await dispatchWebhookAlertAsync(
{
enabled: true,
url: webhookUrl,
secret: "webhook_secret_xyz",
cooldownSec: 120,
},
{
event: "CLIENT_QUOTA_EXCEEDED",
severity: "warning",
title: "Client Quota Exceeded",
message: "Quota reached again for svc_alpha",
clientId: "svc_alpha",
},
);
assert.equal(second.ok, true);
assert.equal(second.suppressedByCooldown, true);
assert.equal(receivedRequests.length, 1);
// 3. Test POST /api/settings/webhook-test route (200 & 502 WEBHOOK_DELIVERY_FAILED V2-E06)
const adminCookie = await makeAdminCookie();
const okTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", {
method: "POST",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
webhookUrl,
webhookSecret: "test_route_hmac_secret",
}),
});
const okTestRes = await webhookTestPost(okTestReq);
assert.equal(okTestRes.status, 200);
const okTestData = await okTestRes.json();
assert.equal(okTestData.success, true);
assert.equal(receivedRequests.length, 2);
assert.equal(receivedRequests[1].headers["x-datastack-event"], "TEST_ALERT");
// Simulate upstream 500 error -> 502 WEBHOOK_DELIVERY_FAILED (V2-E06)
responseStatus = 500;
const failTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", {
method: "POST",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
webhookUrl,
}),
});
const failTestRes = await webhookTestPost(failTestReq);
assert.equal(failTestRes.status, 502);
const failTestData = await failTestRes.json();
assert.equal(failTestData.code, "WEBHOOK_DELIVERY_FAILED");
} finally {
await new Promise<void>((resolve) => mockWebhookServer.close(() => resolve()));
}
});
test("V2-Phase-02 [HTTP & Socket Ingestion Security]: Per-client IP allowlist (403 V2-E07) & Token expiration (401 V2-E08)", async () => {
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-sec-"));
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
process.env.DATASTACK_STORAGE_DIR = tmpDir;
CSVStorageManager.resetForTests(tmpDir);
await CSVStorageManager.initializeAsync();
clearWebhookCooldownCacheForTests();
try {
const adminCookie = await makeAdminCookie();
// 1. Create client with strict IP allowlist (only 10.10.0.0/16) and future expiration
const futureIso = new Date(Date.now() + 3600_000).toISOString();
const createRes = await clientsPost(
new NextRequest("http://localhost:3000/api/clients", {
method: "POST",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
name: "IP Restricted Client",
rateLimit: 100,
storageQuotaMb: 0,
allowedIps: "10.10.0.0/16",
expiresAt: futureIso,
}),
}),
);
assert.equal(createRes.status, 200);
const createData = await createRes.json();
const client = createData.client;
assert.equal(client.allowedIps, "10.10.0.0/16");
assert.equal(client.expiresAt, futureIso);
// 2. Request from forbidden IP 192.168.1.25 -> 403 CLIENT_IP_FORBIDDEN (V2-E07)
const forbiddenIpRes = await eventsPost(
new NextRequest("http://localhost:3000/api/events", {
method: "POST",
headers: {
authorization: `Bearer ${client.token}`,
"content-type": "application/json",
"x-forwarded-for": "192.168.1.25",
},
body: JSON.stringify({
eventId: "evt_ip_denied",
payload: { step: 1 },
}),
}),
);
assert.equal(forbiddenIpRes.status, 403);
const forbiddenIpJson = await forbiddenIpRes.json();
assert.equal(forbiddenIpJson.code, "CLIENT_IP_FORBIDDEN");
// 3. Request from allowed CIDR IP 10.10.42.9 -> 200 OK
const allowedIpRes = await eventsPost(
new NextRequest("http://localhost:3000/api/events", {
method: "POST",
headers: {
authorization: `Bearer ${client.token}`,
"content-type": "application/json",
"x-forwarded-for": "10.10.42.9",
},
body: JSON.stringify({
eventId: "evt_ip_allowed",
payload: { step: 2 },
}),
}),
);
assert.equal(allowedIpRes.status, 200);
// 4. Patch client with past expiresAt -> 401 CLIENT_TOKEN_EXPIRED (V2-E08)
const pastIso = new Date(Date.now() - 60_000).toISOString();
const patchRes = await patchClientRoute(
new NextRequest(
`http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`,
{
method: "PATCH",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
expiresAt: pastIso,
}),
},
),
{ params: Promise.resolve({ id: client.clientId }) },
);
assert.equal(patchRes.status, 200);
const patchData = await patchRes.json();
assert.equal(patchData.client.expiresAt, pastIso);
const expiredRes = await eventsPost(
new NextRequest("http://localhost:3000/api/events", {
method: "POST",
headers: {
authorization: `Bearer ${client.token}`,
"content-type": "application/json",
"x-forwarded-for": "10.10.42.9",
},
body: JSON.stringify({
eventId: "evt_expired",
payload: { step: 3 },
}),
}),
);
assert.equal(expiredRes.status, 401);
const expiredJson = await expiredRes.json();
assert.equal(expiredJson.code, "CLIENT_TOKEN_EXPIRED");
// 5. Verify Port 4000 Socket.IO server enforces IP allowlist & token expiry
process.env.DATASTACK_DISABLE_LISTEN = "1";
const ingestionApp = require("../server/index.js");
// Sync expired client to Port 4000 in-memory cache and verify verifyClientInDb returns CLIENT_TOKEN_EXPIRED
await new Promise<void>((resolve) =>
ingestionApp.server.listen(0, "127.0.0.1", () => resolve()),
);
const port4000 = (ingestionApp.server.address() as AddressInfo).port;
try {
const expiredCheck = await ingestionApp.verifyClientInDb(
client.clientId,
client.token,
true,
"10.10.42.9",
);
assert.equal(expiredCheck.ok, false);
assert.equal(expiredCheck.reason, "CLIENT_TOKEN_EXPIRED");
// Clear expiration via PATCH, keep IP allowlist 10.10.0.0/16, and check Socket.IO handshake from 127.0.0.1
await patchClientRoute(
new NextRequest(
`http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`,
{
method: "PATCH",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
expiresAt: null,
}),
},
),
{ params: Promise.resolve({ id: client.clientId }) },
);
const ipForbiddenCheck = await ingestionApp.verifyClientInDb(
client.clientId,
client.token,
true,
"127.0.0.1",
);
assert.equal(ipForbiddenCheck.ok, false);
assert.equal(ipForbiddenCheck.reason, "CLIENT_IP_FORBIDDEN");
// Socket.IO connection from 127.0.0.1 should fail handshake with CLIENT_IP_FORBIDDEN
const socketError = await new Promise<string>((resolve) => {
const sock: Socket = ioClient(`http://127.0.0.1:${port4000}`, {
auth: { clientId: client.clientId, token: client.token },
transports: ["websocket"],
reconnection: false,
});
sock.on("connect_error", (err) => {
sock.close();
resolve(err.message);
});
sock.on("connect", () => {
sock.close();
resolve("CONNECTED_UNEXPECTEDLY");
});
});
assert.ok(socketError.includes("CLIENT_IP_FORBIDDEN"));
} finally {
await new Promise<void>((resolve) =>
ingestionApp.server.close(() => resolve()),
);
}
} finally {
await DataBufferEngine.flushAllAsync();
SDKManager.stopActivityFlushTimerForTests();
if (prevStorageDir === undefined) {
delete process.env.DATASTACK_STORAGE_DIR;
} else {
process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
}
CSVStorageManager.resetForTests();
await fs.rm(tmpDir, { recursive: true, force: true });
}
});
test("V2-Phase-02 [Prometheus Metrics & NDJSON Streaming Export]: GET /api/prometheus (V2-FEAT-04 / V2-E09) & GET /api/export?format=ndjson (V2-FEAT-06)", async () => {
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-prom-"));
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
process.env.DATASTACK_STORAGE_DIR = tmpDir;
CSVStorageManager.resetForTests(tmpDir);
await CSVStorageManager.initializeAsync();
try {
const adminCookie = await makeAdminCookie();
// 1. Unauthenticated scrape when Prometheus is disabled -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09)
const unauthPromRes = await prometheusGet(
new NextRequest("http://localhost:3000/api/prometheus"),
);
assert.equal(unauthPromRes.status, 401);
const unauthPromJson = await unauthPromRes.json();
assert.equal(unauthPromJson.code, "PROMETHEUS_UNAUTHORIZED");
// 2. Configure Webhook & Prometheus settings via POST /api/settings
const saveSettingsRes = await settingsPost(
new NextRequest("http://localhost:3000/api/settings", {
method: "POST",
headers: {
cookie: adminCookie,
"content-type": "application/json",
},
body: JSON.stringify({
enableWebhookAlerts: true,
webhookUrl: "https://hooks.example.com/datastack",
webhookSecret: "super_secret_webhook_hmac_9988",
webhookCooldownSec: 180,
enablePrometheusMetrics: true,
prometheusMetricsToken: "prom_scrape_bearer_token_7766",
}),
}),
);
assert.equal(saveSettingsRes.status, 200);
const savedData = await saveSettingsRes.json();
assert.equal(savedData.settings.enableWebhookAlerts, true);
assert.equal(savedData.settings.webhookCooldownSec, 180);
assert.equal(savedData.settings.hasWebhookSecret, true);
assert.equal(savedData.settings.webhookSecretMask, "••••9988");
assert.equal("webhookSecretEnc" in savedData.settings, false);
assert.equal(savedData.settings.enablePrometheusMetrics, true);
assert.equal(savedData.settings.hasPrometheusToken, true);
assert.equal(savedData.settings.prometheusTokenMask, "••••7766");
assert.equal("prometheusMetricsTokenEnc" in savedData.settings, false);
// 3. Wrong Bearer token -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09)
const wrongTokenRes = await prometheusGet(
new NextRequest("http://localhost:3000/api/prometheus", {
headers: {
authorization: "Bearer wrong_token_value",
},
}),
);
assert.equal(wrongTokenRes.status, 401);
// 4. Valid Bearer token -> 200 Prometheus text exposition format
const validTokenRes = await prometheusGet(
new NextRequest("http://localhost:3000/api/prometheus", {
headers: {
authorization: "Bearer prom_scrape_bearer_token_7766",
},
}),
);
assert.equal(validTokenRes.status, 200);
assert.ok(
validTokenRes.headers
.get("content-type")
?.includes("text/plain; version=0.0.4"),
);
const metricsBody = await validTokenRes.text();
assert.ok(metricsBody.includes("datastack_up 1"));
assert.ok(metricsBody.includes("datastack_events_total"));
assert.ok(metricsBody.includes("datastack_storage_bytes_total"));
assert.ok(metricsBody.includes("datastack_disk_used_percent"));
// 5. Seed events and export as streaming NDJSON (GET /api/export?format=ndjson) (V2-FEAT-06)
await CSVStorageManager.appendBatchAsync("client_ndjson", [
{
clientId: "client_ndjson",
eventId: "evt_nd_1",
timestamp: "2026-11-19T10:00:00.000Z",
payload: JSON.stringify({ action: "purchase", amount: 49 }),
},
{
clientId: "client_ndjson",
eventId: "evt_nd_2",
timestamp: "2026-11-19T10:01:00.000Z",
payload: JSON.stringify({ action: "refund", amount: 12 }),
},
]);
const ndjsonRes = await exportGet(
new NextRequest(
"http://localhost:3000/api/export?format=ndjson&clientId=client_ndjson",
{
headers: { cookie: adminCookie },
},
),
);
assert.equal(ndjsonRes.status, 200);
assert.ok(
ndjsonRes.headers
.get("content-type")
?.includes("application/x-ndjson"),
);
assert.ok(
ndjsonRes.headers
.get("content-disposition")
?.includes("datastack-client_ndjson.ndjson"),
);
const ndjsonText = await ndjsonRes.text();
const lines = ndjsonText
.trim()
.split("\n")
.map((line) => JSON.parse(line));
assert.equal(lines.length, 2);
assert.equal(lines[0].clientId, "client_ndjson");
assert.equal(lines[0].eventId, "evt_nd_1");
assert.deepEqual(lines[0].payload, { action: "purchase", amount: 49 });
assert.equal(lines[1].eventId, "evt_nd_2");
assert.deepEqual(lines[1].payload, { action: "refund", amount: 12 });
// 6. Verify UI helper validation for webhookCooldownSec & normalizeAlertingSettings
assert.equal(
validateSingleNumericSetting("webhookCooldownSec", "5").valid,
false,
);
assert.equal(
validateSingleNumericSetting("webhookCooldownSec", "300").valid,
true,
);
const drafts = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS);
const validatedDraft = validateEditableSettingsDraft(drafts, true);
assert.equal(validatedDraft.valid, true);
assert.deepEqual(validatedDraft.values, DEFAULT_EDITABLE_SETTINGS);
assert.deepEqual(
normalizeAlertingSettings(undefined),
DEFAULT_ALERTING_SETTINGS,
);
} finally {
await DataBufferEngine.flushAllAsync();
SDKManager.stopActivityFlushTimerForTests();
await SettingsManager.updateSettingsAsync({
enableWebhookAlerts: false,
webhookUrl: "",
clearWebhookSecret: true,
webhookCooldownSec: 300,
enablePrometheusMetrics: false,
clearPrometheusToken: true,
});
if (prevStorageDir === undefined) {
delete process.env.DATASTACK_STORAGE_DIR;
} else {
process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
}
CSVStorageManager.resetForTests();
await fs.rm(tmpDir, { recursive: true, force: true });
}
});