feat(client): implement per-client storage and security controls
- Added storage quota, IP allowlisting, and expiration logic to client management APIs. - Updated CSV storage manager to support auto-compression and improved partition file parsing. - Integrated security checks into event ingestion and data retrieval routes.
This commit is contained in:
1 parent
89e4d68690
commit
2873b9904a
32 files changed
+6075
-241
No files matched your search
@@ -1,7 +1,7 @@
|
|||||||
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory
|
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory
|
||||||
|
|
||||||
> **Last Synchronized**: 2026-11-19
|
> **Last Synchronized**: 2026-11-19
|
||||||
> **Status**: **Production & Deployment Hardening Complete (`BASELINE-2026-11-19-R2` + Deployment Hardening) — `4 / 4` R2 Phases (`18/18` bugs) + `5/5` Docker/Coolify Redeploy Fixes (`DEPLOY-BUG-01`–`05`) Verified (`85/85` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
|
> **Status**: **V2-Phase-01 Complete & Verified (`BASELINE-2026-11-19-V2`) — `1 / 4` V2 Phases Done (Storage `.csv.gz` Auto-Compression, S3/R2 Off-Site Archive with AES-256-GCM Secret Vault, Per-Client `storageQuotaMb` & VPS Disk Pause Guard `HTTP 507`) — Verified (`89/89` TS tests, `10/10` Python tests, ESLint-enabled build & lint `0` errors / `0` warnings).**
|
||||||
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
|
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -300,6 +300,21 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **
|
|||||||
- **`DEPLOY-BUG-05`**: Clarified Coolify `Base Directory: /` and `Docker Compose Location: /docker/docker-compose.yml` in `docs/users/installation.md` and `README.md`.
|
- **`DEPLOY-BUG-05`**: Clarified Coolify `Base Directory: /` and `Docker Compose Location: /docker/docker-compose.yml` in `docs/users/installation.md` and `README.md`.
|
||||||
- Verified: `85/85` full TS suite pass (`11/11` in `tests/phase07-deployment.test.ts`), `10/10` Python SDK tests pass, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings).
|
- Verified: `85/85` full TS suite pass (`11/11` in `tests/phase07-deployment.test.ts`), `10/10` Python SDK tests pass, `npm run build` clean, `npm run lint` (`0` errors, `0` warnings).
|
||||||
|
|
||||||
|
- **V2 Baseline Freeze & Governance Planning Complete (`BASELINE-2026-11-19-V2`)**:
|
||||||
|
- Froze the clean, Deployment-Hardened runtime state (`85/85` TS tests, `10/10` Python tests, clean `npm run build` & `eslint .`, zero runtime code changes) as **`BASELINE-2026-11-19-V2`**.
|
||||||
|
- Created raw-source verified V2 planning & contract documents in `notes/`:
|
||||||
|
- `notes/V2_FEATURE_GAP_AUDIT.md` (`V2-FEAT-01`–`V2-FEAT-07` gap analysis + full `/vcon/*` control mapping).
|
||||||
|
- `notes/V2_DATABASE_AND_SETTINGS_CONTRACT.md` (zero-loss additive PostgreSQL DDL rules preserving `admin_users` `id = 1 CHECK (id = 1)`, AES-256-GCM DB secret encryption + masked `GET /api/settings`, Port `3000`↔`4000` live sync).
|
||||||
|
- `notes/BASELINE_FREEZE.md`, `notes/PRODUCTION_READINESS_SCOPE_LOCK.md` (Doc `001` Scope Locks for `V2-Phase-01`–`04`), `notes/PRODUCTION_ROADMAP.md` (Doc `002`), `notes/PHASE_EXECUTION_LOG.md` (Doc `003`, `0 / 4` V2 phases done), `notes/ERROR_HANDLING_MATRIX.md` (Doc `004`, including `V2-E01`–`V2-E12`), `notes/ACTUAL_IMPLEMENTATION_STATUS.md`, and `notes/README.md`.
|
||||||
|
- Roadmap grouped into 4 Scope-Locked V2 Phases (`V2-Phase-01` Archival/Gzip + Quota/Disk Guard; `V2-Phase-02` Webhooks/Alerts + IP Allowlist/Expiry + Prometheus/NDJSON; `V2-Phase-03` Multi-Admin RBAC + Audit Log; `V2-Phase-04` Node.js/TypeScript SDK + `/vcon/sdk` guide).
|
||||||
|
- **`V2-Phase-01` Complete & Verified (`2026-11-19`)** (`V2-FEAT-01`, `V2-FEAT-03`, `V2-E01`–`V2-E05`):
|
||||||
|
- **AES-256-GCM Secret Vault (`core/Security/secretVault.ts`)**: Deterministic HKDF-SHA256 key derivation (`encryptSecret`, `decryptSecret`, `getMaskedHintFromEncrypted`, `SecretVaultError`) so `s3_secret_access_key_enc` is encrypted at rest in PostgreSQL and never leaked in plaintext by `GET /api/settings` (`hasS3SecretKey` + `s3SecretKeyMask: "••••xxxx"`).
|
||||||
|
- **S3 / Cloudflare R2 / MinIO SigV4 Client (`core/Storage/s3Archive.ts`)**: Pure Node `node:crypto` + `fetch` AWS Signature v4 client (`testS3Connection`, `uploadFileToS3`, `buildS3ObjectKey`, `normalizeS3EndpointUrl`) with `POST /api/storage/s3-test` and `POST /api/storage/archive`.
|
||||||
|
- **Transparent `.csv.gz` Auto-Compression & Safe Archive-Before-Prune (`core/Storage/csvStorage.ts`, `server/workers/csvWriter.js`)**: Cooperatively locked atomic `.csv` → `.csv.gz` compression (`compressOldPartitionsAsync` / `compressOldPartitions`) excluding today's active UTC partition; transparent `.csv` + `.csv.gz` reading/streaming across `GET /api/data` (surfacing `500 Corrupt .csv.gz partition` on corrupt gzip — `V2-E03`), `GET /api/events`, and `GET /api/export` (`application/gzip` for raw single `.csv.gz` download); fail-safe `beforeDeleteFile` S3 upload hook (`V2-E02`) that never deletes local partitions if S3 upload fails (`502 S3_ARCHIVE_FAILED`).
|
||||||
|
- **Per-Client Storage Quota (`storageQuotaMb`) & VPS Disk Pause Guard (`diskPauseThresholdPct`)**: Additive PostgreSQL columns (`client_sdks.storage_quota_mb` + 9 `admin_settings` columns in `core/Database/database.ts`, synced in `server/index.js` while preserving `ddlCount === 8`), `30s` TTL cached `statfs` + directory byte counters, and HTTP `507 Insufficient Storage` (`DISK_SPACE_LOW` / `STORAGE_QUOTA_EXCEEDED` — `V2-E04`, `V2-E05`) on `POST /api/events` and Port `4000` `sync_data`/`batch_sync` while keeping all Admin read/export/settings routes online.
|
||||||
|
- **Admin UI Controls (`SettingsWorkspace.tsx`, `NewClientModal.tsx`, `ClientsTable.tsx`, `lib/vconUiHelpers.ts`)**: Added live VPS disk usage bar, `Auto-compress (.csv → .csv.gz)`, `Disk pause threshold (0 or 50–99%)`, full `S3 / R2 off-site archive` configuration with `Test S3 Connection` & `Archive Now`, and per-client `Storage quota (MB, 0 = unlimited)` in modal and table.
|
||||||
|
- **Verified**: `89/89` TS tests (`4/4` in `tests/v2-phase01-storage-quotas.test.ts`), `10/10` Python SDK tests, `npm run build` clean, `npm run lint` (`0` errors / `0` warnings), and Chromium E2E across `/vcon/settings` (Storage tab) and `/vcon/clients` (`0` console errors).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Known Constraints & Operational Notes for Future Contributors
|
## 5. Known Constraints & Operational Notes for Future Contributors
|
||||||
|
|||||||
@@ -1,9 +1,21 @@
|
|||||||
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
||||||
import { SDKManager } from "@/core/sdk/sdkManager";
|
import { SDKManager } from "@/core/sdk/sdkManager";
|
||||||
|
import {
|
||||||
|
validateAllowedIpsInput,
|
||||||
|
validateClientExpiresAtInput,
|
||||||
|
} from "@/core/Security/ipAllowlist";
|
||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
const SESSION_COOKIE = "datastack_admin_session";
|
const SESSION_COOKIE = "datastack_admin_session";
|
||||||
const ALLOWED_PATCH_KEYS = new Set(["action", "name", "rateLimit", "status"]);
|
const ALLOWED_PATCH_KEYS = new Set([
|
||||||
|
"action",
|
||||||
|
"name",
|
||||||
|
"rateLimit",
|
||||||
|
"storageQuotaMb",
|
||||||
|
"allowedIps",
|
||||||
|
"expiresAt",
|
||||||
|
"status",
|
||||||
|
]);
|
||||||
|
|
||||||
export async function GET(
|
export async function GET(
|
||||||
req: NextRequest,
|
req: NextRequest,
|
||||||
@@ -125,7 +137,14 @@ export async function PATCH(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const updates: { name?: string; rateLimit?: number; status?: "active" | "revoked" } = {};
|
const updates: {
|
||||||
|
name?: string;
|
||||||
|
rateLimit?: number;
|
||||||
|
storageQuotaMb?: number;
|
||||||
|
allowedIps?: string;
|
||||||
|
expiresAt?: string | null;
|
||||||
|
status?: "active" | "revoked";
|
||||||
|
} = {};
|
||||||
if (body.name !== undefined) {
|
if (body.name !== undefined) {
|
||||||
if (
|
if (
|
||||||
typeof body.name !== "string" ||
|
typeof body.name !== "string" ||
|
||||||
@@ -153,6 +172,46 @@ export async function PATCH(
|
|||||||
}
|
}
|
||||||
updates.rateLimit = body.rateLimit;
|
updates.rateLimit = body.rateLimit;
|
||||||
}
|
}
|
||||||
|
if (body.storageQuotaMb !== undefined) {
|
||||||
|
if (
|
||||||
|
typeof body.storageQuotaMb !== "number" ||
|
||||||
|
!Number.isInteger(body.storageQuotaMb) ||
|
||||||
|
body.storageQuotaMb < 0 ||
|
||||||
|
body.storageQuotaMb > 1048576
|
||||||
|
) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
updates.storageQuotaMb = body.storageQuotaMb;
|
||||||
|
}
|
||||||
|
if (body.allowedIps !== undefined) {
|
||||||
|
if (typeof body.allowedIps !== "string") {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "allowedIps must be a string.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const ipValidation = validateAllowedIpsInput(body.allowedIps);
|
||||||
|
if (!ipValidation.valid) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
updates.allowedIps = ipValidation.normalized;
|
||||||
|
}
|
||||||
|
if (body.expiresAt !== undefined) {
|
||||||
|
const expValidation = validateClientExpiresAtInput(body.expiresAt);
|
||||||
|
if (!expValidation.valid) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
updates.expiresAt = expValidation.normalized;
|
||||||
|
}
|
||||||
if (body.status !== undefined) {
|
if (body.status !== undefined) {
|
||||||
if (body.status !== "active" && body.status !== "revoked") {
|
if (body.status !== "active" && body.status !== "revoked") {
|
||||||
return NextResponse.json(
|
return NextResponse.json(
|
||||||
|
|||||||
@@ -1,5 +1,9 @@
|
|||||||
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
||||||
import { SDKManager } from "@/core/sdk/sdkManager";
|
import { SDKManager } from "@/core/sdk/sdkManager";
|
||||||
|
import {
|
||||||
|
validateAllowedIpsInput,
|
||||||
|
validateClientExpiresAtInput,
|
||||||
|
} from "@/core/Security/ipAllowlist";
|
||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
const SESSION_COOKIE = "datastack_admin_session";
|
const SESSION_COOKIE = "datastack_admin_session";
|
||||||
@@ -58,7 +62,14 @@ export async function POST(req: NextRequest) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const { clientId, name, rateLimit } = body;
|
const {
|
||||||
|
clientId,
|
||||||
|
name,
|
||||||
|
rateLimit,
|
||||||
|
storageQuotaMb,
|
||||||
|
allowedIps,
|
||||||
|
expiresAt,
|
||||||
|
} = body;
|
||||||
|
|
||||||
if (typeof clientId !== "string" || typeof name !== "string" || !clientId.trim() || !name.trim()) {
|
if (typeof clientId !== "string" || typeof name !== "string" || !clientId.trim() || !name.trim()) {
|
||||||
return NextResponse.json(
|
return NextResponse.json(
|
||||||
@@ -83,6 +94,44 @@ export async function POST(req: NextRequest) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const parsedQuotaMb = storageQuotaMb === undefined ? 0 : Number(storageQuotaMb);
|
||||||
|
if (!Number.isInteger(parsedQuotaMb) || parsedQuotaMb < 0 || parsedQuotaMb > 1048576) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "storageQuotaMb must be an integer between 0 and 1048576.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let normalizedAllowedIps = "";
|
||||||
|
if (allowedIps !== undefined && allowedIps !== null) {
|
||||||
|
if (typeof allowedIps !== "string") {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "allowedIps must be a string.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const ipValidation = validateAllowedIpsInput(allowedIps);
|
||||||
|
if (!ipValidation.valid) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: ipValidation.error || "Invalid allowedIps.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
normalizedAllowedIps = ipValidation.normalized;
|
||||||
|
}
|
||||||
|
|
||||||
|
let normalizedExpiresAt: string | null = null;
|
||||||
|
if (expiresAt !== undefined) {
|
||||||
|
const expValidation = validateClientExpiresAtInput(expiresAt);
|
||||||
|
if (!expValidation.valid) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: expValidation.error || "Invalid expiresAt.", code: "VALIDATION_ERROR" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
normalizedExpiresAt = expValidation.normalized;
|
||||||
|
}
|
||||||
|
|
||||||
const existing = await SDKManager.getClientAsync(safeClientId);
|
const existing = await SDKManager.getClientAsync(safeClientId);
|
||||||
if (existing) {
|
if (existing) {
|
||||||
return NextResponse.json(
|
return NextResponse.json(
|
||||||
@@ -95,6 +144,9 @@ export async function POST(req: NextRequest) {
|
|||||||
clientId: safeClientId,
|
clientId: safeClientId,
|
||||||
name: name.trim().slice(0, 120),
|
name: name.trim().slice(0, 120),
|
||||||
rateLimit: parsedRateLimit,
|
rateLimit: parsedRateLimit,
|
||||||
|
storageQuotaMb: parsedQuotaMb,
|
||||||
|
allowedIps: normalizedAllowedIps,
|
||||||
|
expiresAt: normalizedExpiresAt,
|
||||||
});
|
});
|
||||||
|
|
||||||
return NextResponse.json({ client: newClient }, { status: 201 });
|
return NextResponse.json({ client: newClient }, { status: 201 });
|
||||||
|
|||||||
+12
-6
@@ -6,7 +6,6 @@ import { CSVStorageManager } from "@/core/Storage/csvStorage";
|
|||||||
import { requireAdminApiSession } from "@/lib/adminAuth";
|
import { requireAdminApiSession } from "@/lib/adminAuth";
|
||||||
import { parse } from "csv-parse";
|
import { parse } from "csv-parse";
|
||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
import { createReadStream } from "node:fs";
|
|
||||||
import { lstat, readdir } from "node:fs/promises";
|
import { lstat, readdir } from "node:fs/promises";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
|
|
||||||
@@ -135,10 +134,10 @@ export async function GET(request: NextRequest) {
|
|||||||
try {
|
try {
|
||||||
filenames = (await readdir(clientDir))
|
filenames = (await readdir(clientDir))
|
||||||
.filter((filename) => {
|
.filter((filename) => {
|
||||||
if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?\.csv$/.test(filename)) {
|
if (!/^\d{4}-\d{2}-\d{2}(?:-part\d+)?(?:-gz\d+)?\.csv(?:\.gz)?$/.test(filename)) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
const fileDate = filename.slice(0, 10);
|
const fileDate = CSVStorageManager.extractDateFromFilename(filename);
|
||||||
if ((from && fileDate < from) || (to && fileDate > to)) {
|
if ((from && fileDate < from) || (to && fileDate > to)) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -150,7 +149,7 @@ export async function GET(request: NextRequest) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for (const filename of filenames) {
|
for (const filename of filenames) {
|
||||||
const fileDate = filename.slice(0, 10);
|
const fileDate = CSVStorageManager.extractDateFromFilename(filename);
|
||||||
|
|
||||||
const filePath = path.join(clientDir, filename);
|
const filePath = path.join(clientDir, filename);
|
||||||
let fileLstat;
|
let fileLstat;
|
||||||
@@ -161,10 +160,13 @@ export async function GET(request: NextRequest) {
|
|||||||
}
|
}
|
||||||
if (fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue;
|
if (fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue;
|
||||||
|
|
||||||
const fileStream = createReadStream(filePath);
|
const fileStream = CSVStorageManager.openPartitionTextStream(filePath);
|
||||||
const parser = fileStream.pipe(
|
const parser = fileStream.pipe(
|
||||||
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
||||||
);
|
);
|
||||||
|
fileStream.on("error", (err) => {
|
||||||
|
if (!parser.destroyed) parser.destroy(err);
|
||||||
|
});
|
||||||
|
|
||||||
try {
|
try {
|
||||||
for await (const record of parser as AsyncIterable<
|
for await (const record of parser as AsyncIterable<
|
||||||
@@ -180,7 +182,11 @@ export async function GET(request: NextRequest) {
|
|||||||
if (event.eventId) addIfMatching(event);
|
if (event.eventId) addIfMatching(event);
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.warn(`[API /data] Failed reading CSV partition ${filePath}:`, err);
|
console.warn(`[API /data] Failed reading partition ${filePath}:`, err);
|
||||||
|
if (filename.endsWith(".csv.gz")) {
|
||||||
|
const detail = err instanceof Error ? err.message : String(err);
|
||||||
|
throw new Error(`Corrupt .csv.gz partition (${filename}): ${detail}`);
|
||||||
|
}
|
||||||
} finally {
|
} finally {
|
||||||
if (!parser.destroyed) parser.destroy();
|
if (!parser.destroyed) parser.destroy();
|
||||||
if (!fileStream.destroyed) fileStream.destroy();
|
if (!fileStream.destroyed) fileStream.destroy();
|
||||||
|
|||||||
+81
-4
@@ -3,6 +3,10 @@ import { verifyRequestAdminSession } from '@/core/auth/adminUsers';
|
|||||||
import { AuthService } from '@/core/auth/jwt';
|
import { AuthService } from '@/core/auth/jwt';
|
||||||
import { DataBufferEngine, REDIS_BUFFER_KEY } from '@/core/data/buffer';
|
import { DataBufferEngine, REDIS_BUFFER_KEY } from '@/core/data/buffer';
|
||||||
import { SDKManager } from '@/core/sdk/sdkManager';
|
import { SDKManager } from '@/core/sdk/sdkManager';
|
||||||
|
import {
|
||||||
|
isClientIpAllowed,
|
||||||
|
isClientTokenExpired,
|
||||||
|
} from '@/core/Security/ipAllowlist';
|
||||||
import {
|
import {
|
||||||
SecurityEngine,
|
SecurityEngine,
|
||||||
getInternalIngestionHeaders,
|
getInternalIngestionHeaders,
|
||||||
@@ -15,8 +19,7 @@ import {
|
|||||||
import { parse } from 'csv-parse';
|
import { parse } from 'csv-parse';
|
||||||
import { NextRequest, NextResponse } from 'next/server';
|
import { NextRequest, NextResponse } from 'next/server';
|
||||||
import crypto from 'node:crypto';
|
import crypto from 'node:crypto';
|
||||||
import { createReadStream } from 'node:fs';
|
import { extractClientIp, requireAdminApiSession } from '@/lib/adminAuth';
|
||||||
import { requireAdminApiSession } from '@/lib/adminAuth';
|
|
||||||
import type { BufferedBucket, BufferedSummary } from '@/lib/vconUiHelpers';
|
import type { BufferedBucket, BufferedSummary } from '@/lib/vconUiHelpers';
|
||||||
|
|
||||||
const SESSION_COOKIE = 'datastack_admin_session';
|
const SESSION_COOKIE = 'datastack_admin_session';
|
||||||
@@ -233,10 +236,13 @@ async function getRecentStoredEvents(
|
|||||||
|
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
const fileTail: DataEvent[] = [];
|
const fileTail: DataEvent[] = [];
|
||||||
const fileStream = createReadStream(file.filePath);
|
const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath);
|
||||||
const parser = fileStream.pipe(
|
const parser = fileStream.pipe(
|
||||||
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
||||||
);
|
);
|
||||||
|
fileStream.on('error', (err) => {
|
||||||
|
if (!parser.destroyed) parser.destroy(err);
|
||||||
|
});
|
||||||
|
|
||||||
try {
|
try {
|
||||||
for await (const record of parser as AsyncIterable<
|
for await (const record of parser as AsyncIterable<
|
||||||
@@ -254,7 +260,7 @@ async function getRecentStoredEvents(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.warn(`[API /events] Failed reading CSV partition ${file.filePath}:`, err);
|
console.warn(`[API /events] Failed reading partition ${file.filePath}:`, err);
|
||||||
} finally {
|
} finally {
|
||||||
if (!parser.destroyed) parser.destroy();
|
if (!parser.destroyed) parser.destroy();
|
||||||
if (!fileStream.destroyed) fileStream.destroy();
|
if (!fileStream.destroyed) fileStream.destroy();
|
||||||
@@ -476,6 +482,77 @@ export async function POST(req: NextRequest) {
|
|||||||
{ status: 401 },
|
{ status: 401 },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if (isClientTokenExpired(registeredClient.expiresAt)) {
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: 'CLIENT_TOKEN_EXPIRED',
|
||||||
|
severity: 'warning',
|
||||||
|
title: 'Expired Client Token Rejected',
|
||||||
|
message: `Client '${clientId}' attempted HTTP ingestion with an expired token (expiresAt=${registeredClient.expiresAt}).`,
|
||||||
|
clientId,
|
||||||
|
details: { expiresAt: registeredClient.expiresAt },
|
||||||
|
});
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
error: `CLIENT_TOKEN_EXPIRED: Client '${clientId}' token expired at ${registeredClient.expiresAt}. Rotate or extend client expiration.`,
|
||||||
|
code: 'CLIENT_TOKEN_EXPIRED',
|
||||||
|
},
|
||||||
|
{ status: 401 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (registeredClient.allowedIps && registeredClient.allowedIps.trim()) {
|
||||||
|
const clientIp = extractClientIp(req);
|
||||||
|
if (!isClientIpAllowed(clientIp, registeredClient.allowedIps)) {
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: 'CLIENT_IP_FORBIDDEN',
|
||||||
|
severity: 'warning',
|
||||||
|
title: 'Unauthorized Client IP Blocked',
|
||||||
|
message: `Client '${clientId}' request from IP '${clientIp}' was blocked by allowedIps policy.`,
|
||||||
|
clientId,
|
||||||
|
details: {
|
||||||
|
clientIp,
|
||||||
|
allowedIps: registeredClient.allowedIps,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
error: `CLIENT_IP_FORBIDDEN: Request IP '${clientIp}' is not authorized for client '${clientId}'.`,
|
||||||
|
code: 'CLIENT_IP_FORBIDDEN',
|
||||||
|
},
|
||||||
|
{ status: 403 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const storageAdmission = await CSVStorageManager.checkStorageAdmissionAsync(
|
||||||
|
clientId,
|
||||||
|
registeredClient.storageQuotaMb,
|
||||||
|
settings.diskPauseThresholdPct,
|
||||||
|
);
|
||||||
|
if (!storageAdmission.allowed) {
|
||||||
|
if (storageAdmission.code) {
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: storageAdmission.code,
|
||||||
|
severity:
|
||||||
|
storageAdmission.code === 'DISK_SPACE_LOW' ? 'critical' : 'warning',
|
||||||
|
title:
|
||||||
|
storageAdmission.code === 'DISK_SPACE_LOW'
|
||||||
|
? 'VPS Disk Pause Threshold Reached'
|
||||||
|
: 'Client Storage Quota Exceeded',
|
||||||
|
message:
|
||||||
|
storageAdmission.reason ||
|
||||||
|
`Storage admission paused (${storageAdmission.code}) for client '${clientId}'.`,
|
||||||
|
clientId,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
error: storageAdmission.reason,
|
||||||
|
code: storageAdmission.code,
|
||||||
|
},
|
||||||
|
{ status: 507 },
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const effectiveClientRate =
|
const effectiveClientRate =
|
||||||
|
|||||||
+44
-16
@@ -12,7 +12,7 @@ import { NextRequest, NextResponse } from "next/server";
|
|||||||
import { createReadStream } from "node:fs";
|
import { createReadStream } from "node:fs";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
|
|
||||||
type ExportFormat = "csv" | "json";
|
type ExportFormat = "csv" | "json" | "ndjson";
|
||||||
type CsvRecord = Record<string, string>;
|
type CsvRecord = Record<string, string>;
|
||||||
type StreamChunk = string | Uint8Array;
|
type StreamChunk = string | Uint8Array;
|
||||||
|
|
||||||
@@ -46,10 +46,13 @@ function toByteStream(chunks: AsyncIterable<StreamChunk>) {
|
|||||||
async function* readFileRecords(
|
async function* readFileRecords(
|
||||||
file: StorageFileMeta,
|
file: StorageFileMeta,
|
||||||
): AsyncGenerator<CsvRecord> {
|
): AsyncGenerator<CsvRecord> {
|
||||||
const fileStream = createReadStream(file.filePath);
|
const fileStream = CSVStorageManager.openPartitionTextStream(file.filePath);
|
||||||
const parser = fileStream.pipe(
|
const parser = fileStream.pipe(
|
||||||
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
parse({ columns: true, skip_empty_lines: true, bom: true }),
|
||||||
);
|
);
|
||||||
|
fileStream.on("error", (err) => {
|
||||||
|
if (!parser.destroyed) parser.destroy(err);
|
||||||
|
});
|
||||||
|
|
||||||
try {
|
try {
|
||||||
for await (const record of parser as AsyncIterable<CsvRecord>) {
|
for await (const record of parser as AsyncIterable<CsvRecord>) {
|
||||||
@@ -72,13 +75,13 @@ function makeExportStream(
|
|||||||
|
|
||||||
if (format === "json") {
|
if (format === "json") {
|
||||||
yield "[";
|
yield "[";
|
||||||
} else if (includeHeader) {
|
} else if (format === "csv" && includeHeader) {
|
||||||
yield `${fields.join(",")}\n`;
|
yield `${fields.join(",")}\n`;
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
for await (const record of readFileRecords(file)) {
|
for await (const record of readFileRecords(file)) {
|
||||||
if (format === "json") {
|
if (format === "json" || format === "ndjson") {
|
||||||
const payload = record.payload ?? "";
|
const payload = record.payload ?? "";
|
||||||
let parsedPayload: unknown = payload;
|
let parsedPayload: unknown = payload;
|
||||||
try {
|
try {
|
||||||
@@ -86,7 +89,15 @@ function makeExportStream(
|
|||||||
} catch {
|
} catch {
|
||||||
// Keep non-JSON payloads as strings.
|
// Keep non-JSON payloads as strings.
|
||||||
}
|
}
|
||||||
yield `${isFirst ? "" : ","}${JSON.stringify({ ...record, payload: parsedPayload })}`;
|
const serialized = JSON.stringify({
|
||||||
|
...record,
|
||||||
|
payload: parsedPayload,
|
||||||
|
});
|
||||||
|
if (format === "ndjson") {
|
||||||
|
yield `${serialized}\n`;
|
||||||
|
} else {
|
||||||
|
yield `${isFirst ? "" : ","}${serialized}`;
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
const line = fields
|
const line = fields
|
||||||
.map((field) => `"${String(record[field] ?? "").replaceAll('"', '""')}"`)
|
.map((field) => `"${String(record[field] ?? "").replaceAll('"', '""')}"`)
|
||||||
@@ -224,9 +235,14 @@ export async function GET(request: NextRequest) {
|
|||||||
: SDKManager.normalizeClientId(rawClientId) || "all";
|
: SDKManager.normalizeClientId(rawClientId) || "all";
|
||||||
const sanitizedClientId =
|
const sanitizedClientId =
|
||||||
clientId === "all" ? "all" : CSVStorageManager.sanitizeClientId(clientId);
|
clientId === "all" ? "all" : CSVStorageManager.sanitizeClientId(clientId);
|
||||||
const filename = searchParams.get("filename");
|
const filename = searchParams.get("filename") || searchParams.get("file");
|
||||||
const rawFormat = searchParams.get("format");
|
const rawFormat = (searchParams.get("format") || "csv").toLowerCase();
|
||||||
const format: ExportFormat = rawFormat === "json" ? "json" : "csv";
|
const format: ExportFormat =
|
||||||
|
rawFormat === "json"
|
||||||
|
? "json"
|
||||||
|
: rawFormat === "ndjson"
|
||||||
|
? "ndjson"
|
||||||
|
: "csv";
|
||||||
const includeHeader = searchParams.get("includeHeader") !== "false";
|
const includeHeader = searchParams.get("includeHeader") !== "false";
|
||||||
|
|
||||||
let files = await CSVStorageManager.listAllFilesAsync();
|
let files = await CSVStorageManager.listAllFilesAsync();
|
||||||
@@ -259,21 +275,31 @@ export async function GET(request: NextRequest) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
const file = matches[0];
|
const file = matches[0];
|
||||||
|
const baseStem = file.filename.replace(/\.csv(?:\.gz)?$/i, "");
|
||||||
const downloadName =
|
const downloadName =
|
||||||
format === "json"
|
format === "json"
|
||||||
? `${file.filename.replace(/\.csv$/i, "")}.json`
|
? `${baseStem}.json`
|
||||||
: file.filename;
|
: format === "ndjson"
|
||||||
|
? `${baseStem}.ndjson`
|
||||||
|
: !includeHeader && file.compressed
|
||||||
|
? `${baseStem}.csv`
|
||||||
|
: file.filename;
|
||||||
const bodyStream =
|
const bodyStream =
|
||||||
format === "json" || !includeHeader
|
format === "json" || format === "ndjson" || !includeHeader
|
||||||
? makeExportStream([file], format, includeHeader)
|
? makeExportStream([file], format, includeHeader)
|
||||||
: streamFile(file);
|
: streamFile(file);
|
||||||
|
const contentType =
|
||||||
|
format === "json"
|
||||||
|
? "application/json; charset=utf-8"
|
||||||
|
: format === "ndjson"
|
||||||
|
? "application/x-ndjson; charset=utf-8"
|
||||||
|
: file.compressed && includeHeader
|
||||||
|
? "application/gzip"
|
||||||
|
: "text/csv; charset=utf-8";
|
||||||
return new NextResponse(bodyStream, {
|
return new NextResponse(bodyStream, {
|
||||||
status: 200,
|
status: 200,
|
||||||
headers: {
|
headers: {
|
||||||
"Content-Type":
|
"Content-Type": contentType,
|
||||||
format === "json"
|
|
||||||
? "application/json; charset=utf-8"
|
|
||||||
: "text/csv; charset=utf-8",
|
|
||||||
"Content-Disposition": `attachment; filename="${downloadName.replace(/"/g, "")}"`,
|
"Content-Disposition": `attachment; filename="${downloadName.replace(/"/g, "")}"`,
|
||||||
"Cache-Control": "no-store",
|
"Cache-Control": "no-store",
|
||||||
},
|
},
|
||||||
@@ -294,7 +320,9 @@ export async function GET(request: NextRequest) {
|
|||||||
"Content-Type":
|
"Content-Type":
|
||||||
format === "json"
|
format === "json"
|
||||||
? "application/json; charset=utf-8"
|
? "application/json; charset=utf-8"
|
||||||
: "text/csv; charset=utf-8",
|
: format === "ndjson"
|
||||||
|
? "application/x-ndjson; charset=utf-8"
|
||||||
|
: "text/csv; charset=utf-8",
|
||||||
"Content-Disposition": `attachment; filename="${baseName}_${datePart}.${format}"`,
|
"Content-Disposition": `attachment; filename="${baseName}_${datePart}.${format}"`,
|
||||||
"Cache-Control": "no-store",
|
"Cache-Control": "no-store",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -0,0 +1,207 @@
|
|||||||
|
import { SettingsManager } from "@/core/adminSettings/settingsManager";
|
||||||
|
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
||||||
|
import { DataBufferEngine } from "@/core/data/buffer";
|
||||||
|
import { SDKManager } from "@/core/sdk/sdkManager";
|
||||||
|
import { getInternalIngestionHeaders } from "@/core/Security/security";
|
||||||
|
import { CSVStorageManager } from "@/core/Storage/csvStorage";
|
||||||
|
import { ADMIN_SESSION_COOKIE } from "@/lib/adminAuth";
|
||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { createHash, timingSafeEqual } from "node:crypto";
|
||||||
|
|
||||||
|
function verifyBearerTokenConstantTime(
|
||||||
|
expectedToken: string,
|
||||||
|
providedToken: string,
|
||||||
|
): boolean {
|
||||||
|
const expected = expectedToken.trim();
|
||||||
|
const provided = providedToken.trim();
|
||||||
|
if (!expected || !provided) return false;
|
||||||
|
const expectedDigest = createHash("sha256")
|
||||||
|
.update(expected, "utf8")
|
||||||
|
.digest();
|
||||||
|
const providedDigest = createHash("sha256")
|
||||||
|
.update(provided, "utf8")
|
||||||
|
.digest();
|
||||||
|
return timingSafeEqual(expectedDigest, providedDigest);
|
||||||
|
}
|
||||||
|
|
||||||
|
function escapePrometheusLabel(raw: string): string {
|
||||||
|
return String(raw || "")
|
||||||
|
.replace(/\\/g, "\\\\")
|
||||||
|
.replace(/\n/g, "\\n")
|
||||||
|
.replace(/"/g, '\\"');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function GET(req: NextRequest) {
|
||||||
|
try {
|
||||||
|
let authorized = false;
|
||||||
|
|
||||||
|
// 1. Check Bearer token (or ?token= query param) against encrypted Prometheus token
|
||||||
|
const authHeader = req.headers.get("authorization") || "";
|
||||||
|
const bearerCandidate = authHeader.toLowerCase().startsWith("bearer ")
|
||||||
|
? authHeader.slice(7).trim()
|
||||||
|
: req.nextUrl.searchParams.get("token")?.trim() || "";
|
||||||
|
|
||||||
|
const promConfig = await SettingsManager.getDecryptedPrometheusConfigAsync();
|
||||||
|
if (
|
||||||
|
promConfig.enabled &&
|
||||||
|
promConfig.token.trim().length > 0 &&
|
||||||
|
bearerCandidate.length > 0 &&
|
||||||
|
verifyBearerTokenConstantTime(promConfig.token, bearerCandidate)
|
||||||
|
) {
|
||||||
|
authorized = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Fallback: valid Admin Console session cookie
|
||||||
|
if (!authorized) {
|
||||||
|
const sessionCookie = req.cookies.get(ADMIN_SESSION_COOKIE)?.value;
|
||||||
|
if (sessionCookie) {
|
||||||
|
const adminAuth = await verifyRequestAdminSession(sessionCookie);
|
||||||
|
if (adminAuth.valid) {
|
||||||
|
authorized = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!authorized) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
error:
|
||||||
|
"PROMETHEUS_UNAUTHORIZED: Valid bearer scrape token or admin session required.",
|
||||||
|
code: "PROMETHEUS_UNAUTHORIZED",
|
||||||
|
},
|
||||||
|
{ status: 401 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const [clients, storageStats, diskHealth, bufferStats] = await Promise.all([
|
||||||
|
SDKManager.listClientsAsync(),
|
||||||
|
CSVStorageManager.getStatsAsync(),
|
||||||
|
CSVStorageManager.getDiskSpaceHealthAsync(),
|
||||||
|
DataBufferEngine.getBufferStatsAsync(),
|
||||||
|
]);
|
||||||
|
|
||||||
|
let activeSockets = 0;
|
||||||
|
let bridgeUp = 0;
|
||||||
|
let remoteBufferSize = 0;
|
||||||
|
|
||||||
|
const headers = getInternalIngestionHeaders();
|
||||||
|
if (headers["x-datastack-internal-secret"]) {
|
||||||
|
const ingestionBaseUrl = (
|
||||||
|
process.env.INGESTION_INTERNAL_URL || "http://127.0.0.1:4000"
|
||||||
|
).replace(/\/$/, "");
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${ingestionBaseUrl}/internal/metrics`, {
|
||||||
|
headers,
|
||||||
|
cache: "no-store",
|
||||||
|
signal: AbortSignal.timeout(1000),
|
||||||
|
});
|
||||||
|
if (res.ok) {
|
||||||
|
bridgeUp = 1;
|
||||||
|
const data = (await res.json()) as {
|
||||||
|
activeSockets?: number;
|
||||||
|
bufferSize?: number;
|
||||||
|
};
|
||||||
|
activeSockets = Math.max(0, Number(data.activeSockets) || 0);
|
||||||
|
remoteBufferSize = Math.max(0, Number(data.bufferSize) || 0);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
bridgeUp = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const totalBufferedEvents =
|
||||||
|
bufferStats.localBufferSize +
|
||||||
|
(bufferStats.redisEnabled && bufferStats.redisConnected
|
||||||
|
? bufferStats.redisBufferSize
|
||||||
|
: remoteBufferSize);
|
||||||
|
|
||||||
|
let activeClientsCount = 0;
|
||||||
|
let revokedClientsCount = 0;
|
||||||
|
for (const c of clients) {
|
||||||
|
if (c.status === "revoked") {
|
||||||
|
revokedClientsCount += 1;
|
||||||
|
} else {
|
||||||
|
activeClientsCount += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const lines: string[] = [
|
||||||
|
"# HELP datastack_up Whether the DataStack control plane is up (1 = healthy).",
|
||||||
|
"# TYPE datastack_up gauge",
|
||||||
|
"datastack_up 1",
|
||||||
|
"# HELP datastack_ingestion_bridge_up Whether the Port 4000 Socket.IO ingestion engine is reachable (1 = reachable, 0 = unreachable).",
|
||||||
|
"# TYPE datastack_ingestion_bridge_up gauge",
|
||||||
|
`datastack_ingestion_bridge_up ${bridgeUp}`,
|
||||||
|
"# HELP datastack_active_sockets Currently connected Socket.IO SDK clients on Port 4000.",
|
||||||
|
"# TYPE datastack_active_sockets gauge",
|
||||||
|
`datastack_active_sockets ${activeSockets}`,
|
||||||
|
"# HELP datastack_buffer_events_total Total events currently queued in memory or Redis awaiting CSV flush.",
|
||||||
|
"# TYPE datastack_buffer_events_total gauge",
|
||||||
|
`datastack_buffer_events_total ${totalBufferedEvents}`,
|
||||||
|
"# HELP datastack_storage_files_total Total CSV and gzipped CSV partition files on disk.",
|
||||||
|
"# TYPE datastack_storage_files_total gauge",
|
||||||
|
`datastack_storage_files_total ${storageStats.totalFiles}`,
|
||||||
|
"# HELP datastack_storage_bytes_total Total bytes occupied by CSV and gzipped CSV partition files on disk.",
|
||||||
|
"# TYPE datastack_storage_bytes_total gauge",
|
||||||
|
`datastack_storage_bytes_total ${storageStats.totalSizeBytes}`,
|
||||||
|
"# HELP datastack_storage_rows_total Total telemetry rows persisted across all partition files.",
|
||||||
|
"# TYPE datastack_storage_rows_total gauge",
|
||||||
|
`datastack_storage_rows_total ${storageStats.totalRows}`,
|
||||||
|
"# HELP datastack_disk_used_percent Percentage of underlying filesystem capacity currently used.",
|
||||||
|
"# TYPE datastack_disk_used_percent gauge",
|
||||||
|
`datastack_disk_used_percent ${diskHealth.usedPct}`,
|
||||||
|
"# HELP datastack_disk_free_bytes Free bytes available on the underlying storage volume.",
|
||||||
|
"# TYPE datastack_disk_free_bytes gauge",
|
||||||
|
`datastack_disk_free_bytes ${diskHealth.freeBytes}`,
|
||||||
|
"# HELP datastack_disk_total_bytes Total capacity in bytes of the underlying storage volume.",
|
||||||
|
"# TYPE datastack_disk_total_bytes gauge",
|
||||||
|
`datastack_disk_total_bytes ${diskHealth.totalBytes}`,
|
||||||
|
"# HELP datastack_clients_total Number of registered SDK clients partitioned by status.",
|
||||||
|
"# TYPE datastack_clients_total gauge",
|
||||||
|
`datastack_clients_total{status="active"} ${activeClientsCount}`,
|
||||||
|
`datastack_clients_total{status="revoked"} ${revokedClientsCount}`,
|
||||||
|
"# HELP datastack_client_events_total Cumulative ingested events per registered client.",
|
||||||
|
"# TYPE datastack_client_events_total counter",
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const c of clients) {
|
||||||
|
const safeId = escapePrometheusLabel(c.clientId);
|
||||||
|
lines.push(
|
||||||
|
`datastack_client_events_total{client_id="${safeId}"} ${Math.max(0, Number(c.totalEvents) || 0)}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
lines.push(
|
||||||
|
"# HELP datastack_client_storage_bytes Disk usage in bytes per registered client.",
|
||||||
|
"# TYPE datastack_client_storage_bytes gauge",
|
||||||
|
);
|
||||||
|
|
||||||
|
const perClientBytes = new Map<string, number>();
|
||||||
|
for (const f of storageStats.files) {
|
||||||
|
perClientBytes.set(
|
||||||
|
f.clientId,
|
||||||
|
(perClientBytes.get(f.clientId) || 0) + f.sizeBytes,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for (const c of clients) {
|
||||||
|
const safeId = escapePrometheusLabel(c.clientId);
|
||||||
|
const usedBytes = perClientBytes.get(c.clientId) || 0;
|
||||||
|
lines.push(
|
||||||
|
`datastack_client_storage_bytes{client_id="${safeId}"} ${usedBytes}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const body = `${lines.join("\n")}\n`;
|
||||||
|
return new NextResponse(body, {
|
||||||
|
status: 200,
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "text/plain; version=0.0.4; charset=utf-8",
|
||||||
|
"Cache-Control": "no-store, no-cache, must-revalidate",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
} catch (err: unknown) {
|
||||||
|
const message =
|
||||||
|
err instanceof Error ? err.message : "Failed to generate Prometheus metrics";
|
||||||
|
return NextResponse.json({ error: message }, { status: 500 });
|
||||||
|
}
|
||||||
|
}
|
||||||
+190
-3
@@ -1,8 +1,9 @@
|
|||||||
import {
|
import {
|
||||||
SettingsManager,
|
SettingsManager,
|
||||||
type AdminSettings,
|
type AdminSettingsUpdateInput,
|
||||||
} from "@/core/adminSettings/settingsManager";
|
} from "@/core/adminSettings/settingsManager";
|
||||||
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
import { verifyRequestAdminSession } from "@/core/auth/adminUsers";
|
||||||
|
import { SecretVaultError } from "@/core/Security/secretVault";
|
||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
const SESSION_COOKIE = "datastack_admin_session";
|
const SESSION_COOKIE = "datastack_admin_session";
|
||||||
@@ -16,6 +17,24 @@ const ALLOWED_SETTING_KEYS = new Set([
|
|||||||
"enableIdempotency",
|
"enableIdempotency",
|
||||||
"autoPruneDays",
|
"autoPruneDays",
|
||||||
"jwtSecret",
|
"jwtSecret",
|
||||||
|
"compressAfterDays",
|
||||||
|
"enableS3Backup",
|
||||||
|
"s3Endpoint",
|
||||||
|
"s3Region",
|
||||||
|
"s3Bucket",
|
||||||
|
"s3Prefix",
|
||||||
|
"s3AccessKeyId",
|
||||||
|
"s3SecretAccessKey",
|
||||||
|
"clearS3SecretKey",
|
||||||
|
"diskPauseThresholdPct",
|
||||||
|
"enableWebhookAlerts",
|
||||||
|
"webhookUrl",
|
||||||
|
"webhookSecret",
|
||||||
|
"clearWebhookSecret",
|
||||||
|
"webhookCooldownSec",
|
||||||
|
"enablePrometheusMetrics",
|
||||||
|
"prometheusMetricsToken",
|
||||||
|
"clearPrometheusToken",
|
||||||
]);
|
]);
|
||||||
|
|
||||||
function parseBoundedInt(
|
function parseBoundedInt(
|
||||||
@@ -33,6 +52,21 @@ function parseBoundedInt(
|
|||||||
return raw;
|
return raw;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function parseOptionalString(
|
||||||
|
raw: unknown,
|
||||||
|
field: string,
|
||||||
|
maxLength: number,
|
||||||
|
): string {
|
||||||
|
if (typeof raw !== "string") {
|
||||||
|
throw new Error(`${field} must be a string.`);
|
||||||
|
}
|
||||||
|
const trimmed = raw.trim();
|
||||||
|
if (trimmed.length > maxLength) {
|
||||||
|
throw new Error(`${field} must be at most ${maxLength} characters.`);
|
||||||
|
}
|
||||||
|
return trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
export async function GET(req: NextRequest) {
|
export async function GET(req: NextRequest) {
|
||||||
try {
|
try {
|
||||||
const auth = await verifyRequestAdminSession(
|
const auth = await verifyRequestAdminSession(
|
||||||
@@ -109,7 +143,7 @@ export async function POST(req: NextRequest) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const validated: Partial<AdminSettings> = {};
|
const validated: AdminSettingsUpdateInput = {};
|
||||||
try {
|
try {
|
||||||
if (body.maxEventsPerSec !== undefined) {
|
if (body.maxEventsPerSec !== undefined) {
|
||||||
validated.maxEventsPerSec = parseBoundedInt(
|
validated.maxEventsPerSec = parseBoundedInt(
|
||||||
@@ -159,14 +193,161 @@ export async function POST(req: NextRequest) {
|
|||||||
3650,
|
3650,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if (body.compressAfterDays !== undefined) {
|
||||||
|
validated.compressAfterDays = parseBoundedInt(
|
||||||
|
body.compressAfterDays,
|
||||||
|
"compressAfterDays",
|
||||||
|
0,
|
||||||
|
3650,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.diskPauseThresholdPct !== undefined) {
|
||||||
|
const pct = parseBoundedInt(
|
||||||
|
body.diskPauseThresholdPct,
|
||||||
|
"diskPauseThresholdPct",
|
||||||
|
0,
|
||||||
|
99,
|
||||||
|
);
|
||||||
|
if (pct !== 0 && pct < 50) {
|
||||||
|
throw new Error(
|
||||||
|
"diskPauseThresholdPct must be 0 (disabled) or between 50 and 99.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
validated.diskPauseThresholdPct = pct;
|
||||||
|
}
|
||||||
if (body.enableIdempotency !== undefined) {
|
if (body.enableIdempotency !== undefined) {
|
||||||
if (typeof body.enableIdempotency !== "boolean") {
|
if (typeof body.enableIdempotency !== "boolean") {
|
||||||
throw new Error("enableIdempotency must be a boolean.");
|
throw new Error("enableIdempotency must be a boolean.");
|
||||||
}
|
}
|
||||||
validated.enableIdempotency = body.enableIdempotency;
|
validated.enableIdempotency = body.enableIdempotency;
|
||||||
}
|
}
|
||||||
|
if (body.enableS3Backup !== undefined) {
|
||||||
|
if (typeof body.enableS3Backup !== "boolean") {
|
||||||
|
throw new Error("enableS3Backup must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.enableS3Backup = body.enableS3Backup;
|
||||||
|
}
|
||||||
|
if (body.s3Endpoint !== undefined) {
|
||||||
|
const ep = parseOptionalString(body.s3Endpoint, "s3Endpoint", 512);
|
||||||
|
if (ep) {
|
||||||
|
let parsedEp: URL;
|
||||||
|
try {
|
||||||
|
parsedEp = new URL(
|
||||||
|
/^https?:\/\//i.test(ep) ? ep : `https://${ep}`,
|
||||||
|
);
|
||||||
|
} catch {
|
||||||
|
throw new Error("s3Endpoint must be a valid http(s) URL.");
|
||||||
|
}
|
||||||
|
if (parsedEp.protocol !== "http:" && parsedEp.protocol !== "https:") {
|
||||||
|
throw new Error("s3Endpoint must use http:// or https://.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
validated.s3Endpoint = ep;
|
||||||
|
}
|
||||||
|
if (body.s3Region !== undefined) {
|
||||||
|
validated.s3Region =
|
||||||
|
parseOptionalString(body.s3Region, "s3Region", 64) || "auto";
|
||||||
|
}
|
||||||
|
if (body.s3Bucket !== undefined) {
|
||||||
|
validated.s3Bucket = parseOptionalString(
|
||||||
|
body.s3Bucket,
|
||||||
|
"s3Bucket",
|
||||||
|
128,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.s3Prefix !== undefined) {
|
||||||
|
validated.s3Prefix = parseOptionalString(
|
||||||
|
body.s3Prefix,
|
||||||
|
"s3Prefix",
|
||||||
|
256,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.s3AccessKeyId !== undefined) {
|
||||||
|
validated.s3AccessKeyId = parseOptionalString(
|
||||||
|
body.s3AccessKeyId,
|
||||||
|
"s3AccessKeyId",
|
||||||
|
256,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.s3SecretAccessKey !== undefined) {
|
||||||
|
validated.s3SecretAccessKey = parseOptionalString(
|
||||||
|
body.s3SecretAccessKey,
|
||||||
|
"s3SecretAccessKey",
|
||||||
|
512,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.clearS3SecretKey !== undefined) {
|
||||||
|
if (typeof body.clearS3SecretKey !== "boolean") {
|
||||||
|
throw new Error("clearS3SecretKey must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.clearS3SecretKey = body.clearS3SecretKey;
|
||||||
|
}
|
||||||
|
if (body.enableWebhookAlerts !== undefined) {
|
||||||
|
if (typeof body.enableWebhookAlerts !== "boolean") {
|
||||||
|
throw new Error("enableWebhookAlerts must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.enableWebhookAlerts = body.enableWebhookAlerts;
|
||||||
|
}
|
||||||
|
if (body.webhookUrl !== undefined) {
|
||||||
|
const wh = parseOptionalString(body.webhookUrl, "webhookUrl", 1024);
|
||||||
|
if (wh) {
|
||||||
|
let parsedWh: URL;
|
||||||
|
try {
|
||||||
|
parsedWh = new URL(wh);
|
||||||
|
} catch {
|
||||||
|
throw new Error("webhookUrl must be a valid http(s) URL.");
|
||||||
|
}
|
||||||
|
if (parsedWh.protocol !== "http:" && parsedWh.protocol !== "https:") {
|
||||||
|
throw new Error("webhookUrl must use http:// or https://.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
validated.webhookUrl = wh;
|
||||||
|
}
|
||||||
|
if (body.webhookSecret !== undefined) {
|
||||||
|
validated.webhookSecret = parseOptionalString(
|
||||||
|
body.webhookSecret,
|
||||||
|
"webhookSecret",
|
||||||
|
512,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.clearWebhookSecret !== undefined) {
|
||||||
|
if (typeof body.clearWebhookSecret !== "boolean") {
|
||||||
|
throw new Error("clearWebhookSecret must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.clearWebhookSecret = body.clearWebhookSecret;
|
||||||
|
}
|
||||||
|
if (body.webhookCooldownSec !== undefined) {
|
||||||
|
validated.webhookCooldownSec = parseBoundedInt(
|
||||||
|
body.webhookCooldownSec,
|
||||||
|
"webhookCooldownSec",
|
||||||
|
10,
|
||||||
|
86400,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.enablePrometheusMetrics !== undefined) {
|
||||||
|
if (typeof body.enablePrometheusMetrics !== "boolean") {
|
||||||
|
throw new Error("enablePrometheusMetrics must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.enablePrometheusMetrics = body.enablePrometheusMetrics;
|
||||||
|
}
|
||||||
|
if (body.prometheusMetricsToken !== undefined) {
|
||||||
|
validated.prometheusMetricsToken = parseOptionalString(
|
||||||
|
body.prometheusMetricsToken,
|
||||||
|
"prometheusMetricsToken",
|
||||||
|
512,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (body.clearPrometheusToken !== undefined) {
|
||||||
|
if (typeof body.clearPrometheusToken !== "boolean") {
|
||||||
|
throw new Error("clearPrometheusToken must be a boolean.");
|
||||||
|
}
|
||||||
|
validated.clearPrometheusToken = body.clearPrometheusToken;
|
||||||
|
}
|
||||||
if (body.jwtSecret !== undefined) {
|
if (body.jwtSecret !== undefined) {
|
||||||
if (typeof body.jwtSecret !== "string" || body.jwtSecret.trim().length < 32) {
|
if (
|
||||||
|
typeof body.jwtSecret !== "string" ||
|
||||||
|
body.jwtSecret.trim().length < 32
|
||||||
|
) {
|
||||||
throw new Error(
|
throw new Error(
|
||||||
"jwtSecret must be a string with at least 32 characters.",
|
"jwtSecret must be a string with at least 32 characters.",
|
||||||
);
|
);
|
||||||
@@ -208,6 +389,12 @@ export async function POST(req: NextRequest) {
|
|||||||
: {}),
|
: {}),
|
||||||
});
|
});
|
||||||
} catch (err: unknown) {
|
} catch (err: unknown) {
|
||||||
|
if (err instanceof SecretVaultError) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: err.message, code: err.code },
|
||||||
|
{ status: 500 },
|
||||||
|
);
|
||||||
|
}
|
||||||
const message =
|
const message =
|
||||||
err instanceof Error ? err.message : "Settings update failed";
|
err instanceof Error ? err.message : "Settings update failed";
|
||||||
return NextResponse.json({ error: message }, { status: 500 });
|
return NextResponse.json({ error: message }, { status: 500 });
|
||||||
|
|||||||
@@ -0,0 +1,119 @@
|
|||||||
|
import { SettingsManager } from "@/core/adminSettings/settingsManager";
|
||||||
|
import { dispatchWebhookAlertAsync } from "@/core/alerts/webhookNotifier";
|
||||||
|
import { SecretVaultError } from "@/core/Security/secretVault";
|
||||||
|
import { requireAdminApiSession } from "@/lib/adminAuth";
|
||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest) {
|
||||||
|
const auth = await requireAdminApiSession(req);
|
||||||
|
if (!auth.valid) {
|
||||||
|
return auth.response;
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: Record<string, unknown> = {};
|
||||||
|
try {
|
||||||
|
const text = await req.text();
|
||||||
|
if (text.trim()) {
|
||||||
|
const parsed = JSON.parse(text);
|
||||||
|
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
||||||
|
body = parsed as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const savedConfig = await SettingsManager.getDecryptedWebhookConfigAsync();
|
||||||
|
const webhookUrl =
|
||||||
|
typeof body.webhookUrl === "string"
|
||||||
|
? body.webhookUrl.trim()
|
||||||
|
: savedConfig.webhookUrl.trim();
|
||||||
|
const clearWebhookSecret = body.clearWebhookSecret === true;
|
||||||
|
const webhookSecret = clearWebhookSecret
|
||||||
|
? ""
|
||||||
|
: typeof body.webhookSecret === "string" && body.webhookSecret.trim()
|
||||||
|
? body.webhookSecret.trim()
|
||||||
|
: savedConfig.webhookSecret;
|
||||||
|
|
||||||
|
if (!webhookUrl) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "VALIDATION_ERROR",
|
||||||
|
error: "Webhook URL is required to send a test alert.",
|
||||||
|
},
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const parsedUrl = new URL(webhookUrl);
|
||||||
|
if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") {
|
||||||
|
throw new Error("Webhook URL must use http:// or https://.");
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "VALIDATION_ERROR",
|
||||||
|
error: "Webhook URL must be a valid http:// or https:// endpoint.",
|
||||||
|
},
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const result = await dispatchWebhookAlertAsync(
|
||||||
|
{
|
||||||
|
event: "TEST_ALERT",
|
||||||
|
severity: "info",
|
||||||
|
title: "Webhook Verification",
|
||||||
|
message: "Test alert dispatched from DataStack Admin Console.",
|
||||||
|
details: {
|
||||||
|
triggeredBy: auth.auth.user?.email || "admin",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
enabled: true,
|
||||||
|
webhookUrl,
|
||||||
|
webhookSecret,
|
||||||
|
cooldownSec: savedConfig.cooldownSec,
|
||||||
|
},
|
||||||
|
{ bypassCooldown: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!result.ok) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "WEBHOOK_DELIVERY_FAILED",
|
||||||
|
error: result.error || "Webhook target failed to accept test alert.",
|
||||||
|
status: result.status,
|
||||||
|
},
|
||||||
|
{ status: 502 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({
|
||||||
|
success: true,
|
||||||
|
status: result.status,
|
||||||
|
message: `Webhook test alert delivered (HTTP ${result.status ?? 200}).`,
|
||||||
|
});
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if (err instanceof SecretVaultError) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: err.code, error: err.message },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const message =
|
||||||
|
err instanceof Error ? err.message : "Webhook test alert failed";
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: "WEBHOOK_DELIVERY_FAILED", error: message },
|
||||||
|
{ status: 502 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
import { SettingsManager } from "@/core/adminSettings/settingsManager";
|
||||||
|
import { SecretVaultError } from "@/core/Security/secretVault";
|
||||||
|
import { CSVStorageManager } from "@/core/Storage/csvStorage";
|
||||||
|
import {
|
||||||
|
buildS3ObjectKey,
|
||||||
|
normalizeS3EndpointUrl,
|
||||||
|
uploadFileToS3,
|
||||||
|
} from "@/core/Storage/s3Archive";
|
||||||
|
import { requireAdminApiSession } from "@/lib/adminAuth";
|
||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest) {
|
||||||
|
const auth = await requireAdminApiSession(req);
|
||||||
|
if (!auth.valid) {
|
||||||
|
return auth.response;
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: Record<string, unknown> = {};
|
||||||
|
try {
|
||||||
|
const text = await req.text();
|
||||||
|
if (text.trim()) {
|
||||||
|
const parsed = JSON.parse(text);
|
||||||
|
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
||||||
|
body = parsed as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const settings = await SettingsManager.getSettingsAsync();
|
||||||
|
const s3Config = await SettingsManager.getDecryptedS3ConfigAsync();
|
||||||
|
|
||||||
|
if (!s3Config.bucket.trim() || !s3Config.accessKeyId.trim() || !s3Config.secretAccessKey.trim()) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "S3_INVALID_CONFIG",
|
||||||
|
error: "Configure S3 Bucket, Access Key ID, and Secret Access Key before archiving.",
|
||||||
|
},
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
normalizeS3EndpointUrl(s3Config.endpoint, s3Config.region);
|
||||||
|
} catch (urlErr: unknown) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "S3_INVALID_CONFIG",
|
||||||
|
error: urlErr instanceof Error ? urlErr.message : "Invalid S3 endpoint URL.",
|
||||||
|
},
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const pruneAfterUpload = body.pruneAfterUpload === true;
|
||||||
|
const rawOlderThanDays =
|
||||||
|
typeof body.olderThanDays === "number" && Number.isFinite(body.olderThanDays)
|
||||||
|
? Math.max(0, Math.floor(body.olderThanDays))
|
||||||
|
: settings.autoPruneDays > 0
|
||||||
|
? settings.autoPruneDays
|
||||||
|
: 0;
|
||||||
|
const shouldCompressFirst =
|
||||||
|
body.compressFirst !== false && settings.compressAfterDays > 0;
|
||||||
|
|
||||||
|
let compressedCount = 0;
|
||||||
|
if (shouldCompressFirst) {
|
||||||
|
compressedCount = await CSVStorageManager.compressOldPartitionsAsync(
|
||||||
|
settings.compressAfterDays,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const archiveResult =
|
||||||
|
await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync(
|
||||||
|
settings.autoPruneDays,
|
||||||
|
{
|
||||||
|
archiveOnlyWithoutDelete: !pruneAfterUpload,
|
||||||
|
maxCutoffDays: rawOlderThanDays,
|
||||||
|
beforeDeleteFile: async (file) => {
|
||||||
|
const key = buildS3ObjectKey(
|
||||||
|
s3Config.prefix,
|
||||||
|
file.clientId,
|
||||||
|
file.filename,
|
||||||
|
);
|
||||||
|
const uploadRes = await uploadFileToS3(
|
||||||
|
s3Config,
|
||||||
|
file.filePath,
|
||||||
|
key,
|
||||||
|
);
|
||||||
|
if (!uploadRes.ok) {
|
||||||
|
throw new Error(
|
||||||
|
uploadRes.error || `Failed to upload ${file.filename}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
if (
|
||||||
|
archiveResult.errors.length > 0 &&
|
||||||
|
archiveResult.archivedCount === 0
|
||||||
|
) {
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: "S3_ARCHIVE_FAILED",
|
||||||
|
severity: "critical",
|
||||||
|
title: "S3 Cold Archive Upload Failed",
|
||||||
|
message: `Manual S3 archive failed: ${archiveResult.errors[0]}`,
|
||||||
|
details: { errorCount: archiveResult.errors.length },
|
||||||
|
});
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: "S3_ARCHIVE_FAILED",
|
||||||
|
error: archiveResult.errors[0],
|
||||||
|
compressedCount,
|
||||||
|
...archiveResult,
|
||||||
|
},
|
||||||
|
{ status: 502 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (archiveResult.errors.length > 0) {
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: "S3_ARCHIVE_FAILED",
|
||||||
|
severity: "warning",
|
||||||
|
title: "Partial S3 Cold Archive Errors",
|
||||||
|
message: `S3 archive encountered ${archiveResult.errors.length} error(s): ${archiveResult.errors[0]}`,
|
||||||
|
details: { errorCount: archiveResult.errors.length },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({
|
||||||
|
success: archiveResult.errors.length === 0,
|
||||||
|
compressedCount,
|
||||||
|
...archiveResult,
|
||||||
|
});
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if (err instanceof SecretVaultError) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: err.code, error: err.message },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const message =
|
||||||
|
err instanceof Error ? err.message : "Manual S3 archive failed";
|
||||||
|
SettingsManager.triggerWebhookAlert({
|
||||||
|
event: "S3_ARCHIVE_FAILED",
|
||||||
|
severity: "critical",
|
||||||
|
title: "S3 Cold Archive Upload Failed",
|
||||||
|
message,
|
||||||
|
});
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: "S3_ARCHIVE_FAILED", error: message },
|
||||||
|
{ status: 500 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -23,7 +23,10 @@ export async function GET(req: NextRequest) {
|
|||||||
? CSVStorageManager.sanitizeClientId(clientId)
|
? CSVStorageManager.sanitizeClientId(clientId)
|
||||||
: undefined;
|
: undefined;
|
||||||
|
|
||||||
const allFiles = await CSVStorageManager.listAllFilesAsync();
|
const [allFiles, diskUsage] = await Promise.all([
|
||||||
|
CSVStorageManager.listAllFilesAsync(),
|
||||||
|
CSVStorageManager.getDiskUsageAsync(),
|
||||||
|
]);
|
||||||
let files = allFiles;
|
let files = allFiles;
|
||||||
|
|
||||||
if (clientId) {
|
if (clientId) {
|
||||||
@@ -39,7 +42,7 @@ export async function GET(req: NextRequest) {
|
|||||||
|
|
||||||
const stats = CSVStorageManager.formatStorageStats(files);
|
const stats = CSVStorageManager.formatStorageStats(files);
|
||||||
|
|
||||||
return NextResponse.json({ stats, files });
|
return NextResponse.json({ stats, files, diskUsage });
|
||||||
} catch (err: unknown) {
|
} catch (err: unknown) {
|
||||||
const message = err instanceof Error ? err.message : 'Storage stats failed';
|
const message = err instanceof Error ? err.message : 'Storage stats failed';
|
||||||
return NextResponse.json({ error: message }, { status: 500 });
|
return NextResponse.json({ error: message }, { status: 500 });
|
||||||
|
|||||||
@@ -0,0 +1,105 @@
|
|||||||
|
import { SettingsManager } from "@/core/adminSettings/settingsManager";
|
||||||
|
import {
|
||||||
|
decryptSecret,
|
||||||
|
SecretVaultError,
|
||||||
|
} from "@/core/Security/secretVault";
|
||||||
|
import {
|
||||||
|
testS3Connection,
|
||||||
|
type S3ArchiveConfig,
|
||||||
|
} from "@/core/Storage/s3Archive";
|
||||||
|
import { requireAdminApiSession } from "@/lib/adminAuth";
|
||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest) {
|
||||||
|
const auth = await requireAdminApiSession(req);
|
||||||
|
if (!auth.valid) {
|
||||||
|
return auth.response;
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: Record<string, unknown> = {};
|
||||||
|
try {
|
||||||
|
const text = await req.text();
|
||||||
|
if (text.trim()) {
|
||||||
|
const parsed = JSON.parse(text);
|
||||||
|
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
||||||
|
body = parsed as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "Invalid JSON request payload.", code: "INVALID_JSON" },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const saved = await SettingsManager.getSettingsAsync();
|
||||||
|
const draftSecret =
|
||||||
|
typeof body.s3SecretAccessKey === "string" &&
|
||||||
|
body.s3SecretAccessKey.trim().length > 0
|
||||||
|
? body.s3SecretAccessKey.trim()
|
||||||
|
: "";
|
||||||
|
|
||||||
|
let effectiveSecret = draftSecret;
|
||||||
|
if (!effectiveSecret && !body.clearS3SecretKey) {
|
||||||
|
effectiveSecret = decryptSecret(saved.s3SecretAccessKeyEnc);
|
||||||
|
}
|
||||||
|
|
||||||
|
const config: S3ArchiveConfig = {
|
||||||
|
endpoint:
|
||||||
|
typeof body.s3Endpoint === "string"
|
||||||
|
? body.s3Endpoint.trim()
|
||||||
|
: saved.s3Endpoint,
|
||||||
|
region:
|
||||||
|
typeof body.s3Region === "string" && body.s3Region.trim()
|
||||||
|
? body.s3Region.trim()
|
||||||
|
: saved.s3Region || "auto",
|
||||||
|
bucket:
|
||||||
|
typeof body.s3Bucket === "string"
|
||||||
|
? body.s3Bucket.trim()
|
||||||
|
: saved.s3Bucket,
|
||||||
|
prefix:
|
||||||
|
typeof body.s3Prefix === "string"
|
||||||
|
? body.s3Prefix.trim()
|
||||||
|
: saved.s3Prefix,
|
||||||
|
accessKeyId:
|
||||||
|
typeof body.s3AccessKeyId === "string"
|
||||||
|
? body.s3AccessKeyId.trim()
|
||||||
|
: saved.s3AccessKeyId,
|
||||||
|
secretAccessKey: effectiveSecret,
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = await testS3Connection(config);
|
||||||
|
if (!result.ok) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
success: false,
|
||||||
|
code: result.code || "S3_CONNECTION_FAILED",
|
||||||
|
error: result.error || "S3 connectivity check failed.",
|
||||||
|
...(result.status !== undefined ? { s3Status: result.status } : {}),
|
||||||
|
},
|
||||||
|
{ status: result.code === "S3_INVALID_CONFIG" ? 400 : 502 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({
|
||||||
|
success: true,
|
||||||
|
endpoint: result.endpoint,
|
||||||
|
bucket: result.bucket,
|
||||||
|
status: result.status,
|
||||||
|
});
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if (err instanceof SecretVaultError) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: err.code, error: err.message },
|
||||||
|
{ status: 400 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const message =
|
||||||
|
err instanceof Error ? err.message : "S3 connectivity check failed";
|
||||||
|
return NextResponse.json(
|
||||||
|
{ success: false, code: "S3_CONNECTION_FAILED", error: message },
|
||||||
|
{ status: 500 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -47,6 +47,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
const [editClient, setEditClient] = useState<ClientItem | null>(null);
|
const [editClient, setEditClient] = useState<ClientItem | null>(null);
|
||||||
const [editName, setEditName] = useState('');
|
const [editName, setEditName] = useState('');
|
||||||
const [editRateLimit, setEditRateLimit] = useState('100');
|
const [editRateLimit, setEditRateLimit] = useState('100');
|
||||||
|
const [editStorageQuotaMb, setEditStorageQuotaMb] = useState('0');
|
||||||
|
const [editAllowedIps, setEditAllowedIps] = useState('');
|
||||||
|
const [editExpiresAt, setEditExpiresAt] = useState('');
|
||||||
const [editError, setEditError] = useState('');
|
const [editError, setEditError] = useState('');
|
||||||
const [processing, setProcessing] = useState(false);
|
const [processing, setProcessing] = useState(false);
|
||||||
|
|
||||||
@@ -59,6 +62,18 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
setEditClient(client);
|
setEditClient(client);
|
||||||
setEditName(client.name);
|
setEditName(client.name);
|
||||||
setEditRateLimit(String(client.rateLimit || 100));
|
setEditRateLimit(String(client.rateLimit || 100));
|
||||||
|
setEditStorageQuotaMb(String(client.storageQuotaMb ?? 0));
|
||||||
|
setEditAllowedIps(client.allowedIps || '');
|
||||||
|
if (client.expiresAt) {
|
||||||
|
const parsedDate = new Date(client.expiresAt);
|
||||||
|
setEditExpiresAt(
|
||||||
|
!Number.isNaN(parsedDate.getTime())
|
||||||
|
? parsedDate.toISOString().slice(0, 16)
|
||||||
|
: '',
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
setEditExpiresAt('');
|
||||||
|
}
|
||||||
setEditError('');
|
setEditError('');
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -68,6 +83,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
|
|
||||||
const trimmedName = editName.trim();
|
const trimmedName = editName.trim();
|
||||||
const numericRateLimit = Number(editRateLimit);
|
const numericRateLimit = Number(editRateLimit);
|
||||||
|
const numericQuotaMb = Number(
|
||||||
|
editStorageQuotaMb.trim() === '' ? '0' : editStorageQuotaMb,
|
||||||
|
);
|
||||||
if (!trimmedName) {
|
if (!trimmedName) {
|
||||||
setEditError('Display name is required.');
|
setEditError('Display name is required.');
|
||||||
return;
|
return;
|
||||||
@@ -76,6 +94,19 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
setEditError('Rate limit must be an integer between 1 and 100,000 events/s.');
|
setEditError('Rate limit must be an integer between 1 and 100,000 events/s.');
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) {
|
||||||
|
setEditError('Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let normalizedExpiresAt: string | null = null;
|
||||||
|
if (editExpiresAt.trim()) {
|
||||||
|
const parsedDate = new Date(editExpiresAt.trim());
|
||||||
|
if (Number.isNaN(parsedDate.getTime())) {
|
||||||
|
setEditError('Expiration date/time is invalid.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
normalizedExpiresAt = parsedDate.toISOString();
|
||||||
|
}
|
||||||
|
|
||||||
setEditError('');
|
setEditError('');
|
||||||
setProcessing(true);
|
setProcessing(true);
|
||||||
@@ -86,6 +117,9 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
body: JSON.stringify({
|
body: JSON.stringify({
|
||||||
name: trimmedName,
|
name: trimmedName,
|
||||||
rateLimit: Math.floor(numericRateLimit),
|
rateLimit: Math.floor(numericRateLimit),
|
||||||
|
storageQuotaMb: Math.floor(numericQuotaMb),
|
||||||
|
allowedIps: editAllowedIps.trim(),
|
||||||
|
expiresAt: normalizedExpiresAt,
|
||||||
}),
|
}),
|
||||||
});
|
});
|
||||||
if (response.status === 401) {
|
if (response.status === 401) {
|
||||||
@@ -224,17 +258,32 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
) : (
|
) : (
|
||||||
clients.map((client) => {
|
clients.map((client) => {
|
||||||
const revoked = client.status === 'revoked';
|
const revoked = client.status === 'revoked';
|
||||||
const online = client.status === 'active' && Boolean(client.lastActive) &&
|
const expired =
|
||||||
|
Boolean(client.expiresAt) &&
|
||||||
|
now >= new Date(client.expiresAt!).getTime();
|
||||||
|
const online =
|
||||||
|
client.status === 'active' &&
|
||||||
|
!expired &&
|
||||||
|
Boolean(client.lastActive) &&
|
||||||
now - new Date(client.lastActive!).getTime() < 60_000;
|
now - new Date(client.lastActive!).getTime() < 60_000;
|
||||||
const pending = client.status === 'active' && !client.lastActive;
|
const pending = client.status === 'active' && !expired && !client.lastActive;
|
||||||
const label = revoked ? 'revoked' : pending ? 'pending' : online ? 'online' : 'offline';
|
const label = revoked
|
||||||
const dotClass = revoked
|
? 'revoked'
|
||||||
? 'bg-[var(--danger)]'
|
: expired
|
||||||
: online
|
? 'expired'
|
||||||
? 'bg-[var(--accent)]'
|
|
||||||
: pending
|
: pending
|
||||||
? 'bg-[var(--warning)]'
|
? 'pending'
|
||||||
: 'bg-[var(--text-faint)]';
|
: online
|
||||||
|
? 'online'
|
||||||
|
: 'offline';
|
||||||
|
const dotClass =
|
||||||
|
revoked || expired
|
||||||
|
? 'bg-[var(--danger)]'
|
||||||
|
: online
|
||||||
|
? 'bg-[var(--accent)]'
|
||||||
|
: pending
|
||||||
|
? 'bg-[var(--warning)]'
|
||||||
|
: 'bg-[var(--text-faint)]';
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<TableRow key={client.clientId}>
|
<TableRow key={client.clientId}>
|
||||||
@@ -249,6 +298,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
revoked
|
revoked
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
|
{expired && (
|
||||||
|
<span className="border border-[var(--danger)] bg-[var(--danger-soft)] px-1 py-0 text-[9px] uppercase tracking-wider text-[var(--danger)]">
|
||||||
|
expired
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
<div className="mt-0.5 truncate font-mono text-[10.5px] text-[var(--text-faint)]">{client.clientId}</div>
|
<div className="mt-0.5 truncate font-mono text-[10.5px] text-[var(--text-faint)]">{client.clientId}</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -259,7 +313,11 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
{client.tokenPreview || 'JWT Active'}
|
{client.tokenPreview || 'JWT Active'}
|
||||||
</div>
|
</div>
|
||||||
<div className="mt-0.5 text-[10px] text-[var(--text-faint)]">
|
<div className="mt-0.5 text-[10px] text-[var(--text-faint)]">
|
||||||
{client.rateLimit}/s limit
|
{client.rateLimit}/s · {(client.storageQuotaMb ?? 0) > 0 ? `${client.storageQuotaMb}MB quota` : 'Unlimited'}
|
||||||
|
{client.allowedIps ? ` · IP: ${client.allowedIps}` : ''}
|
||||||
|
{client.expiresAt
|
||||||
|
? ` · Exp: ${new Date(client.expiresAt).toISOString().slice(0, 10)}`
|
||||||
|
: ''}
|
||||||
</div>
|
</div>
|
||||||
</TableCell>
|
</TableCell>
|
||||||
<TableCell className="text-right font-mono tabular-nums">
|
<TableCell className="text-right font-mono tabular-nums">
|
||||||
@@ -375,6 +433,46 @@ export function ClientsTable({ clients, now, onStatusChange }: ClientsTableProps
|
|||||||
onChange={(event) => setEditRateLimit(event.target.value)}
|
onChange={(event) => setEditRateLimit(event.target.value)}
|
||||||
required
|
required
|
||||||
/>
|
/>
|
||||||
|
<Input
|
||||||
|
label="Storage quota (MB, 0 = unlimited)"
|
||||||
|
compact
|
||||||
|
type="number"
|
||||||
|
min="0"
|
||||||
|
max="1048576"
|
||||||
|
value={editStorageQuotaMb}
|
||||||
|
onChange={(event) => setEditStorageQuotaMb(event.target.value)}
|
||||||
|
required
|
||||||
|
/>
|
||||||
|
<Input
|
||||||
|
label="Allowed IPs / CIDR (blank = allow all)"
|
||||||
|
compact
|
||||||
|
value={editAllowedIps}
|
||||||
|
onChange={(event) => setEditAllowedIps(event.target.value)}
|
||||||
|
placeholder="203.0.113.10, 10.0.0.0/8"
|
||||||
|
className="font-mono text-[11px]"
|
||||||
|
/>
|
||||||
|
<div className="flex items-end gap-1.5">
|
||||||
|
<div className="min-w-0 flex-1">
|
||||||
|
<Input
|
||||||
|
label="Token expiration (optional)"
|
||||||
|
compact
|
||||||
|
type="datetime-local"
|
||||||
|
value={editExpiresAt}
|
||||||
|
onChange={(event) => setEditExpiresAt(event.target.value)}
|
||||||
|
className="font-mono text-[11px]"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
{editExpiresAt && (
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
size="xs"
|
||||||
|
variant="ghost"
|
||||||
|
onClick={() => setEditExpiresAt('')}
|
||||||
|
>
|
||||||
|
Clear
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
{editError && (
|
{editError && (
|
||||||
<p role="alert" className="text-[11px] text-[var(--danger)]">
|
<p role="alert" className="text-[11px] text-[var(--danger)]">
|
||||||
{editError}
|
{editError}
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ interface StorageFile {
|
|||||||
modifiedAt: string;
|
modifiedAt: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
type ExportFormat = 'csv' | 'json';
|
type ExportFormat = 'csv' | 'json' | 'ndjson';
|
||||||
type RangePreset = 'today' | '7d' | '30d' | 'all' | 'custom';
|
type RangePreset = 'today' | '7d' | '30d' | 'all' | 'custom';
|
||||||
|
|
||||||
function formatBytes(bytes: number) {
|
function formatBytes(bytes: number) {
|
||||||
@@ -368,7 +368,7 @@ export function ExportWorkspace() {
|
|||||||
Format
|
Format
|
||||||
</legend>
|
</legend>
|
||||||
<div
|
<div
|
||||||
className="grid grid-cols-2 gap-2"
|
className="grid grid-cols-3 gap-2"
|
||||||
role="radiogroup"
|
role="radiogroup"
|
||||||
aria-label="Export format"
|
aria-label="Export format"
|
||||||
>
|
>
|
||||||
@@ -384,6 +384,12 @@ export function ExportWorkspace() {
|
|||||||
label="JSON"
|
label="JSON"
|
||||||
onSelect={setFormat}
|
onSelect={setFormat}
|
||||||
/>
|
/>
|
||||||
|
<FormatOption
|
||||||
|
value="ndjson"
|
||||||
|
selected={format === 'ndjson'}
|
||||||
|
label="NDJSON"
|
||||||
|
onSelect={setFormat}
|
||||||
|
/>
|
||||||
</div>
|
</div>
|
||||||
</fieldset>
|
</fieldset>
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
|
|||||||
const [clientId, setClientId] = useState('');
|
const [clientId, setClientId] = useState('');
|
||||||
const [name, setName] = useState('');
|
const [name, setName] = useState('');
|
||||||
const [rateLimit, setRateLimit] = useState('100');
|
const [rateLimit, setRateLimit] = useState('100');
|
||||||
|
const [storageQuotaMb, setStorageQuotaMb] = useState('0');
|
||||||
|
const [allowedIps, setAllowedIps] = useState('');
|
||||||
|
const [expiresAt, setExpiresAt] = useState('');
|
||||||
const [submitting, setSubmitting] = useState(false);
|
const [submitting, setSubmitting] = useState(false);
|
||||||
const [credentials, setCredentials] = useState<ClientCredentials | null>(null);
|
const [credentials, setCredentials] = useState<ClientCredentials | null>(null);
|
||||||
|
|
||||||
@@ -35,12 +38,33 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
|
|||||||
error('Validation error', 'Rate limit must be an integer between 1 and 100,000 events/s.');
|
error('Validation error', 'Rate limit must be an integer between 1 and 100,000 events/s.');
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
const numericQuotaMb = Number(storageQuotaMb.trim() === '' ? '0' : storageQuotaMb);
|
||||||
|
if (!Number.isInteger(numericQuotaMb) || numericQuotaMb < 0 || numericQuotaMb > 1048576) {
|
||||||
|
error('Validation error', 'Storage quota must be an integer between 0 and 1,048,576 MB (0 = unlimited).');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let normalizedExpiresAt: string | null = null;
|
||||||
|
if (expiresAt.trim()) {
|
||||||
|
const parsedDate = new Date(expiresAt.trim());
|
||||||
|
if (Number.isNaN(parsedDate.getTime())) {
|
||||||
|
error('Validation error', 'Token expiration must be a valid date/time.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
normalizedExpiresAt = parsedDate.toISOString();
|
||||||
|
}
|
||||||
setSubmitting(true);
|
setSubmitting(true);
|
||||||
try {
|
try {
|
||||||
const response = await fetch('/api/clients', {
|
const response = await fetch('/api/clients', {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
body: JSON.stringify({ clientId: trimmedId, name: trimmedName, rateLimit: numericRate }),
|
body: JSON.stringify({
|
||||||
|
clientId: trimmedId,
|
||||||
|
name: trimmedName,
|
||||||
|
rateLimit: numericRate,
|
||||||
|
storageQuotaMb: numericQuotaMb,
|
||||||
|
allowedIps: allowedIps.trim(),
|
||||||
|
expiresAt: normalizedExpiresAt,
|
||||||
|
}),
|
||||||
});
|
});
|
||||||
if (response.status === 401) {
|
if (response.status === 401) {
|
||||||
window.location.assign('/vcon/login');
|
window.location.assign('/vcon/login');
|
||||||
@@ -53,6 +77,9 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
|
|||||||
setClientId('');
|
setClientId('');
|
||||||
setName('');
|
setName('');
|
||||||
setRateLimit('100');
|
setRateLimit('100');
|
||||||
|
setStorageQuotaMb('0');
|
||||||
|
setAllowedIps('');
|
||||||
|
setExpiresAt('');
|
||||||
onClose();
|
onClose();
|
||||||
success('Client registered', `${data.client.clientId} is ready to connect.`);
|
success('Client registered', `${data.client.clientId} is ready to connect.`);
|
||||||
await onCreated();
|
await onCreated();
|
||||||
@@ -103,6 +130,31 @@ export function NewClientModal({ open, onClose, onCreated }: NewClientModalProps
|
|||||||
value={rateLimit}
|
value={rateLimit}
|
||||||
onChange={(event) => setRateLimit(event.target.value)}
|
onChange={(event) => setRateLimit(event.target.value)}
|
||||||
/>
|
/>
|
||||||
|
<Input
|
||||||
|
compact
|
||||||
|
label="Storage quota (MB, 0 = unlimited)"
|
||||||
|
type="number"
|
||||||
|
min="0"
|
||||||
|
max="1048576"
|
||||||
|
value={storageQuotaMb}
|
||||||
|
onChange={(event) => setStorageQuotaMb(event.target.value)}
|
||||||
|
/>
|
||||||
|
<Input
|
||||||
|
compact
|
||||||
|
label="Allowed IPs / CIDR (blank = allow all)"
|
||||||
|
placeholder="203.0.113.10, 10.0.0.0/8"
|
||||||
|
value={allowedIps}
|
||||||
|
onChange={(event) => setAllowedIps(event.target.value)}
|
||||||
|
className="font-mono text-[11px]"
|
||||||
|
/>
|
||||||
|
<Input
|
||||||
|
compact
|
||||||
|
label="Token expiration (optional)"
|
||||||
|
type="datetime-local"
|
||||||
|
value={expiresAt}
|
||||||
|
onChange={(event) => setExpiresAt(event.target.value)}
|
||||||
|
className="font-mono text-[11px]"
|
||||||
|
/>
|
||||||
</form>
|
</form>
|
||||||
</Modal>
|
</Modal>
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
@@ -2,6 +2,9 @@ export interface ClientItem {
|
|||||||
clientId: string;
|
clientId: string;
|
||||||
name: string;
|
name: string;
|
||||||
rateLimit: number;
|
rateLimit: number;
|
||||||
|
storageQuotaMb?: number;
|
||||||
|
allowedIps?: string;
|
||||||
|
expiresAt?: string | null;
|
||||||
status: "active" | "revoked";
|
status: "active" | "revoked";
|
||||||
totalEvents: number;
|
totalEvents: number;
|
||||||
lastActive?: string;
|
lastActive?: string;
|
||||||
|
|||||||
@@ -17,6 +17,9 @@ export const CORE_SCHEMA_TABLE_DDLS = [
|
|||||||
secret TEXT NOT NULL,
|
secret TEXT NOT NULL,
|
||||||
token TEXT NOT NULL,
|
token TEXT NOT NULL,
|
||||||
rate_limit INTEGER NOT NULL DEFAULT 100,
|
rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||||
|
storage_quota_mb INTEGER NOT NULL DEFAULT 0,
|
||||||
|
allowed_ips TEXT NOT NULL DEFAULT '',
|
||||||
|
expires_at TIMESTAMPTZ,
|
||||||
status TEXT NOT NULL DEFAULT 'active',
|
status TEXT NOT NULL DEFAULT 'active',
|
||||||
total_events BIGINT NOT NULL DEFAULT 0,
|
total_events BIGINT NOT NULL DEFAULT 0,
|
||||||
last_active TIMESTAMPTZ,
|
last_active TIMESTAMPTZ,
|
||||||
@@ -37,6 +40,21 @@ export const CORE_SCHEMA_TABLE_DDLS = [
|
|||||||
jwt_secret TEXT NOT NULL,
|
jwt_secret TEXT NOT NULL,
|
||||||
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
||||||
auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
||||||
|
compress_after_days INTEGER NOT NULL DEFAULT 0,
|
||||||
|
enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
s3_endpoint TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_region TEXT NOT NULL DEFAULT 'auto',
|
||||||
|
s3_bucket TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
|
||||||
|
s3_access_key_id TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
|
||||||
|
enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
webhook_url TEXT NOT NULL DEFAULT '',
|
||||||
|
webhook_secret_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
|
||||||
|
enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
|
||||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
)
|
)
|
||||||
`,
|
`,
|
||||||
@@ -71,6 +89,9 @@ export const CORE_SCHEMA_ALTER_DDLS = [
|
|||||||
`
|
`
|
||||||
ALTER TABLE client_sdks
|
ALTER TABLE client_sdks
|
||||||
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
|
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||||
|
ADD COLUMN IF NOT EXISTS storage_quota_mb INTEGER NOT NULL DEFAULT 0,
|
||||||
|
ADD COLUMN IF NOT EXISTS allowed_ips TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ,
|
||||||
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
|
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
|
||||||
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
|
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
|
||||||
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
|
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
|
||||||
@@ -88,6 +109,21 @@ export const CORE_SCHEMA_ALTER_DDLS = [
|
|||||||
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
|
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
|
||||||
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
||||||
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
||||||
|
ADD COLUMN IF NOT EXISTS compress_after_days INTEGER NOT NULL DEFAULT 0,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_endpoint TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_region TEXT NOT NULL DEFAULT 'auto',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_bucket TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_url TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_secret_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
|
||||||
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
`,
|
`,
|
||||||
`
|
`
|
||||||
|
|||||||
@@ -0,0 +1,219 @@
|
|||||||
|
import net from "node:net";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Per-Client IP / CIDR Allowlist & Token Expiration Validator (V2-Phase-02)
|
||||||
|
*/
|
||||||
|
|
||||||
|
export function stripIpv6MappedPrefix(ip: string): string {
|
||||||
|
const trimmed = String(ip || "").trim();
|
||||||
|
if (trimmed.toLowerCase().startsWith("::ffff:")) {
|
||||||
|
const candidate = trimmed.slice(7);
|
||||||
|
if (net.isIPv4(candidate)) {
|
||||||
|
return candidate;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function ipv4ToUint32(ip: string): number | null {
|
||||||
|
const clean = stripIpv6MappedPrefix(ip);
|
||||||
|
if (!net.isIPv4(clean)) return null;
|
||||||
|
const parts = clean.split(".").map((octet) => Number(octet));
|
||||||
|
if (
|
||||||
|
parts.length !== 4 ||
|
||||||
|
parts.some((n) => !Number.isInteger(n) || n < 0 || n > 255)
|
||||||
|
) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
(((parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3]) >>> 0)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseAllowedIpsList(raw?: string | null): string[] {
|
||||||
|
if (!raw || typeof raw !== "string") return [];
|
||||||
|
return raw
|
||||||
|
.split(/[\s,;]+/)
|
||||||
|
.map((entry) => entry.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function validateAllowedIpsInput(
|
||||||
|
raw: unknown,
|
||||||
|
): { valid: true; normalized: string } | { valid: false; error: string } {
|
||||||
|
if (raw === undefined || raw === null || raw === "") {
|
||||||
|
return { valid: true, normalized: "" };
|
||||||
|
}
|
||||||
|
if (typeof raw !== "string") {
|
||||||
|
return { valid: false, error: "allowedIps must be a string." };
|
||||||
|
}
|
||||||
|
if (raw.length > 2048) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: "allowedIps cannot exceed 2048 characters.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const entries = parseAllowedIpsList(raw);
|
||||||
|
if (entries.length === 0) {
|
||||||
|
return { valid: true, normalized: "" };
|
||||||
|
}
|
||||||
|
if (entries.length > 64) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: "allowedIps cannot contain more than 64 IP/CIDR entries.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const normalizedEntries: string[] = [];
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (entry === "*") {
|
||||||
|
normalizedEntries.push("*");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (entry.includes("/")) {
|
||||||
|
const [ipPart, prefixPart, ...extra] = entry.split("/");
|
||||||
|
if (extra.length > 0 || !ipPart || !prefixPart || !/^\d+$/.test(prefixPart)) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: `Invalid CIDR entry '${entry}'. Expected format like 203.0.113.0/24.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const prefix = Number(prefixPart);
|
||||||
|
const cleanIp = stripIpv6MappedPrefix(ipPart);
|
||||||
|
if (net.isIPv4(cleanIp)) {
|
||||||
|
if (prefix < 0 || prefix > 32) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: `Invalid IPv4 CIDR prefix in '${entry}' (must be 0..32).`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
normalizedEntries.push(`${cleanIp}/${prefix}`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (net.isIPv6(cleanIp)) {
|
||||||
|
if (prefix < 0 || prefix > 128) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: `Invalid IPv6 CIDR prefix in '${entry}' (must be 0..128).`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
normalizedEntries.push(`${cleanIp.toLowerCase()}/${prefix}`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: `Invalid IP address in CIDR entry '${entry}'.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const cleanIp = stripIpv6MappedPrefix(entry);
|
||||||
|
const ipVersion = net.isIP(cleanIp);
|
||||||
|
if (ipVersion === 4) {
|
||||||
|
normalizedEntries.push(cleanIp);
|
||||||
|
} else if (ipVersion === 6) {
|
||||||
|
normalizedEntries.push(cleanIp.toLowerCase());
|
||||||
|
} else {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: `Invalid IP or CIDR entry '${entry}'.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
valid: true,
|
||||||
|
normalized: Array.from(new Set(normalizedEntries)).join(", "),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isClientIpAllowed(
|
||||||
|
clientIp: string,
|
||||||
|
allowedIpsRaw?: string | null,
|
||||||
|
): boolean {
|
||||||
|
const entries = parseAllowedIpsList(allowedIpsRaw);
|
||||||
|
if (entries.length === 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
const cleanClientIp = stripIpv6MappedPrefix(clientIp);
|
||||||
|
if (!cleanClientIp || cleanClientIp === "unknown") {
|
||||||
|
return entries.includes("*");
|
||||||
|
}
|
||||||
|
|
||||||
|
const clientIpv4Int = ipv4ToUint32(cleanClientIp);
|
||||||
|
const isLoopbackClient =
|
||||||
|
cleanClientIp === "127.0.0.1" || cleanClientIp.toLowerCase() === "::1";
|
||||||
|
|
||||||
|
for (const rule of entries) {
|
||||||
|
if (rule === "*") return true;
|
||||||
|
|
||||||
|
if (rule.includes("/")) {
|
||||||
|
const [ruleIpRaw, prefixRaw] = rule.split("/");
|
||||||
|
const ruleIp = stripIpv6MappedPrefix(ruleIpRaw);
|
||||||
|
const prefix = Number(prefixRaw);
|
||||||
|
const ruleIpv4Int = ipv4ToUint32(ruleIp);
|
||||||
|
|
||||||
|
if (clientIpv4Int !== null && ruleIpv4Int !== null && prefix >= 0 && prefix <= 32) {
|
||||||
|
if (prefix === 0) return true;
|
||||||
|
const mask = prefix === 32 ? 0xffffffff : (~((1 << (32 - prefix)) - 1)) >>> 0;
|
||||||
|
if ((clientIpv4Int & mask) >>> 0 === (ruleIpv4Int & mask) >>> 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const cleanRule = stripIpv6MappedPrefix(rule);
|
||||||
|
if (cleanRule.toLowerCase() === cleanClientIp.toLowerCase()) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
isLoopbackClient &&
|
||||||
|
(cleanRule === "127.0.0.1" || cleanRule.toLowerCase() === "::1")
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function validateClientExpiresAtInput(
|
||||||
|
raw: unknown,
|
||||||
|
): { valid: true; iso: string | null } | { valid: false; error: string } {
|
||||||
|
if (raw === undefined || raw === null || raw === "") {
|
||||||
|
return { valid: true, iso: null };
|
||||||
|
}
|
||||||
|
if (typeof raw !== "string") {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: "expiresAt must be an ISO timestamp string or null.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const trimmed = raw.trim();
|
||||||
|
if (!trimmed) {
|
||||||
|
return { valid: true, iso: null };
|
||||||
|
}
|
||||||
|
const parsedMs = Date.parse(trimmed);
|
||||||
|
if (!Number.isFinite(parsedMs)) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: "expiresAt must be a valid ISO-8601 date or timestamp.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { valid: true, iso: new Date(parsedMs).toISOString() };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isClientTokenExpired(
|
||||||
|
expiresAt?: string | null,
|
||||||
|
nowMs = Date.now(),
|
||||||
|
): boolean {
|
||||||
|
if (!expiresAt || typeof expiresAt !== "string") return false;
|
||||||
|
const trimmed = expiresAt.trim();
|
||||||
|
if (!trimmed) return false;
|
||||||
|
const parsedMs = Date.parse(trimmed);
|
||||||
|
if (!Number.isFinite(parsedMs)) return false;
|
||||||
|
return nowMs >= parsedMs;
|
||||||
|
}
|
||||||
@@ -0,0 +1,152 @@
|
|||||||
|
import {
|
||||||
|
createCipheriv,
|
||||||
|
createDecipheriv,
|
||||||
|
hkdfSync,
|
||||||
|
randomBytes,
|
||||||
|
} from "node:crypto";
|
||||||
|
|
||||||
|
const DEV_DEFAULT_SESSION_SECRET =
|
||||||
|
"datastack-admin-session-default-secret-32-chars";
|
||||||
|
const VAULT_PREFIX = "v1";
|
||||||
|
const HKDF_SALT = Buffer.from("datastack-vault-salt-v1", "utf8");
|
||||||
|
const HKDF_INFO = Buffer.from("datastack-aes-256-gcm-v1", "utf8");
|
||||||
|
|
||||||
|
export type SecretVaultErrorCode =
|
||||||
|
| "SECRET_VAULT_KEY_UNAVAILABLE"
|
||||||
|
| "SECRET_VAULT_DECRYPT_FAILED"
|
||||||
|
| "SECRET_VAULT_INVALID_INPUT";
|
||||||
|
|
||||||
|
export class SecretVaultError extends Error {
|
||||||
|
public readonly code: SecretVaultErrorCode;
|
||||||
|
|
||||||
|
constructor(code: SecretVaultErrorCode, message: string) {
|
||||||
|
super(`${code}: ${message}`);
|
||||||
|
this.name = "SecretVaultError";
|
||||||
|
this.code = code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function resolveVaultMasterSecret(secretOverride?: string): string | null {
|
||||||
|
const candidate = (
|
||||||
|
secretOverride ??
|
||||||
|
process.env.DATASTACK_ADMIN_SESSION_SECRET ??
|
||||||
|
""
|
||||||
|
).trim();
|
||||||
|
if (process.env.NODE_ENV === "production") {
|
||||||
|
return candidate.length >= 32 ? candidate : null;
|
||||||
|
}
|
||||||
|
if (candidate.length > 0) {
|
||||||
|
return candidate.length >= 32 ? candidate : null;
|
||||||
|
}
|
||||||
|
return DEV_DEFAULT_SESSION_SECRET;
|
||||||
|
}
|
||||||
|
|
||||||
|
function deriveAesKey(secretOverride?: string): Buffer {
|
||||||
|
const master = resolveVaultMasterSecret(secretOverride);
|
||||||
|
if (!master) {
|
||||||
|
throw new SecretVaultError(
|
||||||
|
"SECRET_VAULT_KEY_UNAVAILABLE",
|
||||||
|
"DATASTACK_ADMIN_SESSION_SECRET (>= 32 characters) is required to encrypt or decrypt stored secrets.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const derived = hkdfSync(
|
||||||
|
"sha256",
|
||||||
|
Buffer.from(master, "utf8"),
|
||||||
|
HKDF_SALT,
|
||||||
|
HKDF_INFO,
|
||||||
|
32,
|
||||||
|
);
|
||||||
|
return Buffer.from(derived);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function encryptSecret(
|
||||||
|
plaintext: string,
|
||||||
|
secretOverride?: string,
|
||||||
|
): string {
|
||||||
|
if (typeof plaintext !== "string") {
|
||||||
|
throw new SecretVaultError(
|
||||||
|
"SECRET_VAULT_INVALID_INPUT",
|
||||||
|
"Secret value must be a string.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (!plaintext) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
const key = deriveAesKey(secretOverride);
|
||||||
|
const iv = randomBytes(12);
|
||||||
|
const cipher = createCipheriv("aes-256-gcm", key, iv);
|
||||||
|
const encrypted = Buffer.concat([
|
||||||
|
cipher.update(plaintext, "utf8"),
|
||||||
|
cipher.final(),
|
||||||
|
]);
|
||||||
|
const authTag = cipher.getAuthTag();
|
||||||
|
return [
|
||||||
|
VAULT_PREFIX,
|
||||||
|
iv.toString("base64url"),
|
||||||
|
authTag.toString("base64url"),
|
||||||
|
encrypted.toString("base64url"),
|
||||||
|
].join(":");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function decryptSecret(
|
||||||
|
ciphertext: string,
|
||||||
|
secretOverride?: string,
|
||||||
|
): string {
|
||||||
|
if (!ciphertext || typeof ciphertext !== "string") {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
const parts = ciphertext.trim().split(":");
|
||||||
|
if (parts.length !== 4 || parts[0] !== VAULT_PREFIX) {
|
||||||
|
throw new SecretVaultError(
|
||||||
|
"SECRET_VAULT_DECRYPT_FAILED",
|
||||||
|
"Stored secret ciphertext has an unrecognized format.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const key = deriveAesKey(secretOverride);
|
||||||
|
try {
|
||||||
|
const iv = Buffer.from(parts[1], "base64url");
|
||||||
|
const authTag = Buffer.from(parts[2], "base64url");
|
||||||
|
const payload = Buffer.from(parts[3], "base64url");
|
||||||
|
if (iv.length !== 12 || authTag.length !== 16) {
|
||||||
|
throw new Error("Invalid IV or auth tag length");
|
||||||
|
}
|
||||||
|
const decipher = createDecipheriv("aes-256-gcm", key, iv);
|
||||||
|
decipher.setAuthTag(authTag);
|
||||||
|
const decrypted = Buffer.concat([
|
||||||
|
decipher.update(payload),
|
||||||
|
decipher.final(),
|
||||||
|
]);
|
||||||
|
return decrypted.toString("utf8");
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if (err instanceof SecretVaultError) {
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
throw new SecretVaultError(
|
||||||
|
"SECRET_VAULT_DECRYPT_FAILED",
|
||||||
|
"Unable to decrypt stored secret. Verify DATASTACK_ADMIN_SESSION_SECRET matches the key used to encrypt it.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function maskSecretHint(plaintext: string): string {
|
||||||
|
const trimmed = String(plaintext || "").trim();
|
||||||
|
if (!trimmed) return "";
|
||||||
|
if (trimmed.length >= 8) {
|
||||||
|
return `••••${trimmed.slice(-4)}`;
|
||||||
|
}
|
||||||
|
return "••••••••";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getMaskedHintFromEncrypted(
|
||||||
|
ciphertext: string,
|
||||||
|
secretOverride?: string,
|
||||||
|
): string {
|
||||||
|
if (!ciphertext || !ciphertext.trim()) return "";
|
||||||
|
try {
|
||||||
|
const plain = decryptSecret(ciphertext, secretOverride);
|
||||||
|
return maskSecretHint(plain);
|
||||||
|
} catch {
|
||||||
|
return "••••(locked)";
|
||||||
|
}
|
||||||
|
}
|
||||||
+444
-59
@@ -1,7 +1,10 @@
|
|||||||
import crypto from 'node:crypto';
|
import crypto from 'node:crypto';
|
||||||
import fs from 'fs';
|
import fs from 'fs';
|
||||||
import path from 'path';
|
import path from 'path';
|
||||||
|
import { Readable } from 'node:stream';
|
||||||
|
import { pipeline } from 'node:stream/promises';
|
||||||
import { StringDecoder } from 'node:string_decoder';
|
import { StringDecoder } from 'node:string_decoder';
|
||||||
|
import zlib from 'node:zlib';
|
||||||
import { Parser } from 'json2csv';
|
import { Parser } from 'json2csv';
|
||||||
|
|
||||||
export interface DataEvent {
|
export interface DataEvent {
|
||||||
@@ -19,8 +22,28 @@ export interface StorageFileMeta {
|
|||||||
sizeBytes: number;
|
sizeBytes: number;
|
||||||
linesCount: number;
|
linesCount: number;
|
||||||
modifiedAt: Date;
|
modifiedAt: Date;
|
||||||
|
compressed?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface DiskUsageSnapshot {
|
||||||
|
usedPct: number;
|
||||||
|
availableBytes: number;
|
||||||
|
totalBytes: number;
|
||||||
|
checkedAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StorageAdmissionResult {
|
||||||
|
allowed: boolean;
|
||||||
|
code?: 'DISK_SPACE_LOW' | 'STORAGE_QUOTA_EXCEEDED';
|
||||||
|
reason?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type StatfsProvider = (dir: string) => Promise<{
|
||||||
|
blocks: number;
|
||||||
|
bavail: number;
|
||||||
|
bsize: number;
|
||||||
|
}>;
|
||||||
|
|
||||||
interface RowCountCacheEntry {
|
interface RowCountCacheEntry {
|
||||||
sizeBytes: number;
|
sizeBytes: number;
|
||||||
mtimeMs: number;
|
mtimeMs: number;
|
||||||
@@ -31,6 +54,7 @@ const LOCK_RETRY_MIN_MS = 5;
|
|||||||
const LOCK_RETRY_MAX_MS = 18;
|
const LOCK_RETRY_MAX_MS = 18;
|
||||||
const LOCK_TIMEOUT_MS = 10_000;
|
const LOCK_TIMEOUT_MS = 10_000;
|
||||||
const STALE_LOCK_AGE_MS = 30_000;
|
const STALE_LOCK_AGE_MS = 30_000;
|
||||||
|
const QUOTA_CACHE_TTL_MS = 30_000;
|
||||||
|
|
||||||
function sleepMs(ms: number): Promise<void> {
|
function sleepMs(ms: number): Promise<void> {
|
||||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||||
@@ -49,11 +73,39 @@ export class CSVStorageManager {
|
|||||||
public static baseStorageDir = resolveDefaultStorageDir();
|
public static baseStorageDir = resolveDefaultStorageDir();
|
||||||
public static maxFileSizeBytes = 100 * 1024 * 1024; // 100MB rotation limit
|
public static maxFileSizeBytes = 100 * 1024 * 1024; // 100MB rotation limit
|
||||||
private static rowCountCache = new Map<string, RowCountCacheEntry>();
|
private static rowCountCache = new Map<string, RowCountCacheEntry>();
|
||||||
|
private static diskUsageCache: DiskUsageSnapshot | null = null;
|
||||||
|
private static clientBytesCache = new Map<
|
||||||
|
string,
|
||||||
|
{ checkedAt: number; bytes: number }
|
||||||
|
>();
|
||||||
|
private static statfsOverrideForTests: StatfsProvider | null = null;
|
||||||
|
|
||||||
|
public static isPartitionFilename(filename: string): boolean {
|
||||||
|
return filename.endsWith('.csv') || filename.endsWith('.csv.gz');
|
||||||
|
}
|
||||||
|
|
||||||
|
public static extractDateFromFilename(filename: string): string {
|
||||||
|
return filename.replace(/\.csv(?:\.gz)?$/, '').split('-part')[0];
|
||||||
|
}
|
||||||
|
|
||||||
public static clearRowCountCache(): void {
|
public static clearRowCountCache(): void {
|
||||||
this.rowCountCache.clear();
|
this.rowCountCache.clear();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static clearQuotaCaches(clientId?: string): void {
|
||||||
|
this.diskUsageCache = null;
|
||||||
|
if (clientId) {
|
||||||
|
this.clientBytesCache.delete(this.sanitizeClientId(clientId));
|
||||||
|
} else {
|
||||||
|
this.clientBytesCache.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static setStatfsOverrideForTests(provider: StatfsProvider | null): void {
|
||||||
|
this.statfsOverrideForTests = provider;
|
||||||
|
this.diskUsageCache = null;
|
||||||
|
}
|
||||||
|
|
||||||
public static sanitizeClientId(clientId: string): string {
|
public static sanitizeClientId(clientId: string): string {
|
||||||
const raw = String(clientId || 'unknown').trim().replace(/[^a-zA-Z0-9_-]/g, '_');
|
const raw = String(clientId || 'unknown').trim().replace(/[^a-zA-Z0-9_-]/g, '_');
|
||||||
return raw || 'unknown';
|
return raw || 'unknown';
|
||||||
@@ -299,6 +351,12 @@ export class CSVStorageManager {
|
|||||||
|
|
||||||
private static countCsvDataRowsChunkedSync(filePath: string): number {
|
private static countCsvDataRowsChunkedSync(filePath: string): number {
|
||||||
const counter = this.createChunkRowCounter();
|
const counter = this.createChunkRowCounter();
|
||||||
|
if (filePath.endsWith('.gz')) {
|
||||||
|
const compressed = fs.readFileSync(filePath);
|
||||||
|
const decompressed = zlib.gunzipSync(compressed);
|
||||||
|
counter.pushChunk(decompressed);
|
||||||
|
return counter.finish();
|
||||||
|
}
|
||||||
const buf = Buffer.allocUnsafe(64 * 1024);
|
const buf = Buffer.allocUnsafe(64 * 1024);
|
||||||
const fd = fs.openSync(filePath, 'r');
|
const fd = fs.openSync(filePath, 'r');
|
||||||
try {
|
try {
|
||||||
@@ -312,14 +370,34 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static openPartitionTextStream(filePath: string): Readable {
|
||||||
|
const fileStream = fs.createReadStream(filePath, {
|
||||||
|
highWaterMark: 64 * 1024,
|
||||||
|
});
|
||||||
|
if (!filePath.endsWith('.gz')) {
|
||||||
|
return fileStream;
|
||||||
|
}
|
||||||
|
const gunzip = zlib.createGunzip();
|
||||||
|
fileStream.on('error', (err) => {
|
||||||
|
if (!gunzip.destroyed) {
|
||||||
|
gunzip.destroy(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
gunzip.on('close', () => {
|
||||||
|
if (!fileStream.destroyed) {
|
||||||
|
fileStream.destroy();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
fileStream.pipe(gunzip);
|
||||||
|
return gunzip;
|
||||||
|
}
|
||||||
|
|
||||||
private static async countCsvDataRowsStreamAsync(
|
private static async countCsvDataRowsStreamAsync(
|
||||||
filePath: string,
|
filePath: string,
|
||||||
): Promise<number> {
|
): Promise<number> {
|
||||||
const counter = this.createChunkRowCounter();
|
const counter = this.createChunkRowCounter();
|
||||||
return new Promise<number>((resolve, reject) => {
|
return new Promise<number>((resolve, reject) => {
|
||||||
const stream = fs.createReadStream(filePath, {
|
const stream = this.openPartitionTextStream(filePath);
|
||||||
highWaterMark: 64 * 1024,
|
|
||||||
});
|
|
||||||
stream.on('data', (chunk) => counter.pushChunk(chunk));
|
stream.on('data', (chunk) => counter.pushChunk(chunk));
|
||||||
stream.on('end', () => resolve(counter.finish()));
|
stream.on('end', () => resolve(counter.finish()));
|
||||||
stream.on('error', (err) => reject(err));
|
stream.on('error', (err) => reject(err));
|
||||||
@@ -437,8 +515,9 @@ export class CSVStorageManager {
|
|||||||
fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
|
fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
|
||||||
}
|
}
|
||||||
|
|
||||||
// Invalidate row cache for modified file immediately
|
// Invalidate row & quota caches for modified file immediately
|
||||||
this.rowCountCache.delete(resolvedPath);
|
this.rowCountCache.delete(resolvedPath);
|
||||||
|
this.clearQuotaCaches(clientId);
|
||||||
return resolvedPath;
|
return resolvedPath;
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -449,7 +528,7 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* List all stored CSV files with cached row counts (invalidated on mtime/size changes)
|
* List all stored CSV and gzipped CSV (.csv / .csv.gz) files with cached row counts
|
||||||
*/
|
*/
|
||||||
public static listAllFiles(): StorageFileMeta[] {
|
public static listAllFiles(): StorageFileMeta[] {
|
||||||
const results: StorageFileMeta[] = [];
|
const results: StorageFileMeta[] = [];
|
||||||
@@ -473,7 +552,7 @@ export class CSVStorageManager {
|
|||||||
|
|
||||||
const files = fs.readdirSync(fullClientPath);
|
const files = fs.readdirSync(fullClientPath);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const fullFilePath = path.join(fullClientPath, file);
|
const fullFilePath = path.join(fullClientPath, file);
|
||||||
let stats: fs.Stats;
|
let stats: fs.Stats;
|
||||||
try {
|
try {
|
||||||
@@ -505,7 +584,7 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
|
|
||||||
results.push({
|
results.push({
|
||||||
clientId: clientDir,
|
clientId: clientDir,
|
||||||
@@ -515,6 +594,7 @@ export class CSVStorageManager {
|
|||||||
sizeBytes: stats.size,
|
sizeBytes: stats.size,
|
||||||
linesCount,
|
linesCount,
|
||||||
modifiedAt: stats.mtime,
|
modifiedAt: stats.mtime,
|
||||||
|
compressed: file.endsWith('.csv.gz'),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -551,7 +631,7 @@ export class CSVStorageManager {
|
|||||||
|
|
||||||
const files = await fs.promises.readdir(fullClientPath);
|
const files = await fs.promises.readdir(fullClientPath);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const fullFilePath = path.join(fullClientPath, file);
|
const fullFilePath = path.join(fullClientPath, file);
|
||||||
const stats = await fs.promises.lstat(fullFilePath).catch(() => null);
|
const stats = await fs.promises.lstat(fullFilePath).catch(() => null);
|
||||||
if (!stats || stats.isSymbolicLink() || !stats.isFile()) continue;
|
if (!stats || stats.isSymbolicLink() || !stats.isFile()) continue;
|
||||||
@@ -578,7 +658,7 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
|
|
||||||
results.push({
|
results.push({
|
||||||
clientId: clientDir,
|
clientId: clientDir,
|
||||||
@@ -588,6 +668,7 @@ export class CSVStorageManager {
|
|||||||
sizeBytes: stats.size,
|
sizeBytes: stats.size,
|
||||||
linesCount,
|
linesCount,
|
||||||
modifiedAt: stats.mtime,
|
modifiedAt: stats.mtime,
|
||||||
|
compressed: file.endsWith('.csv.gz'),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -605,7 +686,121 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Prune CSV partitions older than autoPruneDays under cooperative partition locks
|
* Compress uncompressed .csv partitions older than compressAfterDays into .csv.gz under cooperative partition locks
|
||||||
|
*/
|
||||||
|
public static async compressOldPartitionsAsync(
|
||||||
|
compressAfterDays: number,
|
||||||
|
): Promise<number> {
|
||||||
|
if (!Number.isFinite(compressAfterDays) || compressAfterDays <= 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null);
|
||||||
|
if (!baseStat || !baseStat.isDirectory()) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const cutoffDate = new Date(
|
||||||
|
Date.now() - Math.floor(compressAfterDays) * 86_400_000,
|
||||||
|
)
|
||||||
|
.toISOString()
|
||||||
|
.split('T')[0];
|
||||||
|
|
||||||
|
let compressedCount = 0;
|
||||||
|
try {
|
||||||
|
const clientDirs = await fs.promises.readdir(this.baseStorageDir);
|
||||||
|
for (const clientDir of clientDirs) {
|
||||||
|
const fullClientPath = path.join(this.baseStorageDir, clientDir);
|
||||||
|
const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null);
|
||||||
|
if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const files = await fs.promises
|
||||||
|
.readdir(fullClientPath)
|
||||||
|
.catch(() => [] as string[]);
|
||||||
|
for (const file of files) {
|
||||||
|
if (!file.endsWith('.csv')) continue;
|
||||||
|
const datePart = this.extractDateFromFilename(file);
|
||||||
|
if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const fullPath = path.join(fullClientPath, file);
|
||||||
|
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
||||||
|
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
|
||||||
|
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
|
||||||
|
let acquired = false;
|
||||||
|
const tryAcquireLock = async (): Promise<boolean> => {
|
||||||
|
const handle = await fs.promises.open(lockPath, 'wx');
|
||||||
|
try {
|
||||||
|
await handle.writeFile(lockValue, 'utf8');
|
||||||
|
} finally {
|
||||||
|
await handle.close();
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
acquired = await tryAcquireLock();
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') {
|
||||||
|
await this.removeStaleLockAtomicallyAsync(lockPath);
|
||||||
|
try {
|
||||||
|
acquired = await tryAcquireLock();
|
||||||
|
} catch {
|
||||||
|
acquired = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!acquired) continue;
|
||||||
|
|
||||||
|
let targetGzPath = `${fullPath}.gz`;
|
||||||
|
if (fs.existsSync(targetGzPath)) {
|
||||||
|
const stem = fullPath.replace(/\.csv$/, '');
|
||||||
|
let suffix = 2;
|
||||||
|
while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) {
|
||||||
|
suffix += 1;
|
||||||
|
}
|
||||||
|
targetGzPath = `${stem}-gz${suffix}.csv.gz`;
|
||||||
|
}
|
||||||
|
const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
await pipeline(
|
||||||
|
fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }),
|
||||||
|
zlib.createGzip({ level: 6 }),
|
||||||
|
fs.createWriteStream(tmpGzPath),
|
||||||
|
);
|
||||||
|
await fs.promises
|
||||||
|
.utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime)
|
||||||
|
.catch(() => undefined);
|
||||||
|
await fs.promises.rename(tmpGzPath, targetGzPath);
|
||||||
|
await fs.promises.unlink(fullPath);
|
||||||
|
this.rowCountCache.delete(fullPath);
|
||||||
|
this.rowCountCache.delete(targetGzPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
|
compressedCount += 1;
|
||||||
|
} catch {
|
||||||
|
await fs.promises.unlink(tmpGzPath).catch(() => undefined);
|
||||||
|
} finally {
|
||||||
|
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Ignore directory scan failure
|
||||||
|
}
|
||||||
|
|
||||||
|
return compressedCount;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Prune CSV / CSV.GZ partitions older than autoPruneDays under cooperative partition locks
|
||||||
*/
|
*/
|
||||||
public static pruneExpiredPartitions(autoPruneDays: number): number {
|
public static pruneExpiredPartitions(autoPruneDays: number): number {
|
||||||
if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) {
|
if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) {
|
||||||
@@ -636,8 +831,8 @@ export class CSVStorageManager {
|
|||||||
|
|
||||||
const files = fs.readdirSync(fullClientPath);
|
const files = fs.readdirSync(fullClientPath);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
||||||
const fullPath = path.join(fullClientPath, file);
|
const fullPath = path.join(fullClientPath, file);
|
||||||
let fileLstat: fs.Stats;
|
let fileLstat: fs.Stats;
|
||||||
@@ -683,6 +878,7 @@ export class CSVStorageManager {
|
|||||||
if (fs.existsSync(fullPath)) {
|
if (fs.existsSync(fullPath)) {
|
||||||
fs.unlinkSync(fullPath);
|
fs.unlinkSync(fullPath);
|
||||||
this.rowCountCache.delete(fullPath);
|
this.rowCountCache.delete(fullPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
prunedCount += 1;
|
prunedCount += 1;
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
@@ -701,27 +897,54 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Non-blocking async partition pruner using fs.promises under cooperative partition locks
|
* Archive and/or prune expired partitions under cooperative partition locks.
|
||||||
|
* Never deletes a local partition if options.beforeDeleteFile throws or rejects (V2-E02).
|
||||||
*/
|
*/
|
||||||
public static async pruneExpiredPartitionsAsync(
|
public static async pruneOrArchiveExpiredPartitionsAsync(
|
||||||
autoPruneDays: number,
|
autoPruneDays: number,
|
||||||
): Promise<number> {
|
options?: {
|
||||||
if (!Number.isFinite(autoPruneDays) || autoPruneDays <= 0) {
|
beforeDeleteFile?: (file: {
|
||||||
return 0;
|
clientId: string;
|
||||||
|
filename: string;
|
||||||
|
filePath: string;
|
||||||
|
date: string;
|
||||||
|
}) => Promise<void>;
|
||||||
|
archiveOnlyWithoutDelete?: boolean;
|
||||||
|
maxCutoffDays?: number;
|
||||||
|
},
|
||||||
|
): Promise<{
|
||||||
|
scannedCount: number;
|
||||||
|
archivedCount: number;
|
||||||
|
prunedCount: number;
|
||||||
|
errors: string[];
|
||||||
|
}> {
|
||||||
|
const targetDays =
|
||||||
|
options?.maxCutoffDays !== undefined
|
||||||
|
? options.maxCutoffDays
|
||||||
|
: autoPruneDays;
|
||||||
|
if (!Number.isFinite(targetDays) || targetDays < 0) {
|
||||||
|
return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
|
||||||
|
}
|
||||||
|
if (options?.maxCutoffDays === undefined && targetDays <= 0) {
|
||||||
|
return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
|
||||||
}
|
}
|
||||||
|
|
||||||
const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null);
|
const baseStat = await fs.promises.lstat(this.baseStorageDir).catch(() => null);
|
||||||
if (!baseStat || !baseStat.isDirectory()) {
|
if (!baseStat || !baseStat.isDirectory()) {
|
||||||
return 0;
|
return { scannedCount: 0, archivedCount: 0, prunedCount: 0, errors: [] };
|
||||||
}
|
}
|
||||||
|
|
||||||
const cutoffDate = new Date(
|
const cutoffDate = new Date(
|
||||||
Date.now() - Math.floor(autoPruneDays) * 86_400_000,
|
Date.now() - Math.floor(targetDays) * 86_400_000,
|
||||||
)
|
)
|
||||||
.toISOString()
|
.toISOString()
|
||||||
.split('T')[0];
|
.split('T')[0];
|
||||||
|
|
||||||
|
let scannedCount = 0;
|
||||||
|
let archivedCount = 0;
|
||||||
let prunedCount = 0;
|
let prunedCount = 0;
|
||||||
|
const errors: string[] = [];
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const clientDirs = await fs.promises.readdir(this.baseStorageDir);
|
const clientDirs = await fs.promises.readdir(this.baseStorageDir);
|
||||||
for (const clientDir of clientDirs) {
|
for (const clientDir of clientDirs) {
|
||||||
@@ -731,54 +954,71 @@ export class CSVStorageManager {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
const files = await fs.promises.readdir(fullClientPath).catch(() => [] as string[]);
|
const files = await fs.promises
|
||||||
|
.readdir(fullClientPath)
|
||||||
|
.catch(() => [] as string[]);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) {
|
||||||
const fullPath = path.join(fullClientPath, file);
|
continue;
|
||||||
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
}
|
||||||
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) {
|
const fullPath = path.join(fullClientPath, file);
|
||||||
continue;
|
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
||||||
}
|
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
scannedCount += 1;
|
||||||
|
|
||||||
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
|
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
|
||||||
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
|
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
|
||||||
let acquired = false;
|
let acquired = false;
|
||||||
const tryAcquireLock = async (): Promise<boolean> => {
|
const tryAcquireLock = async (): Promise<boolean> => {
|
||||||
const handle = await fs.promises.open(lockPath, 'wx');
|
const handle = await fs.promises.open(lockPath, 'wx');
|
||||||
|
try {
|
||||||
|
await handle.writeFile(lockValue, 'utf8');
|
||||||
|
} finally {
|
||||||
|
await handle.close();
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
acquired = await tryAcquireLock();
|
||||||
|
} catch (err: unknown) {
|
||||||
|
if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') {
|
||||||
|
await this.removeStaleLockAtomicallyAsync(lockPath);
|
||||||
try {
|
try {
|
||||||
await handle.writeFile(lockValue, 'utf8');
|
acquired = await tryAcquireLock();
|
||||||
} finally {
|
} catch {
|
||||||
await handle.close();
|
acquired = false;
|
||||||
}
|
|
||||||
return true;
|
|
||||||
};
|
|
||||||
|
|
||||||
try {
|
|
||||||
acquired = await tryAcquireLock();
|
|
||||||
} catch (err: unknown) {
|
|
||||||
if ((err as NodeJS.ErrnoException)?.code === 'EEXIST') {
|
|
||||||
await this.removeStaleLockAtomicallyAsync(lockPath);
|
|
||||||
try {
|
|
||||||
acquired = await tryAcquireLock();
|
|
||||||
} catch {
|
|
||||||
acquired = false;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (!acquired) continue;
|
if (!acquired) continue;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
if (options?.beforeDeleteFile) {
|
||||||
|
await options.beforeDeleteFile({
|
||||||
|
clientId: clientDir,
|
||||||
|
filename: file,
|
||||||
|
filePath: fullPath,
|
||||||
|
date: datePart,
|
||||||
|
});
|
||||||
|
archivedCount += 1;
|
||||||
|
}
|
||||||
|
if (!options?.archiveOnlyWithoutDelete) {
|
||||||
await fs.promises.unlink(fullPath);
|
await fs.promises.unlink(fullPath);
|
||||||
this.rowCountCache.delete(fullPath);
|
this.rowCountCache.delete(fullPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
prunedCount += 1;
|
prunedCount += 1;
|
||||||
} catch {
|
|
||||||
// Ignore individual unlink error or concurrent removal
|
|
||||||
} finally {
|
|
||||||
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
|
|
||||||
}
|
}
|
||||||
|
} catch (err: unknown) {
|
||||||
|
const msg = err instanceof Error ? err.message : 'Archive/prune error';
|
||||||
|
errors.push(`${clientDir}/${file}: ${msg}`);
|
||||||
|
} finally {
|
||||||
|
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -786,7 +1026,149 @@ export class CSVStorageManager {
|
|||||||
// Ignore directory scan failure
|
// Ignore directory scan failure
|
||||||
}
|
}
|
||||||
|
|
||||||
return prunedCount;
|
return { scannedCount, archivedCount, prunedCount, errors };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Non-blocking async partition pruner using fs.promises under cooperative partition locks
|
||||||
|
*/
|
||||||
|
public static async pruneExpiredPartitionsAsync(
|
||||||
|
autoPruneDays: number,
|
||||||
|
): Promise<number> {
|
||||||
|
const res = await this.pruneOrArchiveExpiredPartitionsAsync(autoPruneDays);
|
||||||
|
return res.prunedCount;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cached VPS filesystem disk usage snapshot (30s TTL) for disk-full protection
|
||||||
|
*/
|
||||||
|
public static async getDiskUsageAsync(): Promise<DiskUsageSnapshot> {
|
||||||
|
const now = Date.now();
|
||||||
|
if (
|
||||||
|
this.diskUsageCache &&
|
||||||
|
now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS
|
||||||
|
) {
|
||||||
|
return this.diskUsageCache;
|
||||||
|
}
|
||||||
|
|
||||||
|
let probeDir = path.resolve(this.baseStorageDir);
|
||||||
|
while (!fs.existsSync(probeDir)) {
|
||||||
|
const parent = path.dirname(probeDir);
|
||||||
|
if (parent === probeDir) break;
|
||||||
|
probeDir = parent;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const statfsRes = this.statfsOverrideForTests
|
||||||
|
? await this.statfsOverrideForTests(probeDir)
|
||||||
|
: await fs.promises.statfs(probeDir);
|
||||||
|
const bsize = Math.max(1, Number(statfsRes.bsize) || 4096);
|
||||||
|
const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize);
|
||||||
|
const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize);
|
||||||
|
const usedBytes = Math.max(0, totalBytes - availableBytes);
|
||||||
|
const usedPct =
|
||||||
|
totalBytes > 0
|
||||||
|
? Number(
|
||||||
|
Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2),
|
||||||
|
)
|
||||||
|
: 0;
|
||||||
|
const snapshot: DiskUsageSnapshot = {
|
||||||
|
usedPct,
|
||||||
|
availableBytes,
|
||||||
|
totalBytes,
|
||||||
|
checkedAt: now,
|
||||||
|
};
|
||||||
|
this.diskUsageCache = snapshot;
|
||||||
|
return snapshot;
|
||||||
|
} catch {
|
||||||
|
const fallback: DiskUsageSnapshot = {
|
||||||
|
usedPct: 0,
|
||||||
|
availableBytes: 0,
|
||||||
|
totalBytes: 0,
|
||||||
|
checkedAt: now,
|
||||||
|
};
|
||||||
|
this.diskUsageCache = fallback;
|
||||||
|
return fallback;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cached per-client directory byte count (30s TTL)
|
||||||
|
*/
|
||||||
|
public static async getClientStorageBytesAsync(
|
||||||
|
clientId: string,
|
||||||
|
): Promise<number> {
|
||||||
|
const safeClientId = this.sanitizeClientId(clientId);
|
||||||
|
const now = Date.now();
|
||||||
|
const cached = this.clientBytesCache.get(safeClientId);
|
||||||
|
if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) {
|
||||||
|
return cached.bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
const resolvedBase = path.resolve(this.baseStorageDir);
|
||||||
|
const clientDir = path.resolve(resolvedBase, safeClientId);
|
||||||
|
if (!clientDir.startsWith(resolvedBase + path.sep)) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
let totalBytes = 0;
|
||||||
|
try {
|
||||||
|
const dirStat = await fs.promises.lstat(clientDir).catch(() => null);
|
||||||
|
if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) {
|
||||||
|
const entries = await fs.promises
|
||||||
|
.readdir(clientDir)
|
||||||
|
.catch(() => [] as string[]);
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (!this.isPartitionFilename(entry)) continue;
|
||||||
|
const filePath = path.join(clientDir, entry);
|
||||||
|
const stats = await fs.promises.lstat(filePath).catch(() => null);
|
||||||
|
if (stats && !stats.isSymbolicLink() && stats.isFile()) {
|
||||||
|
totalBytes += stats.size;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Fall back to 0 on unreadable directory
|
||||||
|
}
|
||||||
|
|
||||||
|
this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes });
|
||||||
|
return totalBytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Checks global VPS disk threshold and per-client MB quota before admitting new events
|
||||||
|
*/
|
||||||
|
public static async checkStorageAdmissionAsync(
|
||||||
|
clientId: string,
|
||||||
|
clientQuotaMb: number,
|
||||||
|
diskPauseThresholdPct: number,
|
||||||
|
): Promise<StorageAdmissionResult> {
|
||||||
|
const threshold = Number(diskPauseThresholdPct);
|
||||||
|
if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) {
|
||||||
|
const disk = await this.getDiskUsageAsync();
|
||||||
|
if (disk.totalBytes > 0 && disk.usedPct >= threshold) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'DISK_SPACE_LOW',
|
||||||
|
reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const quotaMb = Number(clientQuotaMb);
|
||||||
|
if (Number.isFinite(quotaMb) && quotaMb > 0) {
|
||||||
|
const currentBytes = await this.getClientStorageBytesAsync(clientId);
|
||||||
|
const quotaBytes = Math.floor(quotaMb) * 1024 * 1024;
|
||||||
|
if (currentBytes >= quotaBytes) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'STORAGE_QUOTA_EXCEEDED',
|
||||||
|
reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { allowed: true };
|
||||||
}
|
}
|
||||||
|
|
||||||
public static formatStorageStats(files: StorageFileMeta[]) {
|
public static formatStorageStats(files: StorageFileMeta[]) {
|
||||||
@@ -826,12 +1208,12 @@ export class CSVStorageManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Reads file content for export / download with path traversal protection
|
* Reads file content for export / download with path traversal protection (supports .csv & .csv.gz)
|
||||||
*/
|
*/
|
||||||
public static getFileContent(clientId: string, filename: string): string | null {
|
public static getFileContent(clientId: string, filename: string): string | null {
|
||||||
const safeClientId = this.sanitizeClientId(clientId);
|
const safeClientId = this.sanitizeClientId(clientId);
|
||||||
const safeFilename = path.basename(String(filename || ''));
|
const safeFilename = path.basename(String(filename || ''));
|
||||||
if (!safeClientId || !safeFilename || !safeFilename.endsWith('.csv')) {
|
if (!safeClientId || !safeFilename || !this.isPartitionFilename(safeFilename)) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
const resolvedBase = path.resolve(this.baseStorageDir);
|
const resolvedBase = path.resolve(this.baseStorageDir);
|
||||||
@@ -844,6 +1226,9 @@ export class CSVStorageManager {
|
|||||||
if (stats.isSymbolicLink() || !stats.isFile()) {
|
if (stats.isSymbolicLink() || !stats.isFile()) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
if (safeFilename.endsWith('.gz')) {
|
||||||
|
return zlib.gunzipSync(fs.readFileSync(fullPath)).toString('utf8');
|
||||||
|
}
|
||||||
return fs.readFileSync(fullPath, 'utf8');
|
return fs.readFileSync(fullPath, 'utf8');
|
||||||
}
|
}
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
@@ -0,0 +1,335 @@
|
|||||||
|
import crypto from "node:crypto";
|
||||||
|
import fs from "node:fs";
|
||||||
|
|
||||||
|
export interface S3ArchiveConfig {
|
||||||
|
endpoint: string;
|
||||||
|
region: string;
|
||||||
|
bucket: string;
|
||||||
|
prefix: string;
|
||||||
|
accessKeyId: string;
|
||||||
|
secretAccessKey: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface S3OperationResult {
|
||||||
|
ok: boolean;
|
||||||
|
status?: number;
|
||||||
|
endpoint?: string;
|
||||||
|
bucket?: string;
|
||||||
|
key?: string;
|
||||||
|
etag?: string;
|
||||||
|
code?: "S3_INVALID_CONFIG" | "S3_CONNECTION_FAILED" | "S3_ARCHIVE_FAILED";
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function sha256Hex(data: Buffer | string): string {
|
||||||
|
return crypto.createHash("sha256").update(data).digest("hex");
|
||||||
|
}
|
||||||
|
|
||||||
|
function hmacSha256(key: Buffer | string, data: string): Buffer {
|
||||||
|
return crypto.createHmac("sha256", key).update(data, "utf8").digest();
|
||||||
|
}
|
||||||
|
|
||||||
|
function encodeRfc3986(value: string): string {
|
||||||
|
return encodeURIComponent(value).replace(
|
||||||
|
/[!'()*]/g,
|
||||||
|
(ch) => `%${ch.charCodeAt(0).toString(16).toUpperCase()}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function encodeS3Path(rawPath: string): string {
|
||||||
|
return rawPath
|
||||||
|
.split("/")
|
||||||
|
.map((segment) => encodeRfc3986(segment))
|
||||||
|
.join("/");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildS3ObjectKey(
|
||||||
|
prefix: string,
|
||||||
|
clientId: string,
|
||||||
|
filename: string,
|
||||||
|
): string {
|
||||||
|
const cleanPrefix = String(prefix || "")
|
||||||
|
.trim()
|
||||||
|
.replace(/^\/+|\/+$/g, "");
|
||||||
|
const cleanClient = String(clientId || "unknown")
|
||||||
|
.trim()
|
||||||
|
.replace(/^\/+|\/+$/g, "");
|
||||||
|
const cleanFile = String(filename || "")
|
||||||
|
.trim()
|
||||||
|
.replace(/^\/+/, "");
|
||||||
|
return [cleanPrefix, cleanClient, cleanFile].filter(Boolean).join("/");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeS3EndpointUrl(
|
||||||
|
endpoint: string,
|
||||||
|
region: string,
|
||||||
|
): { url: URL; region: string } {
|
||||||
|
const trimmedRegion = String(region || "").trim() || "auto";
|
||||||
|
const trimmedEndpoint = String(endpoint || "").trim().replace(/\/+$/, "");
|
||||||
|
|
||||||
|
if (trimmedEndpoint) {
|
||||||
|
const withProtocol = /^https?:\/\//i.test(trimmedEndpoint)
|
||||||
|
? trimmedEndpoint
|
||||||
|
: `https://${trimmedEndpoint}`;
|
||||||
|
const parsed = new URL(withProtocol);
|
||||||
|
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
|
||||||
|
throw new Error("S3 endpoint must use http:// or https://");
|
||||||
|
}
|
||||||
|
return { url: parsed, region: trimmedRegion };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (trimmedRegion === "auto") {
|
||||||
|
throw new Error(
|
||||||
|
"S3 Endpoint URL is required when Region is set to 'auto' (e.g. Cloudflare R2 or MinIO).",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
url: new URL(`https://s3.${trimmedRegion}.amazonaws.com`),
|
||||||
|
region: trimmedRegion,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function signS3Request(params: {
|
||||||
|
method: "GET" | "HEAD" | "PUT";
|
||||||
|
endpointUrl: URL;
|
||||||
|
bucket: string;
|
||||||
|
objectKey?: string;
|
||||||
|
queryParams?: Record<string, string>;
|
||||||
|
region: string;
|
||||||
|
accessKeyId: string;
|
||||||
|
secretAccessKey: string;
|
||||||
|
payloadHashHex: string;
|
||||||
|
contentType?: string;
|
||||||
|
now?: Date;
|
||||||
|
}): { targetUrl: string; headers: Record<string, string> } {
|
||||||
|
const now = params.now ?? new Date();
|
||||||
|
const amzDate = now.toISOString().replace(/[:-]|\.\d{3}/g, "");
|
||||||
|
const dateStamp = amzDate.slice(0, 8);
|
||||||
|
|
||||||
|
const basePath = params.endpointUrl.pathname.replace(/\/+$/, "");
|
||||||
|
const bucketPart = encodeRfc3986(params.bucket.trim());
|
||||||
|
const keyPart = params.objectKey ? `/${encodeS3Path(params.objectKey)}` : "";
|
||||||
|
const canonicalUri = `${basePath}/${bucketPart}${keyPart}`.replace(
|
||||||
|
/\/{2,}/g,
|
||||||
|
"/",
|
||||||
|
);
|
||||||
|
|
||||||
|
const queryEntries = Object.entries(params.queryParams ?? {}).sort(([a], [b]) =>
|
||||||
|
a < b ? -1 : a > b ? 1 : 0,
|
||||||
|
);
|
||||||
|
const canonicalQueryString = queryEntries
|
||||||
|
.map(([k, v]) => `${encodeRfc3986(k)}=${encodeRfc3986(v)}`)
|
||||||
|
.join("&");
|
||||||
|
|
||||||
|
const hostHeader = params.endpointUrl.host;
|
||||||
|
const headerMap: Record<string, string> = {
|
||||||
|
host: hostHeader,
|
||||||
|
"x-amz-content-sha256": params.payloadHashHex,
|
||||||
|
"x-amz-date": amzDate,
|
||||||
|
};
|
||||||
|
if (params.contentType) {
|
||||||
|
headerMap["content-type"] = params.contentType;
|
||||||
|
}
|
||||||
|
|
||||||
|
const sortedHeaderKeys = Object.keys(headerMap).sort();
|
||||||
|
const canonicalHeaders = sortedHeaderKeys
|
||||||
|
.map((key) => `${key}:${headerMap[key].trim()}\n`)
|
||||||
|
.join("");
|
||||||
|
const signedHeaders = sortedHeaderKeys.join(";");
|
||||||
|
|
||||||
|
const canonicalRequest = [
|
||||||
|
params.method,
|
||||||
|
canonicalUri,
|
||||||
|
canonicalQueryString,
|
||||||
|
canonicalHeaders,
|
||||||
|
signedHeaders,
|
||||||
|
params.payloadHashHex,
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
const credentialScope = `${dateStamp}/${params.region}/s3/aws4_request`;
|
||||||
|
const stringToSign = [
|
||||||
|
"AWS4-HMAC-SHA256",
|
||||||
|
amzDate,
|
||||||
|
credentialScope,
|
||||||
|
sha256Hex(canonicalRequest),
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
const kDate = hmacSha256(`AWS4${params.secretAccessKey}`, dateStamp);
|
||||||
|
const kRegion = hmacSha256(kDate, params.region);
|
||||||
|
const kService = hmacSha256(kRegion, "s3");
|
||||||
|
const kSigning = hmacSha256(kService, "aws4_request");
|
||||||
|
const signature = crypto
|
||||||
|
.createHmac("sha256", kSigning)
|
||||||
|
.update(stringToSign, "utf8")
|
||||||
|
.digest("hex");
|
||||||
|
|
||||||
|
const authorization = `AWS4-HMAC-SHA256 Credential=${params.accessKeyId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}`;
|
||||||
|
|
||||||
|
const targetUrl = `${params.endpointUrl.origin}${canonicalUri}${
|
||||||
|
canonicalQueryString ? `?${canonicalQueryString}` : ""
|
||||||
|
}`;
|
||||||
|
|
||||||
|
return {
|
||||||
|
targetUrl,
|
||||||
|
headers: {
|
||||||
|
"x-amz-date": amzDate,
|
||||||
|
"x-amz-content-sha256": params.payloadHashHex,
|
||||||
|
...(params.contentType ? { "content-type": params.contentType } : {}),
|
||||||
|
Authorization: authorization,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function validateS3Config(config: S3ArchiveConfig): string | null {
|
||||||
|
if (!config.bucket?.trim()) return "S3 Bucket name is required.";
|
||||||
|
if (!config.accessKeyId?.trim()) return "S3 Access Key ID is required.";
|
||||||
|
if (!config.secretAccessKey?.trim()) return "S3 Secret Access Key is required.";
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function testS3Connection(
|
||||||
|
config: S3ArchiveConfig,
|
||||||
|
): Promise<S3OperationResult> {
|
||||||
|
const validationError = validateS3Config(config);
|
||||||
|
if (validationError) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
code: "S3_INVALID_CONFIG",
|
||||||
|
error: validationError,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const { url, region } = normalizeS3EndpointUrl(
|
||||||
|
config.endpoint,
|
||||||
|
config.region,
|
||||||
|
);
|
||||||
|
const signed = signS3Request({
|
||||||
|
method: "GET",
|
||||||
|
endpointUrl: url,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
queryParams: { "list-type": "2", "max-keys": "1" },
|
||||||
|
region,
|
||||||
|
accessKeyId: config.accessKeyId.trim(),
|
||||||
|
secretAccessKey: config.secretAccessKey,
|
||||||
|
payloadHashHex: sha256Hex(""),
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await fetch(signed.targetUrl, {
|
||||||
|
method: "GET",
|
||||||
|
headers: signed.headers,
|
||||||
|
cache: "no-store",
|
||||||
|
signal: AbortSignal.timeout(8000),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const bodyPreview = await response.text().catch(() => "");
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
status: response.status,
|
||||||
|
endpoint: url.origin,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
code: "S3_CONNECTION_FAILED",
|
||||||
|
error: `S3 endpoint returned HTTP ${response.status}${
|
||||||
|
bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : ""
|
||||||
|
}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
ok: true,
|
||||||
|
status: response.status,
|
||||||
|
endpoint: url.origin,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
};
|
||||||
|
} catch (err: unknown) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
code: "S3_CONNECTION_FAILED",
|
||||||
|
error:
|
||||||
|
err instanceof Error ? err.message : "Unable to reach S3 endpoint",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function uploadFileToS3(
|
||||||
|
config: S3ArchiveConfig,
|
||||||
|
localFilePath: string,
|
||||||
|
objectKey: string,
|
||||||
|
): Promise<S3OperationResult> {
|
||||||
|
const validationError = validateS3Config(config);
|
||||||
|
if (validationError) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
code: "S3_INVALID_CONFIG",
|
||||||
|
error: validationError,
|
||||||
|
key: objectKey,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const { url, region } = normalizeS3EndpointUrl(
|
||||||
|
config.endpoint,
|
||||||
|
config.region,
|
||||||
|
);
|
||||||
|
const bodyBuffer = await fs.promises.readFile(localFilePath);
|
||||||
|
const payloadHashHex = sha256Hex(bodyBuffer);
|
||||||
|
const contentType = localFilePath.endsWith(".gz")
|
||||||
|
? "application/gzip"
|
||||||
|
: "text/csv; charset=utf-8";
|
||||||
|
|
||||||
|
const signed = signS3Request({
|
||||||
|
method: "PUT",
|
||||||
|
endpointUrl: url,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
objectKey,
|
||||||
|
region,
|
||||||
|
accessKeyId: config.accessKeyId.trim(),
|
||||||
|
secretAccessKey: config.secretAccessKey,
|
||||||
|
payloadHashHex,
|
||||||
|
contentType,
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await fetch(signed.targetUrl, {
|
||||||
|
method: "PUT",
|
||||||
|
headers: {
|
||||||
|
...signed.headers,
|
||||||
|
"content-length": String(bodyBuffer.byteLength),
|
||||||
|
},
|
||||||
|
body: new Uint8Array(bodyBuffer),
|
||||||
|
cache: "no-store",
|
||||||
|
signal: AbortSignal.timeout(30000),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const bodyPreview = await response.text().catch(() => "");
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
status: response.status,
|
||||||
|
endpoint: url.origin,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
key: objectKey,
|
||||||
|
code: "S3_ARCHIVE_FAILED",
|
||||||
|
error: `S3 upload failed with HTTP ${response.status}${
|
||||||
|
bodyPreview ? `: ${bodyPreview.slice(0, 180)}` : ""
|
||||||
|
}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
ok: true,
|
||||||
|
status: response.status,
|
||||||
|
endpoint: url.origin,
|
||||||
|
bucket: config.bucket.trim(),
|
||||||
|
key: objectKey,
|
||||||
|
etag: response.headers.get("etag") ?? undefined,
|
||||||
|
};
|
||||||
|
} catch (err: unknown) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
key: objectKey,
|
||||||
|
code: "S3_ARCHIVE_FAILED",
|
||||||
|
error: err instanceof Error ? err.message : "S3 upload request failed",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,12 +1,27 @@
|
|||||||
|
import {
|
||||||
|
dispatchWebhookAlertAsync,
|
||||||
|
type WebhookAlertPayload,
|
||||||
|
type WebhookRuntimeConfig,
|
||||||
|
} from "../alerts/webhookNotifier";
|
||||||
import { AuthService, resolveDefaultJwtSecret } from "../auth/jwt";
|
import { AuthService, resolveDefaultJwtSecret } from "../auth/jwt";
|
||||||
import { DataBufferEngine } from "../data/buffer";
|
import { DataBufferEngine } from "../data/buffer";
|
||||||
import { DatabaseEngine } from "../Database/database";
|
import { DatabaseEngine } from "../Database/database";
|
||||||
import { SDKManager } from "../sdk/sdkManager";
|
import { SDKManager } from "../sdk/sdkManager";
|
||||||
|
import {
|
||||||
|
decryptSecret,
|
||||||
|
encryptSecret,
|
||||||
|
getMaskedHintFromEncrypted,
|
||||||
|
} from "../Security/secretVault";
|
||||||
import {
|
import {
|
||||||
SecurityEngine,
|
SecurityEngine,
|
||||||
getInternalIngestionHeaders,
|
getInternalIngestionHeaders,
|
||||||
} from "../Security/security";
|
} from "../Security/security";
|
||||||
import { CSVStorageManager } from "../Storage/csvStorage";
|
import { CSVStorageManager } from "../Storage/csvStorage";
|
||||||
|
import {
|
||||||
|
buildS3ObjectKey,
|
||||||
|
type S3ArchiveConfig,
|
||||||
|
uploadFileToS3,
|
||||||
|
} from "../Storage/s3Archive";
|
||||||
|
|
||||||
export interface AdminSettings {
|
export interface AdminSettings {
|
||||||
serverPort: number;
|
serverPort: number;
|
||||||
@@ -19,9 +34,49 @@ export interface AdminSettings {
|
|||||||
jwtSecret: string;
|
jwtSecret: string;
|
||||||
enableIdempotency: boolean;
|
enableIdempotency: boolean;
|
||||||
autoPruneDays: number;
|
autoPruneDays: number;
|
||||||
|
compressAfterDays: number;
|
||||||
|
enableS3Backup: boolean;
|
||||||
|
s3Endpoint: string;
|
||||||
|
s3Region: string;
|
||||||
|
s3Bucket: string;
|
||||||
|
s3Prefix: string;
|
||||||
|
s3AccessKeyId: string;
|
||||||
|
s3SecretAccessKeyEnc: string;
|
||||||
|
diskPauseThresholdPct: number;
|
||||||
|
enableWebhookAlerts: boolean;
|
||||||
|
webhookUrl: string;
|
||||||
|
webhookSecretEnc: string;
|
||||||
|
webhookCooldownSec: number;
|
||||||
|
enablePrometheusMetrics: boolean;
|
||||||
|
prometheusMetricsTokenEnc: string;
|
||||||
updatedAt: string;
|
updatedAt: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface PublicAdminSettings
|
||||||
|
extends Omit<
|
||||||
|
AdminSettings,
|
||||||
|
| "jwtSecret"
|
||||||
|
| "s3SecretAccessKeyEnc"
|
||||||
|
| "webhookSecretEnc"
|
||||||
|
| "prometheusMetricsTokenEnc"
|
||||||
|
> {
|
||||||
|
hasS3SecretKey: boolean;
|
||||||
|
s3SecretKeyMask: string;
|
||||||
|
hasWebhookSecret: boolean;
|
||||||
|
webhookSecretMask: string;
|
||||||
|
hasPrometheusToken: boolean;
|
||||||
|
prometheusTokenMask: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type AdminSettingsUpdateInput = Partial<AdminSettings> & {
|
||||||
|
s3SecretAccessKey?: string;
|
||||||
|
clearS3SecretKey?: boolean;
|
||||||
|
webhookSecret?: string;
|
||||||
|
clearWebhookSecret?: boolean;
|
||||||
|
prometheusMetricsToken?: string;
|
||||||
|
clearPrometheusToken?: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
export interface SettingsBridgeSyncStatus {
|
export interface SettingsBridgeSyncStatus {
|
||||||
attempted: boolean;
|
attempted: boolean;
|
||||||
ok: boolean;
|
ok: boolean;
|
||||||
@@ -36,9 +91,11 @@ type SettingsGlobals = typeof globalThis & {
|
|||||||
__dataStackSettingsHydrated?: boolean;
|
__dataStackSettingsHydrated?: boolean;
|
||||||
__dataStackLastSettingsBridgeSync?: SettingsBridgeSyncStatus;
|
__dataStackLastSettingsBridgeSync?: SettingsBridgeSyncStatus;
|
||||||
__dataStackLastPruneMeta?: { days: number; ranAtMs: number };
|
__dataStackLastPruneMeta?: { days: number; ranAtMs: number };
|
||||||
|
__dataStackLastCompressMeta?: { days: number; ranAtMs: number };
|
||||||
};
|
};
|
||||||
|
|
||||||
const PRUNE_COOLDOWN_MS = 60 * 60 * 1000;
|
const PRUNE_COOLDOWN_MS = 60 * 60 * 1000;
|
||||||
|
const COMPRESS_COOLDOWN_MS = 60 * 60 * 1000;
|
||||||
const globals = globalThis as SettingsGlobals;
|
const globals = globalThis as SettingsGlobals;
|
||||||
|
|
||||||
function getDefaultSettings(): AdminSettings {
|
function getDefaultSettings(): AdminSettings {
|
||||||
@@ -53,6 +110,21 @@ function getDefaultSettings(): AdminSettings {
|
|||||||
jwtSecret: resolveDefaultJwtSecret(),
|
jwtSecret: resolveDefaultJwtSecret(),
|
||||||
enableIdempotency: true,
|
enableIdempotency: true,
|
||||||
autoPruneDays: 30,
|
autoPruneDays: 30,
|
||||||
|
compressAfterDays: 0,
|
||||||
|
enableS3Backup: false,
|
||||||
|
s3Endpoint: "",
|
||||||
|
s3Region: "auto",
|
||||||
|
s3Bucket: "",
|
||||||
|
s3Prefix: "datastack/",
|
||||||
|
s3AccessKeyId: "",
|
||||||
|
s3SecretAccessKeyEnc: "",
|
||||||
|
diskPauseThresholdPct: 95,
|
||||||
|
enableWebhookAlerts: false,
|
||||||
|
webhookUrl: "",
|
||||||
|
webhookSecretEnc: "",
|
||||||
|
webhookCooldownSec: 300,
|
||||||
|
enablePrometheusMetrics: false,
|
||||||
|
prometheusMetricsTokenEnc: "",
|
||||||
updatedAt: new Date().toISOString(),
|
updatedAt: new Date().toISOString(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -64,8 +136,132 @@ function getSettingsState(): AdminSettings {
|
|||||||
return globals.__dataStackAdminSettings;
|
return globals.__dataStackAdminSettings;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function toPublicSettings(settings: AdminSettings): PublicAdminSettings {
|
||||||
|
const s3SecretKeyMask = getMaskedHintFromEncrypted(
|
||||||
|
settings.s3SecretAccessKeyEnc,
|
||||||
|
);
|
||||||
|
const webhookSecretMask = getMaskedHintFromEncrypted(
|
||||||
|
settings.webhookSecretEnc,
|
||||||
|
);
|
||||||
|
const prometheusTokenMask = getMaskedHintFromEncrypted(
|
||||||
|
settings.prometheusMetricsTokenEnc,
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
serverPort: settings.serverPort,
|
||||||
|
webPort: settings.webPort,
|
||||||
|
maxEventsPerSec: settings.maxEventsPerSec,
|
||||||
|
maxPayloadSizeKb: settings.maxPayloadSizeKb,
|
||||||
|
batchFlushThreshold: settings.batchFlushThreshold,
|
||||||
|
batchFlushIntervalMs: settings.batchFlushIntervalMs,
|
||||||
|
fileRotationMb: settings.fileRotationMb,
|
||||||
|
enableIdempotency: settings.enableIdempotency,
|
||||||
|
autoPruneDays: settings.autoPruneDays,
|
||||||
|
compressAfterDays: settings.compressAfterDays,
|
||||||
|
enableS3Backup: settings.enableS3Backup,
|
||||||
|
s3Endpoint: settings.s3Endpoint,
|
||||||
|
s3Region: settings.s3Region,
|
||||||
|
s3Bucket: settings.s3Bucket,
|
||||||
|
s3Prefix: settings.s3Prefix,
|
||||||
|
s3AccessKeyId: settings.s3AccessKeyId,
|
||||||
|
hasS3SecretKey: Boolean(settings.s3SecretAccessKeyEnc),
|
||||||
|
s3SecretKeyMask,
|
||||||
|
diskPauseThresholdPct: settings.diskPauseThresholdPct,
|
||||||
|
enableWebhookAlerts: settings.enableWebhookAlerts,
|
||||||
|
webhookUrl: settings.webhookUrl,
|
||||||
|
hasWebhookSecret: Boolean(settings.webhookSecretEnc),
|
||||||
|
webhookSecretMask,
|
||||||
|
webhookCooldownSec: settings.webhookCooldownSec,
|
||||||
|
enablePrometheusMetrics: settings.enablePrometheusMetrics,
|
||||||
|
hasPrometheusToken: Boolean(settings.prometheusMetricsTokenEnc),
|
||||||
|
prometheusTokenMask,
|
||||||
|
updatedAt: settings.updatedAt,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export class SettingsManager {
|
export class SettingsManager {
|
||||||
private static applyRuntimeSettings(settings: AdminSettings, forcePrune = false) {
|
public static triggerWebhookAlert(payload: WebhookAlertPayload): void {
|
||||||
|
const current = getSettingsState();
|
||||||
|
if (!current.enableWebhookAlerts || !current.webhookUrl.trim()) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
void (async () => {
|
||||||
|
try {
|
||||||
|
let secret = "";
|
||||||
|
if (current.webhookSecretEnc) {
|
||||||
|
try {
|
||||||
|
secret = decryptSecret(current.webhookSecretEnc);
|
||||||
|
} catch {
|
||||||
|
secret = "";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await dispatchWebhookAlertAsync(payload, {
|
||||||
|
enabled: current.enableWebhookAlerts,
|
||||||
|
webhookUrl: current.webhookUrl,
|
||||||
|
webhookSecret: secret,
|
||||||
|
cooldownSec: current.webhookCooldownSec,
|
||||||
|
});
|
||||||
|
} catch {
|
||||||
|
// Never allow background webhook dispatch to throw into callers
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async executeArchiveAndPrune(
|
||||||
|
settings: AdminSettings,
|
||||||
|
): Promise<void> {
|
||||||
|
if (!settings.enableS3Backup) {
|
||||||
|
await CSVStorageManager.pruneExpiredPartitionsAsync(
|
||||||
|
settings.autoPruneDays,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const s3Config: S3ArchiveConfig = {
|
||||||
|
endpoint: settings.s3Endpoint,
|
||||||
|
region: settings.s3Region,
|
||||||
|
bucket: settings.s3Bucket,
|
||||||
|
prefix: settings.s3Prefix,
|
||||||
|
accessKeyId: settings.s3AccessKeyId,
|
||||||
|
secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc),
|
||||||
|
};
|
||||||
|
|
||||||
|
await CSVStorageManager.pruneOrArchiveExpiredPartitionsAsync(
|
||||||
|
settings.autoPruneDays,
|
||||||
|
{
|
||||||
|
beforeDeleteFile: async (file) => {
|
||||||
|
const objectKey = buildS3ObjectKey(
|
||||||
|
s3Config.prefix,
|
||||||
|
file.clientId,
|
||||||
|
file.filename,
|
||||||
|
);
|
||||||
|
const result = await uploadFileToS3(
|
||||||
|
s3Config,
|
||||||
|
file.filePath,
|
||||||
|
objectKey,
|
||||||
|
);
|
||||||
|
if (!result.ok) {
|
||||||
|
const errMsg =
|
||||||
|
result.error || `Failed to upload ${file.filename} to S3`;
|
||||||
|
this.triggerWebhookAlert({
|
||||||
|
event: "S3_ARCHIVE_FAILED",
|
||||||
|
severity: "critical",
|
||||||
|
title: "Off-Site S3 Archive Upload Failed",
|
||||||
|
message: errMsg,
|
||||||
|
clientId: file.clientId,
|
||||||
|
details: { filename: file.filename, bucket: s3Config.bucket },
|
||||||
|
});
|
||||||
|
throw new Error(errMsg);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static applyRuntimeSettings(
|
||||||
|
settings: AdminSettings,
|
||||||
|
forcePrune = false,
|
||||||
|
forceCompress = false,
|
||||||
|
) {
|
||||||
SecurityEngine.maxEventsPerSec = settings.maxEventsPerSec;
|
SecurityEngine.maxEventsPerSec = settings.maxEventsPerSec;
|
||||||
SecurityEngine.enableIdempotency = settings.enableIdempotency;
|
SecurityEngine.enableIdempotency = settings.enableIdempotency;
|
||||||
SecurityEngine.maxPayloadSizeBytes = settings.maxPayloadSizeKb * 1024;
|
SecurityEngine.maxPayloadSizeBytes = settings.maxPayloadSizeKb * 1024;
|
||||||
@@ -76,10 +272,34 @@ export class SettingsManager {
|
|||||||
}
|
}
|
||||||
CSVStorageManager.maxFileSizeBytes = settings.fileRotationMb * 1024 * 1024;
|
CSVStorageManager.maxFileSizeBytes = settings.fileRotationMb * 1024 * 1024;
|
||||||
const hasConfiguredDb = Boolean(process.env.DATABASE_URL?.trim());
|
const hasConfiguredDb = Boolean(process.env.DATABASE_URL?.trim());
|
||||||
if (
|
const dbReadyForMaintenance =
|
||||||
settings.autoPruneDays > 0 &&
|
!hasConfiguredDb || globals.__dataStackSettingsHydrated === true;
|
||||||
(!hasConfiguredDb || globals.__dataStackSettingsHydrated === true)
|
|
||||||
) {
|
if (settings.compressAfterDays > 0 && dbReadyForMaintenance) {
|
||||||
|
const prevComp = globals.__dataStackLastCompressMeta;
|
||||||
|
const now = Date.now();
|
||||||
|
const shouldCompress =
|
||||||
|
forceCompress ||
|
||||||
|
!prevComp ||
|
||||||
|
prevComp.days !== settings.compressAfterDays ||
|
||||||
|
now - prevComp.ranAtMs >= COMPRESS_COOLDOWN_MS;
|
||||||
|
if (shouldCompress) {
|
||||||
|
globals.__dataStackLastCompressMeta = {
|
||||||
|
days: settings.compressAfterDays,
|
||||||
|
ranAtMs: now,
|
||||||
|
};
|
||||||
|
void CSVStorageManager.compressOldPartitionsAsync(
|
||||||
|
settings.compressAfterDays,
|
||||||
|
).catch((err) => {
|
||||||
|
console.warn(
|
||||||
|
"[DataStack Storage] Background partition compression failed:",
|
||||||
|
err,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (settings.autoPruneDays > 0 && dbReadyForMaintenance) {
|
||||||
const prev = globals.__dataStackLastPruneMeta;
|
const prev = globals.__dataStackLastPruneMeta;
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
const shouldPrune =
|
const shouldPrune =
|
||||||
@@ -92,11 +312,9 @@ export class SettingsManager {
|
|||||||
days: settings.autoPruneDays,
|
days: settings.autoPruneDays,
|
||||||
ranAtMs: now,
|
ranAtMs: now,
|
||||||
};
|
};
|
||||||
void CSVStorageManager.pruneExpiredPartitionsAsync(
|
void this.executeArchiveAndPrune(settings).catch((err) => {
|
||||||
settings.autoPruneDays,
|
|
||||||
).catch((err) => {
|
|
||||||
console.warn(
|
console.warn(
|
||||||
"[DataStack Storage] Background async partition prune failed:",
|
"[DataStack Storage] Background async partition prune/archive failed:",
|
||||||
err,
|
err,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
@@ -115,7 +333,12 @@ export class SettingsManager {
|
|||||||
const result = await pool.query(
|
const result = await pool.query(
|
||||||
`SELECT server_port, web_port, max_events_per_sec, max_payload_size_kb,
|
`SELECT server_port, web_port, max_events_per_sec, max_payload_size_kb,
|
||||||
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
|
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
|
||||||
jwt_secret, enable_idempotency, auto_prune_days, updated_at
|
jwt_secret, enable_idempotency, auto_prune_days,
|
||||||
|
compress_after_days, enable_s3_backup, s3_endpoint, s3_region,
|
||||||
|
s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc,
|
||||||
|
disk_pause_threshold_pct, enable_webhook_alerts, webhook_url,
|
||||||
|
webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics,
|
||||||
|
prometheus_metrics_token_enc, updated_at
|
||||||
FROM admin_settings
|
FROM admin_settings
|
||||||
WHERE id = 'default'`,
|
WHERE id = 'default'`,
|
||||||
);
|
);
|
||||||
@@ -126,6 +349,21 @@ export class SettingsManager {
|
|||||||
row.auto_prune_days !== undefined && row.auto_prune_days !== null
|
row.auto_prune_days !== undefined && row.auto_prune_days !== null
|
||||||
? Number(row.auto_prune_days)
|
? Number(row.auto_prune_days)
|
||||||
: NaN;
|
: NaN;
|
||||||
|
const parsedCompressDays =
|
||||||
|
row.compress_after_days !== undefined &&
|
||||||
|
row.compress_after_days !== null
|
||||||
|
? Number(row.compress_after_days)
|
||||||
|
: NaN;
|
||||||
|
const parsedDiskThreshold =
|
||||||
|
row.disk_pause_threshold_pct !== undefined &&
|
||||||
|
row.disk_pause_threshold_pct !== null
|
||||||
|
? Number(row.disk_pause_threshold_pct)
|
||||||
|
: NaN;
|
||||||
|
const parsedWebhookCooldown =
|
||||||
|
row.webhook_cooldown_sec !== undefined &&
|
||||||
|
row.webhook_cooldown_sec !== null
|
||||||
|
? Number(row.webhook_cooldown_sec)
|
||||||
|
: NaN;
|
||||||
const envSocketPort = Number(process.env.PORT_SOCKET);
|
const envSocketPort = Number(process.env.PORT_SOCKET);
|
||||||
const envWebPort = Number(process.env.PORT);
|
const envWebPort = Number(process.env.PORT);
|
||||||
const loaded: AdminSettings = {
|
const loaded: AdminSettings = {
|
||||||
@@ -142,9 +380,34 @@ export class SettingsManager {
|
|||||||
batchFlushThreshold: Number(row.batch_flush_threshold) || 500,
|
batchFlushThreshold: Number(row.batch_flush_threshold) || 500,
|
||||||
batchFlushIntervalMs: Number(row.batch_flush_interval_ms) || 2000,
|
batchFlushIntervalMs: Number(row.batch_flush_interval_ms) || 2000,
|
||||||
fileRotationMb: Number(row.file_rotation_mb) || 100,
|
fileRotationMb: Number(row.file_rotation_mb) || 100,
|
||||||
jwtSecret: String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(),
|
jwtSecret:
|
||||||
|
String(row.jwt_secret || "").trim() || resolveDefaultJwtSecret(),
|
||||||
enableIdempotency: Boolean(row.enable_idempotency),
|
enableIdempotency: Boolean(row.enable_idempotency),
|
||||||
autoPruneDays: Number.isFinite(parsedPruneDays) ? parsedPruneDays : 30,
|
autoPruneDays: Number.isFinite(parsedPruneDays) ? parsedPruneDays : 30,
|
||||||
|
compressAfterDays: Number.isFinite(parsedCompressDays)
|
||||||
|
? Math.max(0, parsedCompressDays)
|
||||||
|
: 0,
|
||||||
|
enableS3Backup: Boolean(row.enable_s3_backup),
|
||||||
|
s3Endpoint: String(row.s3_endpoint ?? ""),
|
||||||
|
s3Region: String(row.s3_region || "auto"),
|
||||||
|
s3Bucket: String(row.s3_bucket ?? ""),
|
||||||
|
s3Prefix: String(row.s3_prefix ?? "datastack/"),
|
||||||
|
s3AccessKeyId: String(row.s3_access_key_id ?? ""),
|
||||||
|
s3SecretAccessKeyEnc: String(row.s3_secret_access_key_enc ?? ""),
|
||||||
|
diskPauseThresholdPct: Number.isFinite(parsedDiskThreshold)
|
||||||
|
? parsedDiskThreshold
|
||||||
|
: 95,
|
||||||
|
enableWebhookAlerts: Boolean(row.enable_webhook_alerts),
|
||||||
|
webhookUrl: String(row.webhook_url ?? ""),
|
||||||
|
webhookSecretEnc: String(row.webhook_secret_enc ?? ""),
|
||||||
|
webhookCooldownSec:
|
||||||
|
Number.isFinite(parsedWebhookCooldown) && parsedWebhookCooldown >= 10
|
||||||
|
? Math.floor(parsedWebhookCooldown)
|
||||||
|
: 300,
|
||||||
|
enablePrometheusMetrics: Boolean(row.enable_prometheus_metrics),
|
||||||
|
prometheusMetricsTokenEnc: String(
|
||||||
|
row.prometheus_metrics_token_enc ?? "",
|
||||||
|
),
|
||||||
updatedAt: row.updated_at
|
updatedAt: row.updated_at
|
||||||
? new Date(row.updated_at as string | Date).toISOString()
|
? new Date(row.updated_at as string | Date).toISOString()
|
||||||
: new Date().toISOString(),
|
: new Date().toISOString(),
|
||||||
@@ -198,8 +461,17 @@ export class SettingsManager {
|
|||||||
`INSERT INTO admin_settings (
|
`INSERT INTO admin_settings (
|
||||||
id, server_port, web_port, max_events_per_sec, max_payload_size_kb,
|
id, server_port, web_port, max_events_per_sec, max_payload_size_kb,
|
||||||
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
|
batch_flush_threshold, batch_flush_interval_ms, file_rotation_mb,
|
||||||
jwt_secret, enable_idempotency, auto_prune_days, updated_at
|
jwt_secret, enable_idempotency, auto_prune_days,
|
||||||
) VALUES ('default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, NOW())
|
compress_after_days, enable_s3_backup, s3_endpoint, s3_region,
|
||||||
|
s3_bucket, s3_prefix, s3_access_key_id, s3_secret_access_key_enc,
|
||||||
|
disk_pause_threshold_pct, enable_webhook_alerts, webhook_url,
|
||||||
|
webhook_secret_enc, webhook_cooldown_sec, enable_prometheus_metrics,
|
||||||
|
prometheus_metrics_token_enc, updated_at
|
||||||
|
) VALUES (
|
||||||
|
'default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
|
||||||
|
$11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21,
|
||||||
|
$22, $23, $24, $25, NOW()
|
||||||
|
)
|
||||||
ON CONFLICT (id) DO UPDATE SET
|
ON CONFLICT (id) DO UPDATE SET
|
||||||
server_port = EXCLUDED.server_port,
|
server_port = EXCLUDED.server_port,
|
||||||
web_port = EXCLUDED.web_port,
|
web_port = EXCLUDED.web_port,
|
||||||
@@ -211,6 +483,21 @@ export class SettingsManager {
|
|||||||
jwt_secret = EXCLUDED.jwt_secret,
|
jwt_secret = EXCLUDED.jwt_secret,
|
||||||
enable_idempotency = EXCLUDED.enable_idempotency,
|
enable_idempotency = EXCLUDED.enable_idempotency,
|
||||||
auto_prune_days = EXCLUDED.auto_prune_days,
|
auto_prune_days = EXCLUDED.auto_prune_days,
|
||||||
|
compress_after_days = EXCLUDED.compress_after_days,
|
||||||
|
enable_s3_backup = EXCLUDED.enable_s3_backup,
|
||||||
|
s3_endpoint = EXCLUDED.s3_endpoint,
|
||||||
|
s3_region = EXCLUDED.s3_region,
|
||||||
|
s3_bucket = EXCLUDED.s3_bucket,
|
||||||
|
s3_prefix = EXCLUDED.s3_prefix,
|
||||||
|
s3_access_key_id = EXCLUDED.s3_access_key_id,
|
||||||
|
s3_secret_access_key_enc = EXCLUDED.s3_secret_access_key_enc,
|
||||||
|
disk_pause_threshold_pct = EXCLUDED.disk_pause_threshold_pct,
|
||||||
|
enable_webhook_alerts = EXCLUDED.enable_webhook_alerts,
|
||||||
|
webhook_url = EXCLUDED.webhook_url,
|
||||||
|
webhook_secret_enc = EXCLUDED.webhook_secret_enc,
|
||||||
|
webhook_cooldown_sec = EXCLUDED.webhook_cooldown_sec,
|
||||||
|
enable_prometheus_metrics = EXCLUDED.enable_prometheus_metrics,
|
||||||
|
prometheus_metrics_token_enc = EXCLUDED.prometheus_metrics_token_enc,
|
||||||
updated_at = NOW()`,
|
updated_at = NOW()`,
|
||||||
[
|
[
|
||||||
settings.serverPort,
|
settings.serverPort,
|
||||||
@@ -223,6 +510,21 @@ export class SettingsManager {
|
|||||||
settings.jwtSecret,
|
settings.jwtSecret,
|
||||||
settings.enableIdempotency,
|
settings.enableIdempotency,
|
||||||
settings.autoPruneDays,
|
settings.autoPruneDays,
|
||||||
|
settings.compressAfterDays,
|
||||||
|
settings.enableS3Backup,
|
||||||
|
settings.s3Endpoint,
|
||||||
|
settings.s3Region,
|
||||||
|
settings.s3Bucket,
|
||||||
|
settings.s3Prefix,
|
||||||
|
settings.s3AccessKeyId,
|
||||||
|
settings.s3SecretAccessKeyEnc,
|
||||||
|
settings.diskPauseThresholdPct,
|
||||||
|
settings.enableWebhookAlerts,
|
||||||
|
settings.webhookUrl,
|
||||||
|
settings.webhookSecretEnc,
|
||||||
|
settings.webhookCooldownSec,
|
||||||
|
settings.enablePrometheusMetrics,
|
||||||
|
settings.prometheusMetricsTokenEnc,
|
||||||
],
|
],
|
||||||
);
|
);
|
||||||
} catch (err: unknown) {
|
} catch (err: unknown) {
|
||||||
@@ -250,37 +552,49 @@ export class SettingsManager {
|
|||||||
return { ...current };
|
return { ...current };
|
||||||
}
|
}
|
||||||
|
|
||||||
public static getPublicSettings(): Omit<AdminSettings, "jwtSecret"> {
|
public static getPublicSettings(): PublicAdminSettings {
|
||||||
const settings = this.getSettings();
|
return toPublicSettings(this.getSettings());
|
||||||
|
}
|
||||||
|
|
||||||
|
public static async getPublicSettingsAsync(): Promise<PublicAdminSettings> {
|
||||||
|
const settings = await this.getSettingsAsync();
|
||||||
|
return toPublicSettings(settings);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static async getDecryptedS3ConfigAsync(): Promise<S3ArchiveConfig> {
|
||||||
|
const settings = await this.getSettingsAsync();
|
||||||
return {
|
return {
|
||||||
serverPort: settings.serverPort,
|
endpoint: settings.s3Endpoint,
|
||||||
webPort: settings.webPort,
|
region: settings.s3Region,
|
||||||
maxEventsPerSec: settings.maxEventsPerSec,
|
bucket: settings.s3Bucket,
|
||||||
maxPayloadSizeKb: settings.maxPayloadSizeKb,
|
prefix: settings.s3Prefix,
|
||||||
batchFlushThreshold: settings.batchFlushThreshold,
|
accessKeyId: settings.s3AccessKeyId,
|
||||||
batchFlushIntervalMs: settings.batchFlushIntervalMs,
|
secretAccessKey: decryptSecret(settings.s3SecretAccessKeyEnc),
|
||||||
fileRotationMb: settings.fileRotationMb,
|
|
||||||
enableIdempotency: settings.enableIdempotency,
|
|
||||||
autoPruneDays: settings.autoPruneDays,
|
|
||||||
updatedAt: settings.updatedAt,
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
public static async getPublicSettingsAsync(): Promise<
|
public static async getDecryptedWebhookConfigAsync(): Promise<WebhookRuntimeConfig> {
|
||||||
Omit<AdminSettings, "jwtSecret">
|
|
||||||
> {
|
|
||||||
const settings = await this.getSettingsAsync();
|
const settings = await this.getSettingsAsync();
|
||||||
return {
|
return {
|
||||||
serverPort: settings.serverPort,
|
enabled: settings.enableWebhookAlerts,
|
||||||
webPort: settings.webPort,
|
webhookUrl: settings.webhookUrl,
|
||||||
maxEventsPerSec: settings.maxEventsPerSec,
|
webhookSecret: settings.webhookSecretEnc
|
||||||
maxPayloadSizeKb: settings.maxPayloadSizeKb,
|
? decryptSecret(settings.webhookSecretEnc)
|
||||||
batchFlushThreshold: settings.batchFlushThreshold,
|
: "",
|
||||||
batchFlushIntervalMs: settings.batchFlushIntervalMs,
|
cooldownSec: settings.webhookCooldownSec,
|
||||||
fileRotationMb: settings.fileRotationMb,
|
};
|
||||||
enableIdempotency: settings.enableIdempotency,
|
}
|
||||||
autoPruneDays: settings.autoPruneDays,
|
|
||||||
updatedAt: settings.updatedAt,
|
public static async getDecryptedPrometheusConfigAsync(): Promise<{
|
||||||
|
enabled: boolean;
|
||||||
|
token: string;
|
||||||
|
}> {
|
||||||
|
const settings = await this.getSettingsAsync();
|
||||||
|
return {
|
||||||
|
enabled: settings.enablePrometheusMetrics,
|
||||||
|
token: settings.prometheusMetricsTokenEnc
|
||||||
|
? decryptSecret(settings.prometheusMetricsTokenEnc)
|
||||||
|
: "",
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -326,6 +640,12 @@ export class SettingsManager {
|
|||||||
fileRotationMb: settings.fileRotationMb,
|
fileRotationMb: settings.fileRotationMb,
|
||||||
enableIdempotency: settings.enableIdempotency,
|
enableIdempotency: settings.enableIdempotency,
|
||||||
autoPruneDays: settings.autoPruneDays,
|
autoPruneDays: settings.autoPruneDays,
|
||||||
|
compressAfterDays: settings.compressAfterDays,
|
||||||
|
enableS3Backup: settings.enableS3Backup,
|
||||||
|
diskPauseThresholdPct: settings.diskPauseThresholdPct,
|
||||||
|
enableWebhookAlerts: settings.enableWebhookAlerts,
|
||||||
|
webhookUrl: settings.webhookUrl,
|
||||||
|
webhookCooldownSec: settings.webhookCooldownSec,
|
||||||
...(jwtSecretChanged ? { jwtSecret: settings.jwtSecret } : {}),
|
...(jwtSecretChanged ? { jwtSecret: settings.jwtSecret } : {}),
|
||||||
}),
|
}),
|
||||||
cache: "no-store",
|
cache: "no-store",
|
||||||
@@ -355,7 +675,10 @@ export class SettingsManager {
|
|||||||
globals.__dataStackLastSettingsBridgeSync = result;
|
globals.__dataStackLastSettingsBridgeSync = result;
|
||||||
return result;
|
return result;
|
||||||
} catch (err: unknown) {
|
} catch (err: unknown) {
|
||||||
const message = err instanceof Error ? err.message : "Port 4000 settings-sync unreachable";
|
const message =
|
||||||
|
err instanceof Error
|
||||||
|
? err.message
|
||||||
|
: "Port 4000 settings-sync unreachable";
|
||||||
if (
|
if (
|
||||||
process.env.NODE_ENV === "production" ||
|
process.env.NODE_ENV === "production" ||
|
||||||
Boolean(process.env.INGESTION_INTERNAL_URL)
|
Boolean(process.env.INGESTION_INTERNAL_URL)
|
||||||
@@ -374,18 +697,79 @@ export class SettingsManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public static updateSettings(updates: Partial<AdminSettings>): AdminSettings {
|
private static mergeSettingsUpdate(
|
||||||
|
current: AdminSettings,
|
||||||
|
updates: AdminSettingsUpdateInput,
|
||||||
|
): AdminSettings {
|
||||||
|
const {
|
||||||
|
s3SecretAccessKey,
|
||||||
|
clearS3SecretKey,
|
||||||
|
webhookSecret,
|
||||||
|
clearWebhookSecret,
|
||||||
|
prometheusMetricsToken,
|
||||||
|
clearPrometheusToken,
|
||||||
|
...rest
|
||||||
|
} = updates;
|
||||||
|
let nextEncryptedSecret = current.s3SecretAccessKeyEnc;
|
||||||
|
if (clearS3SecretKey) {
|
||||||
|
nextEncryptedSecret = "";
|
||||||
|
} else if (
|
||||||
|
typeof s3SecretAccessKey === "string" &&
|
||||||
|
s3SecretAccessKey.trim().length > 0
|
||||||
|
) {
|
||||||
|
nextEncryptedSecret = encryptSecret(s3SecretAccessKey.trim());
|
||||||
|
} else if (rest.s3SecretAccessKeyEnc !== undefined) {
|
||||||
|
nextEncryptedSecret = rest.s3SecretAccessKeyEnc;
|
||||||
|
}
|
||||||
|
|
||||||
|
let nextWebhookSecretEnc = current.webhookSecretEnc;
|
||||||
|
if (clearWebhookSecret) {
|
||||||
|
nextWebhookSecretEnc = "";
|
||||||
|
} else if (
|
||||||
|
typeof webhookSecret === "string" &&
|
||||||
|
webhookSecret.trim().length > 0
|
||||||
|
) {
|
||||||
|
nextWebhookSecretEnc = encryptSecret(webhookSecret.trim());
|
||||||
|
} else if (rest.webhookSecretEnc !== undefined) {
|
||||||
|
nextWebhookSecretEnc = rest.webhookSecretEnc;
|
||||||
|
}
|
||||||
|
|
||||||
|
let nextPrometheusTokenEnc = current.prometheusMetricsTokenEnc;
|
||||||
|
if (clearPrometheusToken) {
|
||||||
|
nextPrometheusTokenEnc = "";
|
||||||
|
} else if (
|
||||||
|
typeof prometheusMetricsToken === "string" &&
|
||||||
|
prometheusMetricsToken.trim().length > 0
|
||||||
|
) {
|
||||||
|
nextPrometheusTokenEnc = encryptSecret(prometheusMetricsToken.trim());
|
||||||
|
} else if (rest.prometheusMetricsTokenEnc !== undefined) {
|
||||||
|
nextPrometheusTokenEnc = rest.prometheusMetricsTokenEnc;
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
...current,
|
||||||
|
...rest,
|
||||||
|
s3SecretAccessKeyEnc: nextEncryptedSecret,
|
||||||
|
webhookSecretEnc: nextWebhookSecretEnc,
|
||||||
|
prometheusMetricsTokenEnc: nextPrometheusTokenEnc,
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
public static updateSettings(
|
||||||
|
updates: AdminSettingsUpdateInput,
|
||||||
|
): AdminSettings {
|
||||||
const current = getSettingsState();
|
const current = getSettingsState();
|
||||||
const previousJwtSecret = current.jwtSecret;
|
const previousJwtSecret = current.jwtSecret;
|
||||||
const jwtSecretChanged =
|
const jwtSecretChanged =
|
||||||
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
|
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
|
||||||
const nextSettings: AdminSettings = {
|
const nextSettings = this.mergeSettingsUpdate(current, updates);
|
||||||
...current,
|
|
||||||
...updates,
|
|
||||||
updatedAt: new Date().toISOString(),
|
|
||||||
};
|
|
||||||
globals.__dataStackAdminSettings = nextSettings;
|
globals.__dataStackAdminSettings = nextSettings;
|
||||||
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined);
|
this.applyRuntimeSettings(
|
||||||
|
nextSettings,
|
||||||
|
updates.autoPruneDays !== undefined,
|
||||||
|
updates.compressAfterDays !== undefined,
|
||||||
|
);
|
||||||
if (jwtSecretChanged) {
|
if (jwtSecretChanged) {
|
||||||
SDKManager.resignAllClientTokens();
|
SDKManager.resignAllClientTokens();
|
||||||
}
|
}
|
||||||
@@ -403,17 +787,13 @@ export class SettingsManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
public static async updateSettingsAsync(
|
public static async updateSettingsAsync(
|
||||||
updates: Partial<AdminSettings>,
|
updates: AdminSettingsUpdateInput,
|
||||||
): Promise<AdminSettings> {
|
): Promise<AdminSettings> {
|
||||||
const current = await this.getSettingsAsync();
|
const current = await this.getSettingsAsync();
|
||||||
const previousJwtSecret = current.jwtSecret;
|
const previousJwtSecret = current.jwtSecret;
|
||||||
const jwtSecretChanged =
|
const jwtSecretChanged =
|
||||||
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
|
Boolean(updates.jwtSecret) && updates.jwtSecret !== current.jwtSecret;
|
||||||
const nextSettings: AdminSettings = {
|
const nextSettings = this.mergeSettingsUpdate(current, updates);
|
||||||
...current,
|
|
||||||
...updates,
|
|
||||||
updatedAt: new Date().toISOString(),
|
|
||||||
};
|
|
||||||
|
|
||||||
if (
|
if (
|
||||||
jwtSecretChanged &&
|
jwtSecretChanged &&
|
||||||
@@ -450,11 +830,15 @@ export class SettingsManager {
|
|||||||
try {
|
try {
|
||||||
await this.persistSettingsToDb(nextSettings);
|
await this.persistSettingsToDb(nextSettings);
|
||||||
globals.__dataStackAdminSettings = nextSettings;
|
globals.__dataStackAdminSettings = nextSettings;
|
||||||
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined);
|
this.applyRuntimeSettings(
|
||||||
|
nextSettings,
|
||||||
|
updates.autoPruneDays !== undefined,
|
||||||
|
updates.compressAfterDays !== undefined,
|
||||||
|
);
|
||||||
await SDKManager.resignAllClientTokensAsync();
|
await SDKManager.resignAllClientTokensAsync();
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
globals.__dataStackAdminSettings = current;
|
globals.__dataStackAdminSettings = current;
|
||||||
this.applyRuntimeSettings(current, false);
|
this.applyRuntimeSettings(current, false, false);
|
||||||
await this.persistSettingsToDb(current).catch(() => undefined);
|
await this.persistSettingsToDb(current).catch(() => undefined);
|
||||||
if (preSyncStatus.synced) {
|
if (preSyncStatus.synced) {
|
||||||
await this.notifyIngestionSettingsSync(
|
await this.notifyIngestionSettingsSync(
|
||||||
@@ -482,7 +866,11 @@ export class SettingsManager {
|
|||||||
|
|
||||||
await this.persistSettingsToDb(nextSettings);
|
await this.persistSettingsToDb(nextSettings);
|
||||||
globals.__dataStackAdminSettings = nextSettings;
|
globals.__dataStackAdminSettings = nextSettings;
|
||||||
this.applyRuntimeSettings(nextSettings, updates.autoPruneDays !== undefined);
|
this.applyRuntimeSettings(
|
||||||
|
nextSettings,
|
||||||
|
updates.autoPruneDays !== undefined,
|
||||||
|
updates.compressAfterDays !== undefined,
|
||||||
|
);
|
||||||
await this.notifyIngestionSettingsSync(
|
await this.notifyIngestionSettingsSync(
|
||||||
nextSettings,
|
nextSettings,
|
||||||
previousJwtSecret,
|
previousJwtSecret,
|
||||||
|
|||||||
@@ -0,0 +1,183 @@
|
|||||||
|
import crypto from "node:crypto";
|
||||||
|
|
||||||
|
export type WebhookAlertEventType =
|
||||||
|
| "DISK_SPACE_LOW"
|
||||||
|
| "STORAGE_QUOTA_EXCEEDED"
|
||||||
|
| "S3_ARCHIVE_FAILED"
|
||||||
|
| "CLIENT_IP_FORBIDDEN"
|
||||||
|
| "CLIENT_TOKEN_EXPIRED"
|
||||||
|
| "ADMIN_LOGIN_LOCKOUT"
|
||||||
|
| "TEST_ALERT";
|
||||||
|
|
||||||
|
export interface WebhookAlertPayload {
|
||||||
|
event: WebhookAlertEventType;
|
||||||
|
severity: "info" | "warning" | "critical";
|
||||||
|
title: string;
|
||||||
|
message: string;
|
||||||
|
clientId?: string;
|
||||||
|
details?: Record<string, unknown>;
|
||||||
|
timestamp?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WebhookRuntimeConfig {
|
||||||
|
enabled: boolean;
|
||||||
|
webhookUrl: string;
|
||||||
|
webhookSecret: string;
|
||||||
|
cooldownSec: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WebhookDispatchResult {
|
||||||
|
ok: boolean;
|
||||||
|
skipped?: boolean;
|
||||||
|
reason?: string;
|
||||||
|
status?: number;
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
type WebhookGlobals = typeof globalThis & {
|
||||||
|
__dataStackWebhookCooldownMap?: Map<string, number>;
|
||||||
|
};
|
||||||
|
|
||||||
|
const globals = globalThis as WebhookGlobals;
|
||||||
|
const DEFAULT_WEBHOOK_TIMEOUT_MS = 5000;
|
||||||
|
|
||||||
|
function getCooldownMap(): Map<string, number> {
|
||||||
|
if (!globals.__dataStackWebhookCooldownMap) {
|
||||||
|
globals.__dataStackWebhookCooldownMap = new Map<string, number>();
|
||||||
|
}
|
||||||
|
return globals.__dataStackWebhookCooldownMap;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function clearWebhookCooldownCacheForTests(): void {
|
||||||
|
getCooldownMap().clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildWebhookSignature(
|
||||||
|
rawBody: string,
|
||||||
|
secret: string,
|
||||||
|
): string {
|
||||||
|
const hmac = crypto
|
||||||
|
.createHmac("sha256", secret)
|
||||||
|
.update(rawBody, "utf8")
|
||||||
|
.digest("hex");
|
||||||
|
return `sha256=${hmac}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatOutboundWebhookBody(payload: WebhookAlertPayload): string {
|
||||||
|
const timestamp = payload.timestamp || new Date().toISOString();
|
||||||
|
const prefix =
|
||||||
|
payload.severity === "critical"
|
||||||
|
? "[CRITICAL]"
|
||||||
|
: payload.severity === "warning"
|
||||||
|
? "[WARNING]"
|
||||||
|
: "[INFO]";
|
||||||
|
const clientTag = payload.clientId ? ` (client: ${payload.clientId})` : "";
|
||||||
|
const summaryLine = `${prefix} DataStack ${payload.event}${clientTag} — ${payload.title}: ${payload.message}`;
|
||||||
|
|
||||||
|
return JSON.stringify({
|
||||||
|
instance: "DataStack",
|
||||||
|
event: payload.event,
|
||||||
|
severity: payload.severity,
|
||||||
|
title: payload.title,
|
||||||
|
message: payload.message,
|
||||||
|
...(payload.clientId ? { clientId: payload.clientId } : {}),
|
||||||
|
...(payload.details ? { details: payload.details } : {}),
|
||||||
|
timestamp,
|
||||||
|
// Slack & Discord compatible top-level fields
|
||||||
|
text: summaryLine,
|
||||||
|
content: summaryLine,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function dispatchWebhookAlertAsync(
|
||||||
|
payload: WebhookAlertPayload,
|
||||||
|
config: WebhookRuntimeConfig,
|
||||||
|
options?: { bypassCooldown?: boolean; timeoutMs?: number },
|
||||||
|
): Promise<WebhookDispatchResult> {
|
||||||
|
const webhookUrl = String(config.webhookUrl || "").trim();
|
||||||
|
if (!options?.bypassCooldown && !config.enabled) {
|
||||||
|
return { ok: false, skipped: true, reason: "WEBHOOK_DISABLED" };
|
||||||
|
}
|
||||||
|
if (!webhookUrl) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
skipped: !options?.bypassCooldown,
|
||||||
|
error: "Webhook URL is not configured.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsedUrl: URL;
|
||||||
|
try {
|
||||||
|
parsedUrl = new URL(webhookUrl);
|
||||||
|
if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
error: "Webhook URL must use http:// or https://.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
error: "Webhook URL is not a valid URL.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const dedupeKey = `${payload.event}:${payload.clientId || "global"}`;
|
||||||
|
const cooldownMs = Math.max(0, (Number(config.cooldownSec) || 300) * 1000);
|
||||||
|
const now = Date.now();
|
||||||
|
const cooldownMap = getCooldownMap();
|
||||||
|
|
||||||
|
if (!options?.bypassCooldown && cooldownMs > 0) {
|
||||||
|
const lastSentAt = cooldownMap.get(dedupeKey);
|
||||||
|
if (lastSentAt !== undefined && now - lastSentAt < cooldownMs) {
|
||||||
|
return { ok: true, skipped: true, reason: "COOLDOWN_ACTIVE" };
|
||||||
|
}
|
||||||
|
cooldownMap.set(dedupeKey, now);
|
||||||
|
}
|
||||||
|
|
||||||
|
const timestamp = payload.timestamp || new Date(now).toISOString();
|
||||||
|
const bodyStr = formatOutboundWebhookBody({ ...payload, timestamp });
|
||||||
|
const headers: Record<string, string> = {
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"User-Agent": "DataStack-Webhook-Notifier/2.0",
|
||||||
|
"X-DataStack-Event": payload.event,
|
||||||
|
"X-DataStack-Timestamp": timestamp,
|
||||||
|
};
|
||||||
|
|
||||||
|
const secret = String(config.webhookSecret || "").trim();
|
||||||
|
if (secret) {
|
||||||
|
headers["X-DataStack-Signature"] = buildWebhookSignature(bodyStr, secret);
|
||||||
|
}
|
||||||
|
|
||||||
|
const timeoutMs = Math.max(500, options?.timeoutMs || DEFAULT_WEBHOOK_TIMEOUT_MS);
|
||||||
|
try {
|
||||||
|
const res = await fetch(parsedUrl.toString(), {
|
||||||
|
method: "POST",
|
||||||
|
headers,
|
||||||
|
body: bodyStr,
|
||||||
|
cache: "no-store",
|
||||||
|
signal: AbortSignal.timeout(timeoutMs),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!res.ok) {
|
||||||
|
if (!options?.bypassCooldown) {
|
||||||
|
// Keep cooldown entry so failing webhooks don't DDOS a broken upstream endpoint
|
||||||
|
}
|
||||||
|
const errText = await res.text().catch(() => "");
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
status: res.status,
|
||||||
|
error: `Webhook endpoint returned HTTP ${res.status}${errText ? `: ${errText.slice(0, 160)}` : ""}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return { ok: true, status: res.status };
|
||||||
|
} catch (err: unknown) {
|
||||||
|
const msg =
|
||||||
|
err instanceof Error ? err.message : "Webhook delivery failed";
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
error: `Webhook delivery failed: ${msg}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -13,11 +13,13 @@ export class ApiEngine {
|
|||||||
*/
|
*/
|
||||||
public static async getDashboardMetrics() {
|
public static async getDashboardMetrics() {
|
||||||
const settings = await SettingsManager.getSettingsAsync();
|
const settings = await SettingsManager.getSettingsAsync();
|
||||||
const [storageStats, localBufferStats, clients] = await Promise.all([
|
const [storageStats, diskUsage, localBufferStats, clients] =
|
||||||
CSVStorageManager.getStorageStatsAsync(),
|
await Promise.all([
|
||||||
DataBufferEngine.getBufferStatsAsync(),
|
CSVStorageManager.getStorageStatsAsync(),
|
||||||
SDKManager.listClientsAsync(),
|
CSVStorageManager.getDiskUsageAsync(),
|
||||||
]);
|
DataBufferEngine.getBufferStatsAsync(),
|
||||||
|
SDKManager.listClientsAsync(),
|
||||||
|
]);
|
||||||
const activeClients = clients.filter((c) => c.status === 'active').length;
|
const activeClients = clients.filter((c) => c.status === 'active').length;
|
||||||
let activeSockets = 0;
|
let activeSockets = 0;
|
||||||
let ingestionLocalBufferSize = 0;
|
let ingestionLocalBufferSize = 0;
|
||||||
@@ -129,6 +131,7 @@ export class ApiEngine {
|
|||||||
activeClients,
|
activeClients,
|
||||||
bufferStats,
|
bufferStats,
|
||||||
storageStats,
|
storageStats,
|
||||||
|
diskUsage,
|
||||||
securityStats,
|
securityStats,
|
||||||
runtime: {
|
runtime: {
|
||||||
serverPort: liveServerPort,
|
serverPort: liveServerPort,
|
||||||
|
|||||||
+122
-9
@@ -23,6 +23,9 @@ export interface ClientSDK {
|
|||||||
secret: string;
|
secret: string;
|
||||||
token: string;
|
token: string;
|
||||||
rateLimit: number; // max events per sec
|
rateLimit: number; // max events per sec
|
||||||
|
storageQuotaMb: number; // max storage in MB per client (0 = unlimited)
|
||||||
|
allowedIps: string; // comma-separated IPv4/IPv6/CIDR list ("" = allow all)
|
||||||
|
expiresAt?: string | null; // ISO timestamp or null/undefined if never expires
|
||||||
status: "active" | "revoked";
|
status: "active" | "revoked";
|
||||||
totalEvents: number;
|
totalEvents: number;
|
||||||
lastActive?: string;
|
lastActive?: string;
|
||||||
@@ -103,6 +106,9 @@ interface RemoteInMemoryClient {
|
|||||||
status?: string;
|
status?: string;
|
||||||
token?: string;
|
token?: string;
|
||||||
rateLimit?: number;
|
rateLimit?: number;
|
||||||
|
storageQuotaMb?: number;
|
||||||
|
allowedIps?: string;
|
||||||
|
expiresAt?: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): number {
|
function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): number {
|
||||||
@@ -119,6 +125,9 @@ function hydrateLocalMapFromRemote(remoteClients: RemoteInMemoryClient[]): numbe
|
|||||||
secret: "",
|
secret: "",
|
||||||
token: String(item.token || ""),
|
token: String(item.token || ""),
|
||||||
rateLimit: Math.max(1, Number(item.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(item.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof item.allowedIps === "string" ? item.allowedIps : "",
|
||||||
|
expiresAt: item.expiresAt ? String(item.expiresAt) : null,
|
||||||
status: item.status === "revoked" ? "revoked" : "active",
|
status: item.status === "revoked" ? "revoked" : "active",
|
||||||
totalEvents: 0,
|
totalEvents: 0,
|
||||||
createdAt: new Date().toISOString(),
|
createdAt: new Date().toISOString(),
|
||||||
@@ -135,6 +144,9 @@ function serializeSyncClient(client: ClientSDK) {
|
|||||||
status: client.status,
|
status: client.status,
|
||||||
token: client.token,
|
token: client.token,
|
||||||
rateLimit: client.rateLimit,
|
rateLimit: client.rateLimit,
|
||||||
|
storageQuotaMb: client.storageQuotaMb,
|
||||||
|
allowedIps: client.allowedIps || "",
|
||||||
|
expiresAt: client.expiresAt ?? null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -224,12 +236,26 @@ function notifyIngestionClientSync(
|
|||||||
}
|
}
|
||||||
|
|
||||||
function mapRowToClient(row: Record<string, unknown>): ClientSDK {
|
function mapRowToClient(row: Record<string, unknown>): ClientSDK {
|
||||||
|
const parsedQuota = Number(row.storage_quota_mb);
|
||||||
|
let expiresAt: string | null = null;
|
||||||
|
if (row.expires_at) {
|
||||||
|
const d = new Date(row.expires_at as string | Date);
|
||||||
|
if (!Number.isNaN(d.getTime())) {
|
||||||
|
expiresAt = d.toISOString();
|
||||||
|
}
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
clientId: String(row.client_id),
|
clientId: String(row.client_id),
|
||||||
name: String(row.name),
|
name: String(row.name),
|
||||||
secret: String(row.secret),
|
secret: String(row.secret),
|
||||||
token: String(row.token),
|
token: String(row.token),
|
||||||
rateLimit: Number(row.rate_limit) || 100,
|
rateLimit: Number(row.rate_limit) || 100,
|
||||||
|
storageQuotaMb:
|
||||||
|
Number.isFinite(parsedQuota) && parsedQuota >= 0
|
||||||
|
? Math.floor(parsedQuota)
|
||||||
|
: 0,
|
||||||
|
allowedIps: typeof row.allowed_ips === "string" ? row.allowed_ips : "",
|
||||||
|
expiresAt,
|
||||||
status: row.status === "revoked" ? "revoked" : "active",
|
status: row.status === "revoked" ? "revoked" : "active",
|
||||||
totalEvents: Number(row.total_events) || 0,
|
totalEvents: Number(row.total_events) || 0,
|
||||||
lastActive: row.last_active
|
lastActive: row.last_active
|
||||||
@@ -256,7 +282,7 @@ export class SDKManager {
|
|||||||
|
|
||||||
const [result, deletedResult] = await Promise.all([
|
const [result, deletedResult] = await Promise.all([
|
||||||
pool.query(
|
pool.query(
|
||||||
`SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at
|
`SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at
|
||||||
FROM client_sdks
|
FROM client_sdks
|
||||||
ORDER BY created_at DESC`,
|
ORDER BY created_at DESC`,
|
||||||
),
|
),
|
||||||
@@ -464,7 +490,7 @@ export class SDKManager {
|
|||||||
await this.flushPendingClientActivityAsync();
|
await this.flushPendingClientActivityAsync();
|
||||||
}
|
}
|
||||||
const result = await pool.query(
|
const result = await pool.query(
|
||||||
`SELECT client_id, name, secret, token, rate_limit, status, total_events, last_active, created_at
|
`SELECT client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, last_active, created_at
|
||||||
FROM client_sdks
|
FROM client_sdks
|
||||||
WHERE client_id = $1`,
|
WHERE client_id = $1`,
|
||||||
[normalizedId],
|
[normalizedId],
|
||||||
@@ -500,10 +526,19 @@ export class SDKManager {
|
|||||||
clientId: string;
|
clientId: string;
|
||||||
name: string;
|
name: string;
|
||||||
rateLimit?: number;
|
rateLimit?: number;
|
||||||
|
storageQuotaMb?: number;
|
||||||
|
allowedIps?: string;
|
||||||
|
expiresAt?: string | null;
|
||||||
}): ClientSDK {
|
}): ClientSDK {
|
||||||
const safeClientId = this.normalizeClientId(data.clientId);
|
const safeClientId = this.normalizeClientId(data.clientId);
|
||||||
const secret = this.generateClientSecret(safeClientId);
|
const secret = this.generateClientSecret(safeClientId);
|
||||||
const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
|
const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
|
||||||
|
const storageQuotaMb = Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor(Number(data.storageQuotaMb) || 0),
|
||||||
|
);
|
||||||
|
const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : "";
|
||||||
|
const expiresAt = data.expiresAt ? String(data.expiresAt) : null;
|
||||||
|
|
||||||
const token = AuthService.generateClientToken({
|
const token = AuthService.generateClientToken({
|
||||||
clientId: safeClientId,
|
clientId: safeClientId,
|
||||||
@@ -517,6 +552,9 @@ export class SDKManager {
|
|||||||
secret,
|
secret,
|
||||||
token,
|
token,
|
||||||
rateLimit,
|
rateLimit,
|
||||||
|
storageQuotaMb,
|
||||||
|
allowedIps,
|
||||||
|
expiresAt,
|
||||||
status: "active",
|
status: "active",
|
||||||
totalEvents: 0,
|
totalEvents: 0,
|
||||||
createdAt: new Date().toISOString(),
|
createdAt: new Date().toISOString(),
|
||||||
@@ -548,13 +586,16 @@ export class SDKManager {
|
|||||||
[client.clientId],
|
[client.clientId],
|
||||||
);
|
);
|
||||||
await pool.query(
|
await pool.query(
|
||||||
`INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, status, total_events, created_at, updated_at)
|
`INSERT INTO client_sdks (client_id, name, secret, token, rate_limit, storage_quota_mb, allowed_ips, expires_at, status, total_events, created_at, updated_at)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW())
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW())
|
||||||
ON CONFLICT (client_id) DO UPDATE SET
|
ON CONFLICT (client_id) DO UPDATE SET
|
||||||
name = EXCLUDED.name,
|
name = EXCLUDED.name,
|
||||||
secret = EXCLUDED.secret,
|
secret = EXCLUDED.secret,
|
||||||
token = EXCLUDED.token,
|
token = EXCLUDED.token,
|
||||||
rate_limit = EXCLUDED.rate_limit,
|
rate_limit = EXCLUDED.rate_limit,
|
||||||
|
storage_quota_mb = EXCLUDED.storage_quota_mb,
|
||||||
|
allowed_ips = EXCLUDED.allowed_ips,
|
||||||
|
expires_at = EXCLUDED.expires_at,
|
||||||
status = EXCLUDED.status,
|
status = EXCLUDED.status,
|
||||||
updated_at = NOW()`,
|
updated_at = NOW()`,
|
||||||
[
|
[
|
||||||
@@ -563,6 +604,9 @@ export class SDKManager {
|
|||||||
client.secret,
|
client.secret,
|
||||||
client.token,
|
client.token,
|
||||||
client.rateLimit,
|
client.rateLimit,
|
||||||
|
client.storageQuotaMb,
|
||||||
|
client.allowedIps || "",
|
||||||
|
client.expiresAt ?? null,
|
||||||
client.status,
|
client.status,
|
||||||
client.totalEvents,
|
client.totalEvents,
|
||||||
client.createdAt,
|
client.createdAt,
|
||||||
@@ -578,11 +622,20 @@ export class SDKManager {
|
|||||||
clientId: string;
|
clientId: string;
|
||||||
name: string;
|
name: string;
|
||||||
rateLimit?: number;
|
rateLimit?: number;
|
||||||
|
storageQuotaMb?: number;
|
||||||
|
allowedIps?: string;
|
||||||
|
expiresAt?: string | null;
|
||||||
}): Promise<ClientSDK> {
|
}): Promise<ClientSDK> {
|
||||||
await ensureDbSettingsHydrated();
|
await ensureDbSettingsHydrated();
|
||||||
const safeClientId = this.normalizeClientId(data.clientId);
|
const safeClientId = this.normalizeClientId(data.clientId);
|
||||||
const secret = this.generateClientSecret(safeClientId);
|
const secret = this.generateClientSecret(safeClientId);
|
||||||
const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
|
const rateLimit = Math.max(1, Number(data.rateLimit) || 100);
|
||||||
|
const storageQuotaMb = Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor(Number(data.storageQuotaMb) || 0),
|
||||||
|
);
|
||||||
|
const allowedIps = typeof data.allowedIps === "string" ? data.allowedIps.trim() : "";
|
||||||
|
const expiresAt = data.expiresAt ? String(data.expiresAt) : null;
|
||||||
|
|
||||||
const token = AuthService.generateClientToken({
|
const token = AuthService.generateClientToken({
|
||||||
clientId: safeClientId,
|
clientId: safeClientId,
|
||||||
@@ -596,6 +649,9 @@ export class SDKManager {
|
|||||||
secret,
|
secret,
|
||||||
token,
|
token,
|
||||||
rateLimit,
|
rateLimit,
|
||||||
|
storageQuotaMb,
|
||||||
|
allowedIps,
|
||||||
|
expiresAt,
|
||||||
status: "active",
|
status: "active",
|
||||||
totalEvents: 0,
|
totalEvents: 0,
|
||||||
createdAt: new Date().toISOString(),
|
createdAt: new Date().toISOString(),
|
||||||
@@ -683,7 +739,17 @@ export class SDKManager {
|
|||||||
|
|
||||||
public static updateClient(
|
public static updateClient(
|
||||||
clientId: string,
|
clientId: string,
|
||||||
updates: Partial<Pick<ClientSDK, "name" | "rateLimit" | "status">>,
|
updates: Partial<
|
||||||
|
Pick<
|
||||||
|
ClientSDK,
|
||||||
|
| "name"
|
||||||
|
| "rateLimit"
|
||||||
|
| "storageQuotaMb"
|
||||||
|
| "allowedIps"
|
||||||
|
| "expiresAt"
|
||||||
|
| "status"
|
||||||
|
>
|
||||||
|
>,
|
||||||
): ClientSDK | null {
|
): ClientSDK | null {
|
||||||
const normalizedId = this.normalizeClientId(clientId);
|
const normalizedId = this.normalizeClientId(clientId);
|
||||||
const client = getClientsMap().get(normalizedId);
|
const client = getClientsMap().get(normalizedId);
|
||||||
@@ -695,6 +761,20 @@ export class SDKManager {
|
|||||||
...(updates.rateLimit !== undefined
|
...(updates.rateLimit !== undefined
|
||||||
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
|
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
|
||||||
: {}),
|
: {}),
|
||||||
|
...(updates.storageQuotaMb !== undefined
|
||||||
|
? {
|
||||||
|
storageQuotaMb: Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor(Number(updates.storageQuotaMb) || 0),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
...(updates.allowedIps !== undefined
|
||||||
|
? { allowedIps: String(updates.allowedIps).trim() }
|
||||||
|
: {}),
|
||||||
|
...(updates.expiresAt !== undefined
|
||||||
|
? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null }
|
||||||
|
: {}),
|
||||||
...(updates.status !== undefined ? { status: updates.status } : {}),
|
...(updates.status !== undefined ? { status: updates.status } : {}),
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -722,9 +802,17 @@ export class SDKManager {
|
|||||||
try {
|
try {
|
||||||
await pool.query(
|
await pool.query(
|
||||||
`UPDATE client_sdks
|
`UPDATE client_sdks
|
||||||
SET name = $2, rate_limit = $3, status = $4, updated_at = NOW()
|
SET name = $2, rate_limit = $3, storage_quota_mb = $4, allowed_ips = $5, expires_at = $6, status = $7, updated_at = NOW()
|
||||||
WHERE client_id = $1`,
|
WHERE client_id = $1`,
|
||||||
[client.clientId, client.name, client.rateLimit, client.status],
|
[
|
||||||
|
client.clientId,
|
||||||
|
client.name,
|
||||||
|
client.rateLimit,
|
||||||
|
client.storageQuotaMb,
|
||||||
|
client.allowedIps || "",
|
||||||
|
client.expiresAt ?? null,
|
||||||
|
client.status,
|
||||||
|
],
|
||||||
);
|
);
|
||||||
} catch (err: unknown) {
|
} catch (err: unknown) {
|
||||||
const detail = err instanceof Error ? err.message : "Unknown DB error";
|
const detail = err instanceof Error ? err.message : "Unknown DB error";
|
||||||
@@ -734,7 +822,17 @@ export class SDKManager {
|
|||||||
|
|
||||||
public static async updateClientAsync(
|
public static async updateClientAsync(
|
||||||
clientId: string,
|
clientId: string,
|
||||||
updates: Partial<Pick<ClientSDK, "name" | "rateLimit" | "status">>,
|
updates: Partial<
|
||||||
|
Pick<
|
||||||
|
ClientSDK,
|
||||||
|
| "name"
|
||||||
|
| "rateLimit"
|
||||||
|
| "storageQuotaMb"
|
||||||
|
| "allowedIps"
|
||||||
|
| "expiresAt"
|
||||||
|
| "status"
|
||||||
|
>
|
||||||
|
>,
|
||||||
): Promise<ClientSDK | null> {
|
): Promise<ClientSDK | null> {
|
||||||
await ensureDbSettingsHydrated();
|
await ensureDbSettingsHydrated();
|
||||||
const normalizedId = this.normalizeClientId(clientId);
|
const normalizedId = this.normalizeClientId(clientId);
|
||||||
@@ -747,6 +845,20 @@ export class SDKManager {
|
|||||||
...(updates.rateLimit !== undefined
|
...(updates.rateLimit !== undefined
|
||||||
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
|
? { rateLimit: Math.max(1, Number(updates.rateLimit) || 100) }
|
||||||
: {}),
|
: {}),
|
||||||
|
...(updates.storageQuotaMb !== undefined
|
||||||
|
? {
|
||||||
|
storageQuotaMb: Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor(Number(updates.storageQuotaMb) || 0),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
...(updates.allowedIps !== undefined
|
||||||
|
? { allowedIps: String(updates.allowedIps).trim() }
|
||||||
|
: {}),
|
||||||
|
...(updates.expiresAt !== undefined
|
||||||
|
? { expiresAt: updates.expiresAt ? String(updates.expiresAt) : null }
|
||||||
|
: {}),
|
||||||
...(updates.status !== undefined ? { status: updates.status } : {}),
|
...(updates.status !== undefined ? { status: updates.status } : {}),
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -853,7 +965,7 @@ export class SDKManager {
|
|||||||
const dirStat = lstatSync(targetDir);
|
const dirStat = lstatSync(targetDir);
|
||||||
if (!dirStat.isSymbolicLink() && dirStat.isDirectory()) {
|
if (!dirStat.isSymbolicLink() && dirStat.isDirectory()) {
|
||||||
for (const entry of readdirSync(targetDir)) {
|
for (const entry of readdirSync(targetDir)) {
|
||||||
if (!entry.endsWith(".csv")) continue;
|
if (!CSVStorageManager.isPartitionFilename(entry)) continue;
|
||||||
const entryPath = resolve(targetDir, entry);
|
const entryPath = resolve(targetDir, entry);
|
||||||
try {
|
try {
|
||||||
const entryStat = lstatSync(entryPath);
|
const entryStat = lstatSync(entryPath);
|
||||||
@@ -871,6 +983,7 @@ export class SDKManager {
|
|||||||
|
|
||||||
rmSync(targetDir, { recursive: true, force: true });
|
rmSync(targetDir, { recursive: true, force: true });
|
||||||
CSVStorageManager.clearRowCountCache();
|
CSVStorageManager.clearRowCountCache();
|
||||||
|
CSVStorageManager.clearQuotaCaches(safeId);
|
||||||
return purgedFiles;
|
return purgedFiles;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,8 +6,41 @@ export interface EditableSettings {
|
|||||||
fileRotationMb: number;
|
fileRotationMb: number;
|
||||||
enableIdempotency: boolean;
|
enableIdempotency: boolean;
|
||||||
autoPruneDays: number;
|
autoPruneDays: number;
|
||||||
|
compressAfterDays: number;
|
||||||
|
diskPauseThresholdPct: number;
|
||||||
|
webhookCooldownSec: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface S3SettingsFields {
|
||||||
|
enableS3Backup: boolean;
|
||||||
|
s3Endpoint: string;
|
||||||
|
s3Region: string;
|
||||||
|
s3Bucket: string;
|
||||||
|
s3Prefix: string;
|
||||||
|
s3AccessKeyId: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AlertingAndMetricsSettingsFields {
|
||||||
|
enableWebhookAlerts: boolean;
|
||||||
|
webhookUrl: string;
|
||||||
|
enablePrometheusMetrics: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_S3_SETTINGS: S3SettingsFields = {
|
||||||
|
enableS3Backup: false,
|
||||||
|
s3Endpoint: '',
|
||||||
|
s3Region: 'auto',
|
||||||
|
s3Bucket: '',
|
||||||
|
s3Prefix: 'datastack/',
|
||||||
|
s3AccessKeyId: '',
|
||||||
|
};
|
||||||
|
|
||||||
|
export const DEFAULT_ALERTING_SETTINGS: AlertingAndMetricsSettingsFields = {
|
||||||
|
enableWebhookAlerts: false,
|
||||||
|
webhookUrl: '',
|
||||||
|
enablePrometheusMetrics: false,
|
||||||
|
};
|
||||||
|
|
||||||
export const SETTINGS_LIMITS = {
|
export const SETTINGS_LIMITS = {
|
||||||
maxEventsPerSec: { min: 1, max: 100000, defaultValue: 100 },
|
maxEventsPerSec: { min: 1, max: 100000, defaultValue: 100 },
|
||||||
maxPayloadSizeKb: { min: 1, max: 10240, defaultValue: 100 },
|
maxPayloadSizeKb: { min: 1, max: 10240, defaultValue: 100 },
|
||||||
@@ -15,6 +48,9 @@ export const SETTINGS_LIMITS = {
|
|||||||
batchFlushIntervalMs: { min: 100, max: 60000, defaultValue: 2000 },
|
batchFlushIntervalMs: { min: 100, max: 60000, defaultValue: 2000 },
|
||||||
fileRotationMb: { min: 1, max: 2048, defaultValue: 100 },
|
fileRotationMb: { min: 1, max: 2048, defaultValue: 100 },
|
||||||
autoPruneDays: { min: 0, max: 3650, defaultValue: 30 },
|
autoPruneDays: { min: 0, max: 3650, defaultValue: 30 },
|
||||||
|
compressAfterDays: { min: 0, max: 3650, defaultValue: 0 },
|
||||||
|
diskPauseThresholdPct: { min: 0, max: 99, defaultValue: 95 },
|
||||||
|
webhookCooldownSec: { min: 10, max: 86400, defaultValue: 300 },
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = {
|
export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = {
|
||||||
@@ -25,6 +61,9 @@ export const DEFAULT_EDITABLE_SETTINGS: EditableSettings = {
|
|||||||
fileRotationMb: SETTINGS_LIMITS.fileRotationMb.defaultValue,
|
fileRotationMb: SETTINGS_LIMITS.fileRotationMb.defaultValue,
|
||||||
enableIdempotency: true,
|
enableIdempotency: true,
|
||||||
autoPruneDays: SETTINGS_LIMITS.autoPruneDays.defaultValue,
|
autoPruneDays: SETTINGS_LIMITS.autoPruneDays.defaultValue,
|
||||||
|
compressAfterDays: SETTINGS_LIMITS.compressAfterDays.defaultValue,
|
||||||
|
diskPauseThresholdPct: SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue,
|
||||||
|
webhookCooldownSec: SETTINGS_LIMITS.webhookCooldownSec.defaultValue,
|
||||||
};
|
};
|
||||||
|
|
||||||
function clampInt(raw: unknown, min: number, max: number, fallback: number): number {
|
function clampInt(raw: unknown, min: number, max: number, fallback: number): number {
|
||||||
@@ -34,6 +73,57 @@ function clampInt(raw: unknown, min: number, max: number, fallback: number): num
|
|||||||
return Math.max(min, Math.min(max, rounded));
|
return Math.max(min, Math.min(max, rounded));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function normalizeDiskPauseThresholdPct(raw: unknown): number {
|
||||||
|
const clamped = clampInt(
|
||||||
|
raw,
|
||||||
|
SETTINGS_LIMITS.diskPauseThresholdPct.min,
|
||||||
|
SETTINGS_LIMITS.diskPauseThresholdPct.max,
|
||||||
|
SETTINGS_LIMITS.diskPauseThresholdPct.defaultValue,
|
||||||
|
);
|
||||||
|
if (clamped !== 0 && clamped < 50) {
|
||||||
|
return 50;
|
||||||
|
}
|
||||||
|
return clamped;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeS3Settings(
|
||||||
|
input: Partial<S3SettingsFields> | null | undefined,
|
||||||
|
): S3SettingsFields {
|
||||||
|
return {
|
||||||
|
enableS3Backup:
|
||||||
|
typeof input?.enableS3Backup === 'boolean'
|
||||||
|
? input.enableS3Backup
|
||||||
|
: DEFAULT_S3_SETTINGS.enableS3Backup,
|
||||||
|
s3Endpoint: typeof input?.s3Endpoint === 'string' ? input.s3Endpoint.trim() : '',
|
||||||
|
s3Region:
|
||||||
|
typeof input?.s3Region === 'string' && input.s3Region.trim()
|
||||||
|
? input.s3Region.trim()
|
||||||
|
: DEFAULT_S3_SETTINGS.s3Region,
|
||||||
|
s3Bucket: typeof input?.s3Bucket === 'string' ? input.s3Bucket.trim() : '',
|
||||||
|
s3Prefix:
|
||||||
|
typeof input?.s3Prefix === 'string' ? input.s3Prefix.trim() : DEFAULT_S3_SETTINGS.s3Prefix,
|
||||||
|
s3AccessKeyId:
|
||||||
|
typeof input?.s3AccessKeyId === 'string' ? input.s3AccessKeyId.trim() : '',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeAlertingSettings(
|
||||||
|
input: Partial<AlertingAndMetricsSettingsFields> | null | undefined,
|
||||||
|
): AlertingAndMetricsSettingsFields {
|
||||||
|
return {
|
||||||
|
enableWebhookAlerts:
|
||||||
|
typeof input?.enableWebhookAlerts === 'boolean'
|
||||||
|
? input.enableWebhookAlerts
|
||||||
|
: DEFAULT_ALERTING_SETTINGS.enableWebhookAlerts,
|
||||||
|
webhookUrl:
|
||||||
|
typeof input?.webhookUrl === 'string' ? input.webhookUrl.trim() : '',
|
||||||
|
enablePrometheusMetrics:
|
||||||
|
typeof input?.enablePrometheusMetrics === 'boolean'
|
||||||
|
? input.enablePrometheusMetrics
|
||||||
|
: DEFAULT_ALERTING_SETTINGS.enablePrometheusMetrics,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export function normalizeEditableSettings(
|
export function normalizeEditableSettings(
|
||||||
input: Partial<EditableSettings>,
|
input: Partial<EditableSettings>,
|
||||||
): EditableSettings {
|
): EditableSettings {
|
||||||
@@ -78,6 +168,21 @@ export function normalizeEditableSettings(
|
|||||||
SETTINGS_LIMITS.autoPruneDays.max,
|
SETTINGS_LIMITS.autoPruneDays.max,
|
||||||
SETTINGS_LIMITS.autoPruneDays.defaultValue,
|
SETTINGS_LIMITS.autoPruneDays.defaultValue,
|
||||||
),
|
),
|
||||||
|
compressAfterDays: clampInt(
|
||||||
|
input.compressAfterDays,
|
||||||
|
SETTINGS_LIMITS.compressAfterDays.min,
|
||||||
|
SETTINGS_LIMITS.compressAfterDays.max,
|
||||||
|
SETTINGS_LIMITS.compressAfterDays.defaultValue,
|
||||||
|
),
|
||||||
|
diskPauseThresholdPct: normalizeDiskPauseThresholdPct(
|
||||||
|
input.diskPauseThresholdPct,
|
||||||
|
),
|
||||||
|
webhookCooldownSec: clampInt(
|
||||||
|
input.webhookCooldownSec,
|
||||||
|
SETTINGS_LIMITS.webhookCooldownSec.min,
|
||||||
|
SETTINGS_LIMITS.webhookCooldownSec.max,
|
||||||
|
SETTINGS_LIMITS.webhookCooldownSec.defaultValue,
|
||||||
|
),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -398,6 +503,9 @@ export const NUMERIC_SETTING_KEYS: readonly NumericSettingKey[] = [
|
|||||||
'batchFlushIntervalMs',
|
'batchFlushIntervalMs',
|
||||||
'fileRotationMb',
|
'fileRotationMb',
|
||||||
'autoPruneDays',
|
'autoPruneDays',
|
||||||
|
'compressAfterDays',
|
||||||
|
'diskPauseThresholdPct',
|
||||||
|
'webhookCooldownSec',
|
||||||
] as const;
|
] as const;
|
||||||
|
|
||||||
export function settingsToNumericDrafts(
|
export function settingsToNumericDrafts(
|
||||||
@@ -410,6 +518,9 @@ export function settingsToNumericDrafts(
|
|||||||
batchFlushIntervalMs: String(settings.batchFlushIntervalMs),
|
batchFlushIntervalMs: String(settings.batchFlushIntervalMs),
|
||||||
fileRotationMb: String(settings.fileRotationMb),
|
fileRotationMb: String(settings.fileRotationMb),
|
||||||
autoPruneDays: String(settings.autoPruneDays),
|
autoPruneDays: String(settings.autoPruneDays),
|
||||||
|
compressAfterDays: String(settings.compressAfterDays),
|
||||||
|
diskPauseThresholdPct: String(settings.diskPauseThresholdPct),
|
||||||
|
webhookCooldownSec: String(settings.webhookCooldownSec),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -429,6 +540,15 @@ export function validateSingleNumericSetting(
|
|||||||
if (!Number.isSafeInteger(parsed)) {
|
if (!Number.isSafeInteger(parsed)) {
|
||||||
return { valid: false, error: 'Enter a valid integer.' };
|
return { valid: false, error: 'Enter a valid integer.' };
|
||||||
}
|
}
|
||||||
|
if (key === 'diskPauseThresholdPct') {
|
||||||
|
if (parsed !== 0 && (parsed < 50 || parsed > max)) {
|
||||||
|
return {
|
||||||
|
valid: false,
|
||||||
|
error: 'Must be 0 (disabled) or between 50 and 99.',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { valid: true, value: parsed };
|
||||||
|
}
|
||||||
if (parsed < min || parsed > max) {
|
if (parsed < min || parsed > max) {
|
||||||
return {
|
return {
|
||||||
valid: false,
|
valid: false,
|
||||||
@@ -475,6 +595,9 @@ export function validateEditableSettingsDraft(
|
|||||||
batchFlushIntervalMs: parsedValues.batchFlushIntervalMs!,
|
batchFlushIntervalMs: parsedValues.batchFlushIntervalMs!,
|
||||||
fileRotationMb: parsedValues.fileRotationMb!,
|
fileRotationMb: parsedValues.fileRotationMb!,
|
||||||
autoPruneDays: parsedValues.autoPruneDays!,
|
autoPruneDays: parsedValues.autoPruneDays!,
|
||||||
|
compressAfterDays: parsedValues.compressAfterDays!,
|
||||||
|
diskPauseThresholdPct: parsedValues.diskPauseThresholdPct!,
|
||||||
|
webhookCooldownSec: parsedValues.webhookCooldownSec!,
|
||||||
enableIdempotency: Boolean(enableIdempotency),
|
enableIdempotency: Boolean(enableIdempotency),
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|||||||
+284
-8
@@ -199,6 +199,9 @@ const runtimeConfig = {
|
|||||||
fileRotationMb: 100,
|
fileRotationMb: 100,
|
||||||
enableIdempotency: true,
|
enableIdempotency: true,
|
||||||
autoPruneDays: 30,
|
autoPruneDays: 30,
|
||||||
|
compressAfterDays: 0,
|
||||||
|
enableS3Backup: false,
|
||||||
|
diskPauseThresholdPct: 95,
|
||||||
};
|
};
|
||||||
|
|
||||||
let previousJwtSecret = '';
|
let previousJwtSecret = '';
|
||||||
@@ -287,17 +290,113 @@ const inMemoryClientActivity = new Map();
|
|||||||
let hasSyncedInMemoryClients = false;
|
let hasSyncedInMemoryClients = false;
|
||||||
const CLIENT_CACHE_TTL_MS = 2000;
|
const CLIENT_CACHE_TTL_MS = 2000;
|
||||||
|
|
||||||
async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
function stripIpv6MappedPrefix(rawIp) {
|
||||||
|
let ip = String(rawIp || '').trim();
|
||||||
|
if (!ip) return '';
|
||||||
|
if (ip.startsWith('[') && ip.includes(']')) {
|
||||||
|
ip = ip.slice(1, ip.indexOf(']'));
|
||||||
|
} else if (/^\d{1,3}(?:\.\d{1,3}){3}:\d+$/.test(ip)) {
|
||||||
|
ip = ip.slice(0, ip.lastIndexOf(':'));
|
||||||
|
}
|
||||||
|
const pctIndex = ip.indexOf('%');
|
||||||
|
if (pctIndex !== -1) {
|
||||||
|
ip = ip.slice(0, pctIndex);
|
||||||
|
}
|
||||||
|
if (/^::ffff:\d{1,3}(?:\.\d{1,3}){3}$/i.test(ip)) {
|
||||||
|
return ip.slice(7);
|
||||||
|
}
|
||||||
|
return ip.toLowerCase();
|
||||||
|
}
|
||||||
|
|
||||||
|
function ipv4ToUint32(ip) {
|
||||||
|
const parts = ip.split('.');
|
||||||
|
if (parts.length !== 4) return null;
|
||||||
|
let acc = 0;
|
||||||
|
for (const part of parts) {
|
||||||
|
if (!/^\d{1,3}$/.test(part)) return null;
|
||||||
|
if (part.length > 1 && part.startsWith('0')) return null;
|
||||||
|
const num = Number(part);
|
||||||
|
if (num < 0 || num > 255) return null;
|
||||||
|
acc = ((acc << 8) | num) >>> 0;
|
||||||
|
}
|
||||||
|
return acc >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function matchesIpv4Cidr(clientIp, cidrPattern) {
|
||||||
|
const [networkStr, prefixStr] = cidrPattern.split('/');
|
||||||
|
if (!networkStr || prefixStr === undefined) return false;
|
||||||
|
const prefix = Number(prefixStr);
|
||||||
|
if (!Number.isInteger(prefix) || prefix < 0 || prefix > 32) return false;
|
||||||
|
const clientNum = ipv4ToUint32(clientIp);
|
||||||
|
const networkNum = ipv4ToUint32(networkStr);
|
||||||
|
if (clientNum === null || networkNum === null) return false;
|
||||||
|
if (prefix === 0) return true;
|
||||||
|
const mask = (0xffffffff << (32 - prefix)) >>> 0;
|
||||||
|
return (clientNum & mask) >>> 0 === (networkNum & mask) >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isClientIpAllowed(clientIp, rawAllowedIps) {
|
||||||
|
if (!rawAllowedIps || !String(rawAllowedIps).trim()) return true;
|
||||||
|
const entries = String(rawAllowedIps)
|
||||||
|
.split(/[\s,]+/)
|
||||||
|
.map((entry) => entry.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
if (entries.length === 0) return true;
|
||||||
|
if (entries.includes('*') || entries.includes('0.0.0.0/0') || entries.includes('::/0')) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const normalizedClientIp = stripIpv6MappedPrefix(clientIp);
|
||||||
|
if (!normalizedClientIp || normalizedClientIp === 'unknown') {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
for (const entry of entries) {
|
||||||
|
const normalizedEntry = stripIpv6MappedPrefix(entry);
|
||||||
|
if (normalizedEntry.includes('/')) {
|
||||||
|
if (matchesIpv4Cidr(normalizedClientIp, normalizedEntry)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const [netAddr, prefixStr] = normalizedEntry.split('/');
|
||||||
|
if (Number(prefixStr) === 128 && normalizedClientIp === stripIpv6MappedPrefix(netAddr)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (normalizedClientIp === normalizedEntry) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
(normalizedClientIp === '127.0.0.1' && normalizedEntry === '::1') ||
|
||||||
|
(normalizedClientIp === '::1' && normalizedEntry === '127.0.0.1')
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isClientTokenExpired(expiresAt, nowMs = Date.now()) {
|
||||||
|
if (!expiresAt) return false;
|
||||||
|
const ts =
|
||||||
|
expiresAt instanceof Date
|
||||||
|
? expiresAt.getTime()
|
||||||
|
: new Date(String(expiresAt)).getTime();
|
||||||
|
if (Number.isNaN(ts)) return false;
|
||||||
|
return ts <= nowMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function verifyClientInDb(clientId, presentedToken, bypassCache = false, clientIp = '') {
|
||||||
const normalizedId = normalizeClientId(clientId);
|
const normalizedId = normalizeClientId(clientId);
|
||||||
if (!normalizedId) {
|
if (!normalizedId) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Missing client identifier',
|
reason: 'CLIENT_REVOKED: Missing client identifier',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
if (deletedClientsSet.has(normalizedId)) {
|
if (deletedClientsSet.has(normalizedId)) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been deleted or revoked by admin',
|
reason: 'CLIENT_REVOKED: Client access has been deleted or revoked by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -313,6 +412,7 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
deletedClientsSet.add(normalizedId);
|
deletedClientsSet.add(normalizedId);
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
|
reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -326,6 +426,9 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
status: parsed.status === 'revoked' ? 'revoked' : 'active',
|
status: parsed.status === 'revoked' ? 'revoked' : 'active',
|
||||||
token: String(parsed.token || ''),
|
token: String(parsed.token || ''),
|
||||||
rateLimit: Math.max(1, Number(parsed.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(parsed.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(parsed.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof parsed.allowedIps === 'string' ? parsed.allowedIps : '',
|
||||||
|
expiresAt: parsed.expiresAt ? String(parsed.expiresAt) : null,
|
||||||
};
|
};
|
||||||
inMemoryClientRegistry.set(normalizedId, memClient);
|
inMemoryClientRegistry.set(normalizedId, memClient);
|
||||||
hasSyncedInMemoryClients = true;
|
hasSyncedInMemoryClients = true;
|
||||||
@@ -341,20 +444,41 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
if (memClient.status === 'revoked') {
|
if (memClient.status === 'revoked') {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
if (memClient.token && presentedToken && memClient.token !== presentedToken) {
|
if (memClient.token && presentedToken && memClient.token !== presentedToken) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'TOKEN_ROTATED',
|
||||||
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
return { allowed: true, rateLimit: memClient.rateLimit || 100 };
|
if (isClientTokenExpired(memClient.expiresAt)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_TOKEN_EXPIRED',
|
||||||
|
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (memClient.allowedIps && memClient.allowedIps.trim() && !isClientIpAllowed(clientIp, memClient.allowedIps)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_IP_FORBIDDEN',
|
||||||
|
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
allowed: true,
|
||||||
|
rateLimit: memClient.rateLimit || 100,
|
||||||
|
storageQuotaMb: memClient.storageQuotaMb || 0,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
|
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -365,16 +489,36 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
if (prevCached.status === 'revoked') {
|
if (prevCached.status === 'revoked') {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
if (prevCached.token && presentedToken && prevCached.token !== presentedToken) {
|
if (prevCached.token && presentedToken && prevCached.token !== presentedToken) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'TOKEN_ROTATED',
|
||||||
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
return { allowed: true, rateLimit: prevCached.rateLimit };
|
if (isClientTokenExpired(prevCached.expiresAt, now)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_TOKEN_EXPIRED',
|
||||||
|
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (prevCached.allowedIps && prevCached.allowedIps.trim() && !isClientIpAllowed(clientIp, prevCached.allowedIps)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_IP_FORBIDDEN',
|
||||||
|
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
allowed: true,
|
||||||
|
rateLimit: prevCached.rateLimit,
|
||||||
|
storageQuotaMb: prevCached.storageQuotaMb || 0,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
@@ -387,26 +531,39 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
deletedClientsSet.add(normalizedId);
|
deletedClientsSet.add(normalizedId);
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
|
reason: 'CLIENT_REVOKED: Client access has been deleted by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
const res = await dbPool.query(
|
const res = await dbPool.query(
|
||||||
`SELECT status, token, rate_limit FROM client_sdks WHERE client_id = $1`,
|
`SELECT status, token, rate_limit, storage_quota_mb, allowed_ips, expires_at FROM client_sdks WHERE client_id = $1`,
|
||||||
[normalizedId],
|
[normalizedId],
|
||||||
);
|
);
|
||||||
if (!res.rowCount || res.rowCount === 0) {
|
if (!res.rowCount || res.rowCount === 0) {
|
||||||
clientAuthCache.delete(normalizedId);
|
clientAuthCache.delete(normalizedId);
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
|
reason: 'CLIENT_REVOKED: Client registration not found or has been deleted',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
const row = res.rows[0];
|
const row = res.rows[0];
|
||||||
|
const parsedQuota = Number(row.storage_quota_mb);
|
||||||
|
let expiresAtIso = null;
|
||||||
|
if (row.expires_at) {
|
||||||
|
const d = new Date(row.expires_at);
|
||||||
|
if (!Number.isNaN(d.getTime())) {
|
||||||
|
expiresAtIso = d.toISOString();
|
||||||
|
}
|
||||||
|
}
|
||||||
const entry = {
|
const entry = {
|
||||||
status: row.status,
|
status: row.status,
|
||||||
token: row.token,
|
token: row.token,
|
||||||
rateLimit: Number(row.rate_limit) || 100,
|
rateLimit: Number(row.rate_limit) || 100,
|
||||||
|
storageQuotaMb: Number.isFinite(parsedQuota) && parsedQuota >= 0 ? Math.floor(parsedQuota) : 0,
|
||||||
|
allowedIps: typeof row.allowed_ips === 'string' ? row.allowed_ips : '',
|
||||||
|
expiresAt: expiresAtIso,
|
||||||
checkedAt: now,
|
checkedAt: now,
|
||||||
};
|
};
|
||||||
clientAuthCache.set(normalizedId, entry);
|
clientAuthCache.set(normalizedId, entry);
|
||||||
@@ -414,19 +571,40 @@ async function verifyClientInDb(clientId, presentedToken, bypassCache = false) {
|
|||||||
if (entry.status === 'revoked') {
|
if (entry.status === 'revoked') {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'CLIENT_REVOKED',
|
||||||
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
reason: 'CLIENT_REVOKED: Client access has been revoked by admin',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
if (entry.token && presentedToken && entry.token !== presentedToken) {
|
if (entry.token && presentedToken && entry.token !== presentedToken) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'TOKEN_ROTATED',
|
||||||
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
reason: 'TOKEN_ROTATED: Client credentials have been rotated',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
return { allowed: true, rateLimit: entry.rateLimit };
|
if (isClientTokenExpired(entry.expiresAt, now)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_TOKEN_EXPIRED',
|
||||||
|
reason: `CLIENT_TOKEN_EXPIRED: Client '${normalizedId}' token has expired`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (entry.allowedIps && entry.allowedIps.trim() && !isClientIpAllowed(clientIp, entry.allowedIps)) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'CLIENT_IP_FORBIDDEN',
|
||||||
|
reason: `CLIENT_IP_FORBIDDEN: Client IP '${clientIp || 'unknown'}' is not authorized`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
allowed: true,
|
||||||
|
rateLimit: entry.rateLimit,
|
||||||
|
storageQuotaMb: entry.storageQuotaMb,
|
||||||
|
};
|
||||||
} catch {
|
} catch {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
code: 'AUTH_UNAVAILABLE',
|
||||||
reason: 'AUTH_UNAVAILABLE: Unable to verify client credentials against database',
|
reason: 'AUTH_UNAVAILABLE: Unable to verify client credentials against database',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -888,13 +1066,42 @@ function applyRuntimeSettingsObject(settingsObj) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (settingsObj.enableS3Backup !== undefined) {
|
||||||
|
runtimeConfig.enableS3Backup = Boolean(settingsObj.enableS3Backup);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
settingsObj.diskPauseThresholdPct !== undefined &&
|
||||||
|
settingsObj.diskPauseThresholdPct !== null
|
||||||
|
) {
|
||||||
|
const pct = Number(settingsObj.diskPauseThresholdPct);
|
||||||
|
if (Number.isFinite(pct) && (pct === 0 || (pct >= 50 && pct <= 99))) {
|
||||||
|
runtimeConfig.diskPauseThresholdPct = Math.floor(pct);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
settingsObj.compressAfterDays !== undefined &&
|
||||||
|
settingsObj.compressAfterDays !== null
|
||||||
|
) {
|
||||||
|
const parsedCompress = Number(settingsObj.compressAfterDays);
|
||||||
|
if (Number.isFinite(parsedCompress) && parsedCompress >= 0) {
|
||||||
|
const nextCompressDays = Math.floor(parsedCompress);
|
||||||
|
const changed = nextCompressDays !== runtimeConfig.compressAfterDays;
|
||||||
|
runtimeConfig.compressAfterDays = nextCompressDays;
|
||||||
|
if (changed && nextCompressDays > 0) {
|
||||||
|
void writer.compressOldPartitionsAsync(nextCompressDays).catch(() => undefined);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (settingsObj.autoPruneDays !== undefined && settingsObj.autoPruneDays !== null) {
|
if (settingsObj.autoPruneDays !== undefined && settingsObj.autoPruneDays !== null) {
|
||||||
const parsedPrune = Number(settingsObj.autoPruneDays);
|
const parsedPrune = Number(settingsObj.autoPruneDays);
|
||||||
if (Number.isFinite(parsedPrune) && parsedPrune >= 0) {
|
if (Number.isFinite(parsedPrune) && parsedPrune >= 0) {
|
||||||
const nextPruneDays = Math.floor(parsedPrune);
|
const nextPruneDays = Math.floor(parsedPrune);
|
||||||
const changed = nextPruneDays !== runtimeConfig.autoPruneDays;
|
const changed = nextPruneDays !== runtimeConfig.autoPruneDays;
|
||||||
runtimeConfig.autoPruneDays = nextPruneDays;
|
runtimeConfig.autoPruneDays = nextPruneDays;
|
||||||
if (changed && nextPruneDays > 0) {
|
if (changed && nextPruneDays > 0 && !runtimeConfig.enableS3Backup) {
|
||||||
void writer.pruneExpiredPartitionsAsync(nextPruneDays).catch(() => undefined);
|
void writer.pruneExpiredPartitionsAsync(nextPruneDays).catch(() => undefined);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -948,6 +1155,9 @@ async function ensureDatabaseSchema() {
|
|||||||
secret TEXT NOT NULL,
|
secret TEXT NOT NULL,
|
||||||
token TEXT NOT NULL,
|
token TEXT NOT NULL,
|
||||||
rate_limit INTEGER NOT NULL DEFAULT 100,
|
rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||||
|
storage_quota_mb INTEGER NOT NULL DEFAULT 0,
|
||||||
|
allowed_ips TEXT NOT NULL DEFAULT '',
|
||||||
|
expires_at TIMESTAMPTZ,
|
||||||
status TEXT NOT NULL DEFAULT 'active',
|
status TEXT NOT NULL DEFAULT 'active',
|
||||||
total_events BIGINT NOT NULL DEFAULT 0,
|
total_events BIGINT NOT NULL DEFAULT 0,
|
||||||
last_active TIMESTAMPTZ,
|
last_active TIMESTAMPTZ,
|
||||||
@@ -968,6 +1178,21 @@ async function ensureDatabaseSchema() {
|
|||||||
jwt_secret TEXT NOT NULL,
|
jwt_secret TEXT NOT NULL,
|
||||||
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
||||||
auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
||||||
|
compress_after_days INTEGER NOT NULL DEFAULT 0,
|
||||||
|
enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
s3_endpoint TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_region TEXT NOT NULL DEFAULT 'auto',
|
||||||
|
s3_bucket TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
|
||||||
|
s3_access_key_id TEXT NOT NULL DEFAULT '',
|
||||||
|
s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
|
||||||
|
enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
webhook_url TEXT NOT NULL DEFAULT '',
|
||||||
|
webhook_secret_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
|
||||||
|
enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
|
||||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
)
|
)
|
||||||
`);
|
`);
|
||||||
@@ -999,6 +1224,9 @@ async function ensureDatabaseSchema() {
|
|||||||
await pool.query(`
|
await pool.query(`
|
||||||
ALTER TABLE client_sdks
|
ALTER TABLE client_sdks
|
||||||
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
|
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||||
|
ADD COLUMN IF NOT EXISTS storage_quota_mb INTEGER NOT NULL DEFAULT 0,
|
||||||
|
ADD COLUMN IF NOT EXISTS allowed_ips TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ,
|
||||||
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
|
ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active',
|
||||||
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
|
ADD COLUMN IF NOT EXISTS total_events BIGINT NOT NULL DEFAULT 0,
|
||||||
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
|
ADD COLUMN IF NOT EXISTS last_active TIMESTAMPTZ,
|
||||||
@@ -1016,6 +1244,21 @@ async function ensureDatabaseSchema() {
|
|||||||
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
|
ADD COLUMN IF NOT EXISTS file_rotation_mb INTEGER NOT NULL DEFAULT 100,
|
||||||
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
ADD COLUMN IF NOT EXISTS enable_idempotency BOOLEAN NOT NULL DEFAULT true,
|
||||||
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
ADD COLUMN IF NOT EXISTS auto_prune_days INTEGER NOT NULL DEFAULT 30,
|
||||||
|
ADD COLUMN IF NOT EXISTS compress_after_days INTEGER NOT NULL DEFAULT 0,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_s3_backup BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_endpoint TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_region TEXT NOT NULL DEFAULT 'auto',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_bucket TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_prefix TEXT NOT NULL DEFAULT 'datastack/',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS s3_secret_access_key_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS disk_pause_threshold_pct INTEGER NOT NULL DEFAULT 95,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_webhook_alerts BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_url TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_secret_enc TEXT NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN IF NOT EXISTS webhook_cooldown_sec INTEGER NOT NULL DEFAULT 300,
|
||||||
|
ADD COLUMN IF NOT EXISTS enable_prometheus_metrics BOOLEAN NOT NULL DEFAULT false,
|
||||||
|
ADD COLUMN IF NOT EXISTS prometheus_metrics_token_enc TEXT NOT NULL DEFAULT '',
|
||||||
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
`);
|
`);
|
||||||
await pool.query(`
|
await pool.query(`
|
||||||
@@ -1043,7 +1286,7 @@ async function syncSettingsFromDatabase() {
|
|||||||
const res = await dbPool.query(
|
const res = await dbPool.query(
|
||||||
`SELECT max_events_per_sec, max_payload_size_kb, batch_flush_threshold,
|
`SELECT max_events_per_sec, max_payload_size_kb, batch_flush_threshold,
|
||||||
batch_flush_interval_ms, file_rotation_mb, jwt_secret, enable_idempotency,
|
batch_flush_interval_ms, file_rotation_mb, jwt_secret, enable_idempotency,
|
||||||
auto_prune_days
|
auto_prune_days, compress_after_days, enable_s3_backup, disk_pause_threshold_pct
|
||||||
FROM admin_settings
|
FROM admin_settings
|
||||||
WHERE id = 'default'`,
|
WHERE id = 'default'`,
|
||||||
);
|
);
|
||||||
@@ -1053,6 +1296,14 @@ async function syncSettingsFromDatabase() {
|
|||||||
row.auto_prune_days !== undefined && row.auto_prune_days !== null
|
row.auto_prune_days !== undefined && row.auto_prune_days !== null
|
||||||
? Number(row.auto_prune_days)
|
? Number(row.auto_prune_days)
|
||||||
: NaN;
|
: NaN;
|
||||||
|
const parsedCompress =
|
||||||
|
row.compress_after_days !== undefined && row.compress_after_days !== null
|
||||||
|
? Number(row.compress_after_days)
|
||||||
|
: NaN;
|
||||||
|
const parsedDiskPct =
|
||||||
|
row.disk_pause_threshold_pct !== undefined && row.disk_pause_threshold_pct !== null
|
||||||
|
? Number(row.disk_pause_threshold_pct)
|
||||||
|
: NaN;
|
||||||
|
|
||||||
applyRuntimeSettingsObject({
|
applyRuntimeSettingsObject({
|
||||||
maxEventsPerSec: Number(row.max_events_per_sec) || 100,
|
maxEventsPerSec: Number(row.max_events_per_sec) || 100,
|
||||||
@@ -1065,6 +1316,9 @@ async function syncSettingsFromDatabase() {
|
|||||||
fileRotationMb: Number(row.file_rotation_mb) || 100,
|
fileRotationMb: Number(row.file_rotation_mb) || 100,
|
||||||
enableIdempotency: Boolean(row.enable_idempotency),
|
enableIdempotency: Boolean(row.enable_idempotency),
|
||||||
autoPruneDays: Number.isFinite(parsedPrune) ? parsedPrune : 30,
|
autoPruneDays: Number.isFinite(parsedPrune) ? parsedPrune : 30,
|
||||||
|
compressAfterDays: Number.isFinite(parsedCompress) ? Math.max(0, parsedCompress) : 0,
|
||||||
|
enableS3Backup: Boolean(row.enable_s3_backup),
|
||||||
|
diskPauseThresholdPct: Number.isFinite(parsedDiskPct) ? parsedDiskPct : 95,
|
||||||
jwtSecret: row.jwt_secret ? String(row.jwt_secret).trim() : undefined,
|
jwtSecret: row.jwt_secret ? String(row.jwt_secret).trim() : undefined,
|
||||||
});
|
});
|
||||||
return true;
|
return true;
|
||||||
@@ -1338,6 +1592,9 @@ app.post('/internal/client-sync', (req, res) => {
|
|||||||
status: nextStatus,
|
status: nextStatus,
|
||||||
token: nextToken,
|
token: nextToken,
|
||||||
rateLimit: Math.max(1, Number(item.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(item.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(item.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof item.allowedIps === 'string' ? item.allowedIps : '',
|
||||||
|
expiresAt: item.expiresAt ? String(item.expiresAt) : null,
|
||||||
});
|
});
|
||||||
if (nextStatus === 'revoked' || (prev && nextToken && prev.token !== nextToken)) {
|
if (nextStatus === 'revoked' || (prev && nextToken && prev.token !== nextToken)) {
|
||||||
io.to(`client:${cid}`).disconnectSockets(true);
|
io.to(`client:${cid}`).disconnectSockets(true);
|
||||||
@@ -1380,6 +1637,9 @@ app.post('/internal/client-sync', (req, res) => {
|
|||||||
status: 'revoked',
|
status: 'revoked',
|
||||||
token: String(clientPayload.token || ''),
|
token: String(clientPayload.token || ''),
|
||||||
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
|
||||||
|
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
|
||||||
});
|
});
|
||||||
} else if (existingMem) {
|
} else if (existingMem) {
|
||||||
existingMem.status = 'revoked';
|
existingMem.status = 'revoked';
|
||||||
@@ -1390,6 +1650,9 @@ app.post('/internal/client-sync', (req, res) => {
|
|||||||
status: 'revoked',
|
status: 'revoked',
|
||||||
token: '',
|
token: '',
|
||||||
rateLimit: 100,
|
rateLimit: 100,
|
||||||
|
storageQuotaMb: 0,
|
||||||
|
allowedIps: '',
|
||||||
|
expiresAt: null,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
io.to(`client:${normalizedId}`).disconnectSockets(true);
|
io.to(`client:${normalizedId}`).disconnectSockets(true);
|
||||||
@@ -1402,6 +1665,9 @@ app.post('/internal/client-sync', (req, res) => {
|
|||||||
status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
|
status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
|
||||||
token: String(clientPayload.token || ''),
|
token: String(clientPayload.token || ''),
|
||||||
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
|
||||||
|
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
} else if (action === 'updated') {
|
} else if (action === 'updated') {
|
||||||
@@ -1412,6 +1678,9 @@ app.post('/internal/client-sync', (req, res) => {
|
|||||||
status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
|
status: clientPayload.status === 'revoked' ? 'revoked' : 'active',
|
||||||
token: String(clientPayload.token || ''),
|
token: String(clientPayload.token || ''),
|
||||||
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
rateLimit: Math.max(1, Number(clientPayload.rateLimit) || 100),
|
||||||
|
storageQuotaMb: Math.max(0, Math.floor(Number(clientPayload.storageQuotaMb) || 0)),
|
||||||
|
allowedIps: typeof clientPayload.allowedIps === 'string' ? clientPayload.allowedIps : '',
|
||||||
|
expiresAt: clientPayload.expiresAt ? String(clientPayload.expiresAt) : null,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
io.to(`client:${normalizedId}`).disconnectSockets(true);
|
io.to(`client:${normalizedId}`).disconnectSockets(true);
|
||||||
@@ -1542,7 +1811,12 @@ async function startIngestionServer() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pruneTimer = setInterval(() => {
|
pruneTimer = setInterval(() => {
|
||||||
if (runtimeConfig.autoPruneDays > 0) {
|
if (runtimeConfig.compressAfterDays > 0) {
|
||||||
|
void writer
|
||||||
|
.compressOldPartitionsAsync(runtimeConfig.compressAfterDays)
|
||||||
|
.catch(() => undefined);
|
||||||
|
}
|
||||||
|
if (runtimeConfig.autoPruneDays > 0 && !runtimeConfig.enableS3Backup) {
|
||||||
void writer
|
void writer
|
||||||
.pruneExpiredPartitionsAsync(runtimeConfig.autoPruneDays)
|
.pruneExpiredPartitionsAsync(runtimeConfig.autoPruneDays)
|
||||||
.catch(() => undefined);
|
.catch(() => undefined);
|
||||||
@@ -1594,4 +1868,6 @@ module.exports = {
|
|||||||
parseAllowedCorsOrigins,
|
parseAllowedCorsOrigins,
|
||||||
isOriginAllowedByCors,
|
isOriginAllowedByCors,
|
||||||
corsOriginDelegate,
|
corsOriginDelegate,
|
||||||
|
isClientIpAllowed,
|
||||||
|
isClientTokenExpired,
|
||||||
};
|
};
|
||||||
@@ -26,6 +26,34 @@ function normalizeClientId(clientId) {
|
|||||||
.replace(/[^a-z0-9_-]/g, "_");
|
.replace(/[^a-z0-9_-]/g, "_");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function extractSocketClientIp(socket) {
|
||||||
|
const headers = (socket && socket.handshake && socket.handshake.headers) || {};
|
||||||
|
const realIp =
|
||||||
|
typeof headers["x-real-ip"] === "string" ? headers["x-real-ip"].trim() : "";
|
||||||
|
if (realIp) {
|
||||||
|
return realIp.slice(0, 128);
|
||||||
|
}
|
||||||
|
const forwardedFor =
|
||||||
|
typeof headers["x-forwarded-for"] === "string"
|
||||||
|
? headers["x-forwarded-for"]
|
||||||
|
: "";
|
||||||
|
if (forwardedFor) {
|
||||||
|
const hops = forwardedFor
|
||||||
|
.split(",")
|
||||||
|
.map((part) => part.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
const rightmost = hops[hops.length - 1];
|
||||||
|
if (rightmost) {
|
||||||
|
return rightmost.slice(0, 128);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const addr =
|
||||||
|
(socket && socket.handshake && socket.handshake.address) ||
|
||||||
|
(socket && socket.conn && socket.conn.remoteAddress) ||
|
||||||
|
"";
|
||||||
|
return String(addr).trim() || "127.0.0.1";
|
||||||
|
}
|
||||||
|
|
||||||
function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
||||||
const getRuntimeConfig =
|
const getRuntimeConfig =
|
||||||
typeof options.getRuntimeConfig === "function"
|
typeof options.getRuntimeConfig === "function"
|
||||||
@@ -340,17 +368,20 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
const decoded = jwt.verify(cleanToken, config.jwtSecret);
|
const decoded = jwt.verify(cleanToken, config.jwtSecret);
|
||||||
|
|
||||||
const normalizedClientId = normalizeClientId(decoded.clientId);
|
const normalizedClientId = normalizeClientId(decoded.clientId);
|
||||||
|
const clientIp = extractSocketClientIp(socket);
|
||||||
const dbCheck = await verifyClientInDb(
|
const dbCheck = await verifyClientInDb(
|
||||||
normalizedClientId,
|
normalizedClientId,
|
||||||
cleanToken,
|
cleanToken,
|
||||||
true,
|
true,
|
||||||
|
clientIp,
|
||||||
);
|
);
|
||||||
if (!dbCheck.allowed) {
|
if (!dbCheck.allowed) {
|
||||||
return next(new Error(dbCheck.reason || "CLIENT_REVOKED"));
|
return next(new Error(dbCheck.reason || dbCheck.code || "CLIENT_REVOKED"));
|
||||||
}
|
}
|
||||||
|
|
||||||
socket.data.clientId = normalizedClientId;
|
socket.data.clientId = normalizedClientId;
|
||||||
socket.data.token = cleanToken;
|
socket.data.token = cleanToken;
|
||||||
|
socket.data.clientIp = clientIp;
|
||||||
socket.data.rateLimit =
|
socket.data.rateLimit =
|
||||||
dbCheck.rateLimit || decoded.rateLimit || config.maxEventsPerSec || 100;
|
dbCheck.rateLimit || decoded.rateLimit || config.maxEventsPerSec || 100;
|
||||||
next();
|
next();
|
||||||
@@ -401,16 +432,18 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
// Single event
|
// Single event
|
||||||
socket.on("sync_data", async (payload, callback) => {
|
socket.on("sync_data", async (payload, callback) => {
|
||||||
try {
|
try {
|
||||||
|
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
|
||||||
const dbCheck = await verifyClientInDb(
|
const dbCheck = await verifyClientInDb(
|
||||||
clientId,
|
clientId,
|
||||||
socket.data.token,
|
socket.data.token,
|
||||||
false,
|
false,
|
||||||
|
clientIp,
|
||||||
);
|
);
|
||||||
if (!dbCheck.allowed) {
|
if (!dbCheck.allowed) {
|
||||||
if (callback) {
|
if (callback) {
|
||||||
callback({
|
callback({
|
||||||
success: false,
|
success: false,
|
||||||
code: "CLIENT_REVOKED",
|
code: dbCheck.code || "CLIENT_REVOKED",
|
||||||
error: dbCheck.reason || "CLIENT_REVOKED",
|
error: dbCheck.reason || "CLIENT_REVOKED",
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -418,6 +451,25 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (worker && typeof worker.checkStorageAdmission === "function") {
|
||||||
|
const config = getRuntimeConfig();
|
||||||
|
const admission = await worker.checkStorageAdmission(
|
||||||
|
clientId,
|
||||||
|
dbCheck.storageQuotaMb || 0,
|
||||||
|
config.diskPauseThresholdPct ?? 95,
|
||||||
|
);
|
||||||
|
if (!admission.allowed) {
|
||||||
|
if (callback) {
|
||||||
|
callback({
|
||||||
|
success: false,
|
||||||
|
code: admission.code,
|
||||||
|
error: admission.reason,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const effectiveClientRate =
|
const effectiveClientRate =
|
||||||
dbCheck.rateLimit || socket.data.rateLimit || 100;
|
dbCheck.rateLimit || socket.data.rateLimit || 100;
|
||||||
const safePayload =
|
const safePayload =
|
||||||
@@ -499,10 +551,12 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
|
||||||
const dbCheck = await verifyClientInDb(
|
const dbCheck = await verifyClientInDb(
|
||||||
clientId,
|
clientId,
|
||||||
socket.data.token,
|
socket.data.token,
|
||||||
false,
|
false,
|
||||||
|
clientIp,
|
||||||
);
|
);
|
||||||
if (!dbCheck.allowed) {
|
if (!dbCheck.allowed) {
|
||||||
if (callback) {
|
if (callback) {
|
||||||
@@ -510,7 +564,7 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
success: false,
|
success: false,
|
||||||
processed: 0,
|
processed: 0,
|
||||||
rejected: batch.length,
|
rejected: batch.length,
|
||||||
code: "CLIENT_REVOKED",
|
code: dbCheck.code || "CLIENT_REVOKED",
|
||||||
error: dbCheck.reason || "CLIENT_REVOKED",
|
error: dbCheck.reason || "CLIENT_REVOKED",
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -518,6 +572,27 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (worker && typeof worker.checkStorageAdmission === "function") {
|
||||||
|
const config = getRuntimeConfig();
|
||||||
|
const admission = await worker.checkStorageAdmission(
|
||||||
|
clientId,
|
||||||
|
dbCheck.storageQuotaMb || 0,
|
||||||
|
config.diskPauseThresholdPct ?? 95,
|
||||||
|
);
|
||||||
|
if (!admission.allowed) {
|
||||||
|
if (callback) {
|
||||||
|
callback({
|
||||||
|
success: false,
|
||||||
|
processed: 0,
|
||||||
|
rejected: batch.length,
|
||||||
|
code: admission.code,
|
||||||
|
error: admission.reason,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const effectiveClientRate =
|
const effectiveClientRate =
|
||||||
dbCheck.rateLimit || socket.data.rateLimit || 100;
|
dbCheck.rateLimit || socket.data.rateLimit || 100;
|
||||||
let processed = 0;
|
let processed = 0;
|
||||||
@@ -646,16 +721,19 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
// Heartbeat & Reconnection Verification Event
|
// Heartbeat & Reconnection Verification Event
|
||||||
socket.on("ping_verify", async (_payload, callback) => {
|
socket.on("ping_verify", async (_payload, callback) => {
|
||||||
try {
|
try {
|
||||||
|
const clientIp = socket.data.clientIp || extractSocketClientIp(socket);
|
||||||
const dbCheck = await verifyClientInDb(
|
const dbCheck = await verifyClientInDb(
|
||||||
clientId,
|
clientId,
|
||||||
socket.data.token,
|
socket.data.token,
|
||||||
true,
|
true,
|
||||||
|
clientIp,
|
||||||
);
|
);
|
||||||
if (!dbCheck.allowed) {
|
if (!dbCheck.allowed) {
|
||||||
if (callback) {
|
if (callback) {
|
||||||
callback({
|
callback({
|
||||||
success: false,
|
success: false,
|
||||||
status: "REVOKED",
|
status: "REVOKED",
|
||||||
|
code: dbCheck.code || "CLIENT_REVOKED",
|
||||||
error: dbCheck.reason || "CLIENT_REVOKED",
|
error: dbCheck.reason || "CLIENT_REVOKED",
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -691,4 +769,4 @@ function setupSocketHandlers(io, bufferQueue, worker, options = {}) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
module.exports = { setupSocketHandlers, normalizeClientId };
|
module.exports = { setupSocketHandlers, normalizeClientId, extractSocketClientIp };
|
||||||
+233
-5
@@ -1,12 +1,15 @@
|
|||||||
const crypto = require('node:crypto');
|
const crypto = require('node:crypto');
|
||||||
const fs = require('fs');
|
const fs = require('fs');
|
||||||
const path = require('path');
|
const path = require('path');
|
||||||
|
const { pipeline } = require('node:stream/promises');
|
||||||
|
const zlib = require('node:zlib');
|
||||||
const { Parser } = require('json2csv');
|
const { Parser } = require('json2csv');
|
||||||
|
|
||||||
const LOCK_RETRY_MIN_MS = 5;
|
const LOCK_RETRY_MIN_MS = 5;
|
||||||
const LOCK_RETRY_MAX_MS = 18;
|
const LOCK_RETRY_MAX_MS = 18;
|
||||||
const LOCK_TIMEOUT_MS = 10000;
|
const LOCK_TIMEOUT_MS = 10000;
|
||||||
const STALE_LOCK_AGE_MS = 30000;
|
const STALE_LOCK_AGE_MS = 30000;
|
||||||
|
const QUOTA_CACHE_TTL_MS = 30000;
|
||||||
|
|
||||||
function sleepMs(ms) {
|
function sleepMs(ms) {
|
||||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||||
@@ -23,15 +26,42 @@ function resolveDefaultStorageDir() {
|
|||||||
/**
|
/**
|
||||||
* Background CSV Writer Worker
|
* Background CSV Writer Worker
|
||||||
* Flushes batches of real-time events to partitioned CSV files.
|
* Flushes batches of real-time events to partitioned CSV files.
|
||||||
* Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv
|
* Path convention: /storage/data/{clientId}/{YYYY-MM-DD}.csv(.gz)
|
||||||
*/
|
*/
|
||||||
class CsvWriterWorker {
|
class CsvWriterWorker {
|
||||||
constructor(baseDir = resolveDefaultStorageDir(), maxFileSize = 100 * 1024 * 1024) {
|
constructor(baseDir = resolveDefaultStorageDir(), maxFileSize = 100 * 1024 * 1024) {
|
||||||
this.baseDir = baseDir;
|
this.baseDir = baseDir;
|
||||||
this.maxFileSize = maxFileSize;
|
this.maxFileSize = maxFileSize;
|
||||||
|
this.diskUsageCache = null;
|
||||||
|
this.clientBytesCache = new Map();
|
||||||
|
this.statfsOverrideForTests = null;
|
||||||
this.ensureBaseDir();
|
this.ensureBaseDir();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
isPartitionFilename(filename) {
|
||||||
|
return typeof filename === 'string' && (filename.endsWith('.csv') || filename.endsWith('.csv.gz'));
|
||||||
|
}
|
||||||
|
|
||||||
|
extractDateFromFilename(filename) {
|
||||||
|
return String(filename || '')
|
||||||
|
.replace(/\.csv(?:\.gz)?$/, '')
|
||||||
|
.split('-part')[0];
|
||||||
|
}
|
||||||
|
|
||||||
|
clearQuotaCaches(clientId) {
|
||||||
|
this.diskUsageCache = null;
|
||||||
|
if (clientId) {
|
||||||
|
this.clientBytesCache.delete(this.sanitizeClientId(clientId));
|
||||||
|
} else {
|
||||||
|
this.clientBytesCache.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
setStatfsOverrideForTests(provider) {
|
||||||
|
this.statfsOverrideForTests = typeof provider === 'function' ? provider : null;
|
||||||
|
this.diskUsageCache = null;
|
||||||
|
}
|
||||||
|
|
||||||
setMaxFileSizeMb(megabytes) {
|
setMaxFileSizeMb(megabytes) {
|
||||||
const mb = Number(megabytes);
|
const mb = Number(megabytes);
|
||||||
if (Number.isFinite(mb) && mb >= 1) {
|
if (Number.isFinite(mb) && mb >= 1) {
|
||||||
@@ -282,6 +312,7 @@ class CsvWriterWorker {
|
|||||||
fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
|
fs.writeFileSync(resolvedPath, fullCsv, 'utf8');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
this.clearQuotaCaches(clientId);
|
||||||
return resolvedPath;
|
return resolvedPath;
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -291,6 +322,101 @@ class CsvWriterWorker {
|
|||||||
return { written: events.length, files };
|
return { written: events.length, files };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async compressOldPartitionsAsync(compressAfterDays) {
|
||||||
|
const days = Number(compressAfterDays);
|
||||||
|
if (!Number.isFinite(days) || days <= 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const baseStat = await fs.promises.lstat(this.baseDir).catch(() => null);
|
||||||
|
if (!baseStat || !baseStat.isDirectory()) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const cutoffDate = new Date(Date.now() - Math.floor(days) * 86400000)
|
||||||
|
.toISOString()
|
||||||
|
.split('T')[0];
|
||||||
|
|
||||||
|
let compressedCount = 0;
|
||||||
|
try {
|
||||||
|
const clientDirs = await fs.promises.readdir(this.baseDir);
|
||||||
|
for (const clientDir of clientDirs) {
|
||||||
|
const fullClientPath = path.join(this.baseDir, clientDir);
|
||||||
|
const dirLstat = await fs.promises.lstat(fullClientPath).catch(() => null);
|
||||||
|
if (!dirLstat || dirLstat.isSymbolicLink() || !dirLstat.isDirectory()) continue;
|
||||||
|
|
||||||
|
const files = await fs.promises.readdir(fullClientPath).catch(() => []);
|
||||||
|
for (const file of files) {
|
||||||
|
if (!file.endsWith('.csv')) continue;
|
||||||
|
const datePart = this.extractDateFromFilename(file);
|
||||||
|
if (!(/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate)) continue;
|
||||||
|
const fullPath = path.join(fullClientPath, file);
|
||||||
|
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
||||||
|
if (!fileLstat || fileLstat.isSymbolicLink() || !fileLstat.isFile()) continue;
|
||||||
|
|
||||||
|
const lockPath = path.join(fullClientPath, `.${datePart}.csv.lock`);
|
||||||
|
const lockValue = `${process.pid}:${Date.now()}:${crypto.randomUUID()}`;
|
||||||
|
let acquired = false;
|
||||||
|
const tryAcquireLock = async () => {
|
||||||
|
const handle = await fs.promises.open(lockPath, 'wx');
|
||||||
|
try {
|
||||||
|
await handle.writeFile(lockValue, 'utf8');
|
||||||
|
} finally {
|
||||||
|
await handle.close();
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
acquired = await tryAcquireLock();
|
||||||
|
} catch (err) {
|
||||||
|
if (err && err.code === 'EEXIST') {
|
||||||
|
await this.removeStaleLockAtomicallyAsync(lockPath);
|
||||||
|
try {
|
||||||
|
acquired = await tryAcquireLock();
|
||||||
|
} catch {
|
||||||
|
acquired = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!acquired) continue;
|
||||||
|
|
||||||
|
let targetGzPath = `${fullPath}.gz`;
|
||||||
|
if (fs.existsSync(targetGzPath)) {
|
||||||
|
const stem = fullPath.replace(/\.csv$/, '');
|
||||||
|
let suffix = 2;
|
||||||
|
while (fs.existsSync(`${stem}-gz${suffix}.csv.gz`)) {
|
||||||
|
suffix += 1;
|
||||||
|
}
|
||||||
|
targetGzPath = `${stem}-gz${suffix}.csv.gz`;
|
||||||
|
}
|
||||||
|
const tmpGzPath = `${targetGzPath}.tmp.${crypto.randomUUID()}`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
await pipeline(
|
||||||
|
fs.createReadStream(fullPath, { highWaterMark: 64 * 1024 }),
|
||||||
|
zlib.createGzip({ level: 6 }),
|
||||||
|
fs.createWriteStream(tmpGzPath),
|
||||||
|
);
|
||||||
|
await fs.promises.utimes(tmpGzPath, fileLstat.atime, fileLstat.mtime).catch(() => undefined);
|
||||||
|
await fs.promises.rename(tmpGzPath, targetGzPath);
|
||||||
|
await fs.promises.unlink(fullPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
|
compressedCount += 1;
|
||||||
|
} catch {
|
||||||
|
await fs.promises.unlink(tmpGzPath).catch(() => undefined);
|
||||||
|
} finally {
|
||||||
|
await this.releaseLockIfOwnedAsync(lockPath, lockValue);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Ignore directory scan failure
|
||||||
|
}
|
||||||
|
return compressedCount;
|
||||||
|
}
|
||||||
|
|
||||||
pruneExpiredPartitions(autoPruneDays) {
|
pruneExpiredPartitions(autoPruneDays) {
|
||||||
const days = Number(autoPruneDays);
|
const days = Number(autoPruneDays);
|
||||||
if (!Number.isFinite(days) || days <= 0 || !fs.existsSync(this.baseDir)) {
|
if (!Number.isFinite(days) || days <= 0 || !fs.existsSync(this.baseDir)) {
|
||||||
@@ -316,8 +442,8 @@ class CsvWriterWorker {
|
|||||||
|
|
||||||
const files = fs.readdirSync(fullClientPath);
|
const files = fs.readdirSync(fullClientPath);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
||||||
const fullPath = path.join(fullClientPath, file);
|
const fullPath = path.join(fullClientPath, file);
|
||||||
let fileLstat;
|
let fileLstat;
|
||||||
@@ -361,6 +487,7 @@ class CsvWriterWorker {
|
|||||||
try {
|
try {
|
||||||
if (fs.existsSync(fullPath)) {
|
if (fs.existsSync(fullPath)) {
|
||||||
fs.unlinkSync(fullPath);
|
fs.unlinkSync(fullPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
prunedCount += 1;
|
prunedCount += 1;
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
@@ -402,8 +529,8 @@ class CsvWriterWorker {
|
|||||||
|
|
||||||
const files = await fs.promises.readdir(fullClientPath).catch(() => []);
|
const files = await fs.promises.readdir(fullClientPath).catch(() => []);
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!file.endsWith('.csv')) continue;
|
if (!this.isPartitionFilename(file)) continue;
|
||||||
const datePart = file.replace('.csv', '').split('-part')[0];
|
const datePart = this.extractDateFromFilename(file);
|
||||||
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
if (/^\d{4}-\d{2}-\d{2}$/.test(datePart) && datePart < cutoffDate) {
|
||||||
const fullPath = path.join(fullClientPath, file);
|
const fullPath = path.join(fullClientPath, file);
|
||||||
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
const fileLstat = await fs.promises.lstat(fullPath).catch(() => null);
|
||||||
@@ -439,6 +566,7 @@ class CsvWriterWorker {
|
|||||||
|
|
||||||
try {
|
try {
|
||||||
await fs.promises.unlink(fullPath);
|
await fs.promises.unlink(fullPath);
|
||||||
|
this.clearQuotaCaches(clientDir);
|
||||||
prunedCount += 1;
|
prunedCount += 1;
|
||||||
} catch {
|
} catch {
|
||||||
// Ignore individual file unlink failure
|
// Ignore individual file unlink failure
|
||||||
@@ -453,6 +581,106 @@ class CsvWriterWorker {
|
|||||||
}
|
}
|
||||||
return prunedCount;
|
return prunedCount;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async getDiskUsageAsync() {
|
||||||
|
const now = Date.now();
|
||||||
|
if (this.diskUsageCache && now - this.diskUsageCache.checkedAt < QUOTA_CACHE_TTL_MS) {
|
||||||
|
return this.diskUsageCache;
|
||||||
|
}
|
||||||
|
|
||||||
|
let probeDir = path.resolve(this.baseDir);
|
||||||
|
while (!fs.existsSync(probeDir)) {
|
||||||
|
const parent = path.dirname(probeDir);
|
||||||
|
if (parent === probeDir) break;
|
||||||
|
probeDir = parent;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const statfsRes = this.statfsOverrideForTests
|
||||||
|
? await this.statfsOverrideForTests(probeDir)
|
||||||
|
: await fs.promises.statfs(probeDir);
|
||||||
|
const bsize = Math.max(1, Number(statfsRes.bsize) || 4096);
|
||||||
|
const totalBytes = Math.max(0, Number(statfsRes.blocks) * bsize);
|
||||||
|
const availableBytes = Math.max(0, Number(statfsRes.bavail) * bsize);
|
||||||
|
const usedBytes = Math.max(0, totalBytes - availableBytes);
|
||||||
|
const usedPct =
|
||||||
|
totalBytes > 0
|
||||||
|
? Number(Math.min(100, Math.max(0, (usedBytes / totalBytes) * 100)).toFixed(2))
|
||||||
|
: 0;
|
||||||
|
const snapshot = { usedPct, availableBytes, totalBytes, checkedAt: now };
|
||||||
|
this.diskUsageCache = snapshot;
|
||||||
|
return snapshot;
|
||||||
|
} catch {
|
||||||
|
const fallback = { usedPct: 0, availableBytes: 0, totalBytes: 0, checkedAt: now };
|
||||||
|
this.diskUsageCache = fallback;
|
||||||
|
return fallback;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async getClientStorageBytesAsync(clientId) {
|
||||||
|
const safeClientId = this.sanitizeClientId(clientId);
|
||||||
|
const now = Date.now();
|
||||||
|
const cached = this.clientBytesCache.get(safeClientId);
|
||||||
|
if (cached && now - cached.checkedAt < QUOTA_CACHE_TTL_MS) {
|
||||||
|
return cached.bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
const resolvedBase = path.resolve(this.baseDir);
|
||||||
|
const clientDir = path.resolve(resolvedBase, safeClientId);
|
||||||
|
if (!clientDir.startsWith(resolvedBase + path.sep)) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
let totalBytes = 0;
|
||||||
|
try {
|
||||||
|
const dirStat = await fs.promises.lstat(clientDir).catch(() => null);
|
||||||
|
if (dirStat && !dirStat.isSymbolicLink() && dirStat.isDirectory()) {
|
||||||
|
const entries = await fs.promises.readdir(clientDir).catch(() => []);
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (!this.isPartitionFilename(entry)) continue;
|
||||||
|
const filePath = path.join(clientDir, entry);
|
||||||
|
const stats = await fs.promises.lstat(filePath).catch(() => null);
|
||||||
|
if (stats && !stats.isSymbolicLink() && stats.isFile()) {
|
||||||
|
totalBytes += stats.size;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
|
||||||
|
this.clientBytesCache.set(safeClientId, { checkedAt: now, bytes: totalBytes });
|
||||||
|
return totalBytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
async checkStorageAdmission(clientId, clientQuotaMb, diskPauseThresholdPct) {
|
||||||
|
const threshold = Number(diskPauseThresholdPct);
|
||||||
|
if (Number.isFinite(threshold) && threshold >= 50 && threshold <= 99) {
|
||||||
|
const disk = await this.getDiskUsageAsync();
|
||||||
|
if (disk.totalBytes > 0 && disk.usedPct >= threshold) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'DISK_SPACE_LOW',
|
||||||
|
reason: `DISK_SPACE_LOW: Disk utilization (${disk.usedPct.toFixed(1)}%) reached safety pause threshold (${threshold}%).`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const quotaMb = Number(clientQuotaMb);
|
||||||
|
if (Number.isFinite(quotaMb) && quotaMb > 0) {
|
||||||
|
const currentBytes = await this.getClientStorageBytesAsync(clientId);
|
||||||
|
const quotaBytes = Math.floor(quotaMb) * 1024 * 1024;
|
||||||
|
if (currentBytes >= quotaBytes) {
|
||||||
|
return {
|
||||||
|
allowed: false,
|
||||||
|
code: 'STORAGE_QUOTA_EXCEEDED',
|
||||||
|
reason: `STORAGE_QUOTA_EXCEEDED: Client '${clientId}' used ${(currentBytes / (1024 * 1024)).toFixed(2)} MB of ${quotaMb} MB quota.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { allowed: true };
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module.exports = { CsvWriterWorker, resolveDefaultStorageDir };
|
module.exports = { CsvWriterWorker, resolveDefaultStorageDir };
|
||||||
@@ -0,0 +1,727 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import fs from "node:fs/promises";
|
||||||
|
import http from "node:http";
|
||||||
|
import { createRequire } from "node:module";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import test from "node:test";
|
||||||
|
import zlib from "node:zlib";
|
||||||
|
import { NextRequest } from "next/server";
|
||||||
|
import { io as ioClient, type Socket } from "socket.io-client";
|
||||||
|
import { DELETE as deleteClientRoute, PATCH as patchClientRoute } from "../app/api/clients/[id]/route";
|
||||||
|
import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route";
|
||||||
|
import { GET as dataGet } from "../app/api/data/route";
|
||||||
|
import { GET as eventsGet, POST as eventsPost } from "../app/api/events/route";
|
||||||
|
import { GET as exportGet } from "../app/api/export/route";
|
||||||
|
import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route";
|
||||||
|
import { POST as archivePost } from "../app/api/storage/archive/route";
|
||||||
|
import { GET as storageGet } from "../app/api/storage/route";
|
||||||
|
import { POST as s3TestPost } from "../app/api/storage/s3-test/route";
|
||||||
|
import { SettingsManager } from "../core/adminSettings/settingsManager";
|
||||||
|
import {
|
||||||
|
createAdminSessionToken,
|
||||||
|
getAdminSessionSecret,
|
||||||
|
} from "../core/auth/adminSession";
|
||||||
|
import {
|
||||||
|
createInitialAdmin,
|
||||||
|
resetAdminStateForTests,
|
||||||
|
} from "../core/auth/adminUsers";
|
||||||
|
import { DataBufferEngine } from "../core/data/buffer";
|
||||||
|
import { SDKManager } from "../core/sdk/sdkManager";
|
||||||
|
import {
|
||||||
|
decryptSecret,
|
||||||
|
encryptSecret,
|
||||||
|
getMaskedHintFromEncrypted,
|
||||||
|
SecretVaultError,
|
||||||
|
} from "../core/Security/secretVault";
|
||||||
|
import { CSVStorageManager } from "../core/Storage/csvStorage";
|
||||||
|
import {
|
||||||
|
buildS3ObjectKey,
|
||||||
|
normalizeS3EndpointUrl,
|
||||||
|
testS3Connection,
|
||||||
|
uploadFileToS3,
|
||||||
|
} from "../core/Storage/s3Archive";
|
||||||
|
import {
|
||||||
|
DEFAULT_EDITABLE_SETTINGS,
|
||||||
|
settingsToNumericDrafts,
|
||||||
|
validateEditableSettingsDraft,
|
||||||
|
validateSingleNumericSetting,
|
||||||
|
} from "../lib/vconUiHelpers";
|
||||||
|
|
||||||
|
const require = createRequire(import.meta.url);
|
||||||
|
|
||||||
|
async function makeAdminCookie(): Promise<string> {
|
||||||
|
resetAdminStateForTests();
|
||||||
|
await createInitialAdmin("v2-p1@datastack.dev", "V2Phase01Pass!");
|
||||||
|
const secret = getAdminSessionSecret()!;
|
||||||
|
const token = await createAdminSessionToken(
|
||||||
|
"v2-p1@datastack.dev",
|
||||||
|
secret,
|
||||||
|
Date.now() + 3600_000,
|
||||||
|
1,
|
||||||
|
);
|
||||||
|
return `datastack_admin_session=${token}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
test("V2-Phase-01 [Vault]: encryptSecret/decryptSecret AES-256-GCM roundtrip, tamper detection & masked hints", () => {
|
||||||
|
const secretA = "test-master-secret-key-at-least-32-characters";
|
||||||
|
const secretB = "different-master-secret-key-at-least-32-chars";
|
||||||
|
|
||||||
|
const encrypted = encryptSecret("r2_secret_value_987654321", secretA);
|
||||||
|
assert.ok(encrypted.startsWith("v1:"));
|
||||||
|
assert.equal(decryptSecret(encrypted, secretA), "r2_secret_value_987654321");
|
||||||
|
assert.equal(getMaskedHintFromEncrypted(encrypted, secretA), "••••4321");
|
||||||
|
|
||||||
|
// Tampered ciphertext throws SecretVaultError (V2-E01)
|
||||||
|
const parts = encrypted.split(":");
|
||||||
|
const flippedChar = parts[3].startsWith("A") ? "B" : "A";
|
||||||
|
const tampered = `${parts[0]}:${parts[1]}:${parts[2]}:${flippedChar}${parts[3].slice(1)}`;
|
||||||
|
assert.throws(
|
||||||
|
() => decryptSecret(tampered, secretA),
|
||||||
|
(err: unknown) =>
|
||||||
|
err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED",
|
||||||
|
);
|
||||||
|
|
||||||
|
// Wrong master key throws SecretVaultError
|
||||||
|
assert.throws(
|
||||||
|
() => decryptSecret(encrypted, secretB),
|
||||||
|
(err: unknown) =>
|
||||||
|
err instanceof SecretVaultError && err.code === "SECRET_VAULT_DECRYPT_FAILED",
|
||||||
|
);
|
||||||
|
|
||||||
|
// Too-short master secret rejects key derivation
|
||||||
|
assert.throws(
|
||||||
|
() => encryptSecret("hello", "short"),
|
||||||
|
(err: unknown) =>
|
||||||
|
err instanceof SecretVaultError && err.code === "SECRET_VAULT_KEY_UNAVAILABLE",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-01 [S3 SigV4]: buildS3ObjectKey, normalizeS3EndpointUrl, testS3Connection & uploadFileToS3 against mock S3 server", async () => {
|
||||||
|
assert.equal(
|
||||||
|
buildS3ObjectKey("/datastack/", "client_01", "2026-10-01.csv.gz"),
|
||||||
|
"datastack/client_01/2026-10-01.csv.gz",
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
normalizeS3EndpointUrl("", "us-east-1").url.origin,
|
||||||
|
"https://s3.us-east-1.amazonaws.com",
|
||||||
|
);
|
||||||
|
|
||||||
|
const requests: Array<{
|
||||||
|
method: string;
|
||||||
|
url: string;
|
||||||
|
auth: string;
|
||||||
|
bodyLen: number;
|
||||||
|
}> = [];
|
||||||
|
let failUploads = false;
|
||||||
|
|
||||||
|
const mockS3 = http.createServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (chunk: Buffer) => chunks.push(chunk));
|
||||||
|
req.on("end", () => {
|
||||||
|
const body = Buffer.concat(chunks);
|
||||||
|
requests.push({
|
||||||
|
method: req.method || "",
|
||||||
|
url: req.url || "",
|
||||||
|
auth: String(req.headers.authorization || ""),
|
||||||
|
bodyLen: body.byteLength,
|
||||||
|
});
|
||||||
|
if (req.method === "PUT" && failUploads) {
|
||||||
|
res.writeHead(500, { "content-type": "text/plain" });
|
||||||
|
res.end("Simulated S3 outage");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (req.method === "PUT") {
|
||||||
|
res.writeHead(200, { etag: '"mock-etag-1"' });
|
||||||
|
res.end("");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
res.writeHead(200, { "content-type": "application/xml" });
|
||||||
|
res.end("<ListBucketResult></ListBucketResult>");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => mockS3.listen(0, "127.0.0.1", resolve));
|
||||||
|
const addr = mockS3.address() as AddressInfo;
|
||||||
|
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-s3-"));
|
||||||
|
|
||||||
|
try {
|
||||||
|
const samplePath = path.join(tmpDir, "2026-10-01.csv.gz");
|
||||||
|
const gzData = zlib.gzipSync(
|
||||||
|
Buffer.from('clientId,eventId,timestamp,payload\n"c1","e1","2026-10-01T00:00:00Z","{}"\n'),
|
||||||
|
);
|
||||||
|
await fs.writeFile(samplePath, gzData);
|
||||||
|
|
||||||
|
const cfg = {
|
||||||
|
endpoint: `http://127.0.0.1:${addr.port}`,
|
||||||
|
region: "auto",
|
||||||
|
bucket: "ds-archive",
|
||||||
|
prefix: "datastack/",
|
||||||
|
accessKeyId: "AKIA_TEST_123",
|
||||||
|
secretAccessKey: "secret_access_key_xyz_999",
|
||||||
|
};
|
||||||
|
|
||||||
|
const probe = await testS3Connection(cfg);
|
||||||
|
assert.equal(probe.ok, true);
|
||||||
|
assert.equal(probe.status, 200);
|
||||||
|
assert.match(requests[0].auth, /^AWS4-HMAC-SHA256 Credential=AKIA_TEST_123\//);
|
||||||
|
assert.match(requests[0].url, /^\/ds-archive\?list-type=2&max-keys=1$/);
|
||||||
|
|
||||||
|
const uploadOk = await uploadFileToS3(
|
||||||
|
cfg,
|
||||||
|
samplePath,
|
||||||
|
"datastack/c1/2026-10-01.csv.gz",
|
||||||
|
);
|
||||||
|
assert.equal(uploadOk.ok, true);
|
||||||
|
assert.equal(uploadOk.etag, '"mock-etag-1"');
|
||||||
|
assert.equal(requests[1].method, "PUT");
|
||||||
|
assert.equal(requests[1].url, "/ds-archive/datastack/c1/2026-10-01.csv.gz");
|
||||||
|
assert.equal(requests[1].bodyLen, gzData.byteLength);
|
||||||
|
|
||||||
|
failUploads = true;
|
||||||
|
const uploadErr = await uploadFileToS3(
|
||||||
|
cfg,
|
||||||
|
samplePath,
|
||||||
|
"datastack/c1/2026-10-01.csv.gz",
|
||||||
|
);
|
||||||
|
assert.equal(uploadErr.ok, false);
|
||||||
|
assert.equal(uploadErr.code, "S3_ARCHIVE_FAILED");
|
||||||
|
assert.match(uploadErr.error || "", /HTTP 500/);
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) => mockS3.close(() => resolve()));
|
||||||
|
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-01 [Compression, Export, S3 Safe Prune V2-E02, Corrupt Gzip V2-E03]", async () => {
|
||||||
|
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
const prevDbUrl = process.env.DATABASE_URL;
|
||||||
|
const origBaseStorageDir = CSVStorageManager.baseStorageDir;
|
||||||
|
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-gz-"));
|
||||||
|
delete process.env.DATABASE_URL;
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = tmpDir;
|
||||||
|
CSVStorageManager.baseStorageDir = tmpDir;
|
||||||
|
CSVStorageManager.clearRowCountCache();
|
||||||
|
CSVStorageManager.clearQuotaCaches();
|
||||||
|
|
||||||
|
const uploadedKeys: string[] = [];
|
||||||
|
let failNextUpload = false;
|
||||||
|
const mockS3 = http.createServer((req, res) => {
|
||||||
|
req.on("data", () => undefined);
|
||||||
|
req.on("end", () => {
|
||||||
|
if (req.method === "PUT") {
|
||||||
|
if (failNextUpload) {
|
||||||
|
res.writeHead(502);
|
||||||
|
res.end("Upstream error");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
uploadedKeys.push(req.url || "");
|
||||||
|
res.writeHead(200, { etag: '"ok"' });
|
||||||
|
res.end("");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
res.writeHead(200);
|
||||||
|
res.end("<OK/>");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => mockS3.listen(0, "127.0.0.1", resolve));
|
||||||
|
const s3Addr = mockS3.address() as AddressInfo;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const cookie = await makeAdminCookie();
|
||||||
|
const today = new Date().toISOString().slice(0, 10);
|
||||||
|
const oldDate = new Date(Date.now() - 45 * 86_400_000).toISOString().slice(0, 10);
|
||||||
|
await SettingsManager.updateSettings({
|
||||||
|
autoPruneDays: 90,
|
||||||
|
compressAfterDays: 0,
|
||||||
|
enableS3Backup: false,
|
||||||
|
diskPauseThresholdPct: 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
// 1. Write 2 old rows (45 days ago) and 1 today row
|
||||||
|
await CSVStorageManager.appendBatch([
|
||||||
|
{
|
||||||
|
clientId: "v2_alpha",
|
||||||
|
eventId: "evt_old_1",
|
||||||
|
timestamp: `${oldDate}T10:00:00.000Z`,
|
||||||
|
payload: '{"stage":"old","seq":1}',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
clientId: "v2_alpha",
|
||||||
|
eventId: "evt_old_2",
|
||||||
|
timestamp: `${oldDate}T11:00:00.000Z`,
|
||||||
|
payload: '{"stage":"old","seq":2}',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
clientId: "v2_alpha",
|
||||||
|
eventId: "evt_today_1",
|
||||||
|
timestamp: `${today}T09:00:00.000Z`,
|
||||||
|
payload: '{"stage":"live","seq":3}',
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
|
||||||
|
// 2. Compress partitions older than 30 days -> ${oldDate}.csv becomes .csv.gz; today stays .csv
|
||||||
|
const compressed = await CSVStorageManager.compressOldPartitionsAsync(30);
|
||||||
|
assert.equal(compressed, 1);
|
||||||
|
|
||||||
|
const files = await CSVStorageManager.listAllFilesAsync();
|
||||||
|
assert.equal(files.length, 2);
|
||||||
|
const gzMeta = files.find((f) => f.filename === `${oldDate}.csv.gz`);
|
||||||
|
assert.ok(gzMeta, `Expected compressed ${oldDate}.csv.gz partition`);
|
||||||
|
assert.equal(gzMeta.linesCount, 2);
|
||||||
|
assert.equal(gzMeta.compressed, true);
|
||||||
|
|
||||||
|
// Original uncompressed ${oldDate}.csv was removed after atomic rename
|
||||||
|
const legacyCsv = path.join(tmpDir, "v2_alpha", `${oldDate}.csv`);
|
||||||
|
await assert.rejects(() => fs.stat(legacyCsv));
|
||||||
|
|
||||||
|
// 3. Verify GET /api/data reads .csv.gz transparently
|
||||||
|
const dataRes = await dataGet(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/data?clientId=v2_alpha&from=${oldDate}&to=${oldDate}`,
|
||||||
|
{ headers: { cookie } },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(dataRes.status, 200);
|
||||||
|
const dataJson = await dataRes.json();
|
||||||
|
assert.equal(dataJson.total, 2);
|
||||||
|
assert.equal(dataJson.events.length, 2);
|
||||||
|
assert.equal(dataJson.events[0].eventId, "evt_old_2");
|
||||||
|
assert.equal(dataJson.events[1].eventId, "evt_old_1");
|
||||||
|
|
||||||
|
// 4. Verify GET /api/events reads compressed partition seamlessly
|
||||||
|
const evRes = await eventsGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/events?clientId=v2_alpha&limit=10", {
|
||||||
|
headers: { cookie },
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(evRes.status, 200);
|
||||||
|
const evJson = await evRes.json();
|
||||||
|
const ids = (evJson.events as Array<{ eventId: string }>).map((e) => e.eventId);
|
||||||
|
assert.ok(ids.includes("evt_old_1"));
|
||||||
|
assert.ok(ids.includes("evt_today_1"));
|
||||||
|
|
||||||
|
// 5. Verify GET /api/export CSV, JSON, and raw single .csv.gz download
|
||||||
|
const exportCsvRes = await exportGet(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=csv`,
|
||||||
|
{ headers: { cookie } },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(exportCsvRes.status, 200);
|
||||||
|
const exportedCsvText = await exportCsvRes.text();
|
||||||
|
assert.match(exportedCsvText, /evt_old_1/);
|
||||||
|
assert.match(exportedCsvText, /evt_old_2/);
|
||||||
|
|
||||||
|
const exportJsonRes = await exportGet(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/export?clientId=v2_alpha&from=${oldDate}&to=${oldDate}&format=json`,
|
||||||
|
{ headers: { cookie } },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(exportJsonRes.status, 200);
|
||||||
|
const exportedJson = await exportJsonRes.json();
|
||||||
|
assert.equal(exportedJson.length, 2);
|
||||||
|
|
||||||
|
const exportSingleGzRes = await exportGet(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/export?clientId=v2_alpha&file=${oldDate}.csv.gz`,
|
||||||
|
{ headers: { cookie } },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(exportSingleGzRes.status, 200);
|
||||||
|
assert.equal(exportSingleGzRes.headers.get("content-type"), "application/gzip");
|
||||||
|
const downloadedBuf = Buffer.from(await exportSingleGzRes.arrayBuffer());
|
||||||
|
assert.match(zlib.gunzipSync(downloadedBuf).toString("utf8"), /evt_old_1/);
|
||||||
|
|
||||||
|
// 6. Save S3 settings via POST /api/settings and verify GET /api/settings masks secret (zero plaintext leak)
|
||||||
|
const saveRes = await settingsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/settings", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({
|
||||||
|
compressAfterDays: 30,
|
||||||
|
autoPruneDays: 60,
|
||||||
|
diskPauseThresholdPct: 92,
|
||||||
|
enableS3Backup: true,
|
||||||
|
s3Endpoint: `http://127.0.0.1:${s3Addr.port}`,
|
||||||
|
s3Region: "auto",
|
||||||
|
s3Bucket: "prod-vault",
|
||||||
|
s3Prefix: "archive-test/",
|
||||||
|
s3AccessKeyId: "AKIA_V2_KEY",
|
||||||
|
s3SecretAccessKey: "super_secret_v2_token_7788",
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(saveRes.status, 200);
|
||||||
|
|
||||||
|
const getRes = await settingsGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/settings", { headers: { cookie } }),
|
||||||
|
);
|
||||||
|
assert.equal(getRes.status, 200);
|
||||||
|
const getText = await getRes.text();
|
||||||
|
assert.doesNotMatch(getText, /super_secret_v2_token_7788/);
|
||||||
|
const parsedSettings = JSON.parse(getText).settings;
|
||||||
|
assert.equal(parsedSettings.hasS3SecretKey, true);
|
||||||
|
assert.equal(parsedSettings.s3SecretKeyMask, "••••7788");
|
||||||
|
assert.equal(parsedSettings.compressAfterDays, 30);
|
||||||
|
assert.equal(parsedSettings.diskPauseThresholdPct, 92);
|
||||||
|
|
||||||
|
// 7. POST /api/storage/s3-test succeeds with saved credentials
|
||||||
|
const s3TestRes = await s3TestPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/storage/s3-test", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(s3TestRes.status, 200);
|
||||||
|
assert.equal((await s3TestRes.json()).success, true);
|
||||||
|
|
||||||
|
// 8. V2-E02: When S3 upload fails, POST /api/storage/archive returns 502 and NEVER deletes local partition
|
||||||
|
failNextUpload = true;
|
||||||
|
const failArchiveRes = await archivePost(
|
||||||
|
new NextRequest("http://localhost:3000/api/storage/archive", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({ pruneAfterUpload: true, olderThanDays: 30 }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(failArchiveRes.status, 502);
|
||||||
|
const failArchiveJson = await failArchiveRes.json();
|
||||||
|
assert.equal(failArchiveJson.code, "S3_ARCHIVE_FAILED");
|
||||||
|
assert.equal(failArchiveJson.prunedCount, 0);
|
||||||
|
// File MUST still exist on disk!
|
||||||
|
await fs.stat(gzMeta.filePath);
|
||||||
|
|
||||||
|
// 9. When S3 upload succeeds with pruneAfterUpload: false, local file stays + uploadedKeys receives .csv.gz
|
||||||
|
failNextUpload = false;
|
||||||
|
const okArchiveRes = await archivePost(
|
||||||
|
new NextRequest("http://localhost:3000/api/storage/archive", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({ pruneAfterUpload: false, olderThanDays: 30 }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(okArchiveRes.status, 200);
|
||||||
|
const okArchiveJson = await okArchiveRes.json();
|
||||||
|
assert.equal(okArchiveJson.archivedCount, 1);
|
||||||
|
assert.equal(okArchiveJson.prunedCount, 0);
|
||||||
|
assert.deepEqual(uploadedKeys, [`/prod-vault/archive-test/v2_alpha/${oldDate}.csv.gz`]);
|
||||||
|
await fs.stat(gzMeta.filePath);
|
||||||
|
|
||||||
|
// 10. V2-E03: Corrupt .csv.gz cleanly surfaces error in /api/data instead of crashing
|
||||||
|
await fs.writeFile(
|
||||||
|
path.join(tmpDir, "v2_alpha", "2024-01-01.csv.gz"),
|
||||||
|
Buffer.from("not-valid-gzip-bytes"),
|
||||||
|
);
|
||||||
|
const corruptDataRes = await dataGet(
|
||||||
|
new NextRequest(
|
||||||
|
"http://localhost:3000/api/data?clientId=v2_alpha&from=2024-01-01&to=2024-01-01",
|
||||||
|
{ headers: { cookie } },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(corruptDataRes.status, 500);
|
||||||
|
const corruptJson = await corruptDataRes.json();
|
||||||
|
assert.match(corruptJson.error || "", /Corrupt|\.gz/i);
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) => mockS3.close(() => resolve()));
|
||||||
|
await DataBufferEngine.shutdown();
|
||||||
|
CSVStorageManager.baseStorageDir = origBaseStorageDir;
|
||||||
|
CSVStorageManager.clearRowCountCache();
|
||||||
|
CSVStorageManager.clearQuotaCaches();
|
||||||
|
if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
else process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
|
||||||
|
if (prevDbUrl === undefined) delete process.env.DATABASE_URL;
|
||||||
|
else process.env.DATABASE_URL = prevDbUrl;
|
||||||
|
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||||
|
resetAdminStateForTests();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-01 [Quotas V2-E04 & V2-E05]: HTTP POST /api/events + Port 4000 Socket.IO enforce disk threshold & per-client storageQuotaMb while keeping Admin reads online", async () => {
|
||||||
|
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
const prevDbUrl = process.env.DATABASE_URL;
|
||||||
|
const prevDisableListen = process.env.DATASTACK_DISABLE_LISTEN;
|
||||||
|
const origBaseStorageDir = CSVStorageManager.baseStorageDir;
|
||||||
|
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-quota-"));
|
||||||
|
|
||||||
|
delete process.env.DATABASE_URL;
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = tmpDir;
|
||||||
|
CSVStorageManager.baseStorageDir = tmpDir;
|
||||||
|
process.env.DATASTACK_DISABLE_LISTEN = "1";
|
||||||
|
CSVStorageManager.clearQuotaCaches();
|
||||||
|
await DataBufferEngine.shutdown();
|
||||||
|
DataBufferEngine.setRedisClientForTests(null);
|
||||||
|
|
||||||
|
const serverMod = require("../server/index.js") as {
|
||||||
|
server: http.Server;
|
||||||
|
io: { close: (cb: () => void) => void };
|
||||||
|
writer: {
|
||||||
|
baseDir: string;
|
||||||
|
setStatfsOverrideForTests: (
|
||||||
|
fn: ((dir: string) => Promise<{ blocks: number; bavail: number; bsize: number }>) | null,
|
||||||
|
) => void;
|
||||||
|
clearQuotaCaches: () => void;
|
||||||
|
};
|
||||||
|
runtimeConfig: { diskPauseThresholdPct: number };
|
||||||
|
setDbPoolForTests: (pool: unknown) => void;
|
||||||
|
stopTimers: () => void;
|
||||||
|
};
|
||||||
|
const origWriterBaseDir = serverMod.writer.baseDir;
|
||||||
|
serverMod.writer.baseDir = tmpDir;
|
||||||
|
serverMod.writer.clearQuotaCaches();
|
||||||
|
|
||||||
|
const openSockets: Socket[] = [];
|
||||||
|
let listeningUrl = "";
|
||||||
|
|
||||||
|
try {
|
||||||
|
const cookie = await makeAdminCookie();
|
||||||
|
|
||||||
|
// 1. Validate UI helper & API validation for diskPauseThresholdPct (0 or 50..99)
|
||||||
|
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "0").valid, true);
|
||||||
|
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "95").valid, true);
|
||||||
|
assert.equal(validateSingleNumericSetting("diskPauseThresholdPct", "25").valid, false);
|
||||||
|
const fullDraft = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS);
|
||||||
|
assert.equal(validateEditableSettingsDraft(fullDraft, true).valid, true);
|
||||||
|
|
||||||
|
const badPctRes = await settingsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/settings", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({ diskPauseThresholdPct: 25 }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(badPctRes.status, 400);
|
||||||
|
|
||||||
|
await SettingsManager.updateSettingsAsync({ diskPauseThresholdPct: 90 });
|
||||||
|
|
||||||
|
// 2. Register client via /api/clients with 1 MB quota & edit quota via PATCH /api/clients/[id]
|
||||||
|
const createRes = await clientsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({
|
||||||
|
clientId: "quota_client",
|
||||||
|
name: "Quota Client",
|
||||||
|
rateLimit: 200,
|
||||||
|
storageQuotaMb: 2,
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(createRes.status, 201);
|
||||||
|
const createdClient = (await createRes.json()).client;
|
||||||
|
assert.equal(createdClient.storageQuotaMb, 2);
|
||||||
|
|
||||||
|
const patchRes = await patchClientRoute(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients/quota_client", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({ storageQuotaMb: 1 }),
|
||||||
|
}),
|
||||||
|
{ params: Promise.resolve({ id: "quota_client" }) },
|
||||||
|
);
|
||||||
|
assert.equal(patchRes.status, 200);
|
||||||
|
assert.equal((await patchRes.json()).client.storageQuotaMb, 1);
|
||||||
|
|
||||||
|
const listRes = await clientsGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients", { headers: { cookie } }),
|
||||||
|
);
|
||||||
|
const listed = (await listRes.json()).clients.find(
|
||||||
|
(c: { clientId: string }) => c.clientId === "quota_client",
|
||||||
|
);
|
||||||
|
assert.equal(listed.storageQuotaMb, 1);
|
||||||
|
|
||||||
|
// Seed 1.1 MB into client dir to breach 1 MB quota (V2-E05)
|
||||||
|
const clientDir = path.join(tmpDir, "quota_client");
|
||||||
|
await fs.mkdir(clientDir, { recursive: true });
|
||||||
|
await fs.writeFile(
|
||||||
|
path.join(clientDir, "2026-10-10.csv"),
|
||||||
|
Buffer.alloc(Math.ceil(1.1 * 1024 * 1024), "x"),
|
||||||
|
);
|
||||||
|
CSVStorageManager.clearQuotaCaches("quota_client");
|
||||||
|
|
||||||
|
const httpQuotaRes = await eventsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/events", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"content-type": "application/json",
|
||||||
|
authorization: `Bearer ${createdClient.token}`,
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ eventId: "q_evt_1", data: { ok: true } }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(httpQuotaRes.status, 507);
|
||||||
|
assert.equal((await httpQuotaRes.json()).code, "STORAGE_QUOTA_EXCEEDED");
|
||||||
|
|
||||||
|
// Now test global VPS disk threshold >= 90% (V2-E04) across HTTP + Socket.IO
|
||||||
|
await patchClientRoute(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients/quota_client", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json", cookie },
|
||||||
|
body: JSON.stringify({ storageQuotaMb: 0 }),
|
||||||
|
}),
|
||||||
|
{ params: Promise.resolve({ id: "quota_client" }) },
|
||||||
|
);
|
||||||
|
CSVStorageManager.setStatfsOverrideForTests(async () => ({
|
||||||
|
blocks: 1000,
|
||||||
|
bavail: 50, // 95% used >= 90% threshold
|
||||||
|
bsize: 4096,
|
||||||
|
}));
|
||||||
|
|
||||||
|
const httpDiskRes = await eventsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/events", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"content-type": "application/json",
|
||||||
|
authorization: `Bearer ${createdClient.token}`,
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ eventId: "q_evt_2", data: { ok: true } }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(httpDiskRes.status, 507);
|
||||||
|
assert.equal((await httpDiskRes.json()).code, "DISK_SPACE_LOW");
|
||||||
|
|
||||||
|
// Admin read/storage/export MUST remain 100% operational while ingestion is paused!
|
||||||
|
const storageRes = await storageGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/storage", { headers: { cookie } }),
|
||||||
|
);
|
||||||
|
assert.equal(storageRes.status, 200);
|
||||||
|
const storageJson = await storageRes.json();
|
||||||
|
assert.equal(storageJson.diskUsage.usedPct, 95);
|
||||||
|
|
||||||
|
// Port 4000 Socket.IO enforcement for DISK_SPACE_LOW & STORAGE_QUOTA_EXCEEDED
|
||||||
|
let mockQuotaMb = 0;
|
||||||
|
serverMod.runtimeConfig.diskPauseThresholdPct = 90;
|
||||||
|
serverMod.setDbPoolForTests({
|
||||||
|
query: async (sql: string) => {
|
||||||
|
if (sql.includes("FROM client_sdks")) {
|
||||||
|
return {
|
||||||
|
rowCount: 1,
|
||||||
|
rows: [
|
||||||
|
{
|
||||||
|
client_id: "quota_client",
|
||||||
|
token: createdClient.token,
|
||||||
|
status: "active",
|
||||||
|
rate_limit: 200,
|
||||||
|
storage_quota_mb: mockQuotaMb,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { rowCount: 0, rows: [] };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
serverMod.writer.setStatfsOverrideForTests(async () => ({
|
||||||
|
blocks: 1000,
|
||||||
|
bavail: 50, // 95% used
|
||||||
|
bsize: 4096,
|
||||||
|
}));
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) =>
|
||||||
|
serverMod.server.listen(0, "127.0.0.1", resolve),
|
||||||
|
);
|
||||||
|
const addr = serverMod.server.address() as AddressInfo;
|
||||||
|
listeningUrl = `http://127.0.0.1:${addr.port}`;
|
||||||
|
|
||||||
|
const socket = ioClient(listeningUrl, {
|
||||||
|
auth: { token: createdClient.token },
|
||||||
|
transports: ["websocket"],
|
||||||
|
reconnection: false,
|
||||||
|
timeout: 3000,
|
||||||
|
});
|
||||||
|
openSockets.push(socket);
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
socket.on("connect", () => resolve());
|
||||||
|
socket.on("connect_error", reject);
|
||||||
|
});
|
||||||
|
|
||||||
|
const singleDiskAck = await new Promise<{
|
||||||
|
success: boolean;
|
||||||
|
code?: string;
|
||||||
|
}>((resolve, reject) => {
|
||||||
|
socket.timeout(3000).emit("sync_data", { data: { k: 1 } }, (err: unknown, ack: { success: boolean; code?: string }) => {
|
||||||
|
if (err) reject(err);
|
||||||
|
else resolve(ack);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
assert.equal(singleDiskAck.success, false);
|
||||||
|
assert.equal(singleDiskAck.code, "DISK_SPACE_LOW");
|
||||||
|
|
||||||
|
// Relax disk threshold & activate per-client 1 MB quota in Socket.IO worker
|
||||||
|
serverMod.writer.setStatfsOverrideForTests(async () => ({
|
||||||
|
blocks: 1000,
|
||||||
|
bavail: 800, // 20% used
|
||||||
|
bsize: 4096,
|
||||||
|
}));
|
||||||
|
serverMod.writer.clearQuotaCaches();
|
||||||
|
mockQuotaMb = 1;
|
||||||
|
// Wait out 2s verifyClientInDb cache or update query response cleanly
|
||||||
|
await new Promise((r) => setTimeout(r, 2100));
|
||||||
|
|
||||||
|
const batchQuotaAck = await new Promise<{
|
||||||
|
success: boolean;
|
||||||
|
code?: string;
|
||||||
|
rejected?: number;
|
||||||
|
}>((resolve, reject) => {
|
||||||
|
socket.timeout(3000).emit(
|
||||||
|
"batch_sync",
|
||||||
|
[{ eventId: "b1", data: { k: 1 } }],
|
||||||
|
(err: unknown, ack: { success: boolean; code?: string; rejected?: number }) => {
|
||||||
|
if (err) reject(err);
|
||||||
|
else resolve(ack);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
assert.equal(batchQuotaAck.success, false);
|
||||||
|
assert.equal(batchQuotaAck.code, "STORAGE_QUOTA_EXCEEDED");
|
||||||
|
assert.equal(batchQuotaAck.rejected, 1);
|
||||||
|
|
||||||
|
// Clean delete route check
|
||||||
|
const delRes = await deleteClientRoute(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients/quota_client?purgeFiles=true", {
|
||||||
|
method: "DELETE",
|
||||||
|
headers: { cookie },
|
||||||
|
}),
|
||||||
|
{ params: Promise.resolve({ id: "quota_client" }) },
|
||||||
|
);
|
||||||
|
assert.equal(delRes.status, 200);
|
||||||
|
} finally {
|
||||||
|
for (const s of openSockets) {
|
||||||
|
try {
|
||||||
|
s.disconnect();
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
}
|
||||||
|
serverMod.writer.setStatfsOverrideForTests(null);
|
||||||
|
serverMod.writer.baseDir = origWriterBaseDir;
|
||||||
|
serverMod.writer.clearQuotaCaches();
|
||||||
|
serverMod.setDbPoolForTests(null);
|
||||||
|
serverMod.stopTimers();
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
serverMod.io.close(() => {
|
||||||
|
if (serverMod.server.listening) {
|
||||||
|
serverMod.server.close(() => resolve());
|
||||||
|
} else {
|
||||||
|
resolve();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
SDKManager.stopActivityFlushTimerForTests();
|
||||||
|
CSVStorageManager.setStatfsOverrideForTests(null);
|
||||||
|
CSVStorageManager.clearQuotaCaches();
|
||||||
|
await DataBufferEngine.shutdown();
|
||||||
|
CSVStorageManager.baseStorageDir = origBaseStorageDir;
|
||||||
|
if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
else process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
|
||||||
|
if (prevDisableListen === undefined) delete process.env.DATASTACK_DISABLE_LISTEN;
|
||||||
|
else process.env.DATASTACK_DISABLE_LISTEN = prevDisableListen;
|
||||||
|
if (prevDbUrl === undefined) delete process.env.DATABASE_URL;
|
||||||
|
else process.env.DATABASE_URL = prevDbUrl;
|
||||||
|
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||||
|
resetAdminStateForTests();
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -0,0 +1,631 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import fs from "node:fs/promises";
|
||||||
|
import http from "node:http";
|
||||||
|
import { createRequire } from "node:module";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import test from "node:test";
|
||||||
|
import { NextRequest } from "next/server";
|
||||||
|
import { io as ioClient, type Socket } from "socket.io-client";
|
||||||
|
import { PATCH as patchClientRoute } from "../app/api/clients/[id]/route";
|
||||||
|
import { GET as clientsGet, POST as clientsPost } from "../app/api/clients/route";
|
||||||
|
import { POST as eventsPost } from "../app/api/events/route";
|
||||||
|
import { GET as exportGet } from "../app/api/export/route";
|
||||||
|
import { GET as prometheusGet } from "../app/api/prometheus/route";
|
||||||
|
import { GET as settingsGet, POST as settingsPost } from "../app/api/settings/route";
|
||||||
|
import { POST as webhookTestPost } from "../app/api/settings/webhook-test/route";
|
||||||
|
import { SettingsManager } from "../core/adminSettings/settingsManager";
|
||||||
|
import {
|
||||||
|
buildWebhookSignature,
|
||||||
|
clearWebhookCooldownCacheForTests,
|
||||||
|
dispatchWebhookAlertAsync,
|
||||||
|
formatOutboundWebhookBody,
|
||||||
|
} from "../core/alerts/webhookNotifier";
|
||||||
|
import {
|
||||||
|
createAdminSessionToken,
|
||||||
|
getAdminSessionSecret,
|
||||||
|
} from "../core/auth/adminSession";
|
||||||
|
import {
|
||||||
|
createInitialAdmin,
|
||||||
|
resetAdminStateForTests,
|
||||||
|
} from "../core/auth/adminUsers";
|
||||||
|
import { DataBufferEngine } from "../core/data/buffer";
|
||||||
|
import { SDKManager } from "../core/sdk/sdkManager";
|
||||||
|
import {
|
||||||
|
isClientIpAllowed,
|
||||||
|
isClientTokenExpired,
|
||||||
|
validateAllowedIpsInput,
|
||||||
|
validateClientExpiresAtInput,
|
||||||
|
} from "../core/Security/ipAllowlist";
|
||||||
|
import { CSVStorageManager } from "../core/Storage/csvStorage";
|
||||||
|
import {
|
||||||
|
DEFAULT_ALERTING_SETTINGS,
|
||||||
|
DEFAULT_EDITABLE_SETTINGS,
|
||||||
|
normalizeAlertingSettings,
|
||||||
|
settingsToNumericDrafts,
|
||||||
|
validateEditableSettingsDraft,
|
||||||
|
validateSingleNumericSetting,
|
||||||
|
} from "../lib/vconUiHelpers";
|
||||||
|
|
||||||
|
const require = createRequire(import.meta.url);
|
||||||
|
|
||||||
|
async function makeAdminCookie(): Promise<string> {
|
||||||
|
resetAdminStateForTests();
|
||||||
|
await createInitialAdmin("v2-p2@datastack.dev", "V2Phase02Pass!");
|
||||||
|
const secret = getAdminSessionSecret()!;
|
||||||
|
const token = await createAdminSessionToken(
|
||||||
|
"v2-p2@datastack.dev",
|
||||||
|
secret,
|
||||||
|
Date.now() + 3600_000,
|
||||||
|
1,
|
||||||
|
);
|
||||||
|
return `datastack_admin_session=${token}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
test("V2-Phase-02 [IP Allowlist & Token Expiry]: validateAllowedIpsInput, isClientIpAllowed (IPv4/IPv6/CIDR) & isClientTokenExpired", () => {
|
||||||
|
// Valid empty or wildcard
|
||||||
|
assert.deepEqual(validateAllowedIpsInput(""), { valid: true, normalized: "" });
|
||||||
|
assert.deepEqual(validateAllowedIpsInput(" "), { valid: true, normalized: "" });
|
||||||
|
assert.deepEqual(validateAllowedIpsInput("*"), { valid: true, normalized: "*" });
|
||||||
|
|
||||||
|
// Valid comma-separated IPv4, IPv4 CIDR, IPv6, IPv6 CIDR
|
||||||
|
const validList = validateAllowedIpsInput(
|
||||||
|
"203.0.113.10, 10.0.0.0/8, ::ffff:192.0.2.5, 2001:db8::/32, 203.0.113.10",
|
||||||
|
);
|
||||||
|
assert.equal(validList.valid, true);
|
||||||
|
if (validList.valid) {
|
||||||
|
assert.equal(
|
||||||
|
validList.normalized,
|
||||||
|
"203.0.113.10, 10.0.0.0/8, 192.0.2.5, 2001:db8::/32",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Invalid IP or CIDR prefix rejected
|
||||||
|
const badCidr = validateAllowedIpsInput("10.0.0.0/33");
|
||||||
|
assert.equal(badCidr.valid, false);
|
||||||
|
const badIp = validateAllowedIpsInput("999.888.777.666");
|
||||||
|
assert.equal(badIp.valid, false);
|
||||||
|
|
||||||
|
// Matching behavior
|
||||||
|
assert.equal(isClientIpAllowed("198.51.100.77", ""), true);
|
||||||
|
assert.equal(isClientIpAllowed("198.51.100.77", "*"), true);
|
||||||
|
assert.equal(
|
||||||
|
isClientIpAllowed("203.0.113.10", "203.0.113.10, 10.0.0.0/8"),
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
isClientIpAllowed("::ffff:203.0.113.10", "203.0.113.10, 10.0.0.0/8"),
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
isClientIpAllowed("10.24.99.1", "203.0.113.10, 10.0.0.0/8"),
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
isClientIpAllowed("192.168.1.50", "203.0.113.10, 10.0.0.0/8"),
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
// Loopback equivalence between 127.0.0.1 and ::1
|
||||||
|
assert.equal(isClientIpAllowed("::1", "127.0.0.1"), true);
|
||||||
|
assert.equal(isClientIpAllowed("127.0.0.1", "::1"), true);
|
||||||
|
|
||||||
|
// Expiration validation & evaluation
|
||||||
|
assert.deepEqual(validateClientExpiresAtInput(null), {
|
||||||
|
valid: true,
|
||||||
|
normalized: null,
|
||||||
|
});
|
||||||
|
assert.deepEqual(validateClientExpiresAtInput(""), {
|
||||||
|
valid: true,
|
||||||
|
normalized: null,
|
||||||
|
});
|
||||||
|
const validExp = validateClientExpiresAtInput("2030-06-15T12:00:00.000Z");
|
||||||
|
assert.equal(validExp.valid, true);
|
||||||
|
if (validExp.valid) {
|
||||||
|
assert.equal(validExp.normalized, "2030-06-15T12:00:00.000Z");
|
||||||
|
}
|
||||||
|
const badExp = validateClientExpiresAtInput("not-a-timestamp");
|
||||||
|
assert.equal(badExp.valid, false);
|
||||||
|
|
||||||
|
const nowMs = Date.parse("2026-11-19T12:00:00.000Z");
|
||||||
|
assert.equal(isClientTokenExpired(null, nowMs), false);
|
||||||
|
assert.equal(isClientTokenExpired("2026-11-19T12:00:01.000Z", nowMs), false);
|
||||||
|
assert.equal(isClientTokenExpired("2026-11-19T12:00:00.000Z", nowMs), true);
|
||||||
|
assert.equal(isClientTokenExpired("2026-11-19T11:59:59.000Z", nowMs), true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-02 [Webhook Notifier & /api/settings/webhook-test]: HMAC-SHA256 signing, cooldown suppression & V2-E06 error handling", async () => {
|
||||||
|
clearWebhookCooldownCacheForTests();
|
||||||
|
const receivedRequests: Array<{
|
||||||
|
headers: http.IncomingHttpHeaders;
|
||||||
|
body: string;
|
||||||
|
json: Record<string, unknown>;
|
||||||
|
}> = [];
|
||||||
|
let responseStatus = 200;
|
||||||
|
|
||||||
|
const mockWebhookServer = http.createServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
|
||||||
|
req.on("end", () => {
|
||||||
|
const bodyStr = Buffer.concat(chunks).toString("utf8");
|
||||||
|
let parsed: Record<string, unknown> = {};
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(bodyStr);
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
receivedRequests.push({
|
||||||
|
headers: req.headers,
|
||||||
|
body: bodyStr,
|
||||||
|
json: parsed,
|
||||||
|
});
|
||||||
|
res.writeHead(responseStatus, { "Content-Type": "application/json" });
|
||||||
|
res.end(JSON.stringify({ ok: responseStatus < 400 }));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => mockWebhookServer.listen(0, "127.0.0.1", resolve));
|
||||||
|
const port = (mockWebhookServer.address() as AddressInfo).port;
|
||||||
|
const webhookUrl = `http://127.0.0.1:${port}/hooks/datastack`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
// 1. Format & signature verification
|
||||||
|
const formatted = formatOutboundWebhookBody({
|
||||||
|
event: "DISK_SPACE_LOW",
|
||||||
|
severity: "critical",
|
||||||
|
title: "Disk Threshold Reached",
|
||||||
|
message: "VPS disk is at 95.4%.",
|
||||||
|
clientId: "svc_alpha",
|
||||||
|
timestamp: "2026-11-19T12:00:00.000Z",
|
||||||
|
details: { usedPct: 95.4 },
|
||||||
|
});
|
||||||
|
assert.ok(formatted.text.includes("DISK_SPACE_LOW"));
|
||||||
|
assert.equal(formatted.content, formatted.text);
|
||||||
|
|
||||||
|
const bodyJson = JSON.stringify(formatted);
|
||||||
|
const sig = buildWebhookSignature(bodyJson, "webhook_secret_xyz");
|
||||||
|
assert.ok(sig.startsWith("sha256="));
|
||||||
|
|
||||||
|
// 2. Cooldown deduplication test
|
||||||
|
const first = await dispatchWebhookAlertAsync(
|
||||||
|
{
|
||||||
|
enabled: true,
|
||||||
|
url: webhookUrl,
|
||||||
|
secret: "webhook_secret_xyz",
|
||||||
|
cooldownSec: 120,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
event: "CLIENT_QUOTA_EXCEEDED",
|
||||||
|
severity: "warning",
|
||||||
|
title: "Client Quota Exceeded",
|
||||||
|
message: "Quota reached for svc_alpha",
|
||||||
|
clientId: "svc_alpha",
|
||||||
|
},
|
||||||
|
);
|
||||||
|
assert.equal(first.ok, true);
|
||||||
|
assert.equal(receivedRequests.length, 1);
|
||||||
|
assert.equal(
|
||||||
|
receivedRequests[0].headers["x-datastack-event"],
|
||||||
|
"CLIENT_QUOTA_EXCEEDED",
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
receivedRequests[0].headers["x-datastack-signature"],
|
||||||
|
buildWebhookSignature(receivedRequests[0].body, "webhook_secret_xyz"),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Duplicate event for same client within cooldown window is suppressed
|
||||||
|
const second = await dispatchWebhookAlertAsync(
|
||||||
|
{
|
||||||
|
enabled: true,
|
||||||
|
url: webhookUrl,
|
||||||
|
secret: "webhook_secret_xyz",
|
||||||
|
cooldownSec: 120,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
event: "CLIENT_QUOTA_EXCEEDED",
|
||||||
|
severity: "warning",
|
||||||
|
title: "Client Quota Exceeded",
|
||||||
|
message: "Quota reached again for svc_alpha",
|
||||||
|
clientId: "svc_alpha",
|
||||||
|
},
|
||||||
|
);
|
||||||
|
assert.equal(second.ok, true);
|
||||||
|
assert.equal(second.suppressedByCooldown, true);
|
||||||
|
assert.equal(receivedRequests.length, 1);
|
||||||
|
|
||||||
|
// 3. Test POST /api/settings/webhook-test route (200 & 502 WEBHOOK_DELIVERY_FAILED V2-E06)
|
||||||
|
const adminCookie = await makeAdminCookie();
|
||||||
|
const okTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
webhookUrl,
|
||||||
|
webhookSecret: "test_route_hmac_secret",
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
const okTestRes = await webhookTestPost(okTestReq);
|
||||||
|
assert.equal(okTestRes.status, 200);
|
||||||
|
const okTestData = await okTestRes.json();
|
||||||
|
assert.equal(okTestData.success, true);
|
||||||
|
assert.equal(receivedRequests.length, 2);
|
||||||
|
assert.equal(receivedRequests[1].headers["x-datastack-event"], "TEST_ALERT");
|
||||||
|
|
||||||
|
// Simulate upstream 500 error -> 502 WEBHOOK_DELIVERY_FAILED (V2-E06)
|
||||||
|
responseStatus = 500;
|
||||||
|
const failTestReq = new NextRequest("http://localhost:3000/api/settings/webhook-test", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
webhookUrl,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
const failTestRes = await webhookTestPost(failTestReq);
|
||||||
|
assert.equal(failTestRes.status, 502);
|
||||||
|
const failTestData = await failTestRes.json();
|
||||||
|
assert.equal(failTestData.code, "WEBHOOK_DELIVERY_FAILED");
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) => mockWebhookServer.close(() => resolve()));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-02 [HTTP & Socket Ingestion Security]: Per-client IP allowlist (403 V2-E07) & Token expiration (401 V2-E08)", async () => {
|
||||||
|
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-sec-"));
|
||||||
|
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = tmpDir;
|
||||||
|
CSVStorageManager.resetForTests(tmpDir);
|
||||||
|
await CSVStorageManager.initializeAsync();
|
||||||
|
clearWebhookCooldownCacheForTests();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const adminCookie = await makeAdminCookie();
|
||||||
|
|
||||||
|
// 1. Create client with strict IP allowlist (only 10.10.0.0/16) and future expiration
|
||||||
|
const futureIso = new Date(Date.now() + 3600_000).toISOString();
|
||||||
|
const createRes = await clientsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/clients", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
name: "IP Restricted Client",
|
||||||
|
rateLimit: 100,
|
||||||
|
storageQuotaMb: 0,
|
||||||
|
allowedIps: "10.10.0.0/16",
|
||||||
|
expiresAt: futureIso,
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(createRes.status, 200);
|
||||||
|
const createData = await createRes.json();
|
||||||
|
const client = createData.client;
|
||||||
|
assert.equal(client.allowedIps, "10.10.0.0/16");
|
||||||
|
assert.equal(client.expiresAt, futureIso);
|
||||||
|
|
||||||
|
// 2. Request from forbidden IP 192.168.1.25 -> 403 CLIENT_IP_FORBIDDEN (V2-E07)
|
||||||
|
const forbiddenIpRes = await eventsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/events", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
authorization: `Bearer ${client.token}`,
|
||||||
|
"content-type": "application/json",
|
||||||
|
"x-forwarded-for": "192.168.1.25",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
eventId: "evt_ip_denied",
|
||||||
|
payload: { step: 1 },
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(forbiddenIpRes.status, 403);
|
||||||
|
const forbiddenIpJson = await forbiddenIpRes.json();
|
||||||
|
assert.equal(forbiddenIpJson.code, "CLIENT_IP_FORBIDDEN");
|
||||||
|
|
||||||
|
// 3. Request from allowed CIDR IP 10.10.42.9 -> 200 OK
|
||||||
|
const allowedIpRes = await eventsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/events", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
authorization: `Bearer ${client.token}`,
|
||||||
|
"content-type": "application/json",
|
||||||
|
"x-forwarded-for": "10.10.42.9",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
eventId: "evt_ip_allowed",
|
||||||
|
payload: { step: 2 },
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(allowedIpRes.status, 200);
|
||||||
|
|
||||||
|
// 4. Patch client with past expiresAt -> 401 CLIENT_TOKEN_EXPIRED (V2-E08)
|
||||||
|
const pastIso = new Date(Date.now() - 60_000).toISOString();
|
||||||
|
const patchRes = await patchClientRoute(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`,
|
||||||
|
{
|
||||||
|
method: "PATCH",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
expiresAt: pastIso,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
),
|
||||||
|
{ params: Promise.resolve({ id: client.clientId }) },
|
||||||
|
);
|
||||||
|
assert.equal(patchRes.status, 200);
|
||||||
|
const patchData = await patchRes.json();
|
||||||
|
assert.equal(patchData.client.expiresAt, pastIso);
|
||||||
|
|
||||||
|
const expiredRes = await eventsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/events", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
authorization: `Bearer ${client.token}`,
|
||||||
|
"content-type": "application/json",
|
||||||
|
"x-forwarded-for": "10.10.42.9",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
eventId: "evt_expired",
|
||||||
|
payload: { step: 3 },
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(expiredRes.status, 401);
|
||||||
|
const expiredJson = await expiredRes.json();
|
||||||
|
assert.equal(expiredJson.code, "CLIENT_TOKEN_EXPIRED");
|
||||||
|
|
||||||
|
// 5. Verify Port 4000 Socket.IO server enforces IP allowlist & token expiry
|
||||||
|
process.env.DATASTACK_DISABLE_LISTEN = "1";
|
||||||
|
const ingestionApp = require("../server/index.js");
|
||||||
|
|
||||||
|
// Sync expired client to Port 4000 in-memory cache and verify verifyClientInDb returns CLIENT_TOKEN_EXPIRED
|
||||||
|
await new Promise<void>((resolve) =>
|
||||||
|
ingestionApp.server.listen(0, "127.0.0.1", () => resolve()),
|
||||||
|
);
|
||||||
|
const port4000 = (ingestionApp.server.address() as AddressInfo).port;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const expiredCheck = await ingestionApp.verifyClientInDb(
|
||||||
|
client.clientId,
|
||||||
|
client.token,
|
||||||
|
true,
|
||||||
|
"10.10.42.9",
|
||||||
|
);
|
||||||
|
assert.equal(expiredCheck.ok, false);
|
||||||
|
assert.equal(expiredCheck.reason, "CLIENT_TOKEN_EXPIRED");
|
||||||
|
|
||||||
|
// Clear expiration via PATCH, keep IP allowlist 10.10.0.0/16, and check Socket.IO handshake from 127.0.0.1
|
||||||
|
await patchClientRoute(
|
||||||
|
new NextRequest(
|
||||||
|
`http://localhost:3000/api/clients/${encodeURIComponent(client.clientId)}`,
|
||||||
|
{
|
||||||
|
method: "PATCH",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
expiresAt: null,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
),
|
||||||
|
{ params: Promise.resolve({ id: client.clientId }) },
|
||||||
|
);
|
||||||
|
|
||||||
|
const ipForbiddenCheck = await ingestionApp.verifyClientInDb(
|
||||||
|
client.clientId,
|
||||||
|
client.token,
|
||||||
|
true,
|
||||||
|
"127.0.0.1",
|
||||||
|
);
|
||||||
|
assert.equal(ipForbiddenCheck.ok, false);
|
||||||
|
assert.equal(ipForbiddenCheck.reason, "CLIENT_IP_FORBIDDEN");
|
||||||
|
|
||||||
|
// Socket.IO connection from 127.0.0.1 should fail handshake with CLIENT_IP_FORBIDDEN
|
||||||
|
const socketError = await new Promise<string>((resolve) => {
|
||||||
|
const sock: Socket = ioClient(`http://127.0.0.1:${port4000}`, {
|
||||||
|
auth: { clientId: client.clientId, token: client.token },
|
||||||
|
transports: ["websocket"],
|
||||||
|
reconnection: false,
|
||||||
|
});
|
||||||
|
sock.on("connect_error", (err) => {
|
||||||
|
sock.close();
|
||||||
|
resolve(err.message);
|
||||||
|
});
|
||||||
|
sock.on("connect", () => {
|
||||||
|
sock.close();
|
||||||
|
resolve("CONNECTED_UNEXPECTEDLY");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
assert.ok(socketError.includes("CLIENT_IP_FORBIDDEN"));
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) =>
|
||||||
|
ingestionApp.server.close(() => resolve()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await DataBufferEngine.flushAllAsync();
|
||||||
|
SDKManager.stopActivityFlushTimerForTests();
|
||||||
|
if (prevStorageDir === undefined) {
|
||||||
|
delete process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
} else {
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
|
||||||
|
}
|
||||||
|
CSVStorageManager.resetForTests();
|
||||||
|
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("V2-Phase-02 [Prometheus Metrics & NDJSON Streaming Export]: GET /api/prometheus (V2-FEAT-04 / V2-E09) & GET /api/export?format=ndjson (V2-FEAT-06)", async () => {
|
||||||
|
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-v2-p2-prom-"));
|
||||||
|
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = tmpDir;
|
||||||
|
CSVStorageManager.resetForTests(tmpDir);
|
||||||
|
await CSVStorageManager.initializeAsync();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const adminCookie = await makeAdminCookie();
|
||||||
|
|
||||||
|
// 1. Unauthenticated scrape when Prometheus is disabled -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09)
|
||||||
|
const unauthPromRes = await prometheusGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/prometheus"),
|
||||||
|
);
|
||||||
|
assert.equal(unauthPromRes.status, 401);
|
||||||
|
const unauthPromJson = await unauthPromRes.json();
|
||||||
|
assert.equal(unauthPromJson.code, "PROMETHEUS_UNAUTHORIZED");
|
||||||
|
|
||||||
|
// 2. Configure Webhook & Prometheus settings via POST /api/settings
|
||||||
|
const saveSettingsRes = await settingsPost(
|
||||||
|
new NextRequest("http://localhost:3000/api/settings", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
cookie: adminCookie,
|
||||||
|
"content-type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
enableWebhookAlerts: true,
|
||||||
|
webhookUrl: "https://hooks.example.com/datastack",
|
||||||
|
webhookSecret: "super_secret_webhook_hmac_9988",
|
||||||
|
webhookCooldownSec: 180,
|
||||||
|
enablePrometheusMetrics: true,
|
||||||
|
prometheusMetricsToken: "prom_scrape_bearer_token_7766",
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(saveSettingsRes.status, 200);
|
||||||
|
const savedData = await saveSettingsRes.json();
|
||||||
|
assert.equal(savedData.settings.enableWebhookAlerts, true);
|
||||||
|
assert.equal(savedData.settings.webhookCooldownSec, 180);
|
||||||
|
assert.equal(savedData.settings.hasWebhookSecret, true);
|
||||||
|
assert.equal(savedData.settings.webhookSecretMask, "••••9988");
|
||||||
|
assert.equal("webhookSecretEnc" in savedData.settings, false);
|
||||||
|
assert.equal(savedData.settings.enablePrometheusMetrics, true);
|
||||||
|
assert.equal(savedData.settings.hasPrometheusToken, true);
|
||||||
|
assert.equal(savedData.settings.prometheusTokenMask, "••••7766");
|
||||||
|
assert.equal("prometheusMetricsTokenEnc" in savedData.settings, false);
|
||||||
|
|
||||||
|
// 3. Wrong Bearer token -> 401 PROMETHEUS_UNAUTHORIZED (V2-E09)
|
||||||
|
const wrongTokenRes = await prometheusGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/prometheus", {
|
||||||
|
headers: {
|
||||||
|
authorization: "Bearer wrong_token_value",
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(wrongTokenRes.status, 401);
|
||||||
|
|
||||||
|
// 4. Valid Bearer token -> 200 Prometheus text exposition format
|
||||||
|
const validTokenRes = await prometheusGet(
|
||||||
|
new NextRequest("http://localhost:3000/api/prometheus", {
|
||||||
|
headers: {
|
||||||
|
authorization: "Bearer prom_scrape_bearer_token_7766",
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.equal(validTokenRes.status, 200);
|
||||||
|
assert.ok(
|
||||||
|
validTokenRes.headers
|
||||||
|
.get("content-type")
|
||||||
|
?.includes("text/plain; version=0.0.4"),
|
||||||
|
);
|
||||||
|
const metricsBody = await validTokenRes.text();
|
||||||
|
assert.ok(metricsBody.includes("datastack_up 1"));
|
||||||
|
assert.ok(metricsBody.includes("datastack_events_total"));
|
||||||
|
assert.ok(metricsBody.includes("datastack_storage_bytes_total"));
|
||||||
|
assert.ok(metricsBody.includes("datastack_disk_used_percent"));
|
||||||
|
|
||||||
|
// 5. Seed events and export as streaming NDJSON (GET /api/export?format=ndjson) (V2-FEAT-06)
|
||||||
|
await CSVStorageManager.appendBatchAsync("client_ndjson", [
|
||||||
|
{
|
||||||
|
clientId: "client_ndjson",
|
||||||
|
eventId: "evt_nd_1",
|
||||||
|
timestamp: "2026-11-19T10:00:00.000Z",
|
||||||
|
payload: JSON.stringify({ action: "purchase", amount: 49 }),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
clientId: "client_ndjson",
|
||||||
|
eventId: "evt_nd_2",
|
||||||
|
timestamp: "2026-11-19T10:01:00.000Z",
|
||||||
|
payload: JSON.stringify({ action: "refund", amount: 12 }),
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
|
||||||
|
const ndjsonRes = await exportGet(
|
||||||
|
new NextRequest(
|
||||||
|
"http://localhost:3000/api/export?format=ndjson&clientId=client_ndjson",
|
||||||
|
{
|
||||||
|
headers: { cookie: adminCookie },
|
||||||
|
},
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(ndjsonRes.status, 200);
|
||||||
|
assert.ok(
|
||||||
|
ndjsonRes.headers
|
||||||
|
.get("content-type")
|
||||||
|
?.includes("application/x-ndjson"),
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
ndjsonRes.headers
|
||||||
|
.get("content-disposition")
|
||||||
|
?.includes("datastack-client_ndjson.ndjson"),
|
||||||
|
);
|
||||||
|
const ndjsonText = await ndjsonRes.text();
|
||||||
|
const lines = ndjsonText
|
||||||
|
.trim()
|
||||||
|
.split("\n")
|
||||||
|
.map((line) => JSON.parse(line));
|
||||||
|
assert.equal(lines.length, 2);
|
||||||
|
assert.equal(lines[0].clientId, "client_ndjson");
|
||||||
|
assert.equal(lines[0].eventId, "evt_nd_1");
|
||||||
|
assert.deepEqual(lines[0].payload, { action: "purchase", amount: 49 });
|
||||||
|
assert.equal(lines[1].eventId, "evt_nd_2");
|
||||||
|
assert.deepEqual(lines[1].payload, { action: "refund", amount: 12 });
|
||||||
|
|
||||||
|
// 6. Verify UI helper validation for webhookCooldownSec & normalizeAlertingSettings
|
||||||
|
assert.equal(
|
||||||
|
validateSingleNumericSetting("webhookCooldownSec", "5").valid,
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
validateSingleNumericSetting("webhookCooldownSec", "300").valid,
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
const drafts = settingsToNumericDrafts(DEFAULT_EDITABLE_SETTINGS);
|
||||||
|
const validatedDraft = validateEditableSettingsDraft(drafts, true);
|
||||||
|
assert.equal(validatedDraft.valid, true);
|
||||||
|
assert.deepEqual(validatedDraft.values, DEFAULT_EDITABLE_SETTINGS);
|
||||||
|
assert.deepEqual(
|
||||||
|
normalizeAlertingSettings(undefined),
|
||||||
|
DEFAULT_ALERTING_SETTINGS,
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
await DataBufferEngine.flushAllAsync();
|
||||||
|
SDKManager.stopActivityFlushTimerForTests();
|
||||||
|
await SettingsManager.updateSettingsAsync({
|
||||||
|
enableWebhookAlerts: false,
|
||||||
|
webhookUrl: "",
|
||||||
|
clearWebhookSecret: true,
|
||||||
|
webhookCooldownSec: 300,
|
||||||
|
enablePrometheusMetrics: false,
|
||||||
|
clearPrometheusToken: true,
|
||||||
|
});
|
||||||
|
if (prevStorageDir === undefined) {
|
||||||
|
delete process.env.DATASTACK_STORAGE_DIR;
|
||||||
|
} else {
|
||||||
|
process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
|
||||||
|
}
|
||||||
|
CSVStorageManager.resetForTests();
|
||||||
|
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
Reference in new issue
Block a user