feat(mcp): implement MCP infrastructure

- Add `mcp_settings` table to database schema
- Enable internal MCP API endpoint via middleware header authentication
- Introduce runtime log directory and capture utilities for diagnostics
- Update build pipeline to capture build logs
This commit is contained in:
Team Dark Devil committed 2026-11-21 23:00:29 +06:00
1 parent ae9b554189
commit b2669e2ad5
19 files changed
+3280 -20

No files matched your search

+10 -1
View File
@@ -1,7 +1,7 @@
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory # MEMORY.md — DataStack Cross-IDE Persistent Project Memory
> **Last Synchronized**: 2026-11-21 > **Last Synchronized**: 2026-11-21
> **Status**: **`R4` Production Hardening (`BASELINE-2026-11-20-R4`: `4 / 4` Phases Complete — All `R4-Phase-01`..`04` Complete & Verified (`AUDIT-01`–`20`, `19 / 19` actionable findings resolved)).** > **Status**: **`R5-MCP` (`BASELINE-2026-11-21-R5-MCP`: `1 / 3` Phases Complete — `R5-Phase-01` Complete & Verified; `R4` `4/4` & `PREPUSH-FIX-01` `5/5` Preserved).**
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`). > **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
--- ---
@@ -423,6 +423,15 @@ All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **
- **`DEPLOY-BUG-02`**: Hardened `docker/coolify-nginx.conf` and `nginx/default.conf` with `ngx_http_realip_module` (`set_real_ip_from` loopback/RFC1918/Docker bridges + IPv6 ULA, `real_ip_header X-Forwarded-For`, `real_ip_recursive on`) so `proxy_set_header X-Real-IP $remote_addr` preserves real external client IPs behind Coolify Traefik without permitting untrusted public IP spoofing. - **`DEPLOY-BUG-02`**: Hardened `docker/coolify-nginx.conf` and `nginx/default.conf` with `ngx_http_realip_module` (`set_real_ip_from` loopback/RFC1918/Docker bridges + IPv6 ULA, `real_ip_header X-Forwarded-For`, `real_ip_recursive on`) so `proxy_set_header X-Real-IP $remote_addr` preserves real external client IPs behind Coolify Traefik without permitting untrusted public IP spoofing.
- **`DEPLOY-BUG-03` & `DEPLOY-BUG-04`**: Added `ARG INGESTION_INTERNAL_URL=http://ingestion:4000` + `ENV INGESTION_INTERNAL_URL` in `docker/Dockerfile.web` `builder` stage for standalone Next.js rewrites, and forwarded optional `DATASTACK_ALLOW_PRIVATE_URLS`, `DATASTACK_MAX_BATCH_ITEMS`, `DATASTACK_RECENT_LIMIT`, `DATASTACK_MAINTENANCE_INTERVAL_MS`, and `DATASTACK_PROXY_MODE` env vars in `docker/docker-compose.yml`. - **`DEPLOY-BUG-03` & `DEPLOY-BUG-04`**: Added `ARG INGESTION_INTERNAL_URL=http://ingestion:4000` + `ENV INGESTION_INTERNAL_URL` in `docker/Dockerfile.web` `builder` stage for standalone Next.js rewrites, and forwarded optional `DATASTACK_ALLOW_PRIVATE_URLS`, `DATASTACK_MAX_BATCH_ITEMS`, `DATASTACK_RECENT_LIMIT`, `DATASTACK_MAINTENANCE_INTERVAL_MS`, and `DATASTACK_PROXY_MODE` env vars in `docker/docker-compose.yml`.
- **`DEPLOY-BUG-05`**: Updated `sdk/node/datastack/index.ts` (`emitSocketSingle` and `sendBatchChunkInternal`) to call `this.disconnect()` on terminal `DataStackAuthError` rejections (`TOKEN_ROTATED`, `CLIENT_REVOKED`, etc.) and recompiled `sdk/node/datastack/dist/*` so subsequent `client.connect()` performs a fresh handshake (`tests/r3-phase03-sdk-parity.test.ts` `5/5` PASS). - **`DEPLOY-BUG-05`**: Updated `sdk/node/datastack/index.ts` (`emitSocketSingle` and `sendBatchChunkInternal`) to call `this.disconnect()` on terminal `DataStackAuthError` rejections (`TOKEN_ROTATED`, `CLIENT_REVOKED`, etc.) and recompiled `sdk/node/datastack/dist/*` so subsequent `client.connect()` performs a fresh handshake (`tests/r3-phase03-sdk-parity.test.ts` `5/5` PASS).
- **`R5-MCP` Planning, Foundation & Server Track (`BASELINE-2026-11-21-R5-MCP`)**:
- Froze post-`R4` / post-`PREPUSH-FIX-01` codebase and documented the full 8-item raw-source-verified MCP forensic audit (`MCP-AUDIT-01`–`MCP-AUDIT-08`), Scope Locks, A–Z 3-phase roadmap (`R5-Phase-01`–`03`), Phase Execution Log, `ERR-R5-01`–`09` error contracts, and Actual Status Ledger across `/notes/`.
- **`R5-Phase-01` Complete & Verified (`2026-11-21`)** (`MCP-AUDIT-02` Complete; `MCP-AUDIT-03/04/05/07` Foundation; `ERR-R5-01` & `ERR-R5-02`):
- Added 8th PostgreSQL table `mcp_settings` identically in `core/Database/database.ts` and `server/index.js` (`ensureDatabaseSchema()`).
- Created TS (`core/logs/runtimeLogCapture.ts`) and CJS (`server/runtimeLogCapture.js`) bounded rotating runtime/build log writers & readers (`.runtime-logs/{web,ingestion}.log`, max 2MB + `.1` backup, secret redaction for Bearer/`dsmcp_`/JWT/JSON keys, graceful `LOG_CAPTURE_UNAVAILABLE` fallback), wired into `instrumentation.ts`, `server/index.js`, `scripts/start-all.js`, and `scripts/dev-all.js` (`DATASTACK_LOG_CAPTURED_BY_RUNNER=1` prevents duplicate tee writes).
- Created `scripts/capture-build-log.js` (`package.json` `"build"`) and updated `docker/Dockerfile.web` so standalone Next.js builds emit `.next/standalone/build.log`, `.next/standalone/build-meta.json`, and `.runtime-logs/last-build*`.
- Created `core/mcp/mcpSettings.ts` (`McpSettingsManager` with default `enabled: false`, `17` permission definitions across 4 categories, `dsmcp_` token generation + SHA-256 digest + AES-256-GCM vault encryption, IP CIDR validation, SSRF-guarded `coolifyUrl`, PG/RAM roundtrip, and `POST /internal/mcp-sync` notifier) and added `MCP_*` action types + secret redactions in `core/auth/auditLog.ts`.
- Added `middleware.ts` internal secret pass + timing-safe `/api/internal/mcp` bridge (`app/api/internal/mcp/route.ts`) supporting metrics, runtime/build logs, audit queries, client CRUD, safe settings updates, CSV+buffer data queries, storage stats/flush, and S3 archive.
- **Verified**: `5/5` `tests/r5-phase01-mcp-foundation-logs.test.ts` + `78/78` regression checks across 8 suites (`83/83` PASS), `compile_applet` (`npm run build` -> `.next/standalone/build.log` + `build-meta.json` generated), `lint_applet` (`eslint .` -> `0` errors), and `npm ci --include=dev --dry-run` / `--omit=dev --dry-run` clean. Ready for `Proceed to R5-Phase-02`.
--- ---
File diff suppressed because it is too large. Load diff
+15
View File
@@ -109,6 +109,21 @@ export const CORE_SCHEMA_TABLE_DDLS = [
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
) )
`, `,
`
CREATE TABLE IF NOT EXISTS mcp_settings (
id TEXT PRIMARY KEY DEFAULT 'default',
enabled BOOLEAN NOT NULL DEFAULT false,
token_hash TEXT NOT NULL DEFAULT '',
token_enc TEXT NOT NULL DEFAULT '',
token_preview TEXT NOT NULL DEFAULT '',
allowed_ips TEXT NOT NULL DEFAULT '',
permissions_json TEXT NOT NULL DEFAULT '{}',
coolify_url TEXT NOT NULL DEFAULT '',
coolify_token_enc TEXT NOT NULL DEFAULT '',
coolify_app_uuid TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
)
`,
] as const; ] as const;
export const CORE_SCHEMA_ALTER_DDLS = [ export const CORE_SCHEMA_ALTER_DDLS = [
+11
View File
@@ -22,6 +22,10 @@ export type AuditActionType =
| "WEBHOOK_TEST_TRIGGERED" | "WEBHOOK_TEST_TRIGGERED"
| "FILES_MANAGER_MUTATION" | "FILES_MANAGER_MUTATION"
| "DATA_EXPORT_DOWNLOADED" | "DATA_EXPORT_DOWNLOADED"
| "MCP_SETTINGS_UPDATED"
| "MCP_TOKEN_ROTATED"
| "MCP_TOOL_CALLED"
| "MCP_TOOL_DENIED"
| string; | string;
export interface AuditLogEntry { export interface AuditLogEntry {
@@ -98,6 +102,13 @@ function sanitizeMetadata(
"webhookSecretEnc", "webhookSecretEnc",
"prometheusMetricsToken", "prometheusMetricsToken",
"prometheusMetricsTokenEnc", "prometheusMetricsTokenEnc",
"mcpToken",
"mcpBearerToken",
"issuedToken",
"tokenHash",
"tokenEnc",
"coolifyToken",
"coolifyTokenEnc",
]); ]);
const clean: Record<string, unknown> = {}; const clean: Record<string, unknown> = {};
for (const [key, value] of Object.entries(metadata)) { for (const [key, value] of Object.entries(metadata)) {
+448
View File
@@ -0,0 +1,448 @@
import fs from "node:fs";
import path from "node:path";
const MAX_LOG_FILE_BYTES = 2 * 1024 * 1024;
const DEFAULT_TAIL_LINES = 120;
const MAX_TAIL_LINES = 500;
const MAX_SCAN_TAIL_BYTES = 512 * 1024;
export interface RuntimeLogEntry {
ts: string;
service: string;
level: string;
message: string;
}
export interface RuntimeLogReadResult {
entries: RuntimeLogEntry[];
totalMatched: number;
filesChecked: string[];
logsDir: string;
exists: boolean;
code?: "LOG_CAPTURE_UNAVAILABLE";
warning?: string;
}
export interface BuildLogReadResult {
meta: Record<string, unknown> | null;
logFile?: string | null;
lines: string[];
totalLines: number;
exists: boolean;
code?: "LOG_CAPTURE_UNAVAILABLE";
warning?: string;
}
const SECRET_SCRUB_PATTERNS: Array<[RegExp, string]> = [
[/\b(Bearer\s+)[A-Za-z0-9\-._~+/]+=*/gi, "$1[REDACTED]"],
[/\b(dsmcp_)[A-Za-z0-9_-]{8,}\b/gi, "$1[REDACTED]"],
[
/\b(eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.)[A-Za-z0-9_-]+\b/g,
"$1[REDACTED]",
],
[
/("(?:token|secret|password|jwtSecret|s3SecretAccessKey|webhookSecret|prometheusMetricsToken|coolifyToken|mcpToken|x-datastack-internal-secret)"\s*:\s*")([^"]+)(")/gi,
"$1[REDACTED]$3",
],
];
export function scrubSensitiveLogText(rawText: string): string {
let clean = String(rawText ?? "");
for (const [pattern, replacement] of SECRET_SCRUB_PATTERNS) {
clean = clean.replace(pattern, replacement);
}
return clean;
}
export function resolveRuntimeLogsDir(
env: NodeJS.ProcessEnv = process.env,
): string {
const explicitDir = env.DATASTACK_RUNTIME_LOGS_DIR?.trim() || "";
if (explicitDir) {
return path.resolve(explicitDir);
}
const storageDir = env.DATASTACK_STORAGE_DIR?.trim() || "";
if (storageDir) {
const resolved = path.resolve(storageDir);
if (path.basename(resolved).toLowerCase() === "data") {
return path.join(path.dirname(resolved), ".runtime-logs");
}
return path.join(resolved, ".runtime-logs");
}
return path.resolve(process.cwd(), "storage", ".runtime-logs");
}
function normalizeServiceName(service: string): "web" | "ingestion" {
const s = String(service || "").trim().toLowerCase();
if (s === "ingestion" || s === "server" || s === "socket") {
return "ingestion";
}
return "web";
}
function getServiceLogFilePath(service: "web" | "ingestion", logsDir: string): string {
return path.join(logsDir, `${service}.log`);
}
function rotateLogFileIfNeeded(filePath: string): void {
try {
const stat = fs.statSync(filePath);
if (stat.size >= MAX_LOG_FILE_BYTES) {
const backupPath = `${filePath}.1`;
try {
if (fs.existsSync(backupPath)) {
fs.unlinkSync(backupPath);
}
} catch {
// ignore backup cleanup error
}
fs.renameSync(filePath, backupPath);
}
} catch {
// file does not exist yet
}
}
export function appendRuntimeLogLine(
service: "web" | "ingestion" | string,
streamName: "stdout" | "stderr",
rawChunk: string,
options?: { logsDir?: string; env?: NodeJS.ProcessEnv },
): number {
const logsDir = options?.logsDir || resolveRuntimeLogsDir(options?.env);
const safeService = normalizeServiceName(service);
const text = scrubSensitiveLogText(rawChunk);
if (!text) return 0;
const lines = text.split(/\r?\n/).filter((line) => line.trim().length > 0);
if (lines.length === 0) return 0;
try {
fs.mkdirSync(logsDir, { recursive: true });
const filePath = getServiceLogFilePath(safeService, logsDir);
rotateLogFileIfNeeded(filePath);
const level = streamName === "stderr" ? "error" : "info";
const ts = new Date().toISOString();
const formatted =
lines
.map((line) =>
JSON.stringify({ ts, service: safeService, level, message: line }),
)
.join("\n") + "\n";
fs.appendFileSync(filePath, formatted, "utf8");
return lines.length;
} catch {
return 0;
}
}
export function installRuntimeLogHook(
service: "web" | "ingestion",
options?: { logsDir?: string; env?: NodeJS.ProcessEnv; force?: boolean },
): { installed: boolean; reason?: string; service?: string } {
const safeService = normalizeServiceName(service);
if (process.env.DATASTACK_LOG_CAPTURED_BY_RUNNER === "1" && !options?.force) {
return { installed: false, reason: "captured_by_runner" };
}
const globals = globalThis as Record<string, unknown>;
const guardKey = `__dataStackRuntimeLogHook_${safeService}`;
if (globals[guardKey]) {
return { installed: false, reason: "already_installed" };
}
globals[guardKey] = true;
let isWriting = false;
const wrapStream = (
stream: NodeJS.WriteStream,
streamName: "stdout" | "stderr",
) => {
const originalWrite = stream.write.bind(stream) as (
chunk: Uint8Array | string,
encoding?: BufferEncoding,
cb?: (err?: Error | null) => void,
) => boolean;
stream.write = ((
chunk: Uint8Array | string,
encodingOrCb?: BufferEncoding | ((err?: Error | null) => void),
cb?: (err?: Error | null) => void,
): boolean => {
const encoding =
typeof encodingOrCb === "string" ? encodingOrCb : undefined;
const callback = typeof encodingOrCb === "function" ? encodingOrCb : cb;
const result = originalWrite(chunk, encoding, callback);
if (!isWriting) {
isWriting = true;
try {
const raw =
typeof chunk === "string"
? chunk
: Buffer.isBuffer(chunk)
? chunk.toString(encoding || "utf8")
: Buffer.from(chunk).toString("utf8");
appendRuntimeLogLine(safeService, streamName, raw, options);
} catch {
// Never disrupt terminal output
} finally {
isWriting = false;
}
}
return result;
}) as typeof stream.write;
};
wrapStream(process.stdout, "stdout");
wrapStream(process.stderr, "stderr");
return { installed: true, service: safeService };
}
function readTailLinesFromFile(
filePath: string,
maxBytes = MAX_SCAN_TAIL_BYTES,
): string[] {
if (!fs.existsSync(filePath)) return [];
const stat = fs.statSync(filePath);
if (!stat.isFile() || stat.size === 0) return [];
const readSize = Math.min(stat.size, maxBytes);
const startPos = Math.max(0, stat.size - readSize);
const fd = fs.openSync(filePath, "r");
try {
const buf = Buffer.allocUnsafe(readSize);
const bytesRead = fs.readSync(fd, buf, 0, readSize, startPos);
let content = buf.subarray(0, bytesRead).toString("utf8");
if (startPos > 0) {
const firstNewline = content.indexOf("\n");
content = firstNewline >= 0 ? content.slice(firstNewline + 1) : "";
}
return content.split(/\r?\n/).filter(Boolean);
} finally {
fs.closeSync(fd);
}
}
function parseStoredLogLine(
rawLine: string,
fallbackService: string,
): RuntimeLogEntry {
try {
const parsed = JSON.parse(rawLine) as Record<string, unknown>;
if (parsed && typeof parsed === "object" && typeof parsed.message === "string") {
return {
ts: typeof parsed.ts === "string" ? parsed.ts : "",
service:
typeof parsed.service === "string" ? parsed.service : fallbackService,
level: typeof parsed.level === "string" ? parsed.level : "info",
message: parsed.message,
};
}
} catch {
// Plain-text fallback
}
return {
ts: "",
service: fallbackService,
level: "info",
message: rawLine,
};
}
export function readRuntimeLogs(options?: {
logsDir?: string;
env?: NodeJS.ProcessEnv;
service?: "web" | "ingestion" | "all";
lines?: number;
search?: string;
level?: string;
}): RuntimeLogReadResult {
const logsDir = options?.logsDir || resolveRuntimeLogsDir(options?.env);
const serviceFilter = String(options?.service || "all")
.trim()
.toLowerCase();
const requestedLines = Number(options?.lines);
const limit = Number.isFinite(requestedLines)
? Math.min(MAX_TAIL_LINES, Math.max(1, Math.floor(requestedLines)))
: DEFAULT_TAIL_LINES;
const search = options?.search?.trim().toLowerCase() || "";
const levelFilter = options?.level?.trim().toLowerCase() || "";
const services: Array<"web" | "ingestion"> =
serviceFilter === "web"
? ["web"]
: serviceFilter === "ingestion"
? ["ingestion"]
: ["web", "ingestion"];
const collected: RuntimeLogEntry[] = [];
const filesChecked: string[] = [];
for (const svc of services) {
const mainFile = getServiceLogFilePath(svc, logsDir);
const backupFile = `${mainFile}.1`;
for (const candidate of [backupFile, mainFile]) {
if (fs.existsSync(candidate)) {
filesChecked.push(path.basename(candidate));
const rawLines = readTailLinesFromFile(candidate);
for (const rawLine of rawLines) {
const entry = parseStoredLogLine(rawLine, svc);
if (levelFilter && levelFilter !== "all" && entry.level !== levelFilter) {
continue;
}
if (
search &&
!entry.message.toLowerCase().includes(search) &&
!entry.service.toLowerCase().includes(search)
) {
continue;
}
collected.push(entry);
}
}
}
}
if (filesChecked.length === 0) {
return {
entries: [],
totalMatched: 0,
filesChecked: [],
logsDir,
exists: false,
code: "LOG_CAPTURE_UNAVAILABLE",
warning:
"No runtime logs captured yet in .runtime-logs/. Start services or emit stdout logs first.",
};
}
collected.sort((a, b) => {
if (a.ts && b.ts && a.ts !== b.ts) {
return a.ts.localeCompare(b.ts);
}
return 0;
});
return {
entries: collected.slice(-limit),
totalMatched: collected.length,
filesChecked,
logsDir,
exists: true,
};
}
export function readBuildLogs(options?: {
logsDir?: string;
env?: NodeJS.ProcessEnv;
cwd?: string;
lines?: number;
search?: string;
}): BuildLogReadResult {
const logsDir = options?.logsDir || resolveRuntimeLogsDir(options?.env);
const cwd = options?.cwd || process.cwd();
const requestedLines = Number(options?.lines);
const limit = Number.isFinite(requestedLines)
? Math.min(MAX_TAIL_LINES, Math.max(1, Math.floor(requestedLines)))
: 200;
const search = options?.search?.trim().toLowerCase() || "";
const metaCandidates = [
path.join(logsDir, "last-build-meta.json"),
path.join(cwd, "build-meta.json"),
path.join(cwd, ".next", "standalone", "build-meta.json"),
];
const logCandidates = [
path.join(logsDir, "last-build.log"),
path.join(cwd, "build.log"),
path.join(cwd, ".next", "standalone", "build.log"),
];
let meta: Record<string, unknown> | null = null;
for (const metaPath of metaCandidates) {
try {
if (fs.existsSync(metaPath)) {
meta = JSON.parse(fs.readFileSync(metaPath, "utf8")) as Record<
string,
unknown
>;
break;
}
} catch {
// try next candidate
}
}
let logPathUsed: string | null = null;
let allLines: string[] = [];
for (const logPath of logCandidates) {
try {
if (fs.existsSync(logPath)) {
logPathUsed = path.basename(logPath);
allLines = readTailLinesFromFile(logPath);
break;
}
} catch {
// try next candidate
}
}
if (!meta && !logPathUsed) {
return {
meta: null,
lines: [],
totalLines: 0,
exists: false,
code: "LOG_CAPTURE_UNAVAILABLE",
warning:
"No captured build log or build-meta.json found yet. Run npm run build first.",
};
}
const filtered = search
? allLines.filter((line) => line.toLowerCase().includes(search))
: allLines;
return {
meta,
logFile: logPathUsed,
lines: filtered.slice(-limit),
totalLines: filtered.length,
exists: true,
};
}
export function syncStandaloneBuildArtifactsToLogsDir(options?: {
logsDir?: string;
env?: NodeJS.ProcessEnv;
cwd?: string;
}): { synced: boolean; logsDir?: string } {
const logsDir = options?.logsDir || resolveRuntimeLogsDir(options?.env);
const cwd = options?.cwd || process.cwd();
const buildLogSrc = [
path.join(cwd, "build.log"),
path.join(cwd, ".next", "standalone", "build.log"),
].find((candidate) => fs.existsSync(candidate));
const buildMetaSrc = [
path.join(cwd, "build-meta.json"),
path.join(cwd, ".next", "standalone", "build-meta.json"),
].find((candidate) => fs.existsSync(candidate));
if (!buildLogSrc && !buildMetaSrc) {
return { synced: false };
}
try {
fs.mkdirSync(logsDir, { recursive: true });
if (buildLogSrc) {
fs.copyFileSync(buildLogSrc, path.join(logsDir, "last-build.log"));
}
if (buildMetaSrc) {
fs.copyFileSync(buildMetaSrc, path.join(logsDir, "last-build-meta.json"));
}
return { synced: true, logsDir };
} catch {
return { synced: false };
}
}
+672
View File
@@ -0,0 +1,672 @@
import { createHash, randomBytes, timingSafeEqual } from "node:crypto";
import { DatabaseEngine } from "../Database/database";
import { getInternalIngestionHeaders } from "../Security/security";
import {
decryptSecret,
encryptSecret,
getMaskedHintFromEncrypted,
} from "../Security/secretVault";
import { SSRFBlockedError, validateOutboundUrlSync } from "../Security/ssrfGuard";
import { validateAllowedIpsInput } from "../Security/ipAllowlist";
export type McpPermissionKey =
| "metrics_health_read"
| "runtime_logs_read"
| "build_logs_read"
| "audit_logs_read"
| "ingestion_events_read"
| "clients_read"
| "clients_create"
| "clients_update"
| "clients_revoke_delete"
| "files_list_read"
| "files_download"
| "files_upload_create"
| "files_edit"
| "files_delete"
| "settings_read"
| "settings_write"
| "storage_maintenance";
export type McpPermissions = Record<McpPermissionKey, boolean>;
export interface McpPermissionDefinition {
key: McpPermissionKey;
label: string;
description: string;
category:
| "Observability & Logs"
| "Clients & SDKs"
| "Files & Storage"
| "Admin & Maintenance";
defaultEnabled: boolean;
risk: "read" | "write" | "destructive";
}
export const MCP_PERMISSION_DEFINITIONS: readonly McpPermissionDefinition[] = [
{
key: "metrics_health_read",
label: "Dashboard Metrics & Health",
description: "Inspect dashboard metrics, Redis/PG health, buffer depth, and storage usage.",
category: "Observability & Logs",
defaultEnabled: true,
risk: "read",
},
{
key: "runtime_logs_read",
label: "Runtime Logs (Web & Ingestion)",
description: "Read rotating Next.js (Port 3000) and Ingestion (Port 4000) stdout/stderr logs.",
category: "Observability & Logs",
defaultEnabled: true,
risk: "read",
},
{
key: "build_logs_read",
label: "Build & Coolify Logs",
description: "Read captured Next.js build manifest/logs and optional Coolify deploy logs.",
category: "Observability & Logs",
defaultEnabled: true,
risk: "read",
},
{
key: "audit_logs_read",
label: "Admin Audit Logs",
description: "Search administrative, RBAC, client, and MCP action history.",
category: "Observability & Logs",
defaultEnabled: true,
risk: "read",
},
{
key: "ingestion_events_read",
label: "Telemetry Events Read",
description: "Query buffered recent events and stored CSV/GZ telemetry partitions.",
category: "Observability & Logs",
defaultEnabled: true,
risk: "read",
},
{
key: "clients_read",
label: "List & Inspect Clients",
description: "View registered SDK client IDs, quotas, allowlists, and token previews.",
category: "Clients & SDKs",
defaultEnabled: true,
risk: "read",
},
{
key: "clients_create",
label: "Create SDK Clients",
description: "Register new SDK client credentials from an AI agent session.",
category: "Clients & SDKs",
defaultEnabled: false,
risk: "write",
},
{
key: "clients_update",
label: "Update SDK Clients",
description: "Modify client names, rate limits, storage quotas, allowed IPs, or expiry.",
category: "Clients & SDKs",
defaultEnabled: false,
risk: "write",
},
{
key: "clients_revoke_delete",
label: "Rotate, Revoke & Delete Clients",
description: "Regenerate client JWTs, revoke active client tokens, or permanently delete clients.",
category: "Clients & SDKs",
defaultEnabled: false,
risk: "destructive",
},
{
key: "files_list_read",
label: "Browse & Read Storage Files",
description: "List directories and inspect bounded lines/bytes strictly inside /app/storage.",
category: "Files & Storage",
defaultEnabled: true,
risk: "read",
},
{
key: "files_download",
label: "Download Storage File Chunks",
description: "Fetch bounded text or base64 file chunks inside /app/storage.",
category: "Files & Storage",
defaultEnabled: true,
risk: "read",
},
{
key: "files_upload_create",
label: "Upload & Create Files",
description: "Upload or create new files inside /app/storage (respecting .lock protection).",
category: "Files & Storage",
defaultEnabled: false,
risk: "write",
},
{
key: "files_edit",
label: "Edit Storage Files",
description: "Perform targeted search/replace edits on non-locked files inside /app/storage.",
category: "Files & Storage",
defaultEnabled: false,
risk: "write",
},
{
key: "files_delete",
label: "Delete Storage Files",
description: "Delete non-locked files inside /app/storage.",
category: "Files & Storage",
defaultEnabled: false,
risk: "destructive",
},
{
key: "settings_read",
label: "Read System Settings",
description: "Inspect masked public runtime settings (raw secrets are always hidden).",
category: "Admin & Maintenance",
defaultEnabled: true,
risk: "read",
},
{
key: "settings_write",
label: "Update System Settings",
description: "Modify ingestion/storage thresholds (SSRF-guarded; never touches admin users or JWT keys).",
category: "Admin & Maintenance",
defaultEnabled: false,
risk: "destructive",
},
{
key: "storage_maintenance",
label: "Buffer Flush & S3 Archive",
description: "Trigger manual buffer flushes to CSV or on-demand S3 cold backups.",
category: "Admin & Maintenance",
defaultEnabled: false,
risk: "write",
},
] as const;
export interface McpSettingsRecord {
enabled: boolean;
tokenHash: string;
tokenEnc: string;
tokenPreview: string;
allowedIps: string;
permissions: McpPermissions;
coolifyUrl: string;
coolifyTokenEnc: string;
coolifyAppUuid: string;
updatedAt: string;
}
export interface PublicMcpSettings {
enabled: boolean;
hasToken: boolean;
tokenPreview: string;
allowedIps: string;
permissions: McpPermissions;
coolifyUrl: string;
hasCoolifyToken: boolean;
coolifyTokenMask: string;
coolifyAppUuid: string;
endpointPath: string;
updatedAt: string;
}
export interface McpBridgeSyncResult {
attempted: boolean;
ok: boolean;
synced: boolean;
status?: number;
error?: string;
checkedAt: string;
}
export type McpSettingsUpdateInput = {
enabled?: boolean;
allowedIps?: string;
permissions?: Partial<Record<McpPermissionKey, boolean>>;
rotateToken?: boolean;
revokeToken?: boolean;
coolifyUrl?: string;
coolifyToken?: string;
clearCoolifyToken?: boolean;
coolifyAppUuid?: string;
};
type McpGlobals = typeof globalThis & {
__dataStackMcpSettings?: McpSettingsRecord;
__dataStackLastMcpBridgeSync?: McpBridgeSyncResult;
};
const globals = globalThis as McpGlobals;
export function getDefaultMcpPermissions(): McpPermissions {
const out = {} as McpPermissions;
for (const def of MCP_PERMISSION_DEFINITIONS) {
out[def.key] = def.defaultEnabled;
}
return out;
}
export function normalizeMcpPermissions(
raw: unknown,
base: McpPermissions = getDefaultMcpPermissions(),
): McpPermissions {
const merged: McpPermissions = { ...base };
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
return merged;
}
const source = raw as Record<string, unknown>;
for (const def of MCP_PERMISSION_DEFINITIONS) {
if (typeof source[def.key] === "boolean") {
merged[def.key] = source[def.key] as boolean;
}
}
return merged;
}
export function getDefaultMcpSettings(): McpSettingsRecord {
return {
enabled: false,
tokenHash: "",
tokenEnc: "",
tokenPreview: "",
allowedIps: "",
permissions: getDefaultMcpPermissions(),
coolifyUrl: "",
coolifyTokenEnc: "",
coolifyAppUuid: "",
updatedAt: new Date().toISOString(),
};
}
export function generateMcpBearerToken(): string {
return `dsmcp_${randomBytes(32).toString("hex")}`;
}
export function hashMcpToken(rawToken: string): string {
return createHash("sha256").update(String(rawToken || ""), "utf8").digest("hex");
}
export function formatMcpTokenPreview(rawToken: string): string {
const clean = String(rawToken || "").trim();
if (!clean) return "";
if (clean.length <= 14) return `${clean.slice(0, 4)}...`;
return `${clean.slice(0, 10)}...${clean.slice(-4)}`;
}
export function verifyMcpBearerTokenHash(
presentedToken: string,
expectedHashHex: string,
): boolean {
const cleanPresented = String(presentedToken || "").trim();
const cleanExpected = String(expectedHashHex || "").trim().toLowerCase();
if (!cleanPresented || cleanExpected.length !== 64) {
return false;
}
try {
const presentedBuf = Buffer.from(hashMcpToken(cleanPresented), "hex");
const expectedBuf = Buffer.from(cleanExpected, "hex");
if (presentedBuf.length !== expectedBuf.length) {
return false;
}
return timingSafeEqual(presentedBuf, expectedBuf);
} catch {
return false;
}
}
function getMcpState(): McpSettingsRecord {
if (!globals.__dataStackMcpSettings) {
globals.__dataStackMcpSettings = getDefaultMcpSettings();
}
return globals.__dataStackMcpSettings;
}
export function toPublicMcpSettings(record: McpSettingsRecord): PublicMcpSettings {
return {
enabled: Boolean(record.enabled),
hasToken: Boolean(record.tokenHash),
tokenPreview: record.tokenPreview || "",
allowedIps: record.allowedIps || "",
permissions: normalizeMcpPermissions(record.permissions),
coolifyUrl: record.coolifyUrl || "",
hasCoolifyToken: Boolean(record.coolifyTokenEnc),
coolifyTokenMask: getMaskedHintFromEncrypted(record.coolifyTokenEnc),
coolifyAppUuid: record.coolifyAppUuid || "",
endpointPath: "/mcp",
updatedAt: record.updatedAt || new Date().toISOString(),
};
}
export class McpSettingsManager {
public static getSettings(): McpSettingsRecord {
const current = getMcpState();
return {
...current,
permissions: { ...current.permissions },
};
}
public static async getSettingsAsync(): Promise<McpSettingsRecord> {
const pool = DatabaseEngine.getPool();
if (!pool) {
return this.getSettings();
}
try {
await DatabaseEngine.ensureSchema();
const res = await pool.query<{
enabled: boolean;
token_hash: string;
token_enc: string;
token_preview: string;
allowed_ips: string;
permissions_json: string;
coolify_url: string;
coolify_token_enc: string;
coolify_app_uuid: string;
updated_at: Date | string;
}>(
`SELECT enabled, token_hash, token_enc, token_preview, allowed_ips,
permissions_json, coolify_url, coolify_token_enc, coolify_app_uuid, updated_at
FROM mcp_settings
WHERE id = 'default'`,
);
if (!res.rowCount || res.rowCount === 0) {
return this.getSettings();
}
const row = res.rows[0];
let parsedPermissions: unknown = {};
try {
parsedPermissions = JSON.parse(row.permissions_json || "{}");
} catch {
parsedPermissions = {};
}
const hydrated: McpSettingsRecord = {
enabled: Boolean(row.enabled),
tokenHash: String(row.token_hash || ""),
tokenEnc: String(row.token_enc || ""),
tokenPreview: String(row.token_preview || ""),
allowedIps: String(row.allowed_ips || ""),
permissions: normalizeMcpPermissions(parsedPermissions),
coolifyUrl: String(row.coolify_url || ""),
coolifyTokenEnc: String(row.coolify_token_enc || ""),
coolifyAppUuid: String(row.coolify_app_uuid || ""),
updatedAt:
row.updated_at instanceof Date
? row.updated_at.toISOString()
: String(row.updated_at || new Date().toISOString()),
};
globals.__dataStackMcpSettings = hydrated;
return { ...hydrated, permissions: { ...hydrated.permissions } };
} catch (err) {
console.warn(
"[DataStack MCP] Failed to read mcp_settings from PostgreSQL (using cached/default state):",
err instanceof Error ? err.message : err,
);
return this.getSettings();
}
}
public static async getPublicSettingsAsync(): Promise<PublicMcpSettings> {
const settings = await this.getSettingsAsync();
return toPublicMcpSettings(settings);
}
public static getLastBridgeSyncStatus(): McpBridgeSyncResult | null {
return globals.__dataStackLastMcpBridgeSync ?? null;
}
private static async persistToDb(record: McpSettingsRecord): Promise<void> {
const pool = DatabaseEngine.getPool();
if (!pool) return;
const schemaOk = await DatabaseEngine.ensureSchema();
if (!schemaOk) {
throw new Error(
"DATABASE_WRITE_FAILED: Unable to verify PostgreSQL schema for mcp_settings.",
);
}
await pool.query(
`
INSERT INTO mcp_settings (
id, enabled, token_hash, token_enc, token_preview,
allowed_ips, permissions_json, coolify_url, coolify_token_enc,
coolify_app_uuid, updated_at
)
VALUES ('default', $1, $2, $3, $4, $5, $6, $7, $8, $9, $10::timestamptz)
ON CONFLICT (id) DO UPDATE SET
enabled = EXCLUDED.enabled,
token_hash = EXCLUDED.token_hash,
token_enc = EXCLUDED.token_enc,
token_preview = EXCLUDED.token_preview,
allowed_ips = EXCLUDED.allowed_ips,
permissions_json = EXCLUDED.permissions_json,
coolify_url = EXCLUDED.coolify_url,
coolify_token_enc = EXCLUDED.coolify_token_enc,
coolify_app_uuid = EXCLUDED.coolify_app_uuid,
updated_at = EXCLUDED.updated_at
`,
[
record.enabled,
record.tokenHash,
record.tokenEnc,
record.tokenPreview,
record.allowedIps,
JSON.stringify(record.permissions),
record.coolifyUrl,
record.coolifyTokenEnc,
record.coolifyAppUuid,
record.updatedAt,
],
);
}
private static async notifyIngestionMcpSync(
record: McpSettingsRecord,
tokenRotated: boolean,
): Promise<McpBridgeSyncResult> {
const checkedAt = new Date().toISOString();
const configuredUrl = process.env.INGESTION_INTERNAL_URL?.trim();
if (process.env.DATASTACK_DISABLE_LISTEN === "1" && !configuredUrl) {
const skipped: McpBridgeSyncResult = {
attempted: false,
ok: true,
synced: false,
checkedAt,
};
globals.__dataStackLastMcpBridgeSync = skipped;
return skipped;
}
const headers = getInternalIngestionHeaders();
if (!headers["x-datastack-internal-secret"]) {
const result: McpBridgeSyncResult = {
attempted: true,
ok: false,
synced: false,
error: "Internal bridge secret is not configured",
checkedAt,
};
globals.__dataStackLastMcpBridgeSync = result;
return result;
}
const ingestionBaseUrl = (
configuredUrl || "http://127.0.0.1:4000"
).replace(/\/$/, "");
let coolifyTokenPlain = "";
if (record.coolifyTokenEnc) {
try {
coolifyTokenPlain = decryptSecret(record.coolifyTokenEnc);
} catch {
coolifyTokenPlain = "";
}
}
try {
const res = await fetch(`${ingestionBaseUrl}/internal/mcp-sync`, {
method: "POST",
headers: {
"Content-Type": "application/json",
...headers,
},
body: JSON.stringify({
enabled: record.enabled,
tokenHash: record.tokenHash,
tokenPreview: record.tokenPreview,
allowedIps: record.allowedIps,
permissions: record.permissions,
coolifyUrl: record.coolifyUrl,
coolifyToken: coolifyTokenPlain,
coolifyAppUuid: record.coolifyAppUuid,
updatedAt: record.updatedAt,
tokenRotated,
}),
cache: "no-store",
signal: AbortSignal.timeout(1500),
});
const result: McpBridgeSyncResult = {
attempted: true,
ok: res.ok,
synced: res.ok,
status: res.status,
...(res.ok ? {} : { error: `Port 4000 mcp-sync returned HTTP ${res.status}` }),
checkedAt,
};
globals.__dataStackLastMcpBridgeSync = result;
return result;
} catch (err: unknown) {
const result: McpBridgeSyncResult = {
attempted: true,
ok: false,
synced: false,
error: err instanceof Error ? err.message : "Port 4000 unreachable",
checkedAt,
};
globals.__dataStackLastMcpBridgeSync = result;
return result;
}
}
public static async updateSettingsAsync(
updates: McpSettingsUpdateInput,
): Promise<{
settings: PublicMcpSettings;
issuedToken?: string;
tokenRotated: boolean;
bridgeSync: McpBridgeSyncResult;
}> {
const current = await this.getSettingsAsync();
let nextEnabled =
typeof updates.enabled === "boolean" ? updates.enabled : current.enabled;
let nextAllowedIps = current.allowedIps;
if (updates.allowedIps !== undefined) {
const ipCheck = validateAllowedIpsInput(updates.allowedIps);
if (!ipCheck.valid) {
throw new Error(ipCheck.error || "Invalid allowedIps list.");
}
nextAllowedIps = ipCheck.normalized;
}
let nextCoolifyUrl = current.coolifyUrl;
if (updates.coolifyUrl !== undefined) {
const trimmed = String(updates.coolifyUrl || "").trim();
if (trimmed) {
const ssrf = validateOutboundUrlSync(trimmed, {
fieldName: "coolifyUrl",
});
if (!ssrf.allowed) {
if (ssrf.code === "SSRF_BLOCKED_HOST") {
throw new SSRFBlockedError(
ssrf.error || "coolifyUrl target host is blocked.",
ssrf.hostname || "",
);
}
throw new Error(ssrf.error || "Invalid coolifyUrl.");
}
}
nextCoolifyUrl = trimmed;
}
const nextCoolifyAppUuid =
updates.coolifyAppUuid !== undefined
? String(updates.coolifyAppUuid || "").trim().slice(0, 128)
: current.coolifyAppUuid;
let nextCoolifyTokenEnc = current.coolifyTokenEnc;
if (updates.clearCoolifyToken) {
nextCoolifyTokenEnc = "";
} else if (
typeof updates.coolifyToken === "string" &&
updates.coolifyToken.trim().length > 0
) {
nextCoolifyTokenEnc = encryptSecret(updates.coolifyToken.trim());
}
const nextPermissions = normalizeMcpPermissions(
updates.permissions,
current.permissions,
);
let nextTokenHash = current.tokenHash;
let nextTokenEnc = current.tokenEnc;
let nextTokenPreview = current.tokenPreview;
let issuedToken: string | undefined;
let tokenRotated = false;
if (updates.revokeToken) {
nextTokenHash = "";
nextTokenEnc = "";
nextTokenPreview = "";
nextEnabled = false;
tokenRotated = true;
} else if (
updates.rotateToken ||
(nextEnabled && !current.tokenHash)
) {
const freshToken = generateMcpBearerToken();
issuedToken = freshToken;
nextTokenHash = hashMcpToken(freshToken);
nextTokenEnc = encryptSecret(freshToken);
nextTokenPreview = formatMcpTokenPreview(freshToken);
tokenRotated = true;
}
const nextRecord: McpSettingsRecord = {
enabled: nextEnabled,
tokenHash: nextTokenHash,
tokenEnc: nextTokenEnc,
tokenPreview: nextTokenPreview,
allowedIps: nextAllowedIps,
permissions: nextPermissions,
coolifyUrl: nextCoolifyUrl,
coolifyTokenEnc: nextCoolifyTokenEnc,
coolifyAppUuid: nextCoolifyAppUuid,
updatedAt: new Date().toISOString(),
};
await this.persistToDb(nextRecord);
globals.__dataStackMcpSettings = nextRecord;
const bridgeSync = await this.notifyIngestionMcpSync(
nextRecord,
tokenRotated || updates.enabled === false,
);
return {
settings: toPublicMcpSettings(nextRecord),
...(issuedToken ? { issuedToken } : {}),
tokenRotated,
bridgeSync,
};
}
public static resetForTests(): void {
delete globals.__dataStackMcpSettings;
delete globals.__dataStackLastMcpBridgeSync;
}
}
+2 -2
View File
@@ -21,10 +21,10 @@ ENV NEXT_TELEMETRY_DISABLED=1
ENV PORT=3000 ENV PORT=3000
ENV HOSTNAME=0.0.0.0 ENV HOSTNAME=0.0.0.0
RUN apk add --no-cache su-exec \ RUN apk add --no-cache su-exec \
&& mkdir -p /app/storage/data \ && mkdir -p /app/storage/data /app/storage/.runtime-logs \
&& chown -R node:node /app/storage && chown -R node:node /app/storage
COPY --from=builder --chown=node:node /app/public ./public COPY --from=builder --chown=node:node /app/public ./public
COPY --from=builder --chown=node:node /app/.next/standalone ./ COPY --from=builder --chown=node:node /app/.next/standalone ./
COPY --from=builder --chown=node:node /app/.next/static ./.next/static COPY --from=builder --chown=node:node /app/.next/static ./.next/static
EXPOSE 3000 EXPOSE 3000
CMD ["sh", "-c", "mkdir -p /app/storage/data && NODE_UG=\"$(id -u node):$(id -g node)\" && if [ \"$(stat -c '%u:%g' /app/storage/data)\" != \"$NODE_UG\" ]; then chown -R node:node /app/storage; else chown node:node /app/storage /app/storage/data; fi && exec su-exec node:node node server.js"] CMD ["sh", "-c", "mkdir -p /app/storage/data /app/storage/.runtime-logs && NODE_UG=\"$(id -u node):$(id -g node)\" && if [ \"$(stat -c '%u:%g' /app/storage/data)\" != \"$NODE_UG\" ]; then chown -R node:node /app/storage; else chown node:node /app/storage /app/storage/data /app/storage/.runtime-logs; fi && exec su-exec node:node node server.js"]
+11
View File
@@ -4,6 +4,17 @@ export async function register(): Promise<void> {
return; return;
} }
try {
const {
installRuntimeLogHook,
syncStandaloneBuildArtifactsToLogsDir,
} = await import("@/core/logs/runtimeLogCapture");
installRuntimeLogHook("web");
syncStandaloneBuildArtifactsToLogsDir();
} catch {
// Non-fatal if runtime log hook cannot initialize
}
try { try {
const { SettingsManager } = await import( const { SettingsManager } = await import(
"@/core/adminSettings/settingsManager" "@/core/adminSettings/settingsManager"
+18 -3
View File
@@ -7,12 +7,15 @@ import { NextRequest, NextResponse } from "next/server";
const SESSION_COOKIE = "datastack_admin_session"; const SESSION_COOKIE = "datastack_admin_session";
const SAFE_METHODS = new Set(["GET", "HEAD", "OPTIONS"]); const SAFE_METHODS = new Set(["GET", "HEAD", "OPTIONS"]);
function isProtectedApi(pathname: string, method: string) { function isProtectedApi(pathname: string, method: string, hasInternalSecretHeader: boolean) {
if (pathname === "/api/auth/login") return false; if (pathname === "/api/auth/login") return false;
if (pathname === "/api/auth/setup" && method === "POST") return false; if (pathname === "/api/auth/setup" && method === "POST") return false;
if (pathname === "/api/events" && method === "POST") return false; if (pathname === "/api/events" && method === "POST") return false;
if (pathname === "/api/health" && method === "GET") return false; if (pathname === "/api/health" && method === "GET") return false;
if (pathname === "/api/prometheus" && method === "GET") return false; if (pathname === "/api/prometheus" && method === "GET") return false;
if (pathname === "/api/internal/mcp" && method === "POST" && hasInternalSecretHeader) {
return false;
}
return pathname.startsWith("/api/"); return pathname.startsWith("/api/");
} }
@@ -62,14 +65,26 @@ export async function middleware(request: NextRequest) {
(pathname === "/vcon" || pathname.startsWith("/vcon/")) && (pathname === "/vcon" || pathname.startsWith("/vcon/")) &&
!isLoginPage && !isLoginPage &&
!isSetupPage; !isSetupPage;
const isAdminApi = isProtectedApi(pathname, request.method); const hasInternalSecretHeader = Boolean(
request.headers.get("x-datastack-internal-secret")?.trim(),
);
const isAdminApi = isProtectedApi(
pathname,
request.method,
hasInternalSecretHeader,
);
const hasBearerAuth = Boolean( const hasBearerAuth = Boolean(
request.headers.get("authorization")?.toLowerCase().startsWith("bearer "), request.headers.get("authorization")?.toLowerCase().startsWith("bearer "),
); );
const isStateChangingApi = const isStateChangingApi =
pathname.startsWith("/api/") && pathname.startsWith("/api/") &&
!SAFE_METHODS.has(request.method.toUpperCase()) && !SAFE_METHODS.has(request.method.toUpperCase()) &&
!(pathname === "/api/events" && hasBearerAuth); !(pathname === "/api/events" && hasBearerAuth) &&
!(
pathname === "/api/internal/mcp" &&
request.method.toUpperCase() === "POST" &&
hasInternalSecretHeader
);
const requestHeaders = new Headers(request.headers); const requestHeaders = new Headers(request.headers);
requestHeaders.set("x-datastack-pathname", pathname); requestHeaders.set("x-datastack-pathname", pathname);
+1 -1
View File
@@ -10,7 +10,7 @@
"dev:web": "next dev -p 3000 -H 0.0.0.0", "dev:web": "next dev -p 3000 -H 0.0.0.0",
"dev:all": "node scripts/dev-all.js", "dev:all": "node scripts/dev-all.js",
"server": "node server/index.js", "server": "node server/index.js",
"build": "NODE_ENV=production next build", "build": "node scripts/capture-build-log.js",
"start": "node scripts/start-all.js", "start": "node scripts/start-all.js",
"start:web": "next start", "start:web": "next start",
"start:server": "node server/index.js", "start:server": "node server/index.js",
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env node
/**
* Captures Next.js production build output & metadata into `.next/standalone/build.log`
* and `.next/standalone/build-meta.json` (plus runtime logs dir when writable) so
* build diagnostics are inspectable via MCP `read_build_logs` inside running containers.
*/
const fs = require('node:fs');
const path = require('node:path');
const { spawn } = require('node:child_process');
const { resolveRuntimeLogsDir, scrubSensitiveLogText } = require('../server/runtimeLogCapture');
const MAX_BUILD_LOG_BYTES = 512 * 1024;
const nextBin = require.resolve('next/dist/bin/next');
const pkg = require(path.resolve(__dirname, '..', 'package.json'));
const startedAt = new Date().toISOString();
const startMs = Date.now();
const capturedChunks = [];
let capturedBytes = 0;
function recordChunk(text) {
const clean = scrubSensitiveLogText(text);
capturedChunks.push(clean);
capturedBytes += Buffer.byteLength(clean, 'utf8');
while (capturedBytes > MAX_BUILD_LOG_BYTES && capturedChunks.length > 1) {
const dropped = capturedChunks.shift();
capturedBytes -= Buffer.byteLength(dropped, 'utf8');
}
}
const childEnv = {
...process.env,
NODE_ENV: 'production',
};
const child = spawn(process.execPath, [nextBin, 'build'], {
stdio: ['inherit', 'pipe', 'pipe'],
shell: false,
env: childEnv,
});
child.stdout.on('data', (buf) => {
process.stdout.write(buf);
recordChunk(buf.toString('utf8'));
});
child.stderr.on('data', (buf) => {
process.stderr.write(buf);
recordChunk(buf.toString('utf8'));
});
child.on('error', (err) => {
console.error('[DataStack Build] Failed to spawn Next.js build:', err.message);
process.exit(1);
});
child.on('close', (code) => {
const exitCode = typeof code === 'number' ? code : 1;
const finishedAt = new Date().toISOString();
const durationMs = Date.now() - startMs;
const meta = {
appName: pkg.name || 'datastack',
appVersion: pkg.version || '0.1.0',
nodeVersion: process.version,
nextVersion: (pkg.dependencies && pkg.dependencies.next) || 'unknown',
startedAt,
finishedAt,
durationMs,
exitCode,
status: exitCode === 0 ? 'succeeded' : 'failed',
};
const fullLog = capturedChunks.join('');
const writeArtifactDir = (dir, logName, metaName) => {
try {
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, logName), fullLog, 'utf8');
fs.writeFileSync(path.join(dir, metaName), JSON.stringify(meta, null, 2) + '\n', 'utf8');
} catch {
// Non-fatal if target dir is read-only or unavailable
}
};
const standaloneDir = path.resolve(process.cwd(), '.next', 'standalone');
if (exitCode === 0 && fs.existsSync(standaloneDir)) {
writeArtifactDir(standaloneDir, 'build.log', 'build-meta.json');
}
const runtimeLogsDir = resolveRuntimeLogsDir(process.env);
writeArtifactDir(runtimeLogsDir, 'last-build.log', 'last-build-meta.json');
process.exit(exitCode);
});
+28 -4
View File
@@ -7,6 +7,7 @@
const path = require('node:path'); const path = require('node:path');
const { spawn } = require('node:child_process'); const { spawn } = require('node:child_process');
const { appendRuntimeLogLine } = require('../server/runtimeLogCapture');
const webPort = String(process.env.PORT || '3000'); const webPort = String(process.env.PORT || '3000');
const nextBin = require.resolve('next/dist/bin/next'); const nextBin = require.resolve('next/dist/bin/next');
@@ -19,18 +20,41 @@ console.log('\x1b[36m%s\x1b[0m', ` - Port ${webPort}: Next.js 15 Web & Admin Co
console.log('\x1b[36m%s\x1b[0m', ' - Port 4000: Ingestion & Socket.IO Streaming Engine'); console.log('\x1b[36m%s\x1b[0m', ' - Port 4000: Ingestion & Socket.IO Streaming Engine');
console.log('\x1b[36m%s\x1b[0m', '=======================================================\n'); console.log('\x1b[36m%s\x1b[0m', '=======================================================\n');
const childEnv = {
...process.env,
DATASTACK_LOG_CAPTURED_BY_RUNNER: '1',
};
const nextProc = spawn(process.execPath, [nextBin, 'dev', '-p', webPort, '-H', '0.0.0.0'], { const nextProc = spawn(process.execPath, [nextBin, 'dev', '-p', webPort, '-H', '0.0.0.0'], {
stdio: 'inherit', stdio: ['inherit', 'pipe', 'pipe'],
shell: false, shell: false,
env: { ...process.env }, env: childEnv,
}); });
const serverProc = spawn(process.execPath, [serverEntry], { const serverProc = spawn(process.execPath, [serverEntry], {
stdio: 'inherit', stdio: ['inherit', 'pipe', 'pipe'],
shell: false, shell: false,
env: { ...process.env }, env: childEnv,
}); });
function wireStreamCapture(proc, serviceName) {
if (proc.stdout) {
proc.stdout.on('data', (chunk) => {
process.stdout.write(chunk);
appendRuntimeLogLine(serviceName, 'stdout', chunk.toString('utf8'));
});
}
if (proc.stderr) {
proc.stderr.on('data', (chunk) => {
process.stderr.write(chunk);
appendRuntimeLogLine(serviceName, 'stderr', chunk.toString('utf8'));
});
}
}
wireStreamCapture(nextProc, 'web');
wireStreamCapture(serverProc, 'ingestion');
let shuttingDown = false; let shuttingDown = false;
let exitedChildren = 0; let exitedChildren = 0;
let nextExited = false; let nextExited = false;
+28 -4
View File
@@ -7,28 +7,52 @@
const path = require('node:path'); const path = require('node:path');
const { spawn } = require('node:child_process'); const { spawn } = require('node:child_process');
const { appendRuntimeLogLine } = require('../server/runtimeLogCapture');
const webPort = String(process.env.PORT || '3000'); const webPort = String(process.env.PORT || '3000');
const nextBin = require.resolve('next/dist/bin/next'); const nextBin = require.resolve('next/dist/bin/next');
const serverEntry = path.resolve(__dirname, '..', 'server', 'index.js'); const serverEntry = path.resolve(__dirname, '..', 'server', 'index.js');
const GRACEFUL_SHUTDOWN_TIMEOUT_MS = 7000; const GRACEFUL_SHUTDOWN_TIMEOUT_MS = 7000;
const childEnv = {
...process.env,
DATASTACK_LOG_CAPTURED_BY_RUNNER: '1',
};
const webProc = spawn( const webProc = spawn(
process.execPath, process.execPath,
[nextBin, 'start', '-p', webPort, '-H', '0.0.0.0'], [nextBin, 'start', '-p', webPort, '-H', '0.0.0.0'],
{ {
stdio: 'inherit', stdio: ['inherit', 'pipe', 'pipe'],
shell: false, shell: false,
env: { ...process.env }, env: childEnv,
}, },
); );
const serverProc = spawn(process.execPath, [serverEntry], { const serverProc = spawn(process.execPath, [serverEntry], {
stdio: 'inherit', stdio: ['inherit', 'pipe', 'pipe'],
shell: false, shell: false,
env: { ...process.env }, env: childEnv,
}); });
function wireStreamCapture(proc, serviceName) {
if (proc.stdout) {
proc.stdout.on('data', (chunk) => {
process.stdout.write(chunk);
appendRuntimeLogLine(serviceName, 'stdout', chunk.toString('utf8'));
});
}
if (proc.stderr) {
proc.stderr.on('data', (chunk) => {
process.stderr.write(chunk);
appendRuntimeLogLine(serviceName, 'stderr', chunk.toString('utf8'));
});
}
}
wireStreamCapture(webProc, 'web');
wireStreamCapture(serverProc, 'ingestion');
let shuttingDown = false; let shuttingDown = false;
let exitedChildren = 0; let exitedChildren = 0;
let webExited = false; let webExited = false;
+19
View File
@@ -14,6 +14,7 @@ const {
syncInMemoryAdminSessionState, syncInMemoryAdminSessionState,
clearInMemoryAdminSessionStateForTests, clearInMemoryAdminSessionStateForTests,
} = require('./filesManager'); } = require('./filesManager');
const { installRuntimeLogHook } = require('./runtimeLogCapture');
function parseAllowedCorsOrigins(raw = process.env.DATASTACK_CORS_ORIGINS) { function parseAllowedCorsOrigins(raw = process.env.DATASTACK_CORS_ORIGINS) {
if (typeof raw !== 'string') return null; if (typeof raw !== 'string') return null;
@@ -1735,6 +1736,21 @@ async function ensureDatabaseSchema() {
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
) )
`); `);
await pool.query(`
CREATE TABLE IF NOT EXISTS mcp_settings (
id TEXT PRIMARY KEY DEFAULT 'default',
enabled BOOLEAN NOT NULL DEFAULT false,
token_hash TEXT NOT NULL DEFAULT '',
token_enc TEXT NOT NULL DEFAULT '',
token_preview TEXT NOT NULL DEFAULT '',
allowed_ips TEXT NOT NULL DEFAULT '',
permissions_json TEXT NOT NULL DEFAULT '{}',
coolify_url TEXT NOT NULL DEFAULT '',
coolify_token_enc TEXT NOT NULL DEFAULT '',
coolify_app_uuid TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
)
`);
await pool.query(` await pool.query(`
ALTER TABLE client_sdks ALTER TABLE client_sdks
ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100, ADD COLUMN IF NOT EXISTS rate_limit INTEGER NOT NULL DEFAULT 100,
@@ -2393,6 +2409,9 @@ async function gracefulShutdown(signal, options = {}) {
} }
async function startIngestionServer() { async function startIngestionServer() {
if (process.env.DATASTACK_DISABLE_LISTEN !== '1') {
installRuntimeLogHook('ingestion');
}
if (dbPool) { if (dbPool) {
await Promise.race([ await Promise.race([
syncSettingsFromDatabase().catch(() => false), syncSettingsFromDatabase().catch(() => false),
+371
View File
@@ -0,0 +1,371 @@
const fs = require('node:fs');
const path = require('node:path');
const MAX_LOG_FILE_BYTES = 2 * 1024 * 1024; // 2 MB per service file (+ .1 backup)
const DEFAULT_TAIL_LINES = 120;
const MAX_TAIL_LINES = 500;
const MAX_SCAN_TAIL_BYTES = 512 * 1024; // Bound in-memory file slice reads
const SECRET_SCRUB_PATTERNS = [
[/\b(Bearer\s+)[A-Za-z0-9\-._~+/]+=*/gi, '$1[REDACTED]'],
[/\b(dsmcp_)[A-Za-z0-9_-]{8,}\b/gi, '$1[REDACTED]'],
[/\b(eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.)[A-Za-z0-9_-]+\b/g, '$1[REDACTED]'],
[/("(?:token|secret|password|jwtSecret|s3SecretAccessKey|webhookSecret|prometheusMetricsToken|coolifyToken|mcpToken|x-datastack-internal-secret)"\s*:\s*")([^"]+)(")/gi, '$1[REDACTED]$3'],
];
function scrubSensitiveLogText(rawText) {
let clean = String(rawText ?? '');
for (const [pattern, replacement] of SECRET_SCRUB_PATTERNS) {
clean = clean.replace(pattern, replacement);
}
return clean;
}
function resolveRuntimeLogsDir(env = process.env) {
const explicitDir = typeof env.DATASTACK_RUNTIME_LOGS_DIR === 'string'
? env.DATASTACK_RUNTIME_LOGS_DIR.trim()
: '';
if (explicitDir) {
return path.resolve(explicitDir);
}
const storageDir = typeof env.DATASTACK_STORAGE_DIR === 'string'
? env.DATASTACK_STORAGE_DIR.trim()
: '';
if (storageDir) {
const resolved = path.resolve(storageDir);
if (path.basename(resolved).toLowerCase() === 'data') {
return path.join(path.dirname(resolved), '.runtime-logs');
}
return path.join(resolved, '.runtime-logs');
}
return path.resolve(process.cwd(), 'storage', '.runtime-logs');
}
function normalizeServiceName(service) {
const s = String(service || '').trim().toLowerCase();
if (s === 'web' || s === 'nextjs') return 'web';
if (s === 'ingestion' || s === 'server' || s === 'socket') return 'ingestion';
return 'web';
}
function getServiceLogFilePath(service, logsDir = resolveRuntimeLogsDir()) {
const safeService = normalizeServiceName(service);
return path.join(logsDir, `${safeService}.log`);
}
function rotateLogFileIfNeeded(filePath) {
try {
const stat = fs.statSync(filePath);
if (stat.size >= MAX_LOG_FILE_BYTES) {
const backupPath = `${filePath}.1`;
try {
if (fs.existsSync(backupPath)) {
fs.unlinkSync(backupPath);
}
} catch {
// ignore backup cleanup error
}
fs.renameSync(filePath, backupPath);
}
} catch {
// file does not exist yet
}
}
function appendRuntimeLogLine(service, streamName, rawChunk, options = {}) {
const logsDir = options.logsDir || resolveRuntimeLogsDir(options.env);
const safeService = normalizeServiceName(service);
const text = scrubSensitiveLogText(rawChunk);
if (!text) return 0;
const lines = text.split(/\r?\n/).filter((line) => line.trim().length > 0);
if (lines.length === 0) return 0;
try {
fs.mkdirSync(logsDir, { recursive: true });
const filePath = getServiceLogFilePath(safeService, logsDir);
rotateLogFileIfNeeded(filePath);
const level = streamName === 'stderr' ? 'error' : 'info';
const ts = new Date().toISOString();
const formatted =
lines
.map((line) => JSON.stringify({ ts, service: safeService, level, message: line }))
.join('\n') + '\n';
fs.appendFileSync(filePath, formatted, 'utf8');
return lines.length;
} catch {
return 0;
}
}
function installRuntimeLogHook(service, options = {}) {
const safeService = normalizeServiceName(service);
if (process.env.DATASTACK_LOG_CAPTURED_BY_RUNNER === '1' && !options.force) {
return { installed: false, reason: 'captured_by_runner' };
}
const guardKey = `__dataStackRuntimeLogHook_${safeService}`;
if (globalThis[guardKey]) {
return { installed: false, reason: 'already_installed' };
}
globalThis[guardKey] = true;
let isWriting = false;
const wrapStream = (stream, streamName) => {
const originalWrite = stream.write.bind(stream);
stream.write = function patchedWrite(chunk, encoding, callback) {
const result = originalWrite(chunk, encoding, callback);
if (!isWriting) {
isWriting = true;
try {
const raw =
typeof chunk === 'string'
? chunk
: Buffer.isBuffer(chunk)
? chunk.toString(typeof encoding === 'string' ? encoding : 'utf8')
: String(chunk ?? '');
appendRuntimeLogLine(safeService, streamName, raw, options);
} catch {
// Never interrupt stdout/stderr
} finally {
isWriting = false;
}
}
return result;
};
};
wrapStream(process.stdout, 'stdout');
wrapStream(process.stderr, 'stderr');
return { installed: true, service: safeService };
}
function readTailLinesFromFile(filePath, maxBytes = MAX_SCAN_TAIL_BYTES) {
if (!fs.existsSync(filePath)) return [];
const stat = fs.statSync(filePath);
if (!stat.isFile() || stat.size === 0) return [];
const readSize = Math.min(stat.size, maxBytes);
const startPos = Math.max(0, stat.size - readSize);
const fd = fs.openSync(filePath, 'r');
try {
const buf = Buffer.allocUnsafe(readSize);
const bytesRead = fs.readSync(fd, buf, 0, readSize, startPos);
let content = buf.subarray(0, bytesRead).toString('utf8');
if (startPos > 0) {
const firstNewline = content.indexOf('\n');
content = firstNewline >= 0 ? content.slice(firstNewline + 1) : '';
}
return content.split(/\r?\n/).filter(Boolean);
} finally {
fs.closeSync(fd);
}
}
function parseStoredLogLine(rawLine, fallbackService) {
try {
const parsed = JSON.parse(rawLine);
if (parsed && typeof parsed === 'object' && typeof parsed.message === 'string') {
return {
ts: typeof parsed.ts === 'string' ? parsed.ts : '',
service: typeof parsed.service === 'string' ? parsed.service : fallbackService,
level: typeof parsed.level === 'string' ? parsed.level : 'info',
message: parsed.message,
};
}
} catch {
// Plain-text fallback
}
return {
ts: '',
service: fallbackService,
level: 'info',
message: rawLine,
};
}
function readRuntimeLogs(options = {}) {
const logsDir = options.logsDir || resolveRuntimeLogsDir(options.env);
const serviceFilter = String(options.service || 'all').trim().toLowerCase();
const requestedLines = Number(options.lines);
const limit = Number.isFinite(requestedLines)
? Math.min(MAX_TAIL_LINES, Math.max(1, Math.floor(requestedLines)))
: DEFAULT_TAIL_LINES;
const search = typeof options.search === 'string' ? options.search.trim().toLowerCase() : '';
const levelFilter = typeof options.level === 'string' ? options.level.trim().toLowerCase() : '';
const services =
serviceFilter === 'web'
? ['web']
: serviceFilter === 'ingestion'
? ['ingestion']
: ['web', 'ingestion'];
const collected = [];
const filesChecked = [];
for (const svc of services) {
const mainFile = getServiceLogFilePath(svc, logsDir);
const backupFile = `${mainFile}.1`;
for (const candidate of [backupFile, mainFile]) {
if (fs.existsSync(candidate)) {
filesChecked.push(path.basename(candidate));
const rawLines = readTailLinesFromFile(candidate);
for (const rawLine of rawLines) {
const entry = parseStoredLogLine(rawLine, svc);
if (levelFilter && levelFilter !== 'all' && entry.level !== levelFilter) {
continue;
}
if (
search &&
!entry.message.toLowerCase().includes(search) &&
!entry.service.toLowerCase().includes(search)
) {
continue;
}
collected.push(entry);
}
}
}
}
if (filesChecked.length === 0) {
return {
entries: [],
totalMatched: 0,
filesChecked: [],
logsDir,
exists: false,
code: 'LOG_CAPTURE_UNAVAILABLE',
warning: 'No runtime logs captured yet in .runtime-logs/. Start services or emit stdout logs first.',
};
}
collected.sort((a, b) => {
if (a.ts && b.ts && a.ts !== b.ts) {
return a.ts.localeCompare(b.ts);
}
return 0;
});
const entries = collected.slice(-limit);
return {
entries,
totalMatched: collected.length,
filesChecked,
logsDir,
exists: true,
};
}
function readBuildLogs(options = {}) {
const logsDir = options.logsDir || resolveRuntimeLogsDir(options.env);
const cwd = options.cwd || process.cwd();
const requestedLines = Number(options.lines);
const limit = Number.isFinite(requestedLines)
? Math.min(MAX_TAIL_LINES, Math.max(1, Math.floor(requestedLines)))
: 200;
const search = typeof options.search === 'string' ? options.search.trim().toLowerCase() : '';
const metaCandidates = [
path.join(logsDir, 'last-build-meta.json'),
path.join(cwd, 'build-meta.json'),
path.join(cwd, '.next', 'standalone', 'build-meta.json'),
];
const logCandidates = [
path.join(logsDir, 'last-build.log'),
path.join(cwd, 'build.log'),
path.join(cwd, '.next', 'standalone', 'build.log'),
];
let meta = null;
for (const metaPath of metaCandidates) {
try {
if (fs.existsSync(metaPath)) {
meta = JSON.parse(fs.readFileSync(metaPath, 'utf8'));
break;
}
} catch {
// try next candidate
}
}
let logPathUsed = null;
let allLines = [];
for (const logPath of logCandidates) {
try {
if (fs.existsSync(logPath)) {
logPathUsed = path.basename(logPath);
allLines = readTailLinesFromFile(logPath);
break;
}
} catch {
// try next candidate
}
}
if (!meta && !logPathUsed) {
return {
meta: null,
lines: [],
totalLines: 0,
exists: false,
code: 'LOG_CAPTURE_UNAVAILABLE',
warning: 'No captured build log or build-meta.json found yet. Run npm run build first.',
};
}
const filtered = search
? allLines.filter((line) => line.toLowerCase().includes(search))
: allLines;
return {
meta,
logFile: logPathUsed,
lines: filtered.slice(-limit),
totalLines: filtered.length,
exists: true,
};
}
function syncStandaloneBuildArtifactsToLogsDir(options = {}) {
const logsDir = options.logsDir || resolveRuntimeLogsDir(options.env);
const cwd = options.cwd || process.cwd();
const buildLogSrc = [
path.join(cwd, 'build.log'),
path.join(cwd, '.next', 'standalone', 'build.log'),
].find((candidate) => fs.existsSync(candidate));
const buildMetaSrc = [
path.join(cwd, 'build-meta.json'),
path.join(cwd, '.next', 'standalone', 'build-meta.json'),
].find((candidate) => fs.existsSync(candidate));
if (!buildLogSrc && !buildMetaSrc) {
return { synced: false };
}
try {
fs.mkdirSync(logsDir, { recursive: true });
if (buildLogSrc) {
fs.copyFileSync(buildLogSrc, path.join(logsDir, 'last-build.log'));
}
if (buildMetaSrc) {
fs.copyFileSync(buildMetaSrc, path.join(logsDir, 'last-build-meta.json'));
}
return { synced: true, logsDir };
} catch {
return { synced: false };
}
}
module.exports = {
MAX_LOG_FILE_BYTES,
scrubSensitiveLogText,
resolveRuntimeLogsDir,
getServiceLogFilePath,
appendRuntimeLogLine,
installRuntimeLogHook,
readRuntimeLogs,
readBuildLogs,
syncStandaloneBuildArtifactsToLogsDir,
};
@@ -0,0 +1,11 @@
{
"appName": "datastack",
"appVersion": "0.1.0",
"nodeVersion": "v22.23.2",
"nextVersion": "^15.4.9",
"startedAt": "2026-11-21T16:53:22.930Z",
"finishedAt": "2026-11-21T16:54:54.446Z",
"durationMs": 91516,
"exitCode": 0,
"status": "succeeded"
}
+3 -3
View File
@@ -444,17 +444,17 @@ test("DEPLOY-BUG-02, 03 & 04: Port 4000 memoizes schema DDLs, hydrates rotated J
assert.equal(failedSync, false); assert.equal(failedSync, false);
assert.equal(ddlCount, 1); assert.equal(ddlCount, 1);
// 2. Recovery runs 10 DDLs once (7 CREATE TABLE + 3 ALTER TABLE) and hydrates rotated jwt_secret; repeated calls do zero extra DDLs // 2. Recovery runs 11 DDLs once (8 CREATE TABLE + 3 ALTER TABLE) and hydrates rotated jwt_secret; repeated calls do zero extra DDLs
failDdl = false; failDdl = false;
ddlCount = 0; ddlCount = 0;
assert.equal(await ingestionServer.syncSettingsFromDatabase(), true); assert.equal(await ingestionServer.syncSettingsFromDatabase(), true);
assert.equal(ddlCount, 10); assert.equal(ddlCount, 11);
assert.equal(selectCount, 1); assert.equal(selectCount, 1);
assert.equal(ingestionServer.runtimeConfig.jwtSecret, rotatedSecret); assert.equal(ingestionServer.runtimeConfig.jwtSecret, rotatedSecret);
assert.equal(await ingestionServer.syncSettingsFromDatabase(), true); assert.equal(await ingestionServer.syncSettingsFromDatabase(), true);
assert.equal(await ingestionServer.syncSettingsFromDatabase(), true); assert.equal(await ingestionServer.syncSettingsFromDatabase(), true);
assert.equal(ddlCount, 10, "Repeated 3s sync ticks must NOT re-run CREATE/ALTER DDL statements"); assert.equal(ddlCount, 11, "Repeated 3s sync ticks must NOT re-run CREATE/ALTER DDL statements");
assert.equal(selectCount, 3); assert.equal(selectCount, 3);
// 3. Graceful shutdown flushes buffered events, quits Redis, and closes PostgreSQL pool cleanly // 3. Graceful shutdown flushes buffered events, quits Redis, and closes PostgreSQL pool cleanly
+3 -2
View File
@@ -357,7 +357,7 @@ test("R4-Phase-01 [AUDIT-01 / ERR-R4-01]: API routes (/api/settings, /api/settin
} }
}); });
test("R4-Phase-01 [AUDIT-03 / ERR-R4-02]: Port 4000 ensureDatabaseSchema creates all 7 tables including admin_team_members and admin_audit_logs", async () => { test("R4-Phase-01 [AUDIT-03 / ERR-R4-02]: Port 4000 ensureDatabaseSchema creates all 8 tables including admin_team_members, admin_audit_logs, and mcp_settings", async () => {
const executedSql: string[] = []; const executedSql: string[] = [];
const mockPool = { const mockPool = {
async query(sql: string) { async query(sql: string) {
@@ -375,7 +375,7 @@ test("R4-Phase-01 [AUDIT-03 / ERR-R4-02]: Port 4000 ensureDatabaseSchema creates
assert.equal(synced, true); assert.equal(synced, true);
const createStatements = executedSql.filter((s) => /CREATE TABLE IF NOT EXISTS/i.test(s)); const createStatements = executedSql.filter((s) => /CREATE TABLE IF NOT EXISTS/i.test(s));
assert.equal(createStatements.length, 7, "Port 4000 must create all 7 tables"); assert.equal(createStatements.length, 8, "Port 4000 must create all 8 tables");
const joined = createStatements.join("\n"); const joined = createStatements.join("\n");
for (const tableName of [ for (const tableName of [
"client_sdks", "client_sdks",
@@ -385,6 +385,7 @@ test("R4-Phase-01 [AUDIT-03 / ERR-R4-02]: Port 4000 ensureDatabaseSchema creates
"admin_login_attempts", "admin_login_attempts",
"admin_team_members", "admin_team_members",
"admin_audit_logs", "admin_audit_logs",
"mcp_settings",
]) { ]) {
assert.ok( assert.ok(
joined.includes(tableName), joined.includes(tableName),
@@ -0,0 +1,516 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import http from "node:http";
import { createRequire } from "node:module";
import type { AddressInfo } from "node:net";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import { NextRequest } from "next/server";
process.env.DATASTACK_DISABLE_LISTEN = "1";
import { POST as internalMcpPost } from "../app/api/internal/mcp/route";
import {
clearAuditLogsForTests,
listAdminAuditLogsWithStatusAsync,
recordAdminAuditLogAsync,
} from "../core/auth/auditLog";
import { CORE_SCHEMA_TABLE_DDLS } from "../core/Database/database";
import {
appendRuntimeLogLine,
readBuildLogs,
readRuntimeLogs,
scrubSensitiveLogText,
syncStandaloneBuildArtifactsToLogsDir,
} from "../core/logs/runtimeLogCapture";
import {
MCP_PERMISSION_DEFINITIONS,
McpSettingsManager,
generateMcpBearerToken,
hashMcpToken,
verifyMcpBearerTokenHash,
} from "../core/mcp/mcpSettings";
import { SDKManager } from "../core/sdk/sdkManager";
import { getInternalIngestionHeaders } from "../core/Security/security";
import { SSRFBlockedError } from "../core/Security/ssrfGuard";
import { CSVStorageManager } from "../core/Storage/csvStorage";
import { middleware } from "../middleware";
const require = createRequire(import.meta.url);
const cjsRuntimeLogCapture = require("../server/runtimeLogCapture.js") as {
MAX_LOG_FILE_BYTES: number;
scrubSensitiveLogText: (raw: string) => string;
appendRuntimeLogLine: (
service: string,
streamName: "stdout" | "stderr",
rawChunk: string,
options?: { logsDir?: string; env?: NodeJS.ProcessEnv },
) => number;
readRuntimeLogs: (options?: {
logsDir?: string;
service?: string;
lines?: number;
search?: string;
level?: string;
}) => {
entries: Array<{ ts: string; service: string; level: string; message: string }>;
totalMatched: number;
exists: boolean;
code?: string;
warning?: string;
};
readBuildLogs: (options?: {
logsDir?: string;
cwd?: string;
lines?: number;
search?: string;
}) => {
meta: Record<string, unknown> | null;
lines: string[];
totalLines: number;
exists: boolean;
code?: string;
warning?: string;
};
};
const ingestionServer = require("../server/index.js") as {
setDbPoolForTests: (pool: unknown) => void;
syncSettingsFromDatabase: () => Promise<boolean>;
};
test("R5-Phase-01 [MCP-AUDIT-02 / ERR-R5-02]: Runtime log capture scrubs secrets, rotates at 2MB, filters/tails safely, and handles missing dirs gracefully", async () => {
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-r5-logs-"));
const logsDir = path.join(tmpDir, ".runtime-logs");
try {
// 1. Missing directory returns LOG_CAPTURE_UNAVAILABLE cleanly without throwing
const missingRead = readRuntimeLogs({ logsDir });
assert.equal(missingRead.exists, false);
assert.equal(missingRead.code, "LOG_CAPTURE_UNAVAILABLE");
assert.equal(missingRead.entries.length, 0);
const missingBuild = readBuildLogs({ logsDir, cwd: tmpDir });
assert.equal(missingBuild.exists, false);
assert.equal(missingBuild.code, "LOG_CAPTURE_UNAVAILABLE");
// 2. Secret scrubbing across TS + CJS implementations
const rawLine =
'Auth Bearer secret_jwt_token_123 dsmcp_abcdef1234567890 eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.SuperSecretSignature {"coolifyToken":"cf_live_999","mcpToken":"dsmcp_xyz987654321"}';
const scrubbedTs = scrubSensitiveLogText(rawLine);
const scrubbedCjs = cjsRuntimeLogCapture.scrubSensitiveLogText(rawLine);
assert.equal(scrubbedTs, scrubbedCjs);
assert.ok(!scrubbedTs.includes("secret_jwt_token_123"));
assert.ok(!scrubbedTs.includes("abcdef1234567890"));
assert.ok(!scrubbedTs.includes("SuperSecretSignature"));
assert.ok(!scrubbedTs.includes("cf_live_999"));
assert.ok(scrubbedTs.includes("[REDACTED]"));
// 3. Multi-service append + read filtering
const writtenWeb = appendRuntimeLogLine("web", "stdout", `${rawLine}\n[web] ready on port 3000\n`, {
logsDir,
});
const writtenIng = cjsRuntimeLogCapture.appendRuntimeLogLine(
"ingestion",
"stderr",
"[ingestion] redis reconnect warning\n",
{ logsDir },
);
assert.equal(writtenWeb, 2);
assert.equal(writtenIng, 1);
const allLogs = readRuntimeLogs({ logsDir, service: "all", lines: 50 });
assert.equal(allLogs.exists, true);
assert.equal(allLogs.totalMatched, 3);
assert.equal(allLogs.entries.length, 3);
assert.ok(allLogs.entries.every((e) => !e.message.includes("secret_jwt_token_123")));
const errOnly = cjsRuntimeLogCapture.readRuntimeLogs({
logsDir,
service: "ingestion",
level: "error",
});
assert.equal(errOnly.exists, true);
assert.equal(errOnly.entries.length, 1);
assert.equal(errOnly.entries[0]?.service, "ingestion");
assert.equal(errOnly.entries[0]?.level, "error");
// 4. Rotation to .1 backup when file size >= 2MB
const webLogPath = path.join(logsDir, "web.log");
await fs.writeFile(webLogPath, "x".repeat(cjsRuntimeLogCapture.MAX_LOG_FILE_BYTES + 128), "utf8");
appendRuntimeLogLine("web", "stdout", "[web] post-rotation line\n", { logsDir });
const backupStat = await fs.stat(`${webLogPath}.1`);
assert.ok(backupStat.size >= cjsRuntimeLogCapture.MAX_LOG_FILE_BYTES);
const afterRotate = readRuntimeLogs({ logsDir, service: "web", search: "post-rotation" });
assert.equal(afterRotate.entries.length, 1);
assert.match(afterRotate.entries[0]?.message || "", /post-rotation/);
} finally {
await fs.rm(tmpDir, { recursive: true, force: true });
}
});
test("R5-Phase-01 [MCP-AUDIT-03]: Standalone build.log & build-meta.json sync to .runtime-logs and Dockerfile.web/capture-build-log stay intact", async () => {
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-r5-build-"));
const standaloneDir = path.join(tmpDir, ".next", "standalone");
const logsDir = path.join(tmpDir, "storage", ".runtime-logs");
try {
await fs.mkdir(standaloneDir, { recursive: true });
await fs.writeFile(
path.join(standaloneDir, "build.log"),
"Creating an optimized production build ...\nCompiled successfully in 12.4s\n",
"utf8",
);
await fs.writeFile(
path.join(standaloneDir, "build-meta.json"),
JSON.stringify({ appName: "datastack", status: "succeeded", exitCode: 0 }),
"utf8",
);
const synced = syncStandaloneBuildArtifactsToLogsDir({ cwd: tmpDir, logsDir });
assert.equal(synced.synced, true);
const buildRead = readBuildLogs({ logsDir, cwd: tmpDir, search: "compiled" });
assert.equal(buildRead.exists, true);
assert.equal(buildRead.meta?.status, "succeeded");
assert.equal(buildRead.lines.length, 1);
assert.match(buildRead.lines[0] || "", /Compiled successfully/);
const pkgJson = JSON.parse(await fs.readFile(path.resolve(process.cwd(), "package.json"), "utf8")) as {
scripts?: Record<string, string>;
};
assert.equal(pkgJson.scripts?.build, "node scripts/capture-build-log.js");
const dockerfileWeb = await fs.readFile(path.resolve(process.cwd(), "docker/Dockerfile.web"), "utf8");
assert.match(dockerfileWeb, /\.runtime-logs/);
} finally {
await fs.rm(tmpDir, { recursive: true, force: true });
}
});
test("R5-Phase-01 [MCP-AUDIT-05]: McpSettingsManager enforces OFF default, 17 granular toggles, token hash/vault, CIDR/SSRF validation, DB roundtrip & Port 4000 sync", async () => {
McpSettingsManager.resetForTests();
const prevDbUrl = process.env.DATABASE_URL;
const prevCorePool = (globalThis as Record<string, unknown>).__dataStackCorePool;
const prevCoreSchema = (globalThis as Record<string, unknown>).__dataStackCoreSchema;
const prevIngestionUrl = process.env.INGESTION_INTERNAL_URL;
let persistedRow: Record<string, unknown> | null = null;
const syncPayloads: Array<Record<string, unknown>> = [];
const syncServer = http.createServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", (c) => chunks.push(Buffer.from(c)));
req.on("end", () => {
if (req.url === "/internal/mcp-sync" && req.method === "POST") {
syncPayloads.push(JSON.parse(Buffer.concat(chunks).toString("utf8")) as Record<string, unknown>);
res.statusCode = 200;
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify({ success: true }));
return;
}
res.statusCode = 404;
res.end();
});
});
await new Promise<void>((resolve) => syncServer.listen(0, "127.0.0.1", () => resolve()));
const syncPort = (syncServer.address() as AddressInfo).port;
try {
process.env.INGESTION_INTERNAL_URL = `http://127.0.0.1:${syncPort}`;
assert.equal(MCP_PERMISSION_DEFINITIONS.length, 17);
const mockPool = {
async query(sql: string, params?: unknown[]) {
if (/INSERT INTO mcp_settings/i.test(sql) && params) {
persistedRow = {
enabled: params[0],
token_hash: params[1],
token_enc: params[2],
token_preview: params[3],
allowed_ips: params[4],
permissions_json: params[5],
coolify_url: params[6],
coolify_token_enc: params[7],
coolify_app_uuid: params[8],
updated_at: params[9],
};
return { rowCount: 1, rows: [] };
}
if (/FROM mcp_settings/i.test(sql)) {
return persistedRow
? { rowCount: 1, rows: [persistedRow] }
: { rowCount: 0, rows: [] };
}
return { rowCount: 0, rows: [] };
},
};
process.env.DATABASE_URL = "postgres://datastack:test@127.0.0.1:5432/datastack";
(globalThis as Record<string, unknown>).__dataStackCorePool = mockPool;
(globalThis as Record<string, unknown>).__dataStackCoreSchema = Promise.resolve();
// 1. Default safe-by-default OFF state
const initial = await McpSettingsManager.getPublicSettingsAsync();
assert.equal(initial.enabled, false);
assert.equal(initial.hasToken, false);
assert.equal(initial.permissions.clients_create, false);
assert.equal(initial.permissions.metrics_health_read, true);
// 2. Enabling auto-mints Bearer token + encrypts Coolify token + syncs Port 4000
const enabledRes = await McpSettingsManager.updateSettingsAsync({
enabled: true,
allowedIps: "203.0.113.0/24",
permissions: { clients_create: true, files_edit: true },
coolifyUrl: "https://coolify.example.com",
coolifyToken: "coolify_pat_secret_123",
coolifyAppUuid: "app-uuid-01",
});
assert.equal(enabledRes.settings.enabled, true);
assert.equal(enabledRes.settings.hasToken, true);
assert.ok(enabledRes.issuedToken?.startsWith("dsmcp_"));
assert.equal(enabledRes.issuedToken?.length, 70);
assert.equal(enabledRes.settings.permissions.clients_create, true);
assert.equal(enabledRes.settings.permissions.files_edit, true);
assert.equal(enabledRes.settings.hasCoolifyToken, true);
assert.equal(enabledRes.bridgeSync.ok, true);
assert.equal(syncPayloads.length, 1);
assert.equal(syncPayloads[0]?.coolifyToken, "coolify_pat_secret_123");
// 3. SHA-256 digest & timing-safe verification
const rawToken = generateMcpBearerToken();
const digest = hashMcpToken(rawToken);
assert.equal(verifyMcpBearerTokenHash(rawToken, digest), true);
assert.equal(verifyMcpBearerTokenHash(`${rawToken}x`, digest), false);
// 4. Re-hydrate from mock PG row cleanly
McpSettingsManager.resetForTests();
const rehydrated = await McpSettingsManager.getPublicSettingsAsync();
assert.equal(rehydrated.enabled, true);
assert.equal(rehydrated.allowedIps, "203.0.113.0/24");
assert.equal(rehydrated.permissions.clients_create, true);
assert.equal(rehydrated.hasCoolifyToken, true);
// 5. Reject invalid CIDR & SSRF target for coolifyUrl
await assert.rejects(
() => McpSettingsManager.updateSettingsAsync({ allowedIps: "not-a-cidr" }),
/Invalid/i,
);
await assert.rejects(
() =>
McpSettingsManager.updateSettingsAsync({
coolifyUrl: "http://169.254.169.254/latest/meta-data",
}),
(err: unknown) => err instanceof SSRFBlockedError,
);
// 6. Revoke forces enabled: false and clears token
const revoked = await McpSettingsManager.updateSettingsAsync({ revokeToken: true });
assert.equal(revoked.settings.enabled, false);
assert.equal(revoked.settings.hasToken, false);
} finally {
McpSettingsManager.resetForTests();
if (prevDbUrl === undefined) delete process.env.DATABASE_URL;
else process.env.DATABASE_URL = prevDbUrl;
(globalThis as Record<string, unknown>).__dataStackCorePool = prevCorePool;
(globalThis as Record<string, unknown>).__dataStackCoreSchema = prevCoreSchema;
if (prevIngestionUrl === undefined) delete process.env.INGESTION_INTERNAL_URL;
else process.env.INGESTION_INTERNAL_URL = prevIngestionUrl;
await new Promise<void>((resolve) => syncServer.close(() => resolve()));
}
});
test("R5-Phase-01 [MCP-AUDIT-05 / MCP-AUDIT-07]: Port 3000 & Port 4000 share 8-table mcp_settings DDL parity and auditLog records/redacts MCP actions", async () => {
// 1. Port 3000 DDL contains mcp_settings as 8th table
assert.equal(CORE_SCHEMA_TABLE_DDLS.length, 8);
assert.ok(CORE_SCHEMA_TABLE_DDLS.some((ddl) => /CREATE TABLE IF NOT EXISTS mcp_settings/i.test(ddl)));
// 2. Port 4000 ensureDatabaseSchema creates mcp_settings as well
const executedSql: string[] = [];
try {
ingestionServer.setDbPoolForTests({
async query(sql: string) {
executedSql.push(sql);
return { rowCount: 0, rows: [] };
},
});
const synced = await ingestionServer.syncSettingsFromDatabase();
assert.equal(synced, true);
const createStatements = executedSql.filter((s) => /CREATE TABLE IF NOT EXISTS/i.test(s));
assert.equal(createStatements.length, 8);
assert.ok(createStatements.some((s) => /\bmcp_settings\b/.test(s)));
} finally {
ingestionServer.setDbPoolForTests(null);
}
// 3. MCP_* audit actions & secret redaction
clearAuditLogsForTests();
try {
await recordAdminAuditLogAsync({
actorEmail: "owner@datastack.dev",
actorRole: "owner",
action: "MCP_SETTINGS_UPDATED",
target: "mcp_settings",
metadata: {
mcpToken: "dsmcp_plain_should_redact",
mcpBearerToken: "dsmcp_bearer_should_redact",
issuedToken: "dsmcp_issued_should_redact",
tokenHash: "abc123",
tokenEnc: "v1:enc",
coolifyToken: "cf_secret",
coolifyTokenEnc: "v1:cf_enc",
safeFlag: true,
},
});
await recordAdminAuditLogAsync({
actorEmail: "mcp-agent",
actorRole: "mcp",
action: "MCP_TOOL_DENIED",
target: "clients_create",
metadata: { permission: "clients_create" },
});
const list = await listAdminAuditLogsWithStatusAsync({ limit: 10 });
const settingsEvt = list.logs.find((l) => l.action === "MCP_SETTINGS_UPDATED");
const deniedEvt = list.logs.find((l) => l.action === "MCP_TOOL_DENIED");
assert.ok(settingsEvt);
assert.ok(deniedEvt);
for (const redactedKey of [
"mcpToken",
"mcpBearerToken",
"issuedToken",
"tokenHash",
"tokenEnc",
"coolifyToken",
"coolifyTokenEnc",
]) {
assert.equal(settingsEvt.metadata[redactedKey], "[REDACTED]");
}
assert.equal(settingsEvt.metadata.safeFlag, true);
} finally {
clearAuditLogsForTests();
}
});
test("R5-Phase-01 [MCP-AUDIT-04 / ERR-R5-01]: middleware permits POST /api/internal/mcp only with x-datastack-internal-secret and route enforces timing-safe auth & actions", async () => {
const prevStorageDir = process.env.DATASTACK_STORAGE_DIR;
const tmpStorage = await fs.mkdtemp(path.join(os.tmpdir(), "datastack-r5-bridge-"));
const originalBaseDir = CSVStorageManager.baseStorageDir;
try {
CSVStorageManager.baseStorageDir = path.join(tmpStorage, "data");
const internalHeaders = getInternalIngestionHeaders();
assert.ok(internalHeaders["x-datastack-internal-secret"]);
// 1. Middleware rejects POST /api/internal/mcp without internal secret (cross-origin check -> 403)
const noSecretMwReq = new NextRequest("http://localhost:3000/api/internal/mcp", {
method: "POST",
headers: { host: "localhost:3000" },
});
const noSecretMwRes = await middleware(noSecretMwReq);
assert.equal(noSecretMwRes.status, 403);
// 2. Middleware allows POST /api/internal/mcp to reach route when header is present
const withSecretMwReq = new NextRequest("http://localhost:3000/api/internal/mcp", {
method: "POST",
headers: {
host: "localhost:3000",
"x-datastack-internal-secret": internalHeaders["x-datastack-internal-secret"],
},
});
const withSecretMwRes = await middleware(withSecretMwReq);
assert.equal(withSecretMwRes.status, 200);
// 3. Route rejects wrong/missing internal secret with 401 MCP_BRIDGE_UNAUTHORIZED (ERR-R5-01)
const badSecretReq = new NextRequest("http://localhost:3000/api/internal/mcp", {
method: "POST",
headers: {
"Content-Type": "application/json",
"x-datastack-internal-secret": "wrong-secret",
},
body: JSON.stringify({ action: "metrics_get" }),
});
const badSecretRes = await internalMcpPost(badSecretReq);
assert.equal(badSecretRes.status, 401);
const badJson = (await badSecretRes.json()) as { code: string };
assert.equal(badJson.code, "MCP_BRIDGE_UNAUTHORIZED");
const callBridge = async (action: string, params: Record<string, unknown> = {}) => {
const req = new NextRequest("http://localhost:3000/api/internal/mcp", {
method: "POST",
headers: {
"Content-Type": "application/json",
"x-datastack-internal-secret": internalHeaders["x-datastack-internal-secret"],
},
body: JSON.stringify({ action, params }),
});
const res = await internalMcpPost(req);
return { status: res.status, body: (await res.json()) as Record<string, unknown> };
};
// 4. Client create / list / update / rotate / revoke / delete
const uniqueId = `mcp-test-client-${Date.now()}`;
const created = await callBridge("clients_create", {
clientId: uniqueId,
name: "MCP Test Client",
rateLimit: 250,
});
assert.equal(created.status, 201);
const createdClient = created.body.client as { clientId: string; token?: string; secret?: string };
assert.equal(createdClient.clientId, uniqueId);
assert.ok(createdClient.token);
assert.equal(createdClient.secret, undefined);
const listed = await callBridge("clients_list");
assert.equal(listed.status, 200);
const listArr = listed.body.clients as Array<{ clientId: string; token?: string }>;
const found = listArr.find((c) => c.clientId === uniqueId);
assert.ok(found);
assert.equal(found?.token, undefined);
const updated = await callBridge("clients_update", {
clientId: uniqueId,
rateLimit: 500,
});
assert.equal(updated.status, 200);
assert.equal((updated.body.client as { rateLimit: number }).rateLimit, 500);
// 5. settings_update blocks forbidden sensitive field jwtSecret
const badSettings = await callBridge("settings_update", {
jwtSecret: "attempted-overwrite-secret",
});
assert.equal(badSettings.status, 400);
assert.equal(badSettings.body.code, "VALIDATION_ERROR");
// 6. Metrics, settings_get, data_query (with stored CSV row), storage_stats
const metricsRes = await callBridge("metrics_get");
assert.equal(metricsRes.status, 200);
const settingsRes = await callBridge("settings_get");
assert.equal(settingsRes.status, 200);
const clientDataDir = path.join(CSVStorageManager.baseStorageDir, CSVStorageManager.sanitizeClientId(uniqueId));
await fs.mkdir(clientDataDir, { recursive: true });
await fs.writeFile(
path.join(clientDataDir, "2026-11-21.csv"),
`"clientId","eventId","timestamp","payload"\n"${uniqueId}","evt-mcp-csv-1","2026-11-21T10:00:00.000Z","{""ok"":true}"\n`,
"utf8",
);
const dataRes = await callBridge("data_query", { clientId: uniqueId, limit: 10 });
assert.equal(dataRes.status, 200);
assert.equal(dataRes.body.count, 1);
const statsRes = await callBridge("storage_stats");
assert.equal(statsRes.status, 200);
// Cleanup client
const delRes = await callBridge("clients_delete", { clientId: uniqueId, purgeFiles: true });
assert.equal(delRes.status, 200);
await SDKManager.deleteClientWithStatsAsync(uniqueId, { purgeFiles: true });
} finally {
CSVStorageManager.baseStorageDir = originalBaseDir;
if (prevStorageDir === undefined) delete process.env.DATASTACK_STORAGE_DIR;
else process.env.DATASTACK_STORAGE_DIR = prevStorageDir;
clearAuditLogsForTests();
await fs.rm(tmpStorage, { recursive: true, force: true });
}
});