23 KiB
DataStack — Baseline Freeze, Code Proof & Proposed Patch Index
Frozen source baseline SHA: d725841ebc828d58cbe90f517da9e600f23c4630 (2026-11-22, Asia/Dhaka).
Source verification: exact GitHub/GitLab/Gitea remote main and local HEAD matched this SHA. Local working tree clean; local origin configured to MdNurnobi-Dev/DataStack.git, and its local tracking ref was stale (725d3c5), so ahead 5 alone was NOT evidence GitHub JILLU-Games was behind. GitHub JILLU-Games read-only branch HEAD verified at d725841ebc828d58cbe90f517da9e600f23c4630. No force push, ref rewrite or upstream reset authorized.
A. Proof excerpts (verbatim short source fragments, exact baseline files/lines)
CI-02 — stale lockfile contract (P01)
tests/phase07-deployment.test.ts:648-650:
test("FINAL-DEPLOY-FIX-01: Single npm lockfile, Dockerfile .runtime-logs ownership parity, 8-table prisma sync, and cold-boot MCP bridge hydration", async () => {
assert.equal(existsSync(resolve(process.cwd(), "bun.lock")), false, "Stale bun.lock must stay removed");
git ls-files bun.lock listed the file on baseline. Patch reference: check if any build consumer references it; if none, remove exactly bun.lock while preserving package-lock.json. Re-run this specific test.
CI-04 — stale UI test expectation (P01)
tests/v2-phase01-storage-quotas.test.ts:788-804:
assert.match(clientsPageSrc, /function isClientExpired\(/);
assert.match(clientsPageSrc, /const expired = !revoked && isClientExpired\(client\.expiresAt,\s*now\)/);
app/vcon/clients/page.tsx:70-85:
const counts = useMemo(
() => summarizeUiClientCounts(clients, now),
[clients, now],
);
const computedStatus = classifyUiClientStatus(client, now);
lib/vconUiHelpers.ts:717,730,754 exports isUiClientExpired, classifyUiClientStatus, summarizeUiClientCounts.
Proposed patch shape (planning-only, not applied):
--- a/tests/v2-phase01-storage-quotas.test.ts
+++ b/tests/v2-phase01-storage-quotas.test.ts
@@ obsolete regex-based UI expectations, lines 788-804 @@
- assert.match(clientsPageSrc, /function isClientExpired\(/);
+ assert.match(clientsPageSrc, /summarizeUiClientCounts/);
+ assert.match(clientsPageSrc, /classifyUiClientStatus/);
Also replace the other 3 obsolete implementation-detail regexes with behavioral assertions against exported helper and page filtering/count data; do not just drop coverage. Exact final diff is generated only during approved P01.
CI-03 — test-process hang anchor (P01)
core/data/buffer.ts:347-352,379-391:
state.flushTimer = setInterval(() => {
void this.flush().catch(() => undefined);
}, state.batchFlushIntervalMs);
Run #45 suite exit codes 124: r3-phase04-ui-metrics-rbac, r4-phase03-sdk-deploy-auth, r5-phase01-mcp-foundation-logs, r5-phase03-mcp-ui. Timer is a suspect, not a proven sole root cause. Patch reference: instrument open handles and add narrow teardown in four named tests; core/data/buffer.ts is read-only unless new approval.
CI-05 — compiler-source proof (P01)
tsconfig.json:27:
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"]
Run #45 job 68 examples:
tests/phase01-auth.test.ts(88,17): error TS2540: Cannot assign to 'NODE_ENV' because it is a read-only property.
tests/r5-phase02-mcp-server.test.ts(510,8): error TS18046: 'deniedClientCreate.content' is of type 'unknown'.
tests/v2-phase01-storage-quotas.test.ts(762,64): error TS2554: Expected 0 arguments, but got 1.
Patch reference: exact 9 test files / line map in 001. Use correct typings/type guards and current API signatures, not TS ignore or global tsconfig exclusion.
CI-01 — Windows clean-install lockfile mismatch (P01)
Run #42 job 63:
npm error Missing: @img/sharp-win32-x64@0.35.5 from lock file
npm error Missing: @unrs/resolver-binding-win32-x64-msvc@1.13.0 from lock file
Run #45 job 71 showed success only after temporary runner npm install --package-lock-only, followed by npm ci and Node SDK build. Patch reference: package-lock.json optional/native dependencies, retain pinned package graph; verify unchanged tracked lock under CI on Linux/Windows.
CI-06 — Python runner packages (P03)
Run #45 job 67:
E: Failed to fetch ... python3.9-venv_3.9.2-1+deb11u7_amd64.deb 404 Not Found
.gitea/workflows/datastack-ci.yml Python install step uses apt-get update -qq; apt-get install -y python3 python3-venv python3-pip. Run #48 job 74 executes 14 tests PASS on Windows. Patch reference: CI job Python provisioning only.
CI-07 — Docker web layer failure (P03)
docker/Dockerfile.web:13:
COPY --from=dependencies /app/node_modules ./node_modules
Run #50 job 76:
failed to create diff tar stream: lstat .../app/node_modules/@uiw: no such file or directory
docker/Dockerfile.server produced tagged ingestion image; Docker Compose config --no-interpolate --quiet PASS. Patch reference: .gitea/workflows/datastack-docker-ci.yml runner/cache/BuildKit; Dockerfile modification conditional on independent reproduction and new approval.
CI-08 — npm advisory summary (P03)
Run #50 job 76:
24 vulnerabilities (23 moderate, 1 high)
26 vulnerabilities (25 moderate, 1 high)
No evidence of specific advisory IDs yet. Patch reference package.json/package-lock.json only after isolated npm audit --json with package/CVE/runtime attribution. Never add counts or claim 50 unique advisories.
R-01 — Redis volatile fallback ACK risk (P02, conditional)
server/index.js:165-193:
try {
bufferLen = await redis.rpush(REDIS_BUFFER_KEY, ...serialized);
redisConnected = true;
} catch {
redisConnected = false;
}
// ...
bufferQueue.push(...events);
return bufferQueue.length;
server/socket/handlers.js:583-602:
bufferQueue.push(event);
try {
await onBufferPush();
} catch (persistErr) {
// callback BUFFER_PERSIST_FAILED
}
if (callback) callback({ success: true, eventId });
Potential issue: a successful ACK does not necessarily mean durable storage after Redis failure; test first. Proposed patch is NOT predetermined.
R-02 — CSV before Redis trim (P02, conditional)
server/index.js:981-994:
const res = await writer.writeBatch(parsedBatch);
// ...
await redis.ltrim(REDIS_BUFFER_KEY, rawItems.length, -1);
Potential issue: crash after partial write or before list trim may replay buffered rows. Test with injected partial failures and compare event IDs before prescribing exactly-once guarantee.
R-03 — Health shows service liveness (P02, conditional)
server/index.js:1939-1955:
const redisBufferSize = await getRedisBufferLen();
res.json({ status: 'online', /* ... */ redisConnected, /* ... */ });
app/api/health/route.ts:95-105:
const bufferStats = await DataBufferEngine.getBufferStatsAsync();
return NextResponse.json({ status: 'online', service: 'DataStack Web API', /* ... */ });
Potential issue: liveness is not transactional readiness. Preserve existing public endpoint shape and verify separately.
B. Source-to-patch navigation
- P01:
package-lock.json;bun.lock;tests/phase07-deployment.test.ts:649;tests/v2-phase01-storage-quotas.test.ts:762,783,788-804; 4 timeout tests; 9 TS test files;.gitea/workflows/datastack-ci.yml. - P02 conditional:
server/index.js:165-193,943-994,1939-1956,server/socket/handlers.js:583-602,server/workers/csvWriter.js:265-323,app/api/health/route.ts:95-111; associated test files. Read-only until fault reproduction. - P03:
.gitea/workflows/datastack-ci.yml,datastack-python-ci.yml,datastack-docker-ci.yml,datastack-eslint-ci.yml;package.json/package-lock.jsononly justified dependency changes. Source Dockerfile edits separately approved only if needed.
C. Documentation baseline update record
This file freezes the source baseline, not the future documentation-commit SHA. After adding notes/**, record commit SHA and three remote SHAs in notes/003 and the user-facing result. Subsequent P01 implementation should inspect git diff d725841ebc828d58cbe90f517da9e600f23c4630..HEAD and permit only these notes as intervening changes; any unexpected source edits require a new review.
D. P01 Actual Patch & CI Closure Proof — 2026-11-22
Frozen source baseline remains d725841ebc828d58cbe90f517da9e600f23c4630, docs baseline d8ac96e39dad4ea3b241c1d2f16f959c5215701e. CI-verified implementation snapshot 7d8eebd3dc867e82113343403b63ce1d4ed20584 (Gitea disposable P01 branch). Permanent main closure is single squash commit containing this note; verify post-push three remote refs/content trees.
Lock and package integrity evidence (CI-01 / CI-02)
{
"node_modules/@napi-rs/wasm-runtime/node_modules/@tybys/wasm-util": {
"version": "0.10.5",
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.5.tgz",
"integrity": "sha512-dOGOPxlaNV+G046mZzN+9C0dAcQZeSKcXiVaM0TYy/LM4ttTmgSBDxYfovcYI8Em3+eAbiAYpvWLOv8+ZVRJKA==",
"optional": true,
"dependencies": {"tslib": "^2.4.0"}
}
}
The root @tybys/wasm-util@0.9.0 remains unchanged; 68 optional entries added, 1293 old entries unchanged, none removed/repinned, all added entries with registry tarball and integrity, package.json unchanged. bun.lock removed after checking no active consumers. Windows and Linux npm ci --no-audit --no-fund PASS using committed lock without temporary regeneration.
Test-only teardown correction (CI-03 / CI-05)
tests/r5-phase01-mcp-foundation-logs.test.ts local CommonJS require declaration now explicitly includes stopTimers, matching the existing runtime export; test.after releases ingestion timers and DataBufferEngine.shutdown() without any changes to runtime source:
const ingestionServer = require("../server/index.js") as {
setDbPoolForTests: (pool: unknown) => void;
syncSettingsFromDatabase: () => Promise<boolean>;
stopTimers: () => void;
};
test.after(async () => {
ingestionServer.stopTimers();
SDKManager.stopActivityFlushTimerForTests();
await DataBufferEngine.shutdown();
});
Remaining CI-05 corrections were confined to 9 test file type contracts; CI-04 replaced old isClientExpired page regex expectations with classifyUiClientStatus + summarizeUiClientCounts behavioral assertions. No TS strictness flags disabled, no global any blanket suppression, no SDK/API runtime signature changed.
Exact gate results
- Run #58: failed at lockfile nested dependency before tests; historical and superseded.
- Run #59: overall failed because exactly one test mock TypeScript declaration lacked
stopTimers, but separate job results independently verified 24/24 TypeScript suites exit0, no timeouts, Linux/Windows committed-lock clean npm installs, Node SDK compilation and Next.js standalone build + 22/22 pages all PASS. - Run #60: focused quality-only retry after exact test-type correction, strict
npx tsc --noEmitPASS 0 errors andnpm run lintPASS. - Run #48: previous Python SDK 14/14 PASS, no Python source edits or P01 rerun.
Targeted path/diff audit from prior documentation HEAD: 16 scoped code/test/CI paths plus notes 001–006; git diff --check PASS. P02 remaining R-01…R-03 are conditional reliability risks, not confirmed application bugs. P03 Docker/Python-runner/advisory issues still open. No live app, deployment, database, security defaults or user features changed.
E. P02 Concrete Code/CI Evidence — 2026-11-22
Frozen previous phase: P01 Gitea/GitHub commit 667a140c4e9e75382bfdcdbe1d3e9eb96f6e808a. P02 test-only commit bbe73d91666641743cbec6cf7203f1694094caef adds 103 lines across exactly two approved test paths. Gitea CI #63: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63. No application source modification.
R-01 ACK after Redis RPUSH failure
server/index.js:171-192:
try { bufferLen = await redis.rpush(REDIS_BUFFER_KEY, ...serialized); }
catch { redisConnected = false; }
// ... Redis failure path
bufferQueue.push(...events);
return bufferQueue.length;
server/socket/handlers.js:583-602 invokes await onBufferPush() and then calls callback({success:true,eventId}) unless an exception escapes. Test: tests/phase02-sync.test.ts, P02 R-01 diagnostic, overriding fake Redis rpush to reject; check ACK, absent Redis persistence, volatile RAM, and simulated reset. No real process exit.
R-02 replay after append-before-trim
core/data/buffer.ts:795-809 and Port-4000 server/index.js:981-993 order appendBatch()/writeBatch() before redis.ltrim(). Test: tests/phase03-storage-buffer.test.ts, P02 R-02 diagnostic; inject the first LTRIM failure only after successful append, verify still queued, retry flush and count duplicate eventId in appended rows. Web-side representative fault-injection; not a complete ingestion-crash certificate.
R-03 public status versus dependency readiness
server/index.js:1926-1957 catches Redis LLEN error, sets redisConnected=false, and returns HTTP 200 status:online. Test P02 R-03 diagnostic mocks LLEN failure and recovery. This preserves existing public shape rather than implementing a readiness endpoint. PG startup fail-closed coverage remains source-only/NOT VERIFIED.
Patch scope: modified only tests/phase02-sync.test.ts and tests/phase03-storage-buffer.test.ts (plus notes/001–006). Original local user-staged P01 work preserved. Runtime files examined read-only. No production host, Redis, DB, CSV, API, service, CI workflow, release, or security setting changed.
CI #63 observed P02 results
P02 R-01 diagnostic: Redis RPUSH failure yields positive WS ACK for volatile RAM-only event— ok 15, test suitephase02-syncexit 0. Diagnostic indicates acknowledged event loss on simulated RAM reset when Redis has no record.P02 R-03 diagnostic: Redis query failure leaves public ingestion health 200/online— ok 16, same suite exit 0, Redis flag false while liveness stays online.P02 R-02 diagnostic: post-CSV Redis trim failure replays identical event on retry— ok 18,phase03-storage-buffersuite exit 0; duplicate append after retry reproduced on representative web buffer path. Direct ingestion Port 4000 CSV worker crash coverage remains outstanding and should be requested in remediation approval before guarantee changes.- These three diagnostic tests prove findings, do not close production-readiness. See Gitea https://dev.ivib.cc/vibtools/DataStack/actions/runs/63; never call run successful until terminal conclusion is confirmed. No server-side source fix applied.
F. P02 Remediation Candidate Patch Proof — 2026-11-22
Base: Gitea/GitHub published P02 diagnostic tree fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898; GitLab distinct commit, same previous tree. Local candidate commit 44eee8877e40d1683f44502c2a0ba3d437a03b88 on isolated worktree. Initial exact delta: 5 approved source/test paths, 310 insertions and 22 deletions, no application UI, Python SDK, auth, public health route, CI or deployment edit. git diff --check and node --check on JS files PASS.
server/index.js:pushEventsToRedisOrFallback: catch RPUSH failure, keep previous RAM buffering, throw sanitizedBUFFER_PERSIST_FAILED; no positive ACK for that branch.server/socket/handlers.js:sync_data/batch_sync: negative ACK and best-effort claimed idempotency-key rollback with negative retry guard, no change to legitimate duplicate success path or existing auth/limits/error codes.server/index.js:flushBufferandcore/data/buffer.ts:flush: scoped streaming CSV/CSV.gz read-before-reappend (parsing quoted/multiline rows), exact client/date/event ID match; failures abort before Redis LTRIM. No changes toserver/workers/csvWriter.js.tests/phase02-sync.test.ts: R-01 single/batch outage and duplicate claim ACK negative, successful recovery ACK, and unchanged R-03 liveness.tests/phase03-storage-buffer.test.ts: web-side replay verifies exactly one physically stored row; new direct Port 4000 writer Redis-trim failure/recovery proof. All tests on fake Redis and disposable temp CSV paths.
Unverified: Gitea P02 implementation acceptance was unavailable due inaccessible host/connector. Local npm ci not confirmed successful. Do not invent a CI run ID or claim full PASS. Full cross-process transactional exactly-once and all Redis claim crash windows are out of proof scope and may require exact additional approval. The streaming dedupe scan changes I/O cost per flush; performance validation before production is required.
Blocked acceptance evidence — actual tool outcomes
- JS syntax: local
node --check server/index.jsandnode --check server/socket/handlers.jsPASS. - Source scope: local
git diff --checkPASS; only the five allowed P02 runtime/test paths changed before notes updates. - Gitea: connection returned
UNAVAILABLE: McpServerError: Connection failed; Git remotedev.ivib.ccunresponsive. No Gitea CI run ID exists for candidate44eee8877e40d1683f44502c2a0ba3d437a03b88. - Local Windows npm:
npm cifailedETARGET yaml@2.9.1and cleanupEPERMin the isolated worktree, not a TypeScript/test result. Dependency changes are out of P02 scope. - Therefore neither TS typecheck nor regression-suite remediation acceptance is established; keep P02 BLOCKED and do not advance P03 or alter any protected runtime/source beyond this approval.
F. P02 Final Acceptance Patch Index (current source/CI proof)
- Published diagnosis Gitea/GitHub main:
fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898; previous GitLab main same tracked tree. - Runtime candidate:
76cd972bdeea9ef07b4b38e9d8b5b5ed3312ca5b; Gitea CI #66 terminal SUCCESS across node 24/24, strict TS, lint, Next standalone and Windows Node SDK. - Exact supplemental commit:
bf525f0d4e5e4781de7fddd2f5e7d3d602a80532(3 changed files:server/socket/handlers.js,tests/phase02-sync.test.ts,tests/phase03-storage-buffer.test.ts). Compared with published diagnosis, final candidate also includes runtimeserver/index.js,core/data/buffer.ts, and previously modified tests, all authorized, plusnotes/001–006. - R-01 arithmetic reproduction before patch:
const marker=-t; const now=t+61000; now-marker > 300000is true, whilenow-t > 300000is false.server/socket/handlers.js:142-158cleanupMapsexpires old negative marker;validateSingleEventAsync:293-317sees retained Redis idempotency claim;sync_datareturned duplicate success. Final marker is\u0000failed-persist:<clientId>:<eventId>with positive timestamp, corrected age check, cleanup after true TTL, clear marker after durable ACK. Mixed batches reject overall positive ACK while a failed claim remains. - R-02:
server/index.js:843-897,1043-1058,core/data/buffer.ts:652-711,855-872: stream-parsed same-day client CSV/CSV.gz replay suppression, then Redis trim. Focused tests inject direct Port4000 LTRIM crash and compressed part2 quoted/multiline replay; malformed matching partition causes safe no-trim, fixture repair recovers. Memory streamed, partition disk-I/O remains proportional; not globally exactly-once. - Compat: Redis-free RAM ACK and old public /health unchanged; other HTTP/Socket auth, SDK and quota routes unaffected.
- Final Gitea CI #69 dispatched against exact supplemental commit; until terminal result, phase stays pending and no remote publish.
Final P02 CI #69 failure and exact retry patch
- CI #69:
phase02-sync.test.tssubteststale Redis claim never becomes positive ACK after cleanup windowERR_TEST_FAILURE: operation has timed outafter replacing the globalDate.nowduring Socket.IO ACK. This cannot support a server-failure claim; it was a test harness flaw. Other new replay/compatibility assertions PASS and quality/Windows jobs separately passed. - Discovered source branch in
server/socket/handlers.js:if (volatileDuplicate && processed > 0)allowed an overallsuccess:trueresponse with pending non-durable claim and already-durable duplicate whenprocessed=0,duplicates>0. Revisedif (volatileDuplicate)blocks every mixed response with an uncommitted claim; new test reproduces the exact combination. - Test no longer overrides Date.now; independent simulated arithmetic proves the original negative-marker expiry mistake; live ACK checks run on real clock. Supplemental commit
acca8be01a31f8d3972564cbaec8fd3117a0b551; CI #72 must pass before P02 closure. No modification of package/dependency files, test CI workflow, public API, UI or production.
G. P02 Final Acceptance — authoritative terminal CI / Scope Lock proof
- Source baseline published diagnosis
fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898. Fixed runtime candidate76cd972bdeea9ef07b4b38e9d8b5b5ed3312ca5b, previous #66 SUCCESS. Final corrected source/test SHAacca8be01a31f8d3972564cbaec8fd3117a0b551fully verified in Gitea #72, terminal COMPLETED SUCCESS, NodeCI_SUITE_SUMMARY total=24 failed=0. CI job conclusions: node-regression SUCCESS, quality SUCCESS, windows-compatibility SUCCESS, production-build SUCCESS; python-regression and docker-image-build SKIPPED by narrow dispatch. - Exact reproducible Redis idempotency bug: old
-Date.now()failed timestamp ages incorrectly at +61000ms, while 300000ms Redis claim TTL has not elapsed. New\u0000failed-persist:${claimKey}positive timestamp marker and corrected TTL comparison isolate the failed claim. Live Socket.IO retries remain negative ACK; no process-wide clock monkeypatch. Mixed all-duplicate batch also rejects pending claim instead of sending global positive ACK. - CSV fault evidence:
tests/phase03-storage-buffer.test.tsafter CSV append but before Redis LTRIM, direct Port4000 writer, rotated.csv.gz, JSON multiline-quoted payload, corrupt matching partition no-trim/recovery all PASS. Test assertions cannot prove a global exactly-once protocol or live throughput cap. Full-partition scan is streaming bounded-memory, but data-size-linear disk I/O. - Historical #69 failed one test-only global Date.now-induced Socket.IO timeout, not a demonstrated runtime regression; corrected #72 PASS. No secret or credential introduction, no production data access.
- Final audit
git diff --checkPASS and source JS syntax checks PASS; exact tracked source diff from diagnosis = 11 allowlisted files (3 runtime + 2 test + 6 notes); no unrelated UI/auth/SDK/public health, DB schema, package metadata, Docker, CI workflow or live deployment. - Git release gate: final docs-only commits after tested source SHA add no source/test differences. Gitea PR #3 squashed into one
[skip ci]main commit; GitHub/GitLab tracked Git trees must match Gitea before final CLOSED. If GitLab commit SHA differs because of parent history, compare tree SHA not commit SHA. Store exact remote SHAs/trees in phase close operator report; do not invent them before push.