diff --git a/notes/001-PRODUCTION-FORENSIC-AUDIT.md b/notes/001-PRODUCTION-FORENSIC-AUDIT.md index e96563d..b95f2b4 100644 --- a/notes/001-PRODUCTION-FORENSIC-AUDIT.md +++ b/notes/001-PRODUCTION-FORENSIC-AUDIT.md @@ -111,3 +111,8 @@ Confirmed exact root cause in candidate: `server/socket/handlers.js` stores fail R-02: `server/index.js:843-897,1043-1058` and `core/data/buffer.ts:652-711,855-872` replay-scan existing same-client/day CSV, preserving Redis queue until a successful write/dedupe; fault tests include direct Port 4000 post-CSV LTRIM outage, rotated compressed .csv.gz containing quoted multiline payload and unrelated-date corrupt partition; malformed matching-date CSV must reject and leave Redis queue intact, with recovery after disposable fixture repair. Streaming parsing bounds memory, **not total disk I/O**; scan latency grows with matching partition size. No universal exactly-once guarantee, no live throughput claim and no claim of finite hard disk-I/O cap. Public R-03 health remains historical liveness. Intentionally Redis-free RAM fallback and its historical successful ACK stay unchanged. No app/UI/auth/API schema/SDK/Docker/production-data/deployment changes. Work only in a new clean temp worktree; original PC repository's staged 16 P01 files and a separate pre-existing dirty candidate worktree preserved. No production readiness claim until CI #69 and final audit. + + +### P02 Final Focused Correction — CI #69 to #72 + +Gitea [CI #69](https://dev.ivib.cc/vibtools/DataStack/actions/runs/69) was run against `bf525f0d4e5e4781de7fddd2f5e7d3d602a80532`: the new Redis stale-claim test timed out because it globally replaced `Date.now` during live Socket.IO ACK scheduling. This is a **test-harness failure**, not a proven backend regression; other new CSV.gz, partial-corruption, mixed-new/durable and Redis-free regression cases passed. An additional source-grounded edge case was found: `batch_sync` previously failed to reject a batch containing a non-durable claim PLUS an already accepted duplicate when `processed=0, duplicates>0`. Under exact approved scope, the `server/socket/handlers.js` branch changed from `if (volatileDuplicate && processed > 0)` to `if (volatileDuplicate)`, ensuring false overall ACK is never sent for any such mixture. Test now uses local simulated TTL arithmetic and real-clock Socket.IO retries; no global clock manipulation. Corrected commit `acca8be01a31f8d3972564cbaec8fd3117a0b551` sent to [focused CI #72](https://dev.ivib.cc/vibtools/DataStack/actions/runs/72); still no P02 closure or production deployment until PASS. diff --git a/notes/002-PRODUCTION-UPDATE-ROADMAP.md b/notes/002-PRODUCTION-UPDATE-ROADMAP.md index 0b377da..c1a3f79 100644 --- a/notes/002-PRODUCTION-UPDATE-ROADMAP.md +++ b/notes/002-PRODUCTION-UPDATE-ROADMAP.md @@ -99,3 +99,8 @@ User approved strict P02 final acceptance after `fd57a3518d14bcb8367ce9ab9c3ec30 **Acceptance contract:** ACK durability applies to Redis-backed events when Redis configured. Intentionally Redis-free RAM mode retains its historical best-effort ACK. CSV replay is at-least-once with per-partition event-id duplicate suppression under tested crash windows, not a global transactional exactly-once guarantee. Existing scan uses streaming parsing (bounded memory) and scans entire matching partitions (disk I/O scales with data size; monitor before production promotion). Corrupt matching CSV fails closed. If a hard upper disk-scan budget is required for production traffic, obtain a dedicated storage/index design approval rather than silently breaking ingestion. **Next P03 (ONLY after P02 closure):** CI-06 Python runner environment and 14/14 Python SDK; CI-07 three Docker image builds/isolated BuildKit with no live prune; CI-08 read-only npm advisory IDs/direct-transitive audit, compatible dependency pins only if justified; Compose/standalone/health gate evidence and manual release decision. Exact P03 allowlist remains in this roadmap. No live deploy, migrations, tags or releases without separate approval. + + +### P02 Final CI Exception / Narrow Retry + +The final acceptance test-only clock override in #69 interfered with Socket.IO ACK timing. The source-grounded mixed `batch_sync` all-duplicate edge also required a single conditional correction within the previously approved handler. Commit `acca8be01a31f8d3972564cbaec8fd3117a0b551` changes only `server/socket/handlers.js` and `tests/phase02-sync.test.ts` after initial final candidate; one narrow Gitea CI #72 requested because CI #69 node-regression failed one fixture. Other scope/production invariants remain unchanged. P02 closure requires terminal #72 and exact patch/security evidence. diff --git a/notes/003-PHASE-STATUS-LOG.md b/notes/003-PHASE-STATUS-LOG.md index 6603d8d..12c733f 100644 --- a/notes/003-PHASE-STATUS-LOG.md +++ b/notes/003-PHASE-STATUS-LOG.md @@ -117,3 +117,11 @@ The authorized Windows PC attempted a **local disposable** `npm.cmd ci --ignore- - One focused Gitea CI #69 dispatched for final patch; job terminal outcome and any corrective action must be recorded. **Do not merge until this passes and exact patch/security audit is reviewed.** - Original `D:\VibTools_Workspace\01_Products\14_DataStack` at old `d8ac96e` with 16 staged P01 changes untouched. Separate pre-existing `p02-runtime-remediation-20261122` had three dirty files; left untouched. The new isolated worktree was created clean from candidate SHA. - Source behavior: no Redis configured = historical RAM best-effort acceptance preserved; configured Redis failure = negative ACK; public /health unchanged. Per-CSV replay scanning bounded-memory but linear in stored bytes; no global exactly-once/throughput guarantee. P03 cannot begin until P02 acceptance/remote parity close. + + +### P02 Focused Retry — #69 fixture failure / #72 correction + +- Final acceptance CI #69 on `bf525f0d4e5e4781de7fddd2f5e7d3d602a80532`: node-regression **FAIL** because `phase02-sync` stale-marker test globally mocked `Date.now` and Socket.IO ACK timed out. Other new tests for gzip/rotated CSV, partial CSV corruption, Redis-free RAM and mixed batch with a new persisted event PASS. Strict TypeScript, lint and Windows SDK jobs PASS; full workflow conclusion must be read at terminal. +- New source-grounded all-duplicate mixed batch path (`processed=0`, `duplicates=1`, `rejected>0`, volatile claim present) could return overall success; added focused test and changed handler conditional to reject all `volatileDuplicate` combinations. +- Test correction removes process-wide Date.now override, verifies 61000ms/300000ms cleanup math with static source marker assertion, and checks real Socket.IO negative retries. No unrelated files, CI workflows, SDK, auth or public health modified. +- Commit `acca8be01a31f8d3972564cbaec8fd3117a0b551` submitted to disposable CI #72; **P02 remains pending / NOT CLOSED** until terminal acceptance and 3-way mirror parity. diff --git a/notes/004-ERROR-HANDLING-REGISTER.md b/notes/004-ERROR-HANDLING-REGISTER.md index 3116989..1048597 100644 --- a/notes/004-ERROR-HANDLING-REGISTER.md +++ b/notes/004-ERROR-HANDLING-REGISTER.md @@ -92,3 +92,8 @@ Security/API invariants: leave token/JWT/auth/rate limits/quotas/webhooks/MCP/sa | Public /health & 401/403/429/503/507 | Preserve existing behavior; no new endpoint or error contract beyond already approved BUFFER_PERSIST_FAILED | **Limitations:** no globally atomic Redis+CSV transaction, stale Redis claim after process crash may require TTL/retry policy, and per-partition scan has data-size-proportional I/O even though memory is streamed. No live production reliability or throughput certification. CI #66 SUCCESS; final CI #69 pending. + + +### Additional P02 final-acceptance negative ACK case + +If a `batch_sync` contains only (a) an uncommitted Redis-claim duplicate and (b) an ordinary already-durable duplicate, the old `processed===0 && duplicates===0 && rejected>0` test did not trigger; success ACK incorrectly masked volatile claim. Narrow correction uses `if (volatileDuplicate)` before the generic aggregate success path, independent of `processed` or `duplicates`. Specific test in `tests/phase02-sync.test.ts` injects Redis RPUSH failure and fake stale claim then includes the existing durable duplicate. All pending-claim ACKs must remain `BUFFER_PERSIST_FAILED`, even if some other events already committed. Terminal CI #72 pending; no global 5-minute clock advancement inside sockets. diff --git a/notes/005-ACTUAL-FEATURE-REGISTER.md b/notes/005-ACTUAL-FEATURE-REGISTER.md index c6bab39..2c9a990 100644 --- a/notes/005-ACTUAL-FEATURE-REGISTER.md +++ b/notes/005-ACTUAL-FEATURE-REGISTER.md @@ -65,3 +65,8 @@ Do not mark "actual working" simply because a patch compiled. Distinguish source ## P02 Final Feature/Compatibility Register — 2026-11-22 Existing UI/UX, admin auth, SDK public signatures, health route, database schema, Docker and deployment: **UNCHANGED**. Only existing Socket.IO single/batch persistence failure ACK/idempotency cleanup, Redis->CSV replay duplicate filtering and disposable tests changed during P02. No new production feature, public endpoint, release or migration. CI #66 terminal SUCCESS (24/24 TS suites, strict types, ESLint, Windows Node SDK, Next production build); final CI #69 pending. Client-facing `BUFFER_PERSIST_FAILED` error is used only for previously incorrectly acknowledged Redis-backed failure; intentionally Redis-free RAM mode is preserved. CSV write content/format unchanged, gzip and rotation replay acceptance covered in final test fixture. All live status values are **NOT VERIFIED / NO DEPLOYMENT**. No guarantee of global exactly-once delivery or hard bound on disk scans; future throughput benchmark/design approval if required. P03 still gated on final P02 close. + + +### P02 final acceptance correction checkpoint + +Gitea #69 node-regression observed one **fixture timeout**, not a proven new production feature defect, caused by altering global `Date.now`. A separate source-proven mixed batch all-duplicate ACK bug was corrected only in `server/socket/handlers.js`, test case added in existing allowed TS file. Revised `acca8be01a31f8d3972564cbaec8fd3117a0b551` pending focused Gitea CI #72. Existing UI/SDK/auth/public health remain unchanged. This is not evidence that production has been deployed or live-tested. diff --git a/notes/006-BASELINE-AND-PATCH-EVIDENCE.md b/notes/006-BASELINE-AND-PATCH-EVIDENCE.md index 1fba281..5443085 100644 --- a/notes/006-BASELINE-AND-PATCH-EVIDENCE.md +++ b/notes/006-BASELINE-AND-PATCH-EVIDENCE.md @@ -270,3 +270,10 @@ Base: Gitea/GitHub published P02 diagnostic tree `fd57a3518d14bcb8367ce9ab9c3ec3 - **R-02:** `server/index.js:843-897,1043-1058`, `core/data/buffer.ts:652-711,855-872`: stream-parsed same-day client CSV/CSV.gz replay suppression, then Redis trim. Focused tests inject direct Port4000 LTRIM crash and compressed part2 quoted/multiline replay; malformed matching partition causes safe no-trim, fixture repair recovers. Memory streamed, partition disk-I/O remains proportional; not globally exactly-once. - **Compat:** Redis-free RAM ACK and old public /health unchanged; other HTTP/Socket auth, SDK and quota routes unaffected. - Final [Gitea CI #69](https://dev.ivib.cc/vibtools/DataStack/actions/runs/69) dispatched against exact supplemental commit; until terminal result, phase stays pending and no remote publish. + + +### Final P02 CI #69 failure and exact retry patch + +- CI #69: `phase02-sync.test.ts` subtest `stale Redis claim never becomes positive ACK after cleanup window` `ERR_TEST_FAILURE: operation has timed out` after replacing the **global** `Date.now` during Socket.IO ACK. This cannot support a server-failure claim; it was a test harness flaw. Other new replay/compatibility assertions PASS and quality/Windows jobs separately passed. +- Discovered source branch in `server/socket/handlers.js`: `if (volatileDuplicate && processed > 0)` allowed an overall `success:true` response with pending non-durable claim and already-durable duplicate when `processed=0,duplicates>0`. Revised `if (volatileDuplicate)` blocks every mixed response with an uncommitted claim; new test reproduces the exact combination. +- Test no longer overrides Date.now; independent simulated arithmetic proves the original negative-marker expiry mistake; live ACK checks run on real clock. Supplemental commit `acca8be01a31f8d3972564cbaec8fd3117a0b551`; [CI #72](https://dev.ivib.cc/vibtools/DataStack/actions/runs/72) must pass before P02 closure. No modification of package/dependency files, test CI workflow, public API, UI or production.