docs(p02): record guarded ACK and replay patch plus HTTP scope blocker [skip ci]

This commit is contained in:
victorsteele428 committed 2026-11-22 16:58:00 +06:00
1 parent 5c88088537
commit 6a873cb04f
6 files changed
+70

No files matched your search

+17
View File
@@ -100,3 +100,20 @@ Authorized follow-up baseline: Gitea/GitHub `fd57a3518d14bcb8367ce9ab9c3ec306bcf
**Candidate R-02 behavior:** Port 4000 `flushBuffer` and Port 3000 `DataBufferEngine.flush` stream-parse existing CSV/CSV.gz partitions for same-client/date/event IDs before replaying any buffered Redis or RAM batch, dropping already-persisted IDs while retaining Redis queue until the write/check completes. Scan/read/parse exceptions fail closed before LTRIM. In-scope test adds a direct Port 4000 writer + Redis LTRIM-failure/recovery test and updates the existing web-buffer replay fixture. **Not proven globally exactly-once:** cross-process CSV writers outside the Redis flush lock and unknown archival/partial-corruption cases remain; scanning partitions on each flush also has real disk-I/O cost, requiring throughput evaluation before production promotion. **Candidate R-02 behavior:** Port 4000 `flushBuffer` and Port 3000 `DataBufferEngine.flush` stream-parse existing CSV/CSV.gz partitions for same-client/date/event IDs before replaying any buffered Redis or RAM batch, dropping already-persisted IDs while retaining Redis queue until the write/check completes. Scan/read/parse exceptions fail closed before LTRIM. In-scope test adds a direct Port 4000 writer + Redis LTRIM-failure/recovery test and updates the existing web-buffer replay fixture. **Not proven globally exactly-once:** cross-process CSV writers outside the Redis flush lock and unknown archival/partial-corruption cases remain; scanning partitions on each flush also has real disk-I/O cost, requiring throughput evaluation before production promotion.
**Candidate acceptance:** one disposable Gitea CI with 24/24 suites, strict TS, lint, Linux/Windows clean npm CI and production build; direct Port4000 replay tests; exact allowlist/diff audit and three-remote tree parity. The Gitea MCP/host was unavailable on the implementation attempt, so this is **NOT PASSED** and the P02 phase remains **BLOCKED / NOT CLOSED**. No live deployment or runtime production mutation. **Candidate acceptance:** one disposable Gitea CI with 24/24 suites, strict TS, lint, Linux/Windows clean npm CI and production build; direct Port4000 replay tests; exact allowlist/diff audit and three-remote tree parity. The Gitea MCP/host was unavailable on the implementation attempt, so this is **NOT PASSED** and the P02 phase remains **BLOCKED / NOT CLOSED**. No live deployment or runtime production mutation.
## P02-RUNTIME-REMEDIATION-02 — Scope-locked post-patch forensic audit (2026-11-22)
**Source baseline:** P02 diagnostic `fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898`. **Tested patch branch:** `p02-runtime-remediation-20261122`, final remediation source snapshot `5c880885378ae0f4fa8dd3a145f9c7055dada635`. This appendix supersedes the earlier "Gitea unavailable" candidate note. **Do not claim P02 CLOSED**: a separate HTTP fallback code path remains in scope-lock-excluded files.
**R-01 Port 4000 improvements implemented (not live-deployed):**
- `server/index.js:pushEventsToRedisOrFallback` fails closed with sanitized `BUFFER_PERSIST_FAILED` on Redis RPUSH rejection while retaining RAM for recovery; also deduplicates same-client/eventId retries inside a single push.
- The `onBufferPush` integration synchronously flushes to CSV before a successful ACK when Redis is intentionally absent; failed CSV append returns a negative ACK. No-Redis mode is therefore not an automatic RAM-only success ACK.
- `server/socket/handlers.js` single/batch ACK logic clears failed local idempotency claims, never calls an uncertain Redis claim "durable duplicate", verifies a claimed duplicate against a bounded Redis buffer read or read-only CSV/.csv.gz scan, and denies uncertain mixed-batch ACKs. The proof uses a 512-item Redis window and a bounded CSV scan (256 MiB, 1.8 seconds); lack of proof yields a negative ACK, **not proof of data loss**.
**R-02 Port 4000 + web-buffer improvements implemented:** Both allowed flush methods check same-client/partition/eventId in physically persisted CSV/CSV.gz (including rotation) before retrying after Redis trim failure, and only trim when the disk append/check succeeds. Direct disposable Port 4000 LTRIM failure and gzip-partition replay tests exist. **Not a formal globally atomic/exactly-once guarantee:** concurrent independent writers, data corruption and repeated full-partition scans may introduce performance or recovery limitations.
**Critical residual R-01 HTTP path (outside this approval):** `core/data/buffer.ts:412-442` silently falls back to RAM on Redis RPUSH error; `app/api/events/route.ts:744-754` returns HTTP success after `pushAsync`. `sdk/node/datastack/index.ts:643-666,916-942` routes non-auth WebSocket rejections into HTTP fallback. Thus a negative Socket.IO ACK does **not establish end-to-end SDK/HTTP durability**. A disposable HTTP fault diagnostic was added to the P02 test allowlist; it is not a runtime fix. Fixing HTTP `POST` and the buffer push/idempotency contract needs separate explicit approval, including `core/Security/security.ts:230-375` only if reproduced claim rollback demands it.
**R-03:** public `/health` remains HTTP 200 liveness; no readiness endpoint, UI or auth change.
**CI facts:** Gitea [#66](https://dev.ivib.cc/vibtools/DataStack/actions/runs/66) terminal SUCCESS for initial patch; [#68](https://dev.ivib.cc/vibtools/DataStack/actions/runs/68) diagnosed a same-ID retry assertion regression; the exact local idempotency-claim cleanup was fixed in scope. [#70](https://dev.ivib.cc/vibtools/DataStack/actions/runs/70) is the latest disposable source acceptance (check terminal outcome in notes/003). **No code was deployed to production.**
+9
View File
@@ -90,3 +90,12 @@ The user explicitly approved **only** R-01 Redis failure ACK and R-02 Redis-to-C
Local candidate `44eee8877e40d1683f44502c2a0ba3d437a03b88` is **not release-ready** until Gitea acceptance. Before closure, prove: negative ACK on Redis failure with no sensitive error leakage; ordinary successful Redis ACK preserved; both single and batch retries don't convert volatile failures to success; Redis queued event replays after crash without writing duplicate CSV rows in the Port 4000 worker and Port 3000 web buffer; CSV read/parse failure never causes premature Redis trim; validate bounded scan overhead and edge cases for rotated/gzipped partitions; all historical 401/403/429/507/503 tests unchanged. Local candidate `44eee8877e40d1683f44502c2a0ba3d437a03b88` is **not release-ready** until Gitea acceptance. Before closure, prove: negative ACK on Redis failure with no sensitive error leakage; ordinary successful Redis ACK preserved; both single and batch retries don't convert volatile failures to success; Redis queued event replays after crash without writing duplicate CSV rows in the Port 4000 worker and Port 3000 web buffer; CSV read/parse failure never causes premature Redis trim; validate bounded scan overhead and edge cases for rotated/gzipped partitions; all historical 401/403/429/507/503 tests unchanged.
**Stop rule:** if required fixes need touching `validateSingleEventAsync`, `csvWriter.js` or separate journal/storage scheme, do not broaden the implementation scope; record the residual and request new explicit approval. Existing unrelated files including `MEMORY.md` remain unchanged due exact Scope Lock; this note is the cross-IDE handoff. P03 must wait for P02 CLOSED, not merely implementation committed. **Stop rule:** if required fixes need touching `validateSingleEventAsync`, `csvWriter.js` or separate journal/storage scheme, do not broaden the implementation scope; record the residual and request new explicit approval. Existing unrelated files including `MEMORY.md` remain unchanged due exact Scope Lock; this note is the cross-IDE handoff. P03 must wait for P02 CLOSED, not merely implementation committed.
## P02 Post-implementation approval boundary — 2026-11-22
The separately approved R-01/R-02 **Socket.IO and CSV flush** hardening is implemented on an isolated branch, with all allowed runtime sources/tests only. Existing candidate approval text above is historic; the newest evidence is the appendix in notes/001 and notes/003. P02 must **NOT** be closed solely on P02 branch CI success: the HTTP `POST /api/events` RAM-fallback-success path is source-observed outside the current runtime method allowlist. P03 remains NOT STARTED until P02 acceptance and a new explicit approval.
**Proposed separate Scope Lock approval — P02-HTTP-DURABILITY-01 (NOT YET APPROVED):**
`APPROVED — DataStack P02-HTTP-DURABILITY-01. Use latest verified P02 runtime branch and notes/001–006. Default WRITE only app/api/events/route.ts POST; core/data/buffer.ts writeEventsToRedisOrFallback, pushAsync and pushBatchAsync; tests/phase02-sync.test.ts, tests/phase03-storage-buffer.test.ts, tests/r4-phase02-ingestion-audit-streaming.test.ts; notes/001–006. Allow core/Security/security.ts validateEventAsync/idempotency rollback ONLY if a focused failing test proves stale claims cannot be handled within listed methods, and record exact lines before any edit. Fix HTTP success on volatile RAM after Redis write failure; preserve legitimate Redis-free CSV durable fallback, cross-transport duplicate correctness, current 401/403/429/507 contracts, SDK wire format, auth and user data. Use no production Redis/DB, no live requests, no UI, SDK, Docker, migrations, releases, R-03 health change or unrelated files. One focused disposable Gitea CI, exact patch/security audit, notes update, one Git commit, three-remote content parity. Then close P02 only if the entire acceptance matrix is demonstrated; provide P03 approval command.`
**P03 outline once P02 is truly CLOSED:** CI-06 restore reliable Python CI runner provisioning; CI-07 diagnose Docker/containerd Builder snapshot error and build all three images in disposable CI; CI-08 trace exact npm advisories and apply only proven compatible security upgrades. Preserve UI, auth, backend feature semantics and live data. No automatic deployment.
+10
View File
@@ -107,3 +107,13 @@ Historical baseline code stayed frozen through these CI workflow changes. Curren
### Verification environment blocker — local dependency install ### Verification environment blocker — local dependency install
The authorized Windows PC attempted a **local disposable** `npm.cmd ci --ignore-scripts --no-audit --no-fund --prefer-offline` in the isolated P02 worktree only; it failed with `npm ERR! ETARGET No matching version found for yaml@2.9.1` plus cleanup EPERM warnings in the temp worktree's `node_modules`. Exact output captured in `%TEMP%\DataStack-P02-npm-ci.log` on the authorized PC. This is **not a passed code test nor proof of a repository package defect**, and `package-lock.json` / `package.json` cannot be edited under P02 scope. Gitea MCP remained UNAVAILABLE and the Git remote/HTTP service was not responding when CI was attempted, so the mandatory P02 focused CI was **not dispatched**. Phase remains BLOCKED pending actual CI evidence; no runtime change was pushed to production/main. The authorized Windows PC attempted a **local disposable** `npm.cmd ci --ignore-scripts --no-audit --no-fund --prefer-offline` in the isolated P02 worktree only; it failed with `npm ERR! ETARGET No matching version found for yaml@2.9.1` plus cleanup EPERM warnings in the temp worktree's `node_modules`. Exact output captured in `%TEMP%\DataStack-P02-npm-ci.log` on the authorized PC. This is **not a passed code test nor proof of a repository package defect**, and `package-lock.json` / `package.json` cannot be edited under P02 scope. Gitea MCP remained UNAVAILABLE and the Git remote/HTTP service was not responding when CI was attempted, so the mandatory P02 focused CI was **not dispatched**. Phase remains BLOCKED pending actual CI evidence; no runtime change was pushed to production/main.
## P02-RUNTIME-REMEDIATION-02 — Post-patch evidence/status (2026-11-22)
- **P02 current status: BLOCKED / NOT CLOSED** by a separate Port 3000 HTTP RAM-ACK path (and until terminal CI evidence). Completed implementation phases remain **1/3** (P01 only); P03 not authorized.
- Baseline `fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898`; isolated Gitea branch `p02-runtime-remediation-20261122`; inspected code commit `5c880885378ae0f4fa8dd3a145f9c7055dada635`. Original PC root `main` at earlier docs SHA with 16 user-staged P01 paths remains untouched. No force-reset, stash, production write or deployment.
- **Verified earlier focused CI #66:** 24/24 TS suites, strict TypeScript, ESLint, Windows/npm SDK and Next.js standalone production build success; Python/Docker jobs skipped as separate P03 work.
- **CI #68:** new same-ID retry test failed because an unproven duplicate had left a local claim set; this was corrected in handlers on the subsequent scoped code commit. #68 was cancelled after failure (logs preserved). **CI #70:** revised focused acceptance and added HTTP failure diagnostic; confirm actual terminal job conclusions before claiming PASS. The HTTP diagnostic is intentionally expected to expose an unresolved behavior, not fix it.
- P02 code changes confined to `server/index.js` selected ACK/buffer/flush methods, `server/socket/handlers.js` persistence ACK/duplicate proof methods, `core/data/buffer.ts` flush, and three approved test files; notes/001–006 only. R-03 public health unchanged. All other app/UI/auth/SDK/CI workflow/lockfile/Docker/runtime source unchanged.
- **Residual blocking source evidence:** `app/api/events/route.ts:744-754` returns success following `core/data/buffer.ts:412-442` silent Redis-error RAM fallback; Node SDK may switch to HTTP after negative WS ACK. A separate exact scope exception is required for end-to-end no-positive-ACK-on-volatile-data acceptance. Do not merge the blocked candidate into three `main` refs or assert P02 production ready.
- Performance boundary: replay checking can stream entire CSV partitions per flush (not throughput benchmarked); duplicate proof is capped at last 512 Redis queue items and 256 MiB/1.8s CSV proof (negative ACK on uncertainty). This is safer than a false positive, but compatibility under heavy ingestion must be reviewed before release.
+14
View File
@@ -76,3 +76,17 @@ Security invariants: do not expose secrets in ACK/errors, keep JWT/auth and 401/
| Legacy public health | **UNCHANGED** HTTP 200 liveness contract and Redis-connected diagnostic field; R-03 readiness is not introduced in this approval | Does not prove full DB/Redis transactional readiness | | Legacy public health | **UNCHANGED** HTTP 200 liveness contract and Redis-connected diagnostic field; R-03 readiness is not introduced in this approval | Does not prove full DB/Redis transactional readiness |
Security/API invariants: leave token/JWT/auth/rate limits/quotas/webhooks/MCP/sandbox/UI/SDK unchanged. No production Redis/Postgres/API/CSV mutation. No lockfile/dependency versions or CI workflow changes. CI acceptance remains pending; these are code-level observations, not release certification. Security/API invariants: leave token/JWT/auth/rate limits/quotas/webhooks/MCP/sandbox/UI/SDK unchanged. No production Redis/Postgres/API/CSV mutation. No lockfile/dependency versions or CI workflow changes. CI acceptance remains pending; these are code-level observations, not release certification.
## P02-RUNTIME-REMEDIATION-02 — Updated safety contract evidence
| Scenario | In-scope behavior | Residual |
|---|---|---|
| Port 4000 Redis `RPUSH` failure | `BUFFER_PERSIST_FAILED`, `success:false`; no Redis transport diagnostics disclosed in client ACK | SDK can retry via HTTP; Port 3000 volatile RAM success path remains unpatched |
| Port 4000 without Redis | Await actual CSV flush before positive ACK; disk error yields negative ACK and queue retained | Disk API does not provide fsync-level power-loss guarantees |
| Stale idempotency after restart | Never treat a claim alone as persisted; only accept a duplicate after in-memory acknowledged proof, Redis queued ID or read-only CSV partition proof | Bounded scans may cause temporary negative ACK for otherwise durable old events, rather than unsafe success |
| Mixed `batch_sync` with uncertain claim | Overall negative `BUFFER_PERSIST_FAILED`; newer items may already be persisted, repeat dedupe resolves them | Preserve existing response fields and rate/quota/auth validations |
| Redis `LTRIM` fails after CSV write | On retry, scan CSV/CSV.gz by client/date/event ID and skip already written event; trim Redis only after safe check | Cross-process exact-once not proven; I/O cost needs throughput validation |
| Port 3000 HTTP `pushAsync` / `pushBatchAsync` | **UNCHANGED** (excluded from current approval); Redis error can return HTTP success with data only in volatile RAM | **OPEN R-01 blocker**, requires separate approval; `core/Security/security.ts` precommit idempotency claims may also need scoped rollback |
| Public `/health` | Unchanged 200 liveness, `redisConnected` diagnostic | No dependency readiness probe under P02-R01/R02 authorization |
Do not silently remap 401/403/429/507 errors; no SDK edits or live fault injection. Tests use mock Redis and temp CSV only. Relevant CI #66/#68/#70 in notes/003.
+8
View File
@@ -60,3 +60,11 @@ Do not mark "actual working" simply because a patch compiled. Distinguish source
- **Intended change:** Redis replay flush on both ingestion/Next paths inspects existing client CSV partitions and filters already-stored event IDs before reappend; safe failure leaves Redis queue untrimmed. This prevents the tested post-append/LTRIM replay duplication but does **not** prove transactionally exactly-once delivery or zero performance impact. - **Intended change:** Redis replay flush on both ingestion/Next paths inspects existing client CSV partitions and filters already-stored event IDs before reappend; safe failure leaves Redis queue untrimmed. This prevents the tested post-append/LTRIM replay duplication but does **not** prove transactionally exactly-once delivery or zero performance impact.
- **No changes:** application UI/UX, auth, SDK, database schema, Docker, environment, public R-03 health, other features and production runtime. No deployment or live verification. - **No changes:** application UI/UX, auth, SDK, database schema, Docker, environment, public R-03 health, other features and production runtime. No deployment or live verification.
- **Status:** P02 code candidate locally committed, acceptance CI blocked by Gitea unavailability; **NOT CLOSED**, not production-ready. P01 remains only completed phase. - **Status:** P02 code candidate locally committed, acceptance CI blocked by Gitea unavailability; **NOT CLOSED**, not production-ready. P01 remains only completed phase.
## P02 Remediation Hardening Register — 2026-11-22
- **Changed in source, not deployed:** Port 4000 Socket.IO error ACK behavior on Redis write failure; no-Redis mode now waits for CSV flush before success; duplicate ACK checks actual Redis/CSV durability; mixed batch uncertainty gives negative ACK. EventId retry handling is made fail-closed with in-scope handler logic.
- **Changed in source, not deployed:** Port 4000 and web-buffer CSV replay after Redis LTRIM failure now checks persisted CSV/CSV.gz event IDs. Existing file rotation/compression and disk errors need focused tests; scanner I/O impacts high-volume performance and does not certify global exactly-once semantics.
- **Unchanged:** UI/UX/pages, public R-03 health shape/status, auth/JWT/SDK implementation, database/Prisma, Docker/Compose, environment, unrelated features, production data, live server and deployment.
- **Still NOT working as end-to-end guarantee:** HTTP `POST /api/events` currently may send success after Redis RPUSH fails while only RAM contains the event. This is an additional source-observed release blocker, not something fixed by WS error ACK. **P02 NOT CLOSED; P03 NOT STARTED.**
- **Evidence:** Gitea #66 initial remediation PASS; #68 exposed same-ID retry bug fixed in current branch; #70 final focused run awaiting terminal confirmation. The original PC main workspace's 16 staged P01 paths remain untouched.
+12
View File
@@ -259,3 +259,15 @@ Base: Gitea/GitHub published P02 diagnostic tree `fd57a3518d14bcb8367ce9ab9c3ec3
- Gitea: connection returned `UNAVAILABLE: McpServerError: Connection failed`; Git remote `dev.ivib.cc` unresponsive. No Gitea CI run ID exists for candidate `44eee8877e40d1683f44502c2a0ba3d437a03b88`. - Gitea: connection returned `UNAVAILABLE: McpServerError: Connection failed`; Git remote `dev.ivib.cc` unresponsive. No Gitea CI run ID exists for candidate `44eee8877e40d1683f44502c2a0ba3d437a03b88`.
- Local Windows npm: `npm ci` failed `ETARGET yaml@2.9.1` and cleanup `EPERM` in the isolated worktree, **not** a TypeScript/test result. Dependency changes are out of P02 scope. - Local Windows npm: `npm ci` failed `ETARGET yaml@2.9.1` and cleanup `EPERM` in the isolated worktree, **not** a TypeScript/test result. Dependency changes are out of P02 scope.
- Therefore neither TS typecheck nor regression-suite remediation acceptance is established; keep P02 BLOCKED and do not advance P03 or alter any protected runtime/source beyond this approval. - Therefore neither TS typecheck nor regression-suite remediation acceptance is established; keep P02 BLOCKED and do not advance P03 or alter any protected runtime/source beyond this approval.
## F. P02-RUNTIME-REMEDIATION-02 — Exact edge-case proof and scope audit (2026-11-22)
**Baseline** `fd57a3518d14bcb8367ce9ab9c3ec306bcfc9898`; isolated branch `p02-runtime-remediation-20261122`; patch revision `5c880885378ae0f4fa8dd3a145f9c7055dada635`. Scope: `server/index.js` (RPUSH fallback/onBufferPush/flush), `server/socket/handlers.js` (single/batch ACK + durability verification helper), `core/data/buffer.ts` (flush), `tests/{phase02-sync,phase03-storage-buffer,r4-phase02-ingestion-audit-streaming}.test.ts` and `notes/001–006` only. No other source file/CI workflow/dependency/SDK/UI change.
**Runtime contract and tests:** configured Redis RPUSH rejection -> negative `BUFFER_PERSIST_FAILED`; Redis-free Socket.IO ACK -> synchronous local CSV flush, with negative ACK on disk error; fail closed on previous Redis idempotency claim unless persisted to Redis buffer or matching physical CSV; recover same eventId after Redis reconnection; mixed batch uncertain duplicate remains negative until persistence proven. R-02 test injects failure after Port 4000 CSV append and before Redis trim, verifies subsequent retry does not add another row, including moved/rotated `.csv.gz`. R-03 `/health` remains unchanged.
**Test-only HTTP residual diagnostic:** `tests/phase02-sync.test.ts` adds Redis RPUSH failure against `POST /api/events` on a fake Redis. Expected prior behavior: HTTP 200 `success:true` while `DataBufferEngine.getLocalBufferSnapshot()` has the event and Redis buffered list is empty, then simulated RAM loss. This *diagnostic PASS* would confirm a remaining risk, not a completed fix. Source: `app/api/events/route.ts:744-754`, `core/data/buffer.ts:412-442,481-491`, `core/Security/security.ts:230-375`, Node SDK `sdk/node/datastack/index.ts:643-666,916-942`. Those runtime methods are **READ ONLY under current approval**.
**CI ledger:** [#66](https://dev.ivib.cc/vibtools/DataStack/actions/runs/66) initial in-scope implementation overall SUCCESS; [#68](https://dev.ivib.cc/vibtools/DataStack/actions/runs/68) discovered same-ID retry local claim leak, corrected in subsequent branch commit; [#70](https://dev.ivib.cc/vibtools/DataStack/actions/runs/70) revised source/test acceptance queued/running as of this addendum. Do not assert it PASS without terminal proof.
**Security/limitations:** never expose Redis backend errors or credentials in ACKs; preserve auth, quotas, rate limits and SDK response fields. Duplicate proof reads no more than 512 recent Redis buffered records and scans up to 256 MiB of CSV partition data with an abort deadline of 1800 ms, refusing uncertain duplicate-success. R-02 streaming CSV scan per flush has unmeasured throughput cost; no globally atomic multi-process exactly-once claim. Unmodified production data, no migrations, no live events, no releases/deploys; original local root staged changes untouched. P02 remains BLOCKED until a separate narrow HTTP durability approval or an equivalent tested safe contract.