P02 diagnostics: confirm Redis ACK durability, CSV replay and health readiness gaps (#2)
test(p02): document reproduced durability/replay risk, no runtime patch [skip ci]
This commit was merged in pull request #2.
This commit is contained in:
1 parent
667a140c4e
commit
fd57a3518d
8 files changed
+203
-3
No files matched your search
@@ -78,3 +78,14 @@ NOT production-update-ready until P01, conditional P02 gates, and P03 finish, 24
|
|||||||
| CI-05 | 43 historical test-only TypeScript diagnostics corrected; one additional teardown declaration fixed after #59 typecheck failure. Final #60 strict TypeScript PASS (zero errors) and ESLint PASS. |
|
| CI-05 | 43 historical test-only TypeScript diagnostics corrected; one additional teardown declaration fixed after #59 typecheck failure. Final #60 strict TypeScript PASS (zero errors) and ESLint PASS. |
|
||||||
|
|
||||||
**Acceptance evidence:** #59 Node SDK build PASS, 24/24 isolated TypeScript suites exit 0 (no timeouts), Windows SDK build PASS, Next.js 15.5.29 standalone production build PASS with 22/22 pages. #60 targeted `tsc --noEmit` and `npm run lint` both PASS. Prior canonical Windows Python SDK 14/14 PASS from [#48](https://dev.ivib.cc/vibtools/DataStack/actions/runs/48); Python source unchanged and not rerun in P01. Docker images and live production behavior NOT verified in P01; P02 runtime reliability risks and P03 infrastructure/security gates remain open.
|
**Acceptance evidence:** #59 Node SDK build PASS, 24/24 isolated TypeScript suites exit 0 (no timeouts), Windows SDK build PASS, Next.js 15.5.29 standalone production build PASS with 22/22 pages. #60 targeted `tsc --noEmit` and `npm run lint` both PASS. Prior canonical Windows Python SDK 14/14 PASS from [#48](https://dev.ivib.cc/vibtools/DataStack/actions/runs/48); Python source unchanged and not rerun in P01. Docker images and live production behavior NOT verified in P01; P02 runtime reliability risks and P03 infrastructure/security gates remain open.
|
||||||
|
|
||||||
|
|
||||||
|
## P02 Forensic Fault-Injection Update — 2026-11-22
|
||||||
|
|
||||||
|
Source freeze for P02: Gitea/GitHub main `667a140c4e9e75382bfdcdbe1d3e9eb96f6e808a`, matching GitLab tracked tree `d53b9e6dd7ff6d291b8bab7b841d8684ef89c703`. R-01/R-02/R-03 originally **risks**, not established production incidents. The diagnostic-only P02 patch adds disposable fixture tests in `tests/phase02-sync.test.ts` (R-01/R-03) and `tests/phase03-storage-buffer.test.ts` (R-02); CI: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63. **Do not interpret diagnostic assertion success as a durability fix.** No runtime code, API, UI, data store, SDK, security policy, deployment or production host has been modified.
|
||||||
|
|
||||||
|
- **R-01 durable ACK:** `server/index.js:165-193,1895-1923` catches a failed Redis RPUSH and pushes accepted events into volatile RAM. `server/socket/handlers.js:583-602` then sends `{success:true,eventId}` if `onBufferPush` resolves. In the injected-RPUSH-failure fixture, simulated process-memory reset models loss of the acknowledged event; no actual server is killed. **Potential release blocker: acknowledged data is not necessarily durable.**
|
||||||
|
- **R-02 CSV/replay:** `server/index.js:981-993` and `core/data/buffer.ts:795-809` write CSV before Redis LTRIM. The fixture injects an LTRIM failure after successful CSV append and retries the same queue; a duplicate eventId write would demonstrate at-least-once rather than exactly-once delivery. This fixture is an analogous web-buffer implementation, **not direct Port-4000 writer fault injection**. **A release decision and exact remediation require independent targeted evidence; do not extrapolate no-duplicates.**
|
||||||
|
- **R-03 health:** `server/index.js:1926-1957` returns HTTP 200 `status:online` when Redis LLEN fails, while surfacing `redisConnected:false`; the historical route is **liveness**, not dependency readiness. `app/api/health/route.ts:95-111` likewise reports web liveness/buffer and does not prove database/Redis write readiness. A new readiness endpoint or altered public response requires separate approval.
|
||||||
|
|
||||||
|
Do not auto-fix source based solely on tests. Runtime code is READ-ONLY under the current approval. Any proven R-01/R-02 risk requiring behavior change needs a new exact method/file approval before application changes. P03 remains independent and unstarted.
|
||||||
@@ -65,8 +65,19 @@ Prerequisite for any phase: compare current HEAD and changed paths with this bas
|
|||||||
|
|
||||||
## Sequence and approval gate
|
## Sequence and approval gate
|
||||||
|
|
||||||
P01 **CLOSED** -> P02 **awaiting approval** -> P03 **awaiting P02 closure**. P01 acceptance evidence: Gitea CI #59 (Windows + Linux clean install, 24/24 tests, Node SDK, Next standalone) and #60 (strict TypeScript + ESLint), with historical #59 single test-type error corrected before #60. No actual app/runtime/UI/SDK change or live deployment.
|
P01 **CLOSED** -> P02 **approved diagnostic completed / BLOCKED on R-01 and R-02 remediation** -> P03 **must await P02 closure**. P01 acceptance evidence: Gitea CI #59 (Windows + Linux clean install, 24/24 tests, Node SDK, Next standalone) and #60 (strict TypeScript + ESLint), with historical #59 single test-type error corrected before #60. No actual app/runtime/UI/SDK change or live deployment.
|
||||||
|
|
||||||
Approval command for next turn:
|
Approval command for next turn:
|
||||||
|
|
||||||
`APPROVED — DataStack Production Readiness P02. Follow notes/002 P02 Exact Scope Lock. Verify R-01 Redis ACK durability, R-02 CSV crash/replay duplication, and R-03 liveness/readiness with disposable fault-injection tests. App/runtime source READ-ONLY unless a reproducible bug and separate exact patch approval. Preserve all existing UI, API, auth, SDK, data, and production. One focused CI, forensic diff, notes update, commit, three-remote sync. No deploy, releases, migrations or scope expansion.`
|
`APPROVED — DataStack Production Readiness P02. Follow notes/002 P02 Exact Scope Lock. Verify R-01 Redis ACK durability, R-02 CSV crash/replay duplication, and R-03 liveness/readiness with disposable fault-injection tests. App/runtime source READ-ONLY unless a reproducible bug and separate exact patch approval. Preserve all existing UI, API, auth, SDK, data, and production. One focused CI, forensic diff, notes update, commit, three-remote sync. No deploy, releases, migrations or scope expansion.`
|
||||||
|
|
||||||
|
## P02 Execution Scope Addendum — 2026-11-22
|
||||||
|
|
||||||
|
P02 approved as **diagnostic-only**. Published P01 baseline: `667a140c4e9e75382bfdcdbe1d3e9eb96f6e808a`. Isolated local worktree: `p02-readiness-test-20261122`, preserving previously staged local P01 source. The **only code edits** are `tests/phase02-sync.test.ts` and `tests/phase03-storage-buffer.test.ts`; six `notes/001–006` documents may be amended. CI #63 uses the existing `.gitea/workflows/datastack-ci.yml`; changing workflow files is **outside P02 Scope Lock**. No runtime/source/API/UI/SDK change is approved. If loss or unbounded replay is reproduced, classify as **BLOCKED pending an independently authorized narrow runtime patch**, not as fixed; do not advance P03 as if P02's production-readiness gate passed. Approval after evidence must name precise methods, event ACK contract, deduplication/CSV recovery strategy and allowed tests before editing runtime.
|
||||||
|
|
||||||
|
|
||||||
|
### P02 Conditional Remediation Approval (NOT CURRENTLY APPROVED)
|
||||||
|
|
||||||
|
`APPROVED — DataStack P02-RUNTIME-REMEDIATION. Baseline: use the verified P02 diagnostic commit/remote tree. Fix reproduced R-01 and R-02 only, following exact notes/001–006 proof and scoped server/index.js pushEventsToRedisOrFallback/onBufferPush and flushBuffer, server/socket/handlers.js sync_data/batch_sync ACK, and their three existing P02 test files. First document safe ACK/retry semantics, exactly-once versus at-least-once CSV contract and crash recovery. Add disposable fault tests proving no positive ACK for non-durable data and no unbounded duplicate replay. Runtime sources core/data/buffer.ts and server/workers/csvWriter.js remain READ-ONLY unless a separately justified exact file/line amendment is approved. Preserve all UI/auth/SDK/public health behavior and existing error codes, production data, security, other features and Docker. One focused CI, diff/security audit, notes update, commit and 3-remote parity. No deployment, migrations, tags/releases or scope expansion. If a safe fix cannot be achieved within these exact methods, stop and report the required additional Scope Lock rather than broadening automatically.`
|
||||||
|
|
||||||
|
After separate approved remediation and verified P02 closure, **P03 plan** remains CI-06 Linux Python runner provisioning, CI-07 isolated Docker/BuildKit failure remediation and 3-image build, CI-08 supply-chain advisory triage, and non-mutating readiness evidence. P03 should not modify unrelated UI/runtime/API code by default.
|
||||||
@@ -8,7 +8,7 @@ Scope: documentation-only kickoff. Any future phase must update this log **after
|
|||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| P00 Notes / freeze | COMPLETED (documentation only) | `001–006` docs, source baseline and exact scope | `notes/**` only | 6 Markdown files; balanced fences; staged diff `--check` PASS | Documentation commit containing this file; remote SHA verification recorded in session handoff |
|
| P00 Notes / freeze | COMPLETED (documentation only) | `001–006` docs, source baseline and exact scope | `notes/**` only | 6 Markdown files; balanced fences; staged diff `--check` PASS | Documentation commit containing this file; remote SHA verification recorded in session handoff |
|
||||||
| P01 Tests & lockfile | **CLOSED — VERIFIED** | CI-01…CI-05, no runtime feature changes | 16 exact P01 code/test/CI paths, plus `notes/001–006` records | #59 Linux/Windows clean npm/SDK, 24/24 TS suites, Next standalone PASS; #60 strict TypeScript and ESLint PASS; historical #59 quality mismatch corrected | One permanent P01 main commit containing these notes; confirm remote SHA/tree parity in phase handoff |
|
| P01 Tests & lockfile | **CLOSED — VERIFIED** | CI-01…CI-05, no runtime feature changes | 16 exact P01 code/test/CI paths, plus `notes/001–006` records | #59 Linux/Windows clean npm/SDK, 24/24 TS suites, Next standalone PASS; #60 strict TypeScript and ESLint PASS; historical #59 quality mismatch corrected | One permanent P01 main commit containing these notes; confirm remote SHA/tree parity in phase handoff |
|
||||||
| P02 Ingestion reliability | NOT STARTED | Conditional R-01,02,03 | None | Not run | — |
|
| P02 Ingestion reliability | **BLOCKED — three diagnostic cases reproduced; runtime fix requires separate approval** | R-01 ACK/RAM loss; R-02 CSV duplicate replay; R-03 health liveness vs readiness | Only `tests/phase02-sync.test.ts` and `tests/phase03-storage-buffer.test.ts`, plus notes/001–006 | Gitea #63: all three P02 subtests PASS to reproduce existing behavior; full matrix status recorded in appendix | Diagnostic branch `p02-readiness-test-20261122`, no production runtime changes |
|
||||||
| P03 CI/Docker/security | NOT STARTED | CI-06,07,08; release gates | None | Not run | — |
|
| P03 CI/Docker/security | NOT STARTED | CI-06,07,08; release gates | None | Not run | — |
|
||||||
|
|
||||||
## Existing diagnostic CI snapshot (NOT implementation phase completions)
|
## Existing diagnostic CI snapshot (NOT implementation phase completions)
|
||||||
@@ -72,3 +72,22 @@ Historical baseline code stayed frozen through these CI workflow changes. Curren
|
|||||||
- **Working tree caveat:** original local Windows VARUA device reached its monthly quota during P01; its current local branch/working tree could not be rechecked and should be reconciled against published `main` before new local work. A separate authorized temporary Windows clone was used for diff and Git push verification. Never claim original PC is already updated.
|
- **Working tree caveat:** original local Windows VARUA device reached its monthly quota during P01; its current local branch/working tree could not be rechecked and should be reconciled against published `main` before new local work. A separate authorized temporary Windows clone was used for diff and Git push verification. Never claim original PC is already updated.
|
||||||
- **Next:** P02 read-only initial source audit plus targeted disposable fault-injection tests for R-01 (Redis ACK), R-02 (CSV replay), R-03 (health readiness). Runtime-source change requires reproduced issue and a separate exact method/file approval. P03 only after P02 closure.
|
- **Next:** P02 read-only initial source audit plus targeted disposable fault-injection tests for R-01 (Redis ACK), R-02 (CSV replay), R-03 (health readiness). Runtime-source change requires reproduced issue and a separate exact method/file approval. P03 only after P02 closure.
|
||||||
- **P02 implementation approval command:** `APPROVED — DataStack P02. Implement only notes/002 P02 diagnostic/test write scope for R-01, R-02 and R-03. Keep app/UI/API/auth/SDK/runtime/data unchanged. Runtime files read-only unless a proven failure and separately approved narrow code patch. One focused disposable Gitea CI, exact diff, notes/001–006 update, commit and remote verification. No production deployment or scope expansion.`
|
- **P02 implementation approval command:** `APPROVED — DataStack P02. Implement only notes/002 P02 diagnostic/test write scope for R-01, R-02 and R-03. Keep app/UI/API/auth/SDK/runtime/data unchanged. Runtime files read-only unless a proven failure and separately approved narrow code patch. One focused disposable Gitea CI, exact diff, notes/001–006 update, commit and remote verification. No production deployment or scope expansion.`
|
||||||
|
|
||||||
|
|
||||||
|
## P02 Execution Log — 2026-11-22 (supersedes P02: pending until acceptance)
|
||||||
|
|
||||||
|
- **Status: IN PROGRESS / diagnostic CI pending.** Not CLOSED and not a production-readiness certificate.
|
||||||
|
- P01 published baseline `667a140c4e9e75382bfdcdbe1d3e9eb96f6e808a`. Original PC local `main` still at prior docs HEAD with **16 user-visible staged P01 paths**, preserved untouched. New isolated `p02-readiness-test-20261122` worktree based on published baseline; no reset/stash/force update.
|
||||||
|
- P02 disposable test commit: `bbe73d91666641743cbec6cf7203f1694094caef`; **only two files**: `tests/phase02-sync.test.ts` (Redis negative-path ACK and health fallback), `tests/phase03-storage-buffer.test.ts` (failure between CSV append and Redis LTRIM, replay). Source contracts and auth flows unchanged.
|
||||||
|
- Focused evidence: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63. Record terminal suite/typecheck/build results after completion. Test assertions expose existing behavior; they do not implement durability or readiness fixes.
|
||||||
|
- Scope audited: `server/index.js`, `server/socket/handlers.js`, `server/workers/csvWriter.js`, `core/data/buffer.ts`, `app/api/health/route.ts` all READ-ONLY; `app/**`, `components/**`, `lib/**`, `sdk/**`, Docker, settings, production data and deployments unchanged.
|
||||||
|
- P01 remains CLOSED (1/3). If R-01 or R-02 are confirmed unsafe, P02 must remain BLOCKED for runtime remediation approval. No P03 implementation on this approval.
|
||||||
|
|
||||||
|
|
||||||
|
### P02 Verified Diagnostic Gate (R-01/R-02/R-03)
|
||||||
|
|
||||||
|
- Gitea #63 (test-only commit `bbe73d91666641743cbec6cf7203f1694094caef`): `P02 R-01 diagnostic` PASS, `P02 R-03 diagnostic` PASS, `P02 R-02 diagnostic` PASS. Each PASS validates reproduction of existing behavior, **not a successful reliability remediation**.
|
||||||
|
- `tests/phase02-sync.test.ts` suite exit 0; `tests/phase03-storage-buffer.test.ts` suite exit 0. Other CI jobs may have independent status; do not infer full job or complete workflow PASS from three cases.
|
||||||
|
- **BLOCKING RELEASE FINDINGS:** R-01 success ACK issued for non-durable RAM fallback when Redis RPUSH fails; R-02 identical eventId can append twice on post-CSV Redis-trim failure (web-side representative test). R-03 health remains online on Redis loss; liveness-only behavior is backward-compatible, protected dependency-readiness verification remains missing.
|
||||||
|
- **P02 PHASE = BLOCKED**, diagnostic implementation done, release reliability acceptance NOT satisfied. Runtime remediation requires separate exact approval. 1/3 phases still CLOSED (P01 only); P02 and P03 remain unclosed. No P03 start authorization from this diagnostic completion.
|
||||||
|
- Next approval must explicitly authorize exact narrow server/core patch for ACK durability and CSV replay deduplication, plus concrete no-loss/no-duplicate tests. Any readiness endpoint change needs separate explicit approval. User's initial P02 approval is insufficient for runtime mutation.
|
||||||
@@ -53,3 +53,14 @@ Any new error code, status, SDK public exception or alert flow is **feature/API
|
|||||||
## P01 Closure Verification (2026-11-22)
|
## P01 Closure Verification (2026-11-22)
|
||||||
|
|
||||||
The previous #58 package-lock error was resolved by adding the exact nested optional `@tybys/wasm-util@0.10.5` lock entry without changing the 1293 original locked packages. Gitea #59 clean installs and all **24/24** TypeScript regression suites passed; #59 had a single test-only TS2339 mock-type declaration error, corrected in scope and verified by **#60 strict TypeScript + ESLint PASS**. P01 made **zero changes** to runtime error-handler implementation, response codes, auth, SDK error classification, telemetry behavior or UI. Existing handler inventory remains valid as source observations. P02 R-01…R-03 fault-injection and P03 infrastructure/security remain unverified; do not claim production fault tolerance from P01.
|
The previous #58 package-lock error was resolved by adding the exact nested optional `@tybys/wasm-util@0.10.5` lock entry without changing the 1293 original locked packages. Gitea #59 clean installs and all **24/24** TypeScript regression suites passed; #59 had a single test-only TS2339 mock-type declaration error, corrected in scope and verified by **#60 strict TypeScript + ESLint PASS**. P01 made **zero changes** to runtime error-handler implementation, response codes, auth, SDK error classification, telemetry behavior or UI. Existing handler inventory remains valid as source observations. P02 R-01…R-03 fault-injection and P03 infrastructure/security remain unverified; do not claim production fault tolerance from P01.
|
||||||
|
|
||||||
|
|
||||||
|
## P02 R-01–R-03 Diagnostic Addendum — 2026-11-22
|
||||||
|
|
||||||
|
| Gate | Injected failure / expected observation | Scope / remediation gate |
|
||||||
|
|---|---|---|
|
||||||
|
| R-01 Redis durability | Mock `rpush` rejects; in-process websocket `sync_data` ACK may still return `success:true` while the event exists only in RAM. A simulated RAM reset (not killing live services) shows unrecoverable ACKed event if Redis remained empty. | `server/index.js:165-193,1897-1923`, `server/socket/handlers.js:583-602` **READ ONLY**. A correction to ACK/persist/spooling semantics requires explicit approval. |
|
||||||
|
| R-02 CSV replay | Mock Redis LTRIM fails **after** CSV append. Retrying buffered event may write same `eventId` twice, exposing duplicate amplification under persistent failures. | Test the web-side `DataBufferEngine.flush` CSV-before-trim pattern; direct ingestion writer proof remains separate. Deduplication scheme or exactly-once guarantees require explicit approval. |
|
||||||
|
| R-03 availability | Redis `llen` rejection leaves Express `/health` online with `redisConnected:false`. Recovery flips the flag while HTTP 200 remains unchanged. | Public 200 is liveness by existing contract; a protected dependency readiness probe/503 is **not implemented** and needs explicit approval. |
|
||||||
|
|
||||||
|
Security invariants: do not expose secrets in ACK/errors, keep JWT/auth and 401/403/429/503/507 contracts unchanged, and never run failure-injection against actual Redis, Postgres, production API, live events, or user CSV. Disposable fake Redis and temporary files only. CI #63: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63; results tracked in notes/003.
|
||||||
@@ -42,5 +42,13 @@ Do not mark "actual working" simply because a patch compiled. Distinguish source
|
|||||||
## Change log
|
## Change log
|
||||||
- P00: No feature added/modified. `notes/**` documentation only. 0/3 implementation phases completed.
|
- P00: No feature added/modified. `notes/**` documentation only. 0/3 implementation phases completed.
|
||||||
- P01: **CLOSED — 1/3 implementation phases**. CI-01…05 corrected exclusively in package-lock, obsolete bun.lock removal, scoped regression/type tests and Gitea CI harness. **No actual application, UI, backend, API, SDK or production functionality was added, removed or altered.** Gitea #59: committed-lock Linux/Windows clean install PASS; Node SDK builds PASS; all 24 TS suites exit0 with zero timeouts; Next standalone 22/22 pages PASS. One historical #59 test-only declaration TS error fixed; #60 strict TS zero errors and full ESLint PASS. Previous canonical Python SDK 14/14 (#48) preserved but not rerun. No live endpoint mutation or deployment; live data/Redis/S3/MCP actual operational status is not established.
|
- P01: **CLOSED — 1/3 implementation phases**. CI-01…05 corrected exclusively in package-lock, obsolete bun.lock removal, scoped regression/type tests and Gitea CI harness. **No actual application, UI, backend, API, SDK or production functionality was added, removed or altered.** Gitea #59: committed-lock Linux/Windows clean install PASS; Node SDK builds PASS; all 24 TS suites exit0 with zero timeouts; Next standalone 22/22 pages PASS. One historical #59 test-only declaration TS error fixed; #60 strict TS zero errors and full ESLint PASS. Previous canonical Python SDK 14/14 (#48) preserved but not rerun. No live endpoint mutation or deployment; live data/Redis/S3/MCP actual operational status is not established.
|
||||||
- P02: pending.
|
- P02: **BLOCKED pending separate approved runtime remediation**. Three disposable fault-injection tests demonstrated existing behavior: positive ACK on Redis write failure with RAM-only data (R-01), duplicate appended event ID after Redis trim failure and retry in web-side buffer (R-02), and public health 200/online when Redis is down (R-03). These are **diagnostic tests, not production fixes**. Zero app/API/UI/SDK/runtime changes, no live deploy. CI #63. P01 remains closed (1/3 phases complete).
|
||||||
- P03: pending.
|
- P03: pending.
|
||||||
|
|
||||||
|
## P02 Diagnostic Feature Record — 2026-11-22
|
||||||
|
|
||||||
|
- **New production features: NONE.** This phase adds only three disposable failure-injection scenarios across two existing test suites; tests are not production handlers. Existing source/UI/auth/backend/SDK/MCP, Redis/PG/CSV services, Next/Express health interfaces and deployment remain unmodified.
|
||||||
|
- R-01: existing volatile RAM fallback and WS ACK behavior audited, *not corrected*. Do not mark ACK durability as working merely because a diagnostic test proves current behavior.
|
||||||
|
- R-02: existing CSV-before-Redis-trim order audited, *not corrected*. Exactly-once replay safety is **NOT VERIFIED**.
|
||||||
|
- R-03: current public Express health 200 plus `redisConnected:false` is liveness, not dependency readiness. Separate readiness remains **NOT IMPLEMENTED**.
|
||||||
|
- Gitea disposable CI #63: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63. The previous P01 working results remain valid; no live read/write test or production deploy is included in P02.
|
||||||
@@ -200,3 +200,40 @@ Remaining CI-05 corrections were confined to 9 test file type contracts; CI-04 r
|
|||||||
- [Run #48](https://dev.ivib.cc/vibtools/DataStack/actions/runs/48): previous Python SDK 14/14 PASS, no Python source edits or P01 rerun.
|
- [Run #48](https://dev.ivib.cc/vibtools/DataStack/actions/runs/48): previous Python SDK 14/14 PASS, no Python source edits or P01 rerun.
|
||||||
|
|
||||||
Targeted path/diff audit from prior documentation HEAD: 16 scoped code/test/CI paths plus notes `001–006`; `git diff --check` PASS. P02 remaining R-01…R-03 are conditional reliability risks, not confirmed application bugs. P03 Docker/Python-runner/advisory issues still open. No live app, deployment, database, security defaults or user features changed.
|
Targeted path/diff audit from prior documentation HEAD: 16 scoped code/test/CI paths plus notes `001–006`; `git diff --check` PASS. P02 remaining R-01…R-03 are conditional reliability risks, not confirmed application bugs. P03 Docker/Python-runner/advisory issues still open. No live app, deployment, database, security defaults or user features changed.
|
||||||
|
|
||||||
|
|
||||||
|
## E. P02 Concrete Code/CI Evidence — 2026-11-22
|
||||||
|
|
||||||
|
**Frozen previous phase:** P01 Gitea/GitHub commit `667a140c4e9e75382bfdcdbe1d3e9eb96f6e808a`. P02 test-only commit `bbe73d91666641743cbec6cf7203f1694094caef` adds 103 lines across exactly two approved test paths. Gitea CI #63: https://dev.ivib.cc/vibtools/DataStack/actions/runs/63. No application source modification.
|
||||||
|
|
||||||
|
### R-01 ACK after Redis RPUSH failure
|
||||||
|
|
||||||
|
`server/index.js:171-192`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
try { bufferLen = await redis.rpush(REDIS_BUFFER_KEY, ...serialized); }
|
||||||
|
catch { redisConnected = false; }
|
||||||
|
// ... Redis failure path
|
||||||
|
bufferQueue.push(...events);
|
||||||
|
return bufferQueue.length;
|
||||||
|
```
|
||||||
|
|
||||||
|
`server/socket/handlers.js:583-602` invokes `await onBufferPush()` and then calls `callback({success:true,eventId})` unless an exception escapes. Test: `tests/phase02-sync.test.ts`, `P02 R-01 diagnostic`, overriding fake Redis `rpush` to reject; check ACK, absent Redis persistence, volatile RAM, and simulated reset. No real process exit.
|
||||||
|
|
||||||
|
### R-02 replay after append-before-trim
|
||||||
|
|
||||||
|
`core/data/buffer.ts:795-809` and Port-4000 `server/index.js:981-993` order `appendBatch()/writeBatch()` before `redis.ltrim()`. Test: `tests/phase03-storage-buffer.test.ts`, `P02 R-02 diagnostic`; inject the *first* LTRIM failure only after successful append, verify still queued, retry flush and count duplicate `eventId` in appended rows. Web-side representative fault-injection; not a complete ingestion-crash certificate.
|
||||||
|
|
||||||
|
### R-03 public status versus dependency readiness
|
||||||
|
|
||||||
|
`server/index.js:1926-1957` catches Redis LLEN error, sets `redisConnected=false`, and returns HTTP 200 `status:online`. Test `P02 R-03 diagnostic` mocks LLEN failure and recovery. This preserves existing public shape rather than implementing a readiness endpoint. PG startup fail-closed coverage remains source-only/NOT VERIFIED.
|
||||||
|
|
||||||
|
**Patch scope:** modified only `tests/phase02-sync.test.ts` and `tests/phase03-storage-buffer.test.ts` (plus notes/001–006). Original local user-staged P01 work preserved. Runtime files examined read-only. No production host, Redis, DB, CSV, API, service, CI workflow, release, or security setting changed.
|
||||||
|
|
||||||
|
|
||||||
|
### CI #63 observed P02 results
|
||||||
|
|
||||||
|
- `P02 R-01 diagnostic: Redis RPUSH failure yields positive WS ACK for volatile RAM-only event` — **ok 15**, test suite `phase02-sync` exit 0. Diagnostic indicates acknowledged event loss on simulated RAM reset when Redis has no record.
|
||||||
|
- `P02 R-03 diagnostic: Redis query failure leaves public ingestion health 200/online` — **ok 16**, same suite exit 0, Redis flag false while liveness stays online.
|
||||||
|
- `P02 R-02 diagnostic: post-CSV Redis trim failure replays identical event on retry` — **ok 18**, `phase03-storage-buffer` suite exit 0; duplicate append after retry reproduced on representative web buffer path. **Direct ingestion Port 4000 CSV worker crash coverage remains outstanding** and should be requested in remediation approval before guarantee changes.
|
||||||
|
- These three diagnostic tests **prove findings, do not close production-readiness**. See Gitea https://dev.ivib.cc/vibtools/DataStack/actions/runs/63; never call run successful until terminal conclusion is confirmed. No server-side source fix applied.
|
||||||
@@ -1220,3 +1220,64 @@ test("R2 PHASE-02 BUG-10: GET, PATCH (regenerate_token), and DELETE /api/clients
|
|||||||
SDKManager.deleteClientWithStatsAsync = origDeleteWithStats;
|
SDKManager.deleteClientWithStatsAsync = origDeleteWithStats;
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("P02 R-01 diagnostic: Redis RPUSH failure yields positive WS ACK for volatile RAM-only event", async () => {
|
||||||
|
// Disposable test-only fixture: no production Redis, service, or storage state.
|
||||||
|
const previousThreshold = serverMod.runtimeConfig.batchFlushThreshold;
|
||||||
|
const previousRam = serverMod.bufferQueue.slice();
|
||||||
|
const failingRedis = createSharedMockRedis();
|
||||||
|
failingRedis.rpush = async () => {
|
||||||
|
throw new Error("P02 injected Redis RPUSH outage");
|
||||||
|
};
|
||||||
|
serverMod.setRedisClientForTests(failingRedis);
|
||||||
|
serverMod.runtimeConfig.batchFlushThreshold = 100_000;
|
||||||
|
serverMod.bufferQueue.length = 0;
|
||||||
|
let socket: Socket | undefined;
|
||||||
|
try {
|
||||||
|
const client = await SDKManager.registerClientAsync({
|
||||||
|
clientId: "p02_redis_rpush_failure",
|
||||||
|
name: "P02 Redis RPUSH Failure",
|
||||||
|
rateLimit: 100,
|
||||||
|
});
|
||||||
|
socket = await connectSocket(client.token);
|
||||||
|
const ack = await emitWithAck<{ success: boolean; eventId?: string; code?: string }>(
|
||||||
|
socket, "sync_data",
|
||||||
|
{ eventId: "p02-ack-before-crash-001", data: { value: 42 } },
|
||||||
|
);
|
||||||
|
const committedRows = await failingRedis.lrange("datastack:buffer:events", 0, -1);
|
||||||
|
const volatileRows = serverMod.bufferQueue.filter((e) => e.eventId === "p02-ack-before-crash-001");
|
||||||
|
assert.equal(ack.success, true, "Existing handler currently sends a positive ACK after Redis failure");
|
||||||
|
assert.equal(committedRows.length, 0, "Fake Redis has no durable event");
|
||||||
|
assert.equal(volatileRows.length, 1, "Acknowledged event is RAM-only");
|
||||||
|
// Crash model: restart loses RAM but fake Redis survives. No actual process is killed.
|
||||||
|
serverMod.bufferQueue.length = 0;
|
||||||
|
assert.equal((await failingRedis.lrange("datastack:buffer:events", 0, -1)).length, 0);
|
||||||
|
} finally {
|
||||||
|
socket?.disconnect();
|
||||||
|
serverMod.setRedisClientForTests(undefined);
|
||||||
|
serverMod.runtimeConfig.batchFlushThreshold = previousThreshold;
|
||||||
|
serverMod.bufferQueue.splice(0, serverMod.bufferQueue.length, ...previousRam);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("P02 R-03 diagnostic: Redis query failure leaves public ingestion health 200/online", async () => {
|
||||||
|
const redis = createSharedMockRedis();
|
||||||
|
redis.llen = async () => { throw new Error("P02 simulated Redis health outage"); };
|
||||||
|
serverMod.setRedisClientForTests(redis);
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${ingestionUrl}/health`);
|
||||||
|
assert.equal(res.status, 200, "Legacy liveness status remains backwards-compatible");
|
||||||
|
const body = await res.json() as { status: string; redisConnected: boolean; redisBufferSize: number };
|
||||||
|
assert.equal(body.status, "online");
|
||||||
|
assert.equal(body.redisConnected, false, "Redis failure is represented only in the payload");
|
||||||
|
assert.equal(body.redisBufferSize, 0);
|
||||||
|
const healthyRedis = createSharedMockRedis();
|
||||||
|
serverMod.setRedisClientForTests(healthyRedis);
|
||||||
|
const recovered = await fetch(`${ingestionUrl}/health`);
|
||||||
|
const recoveredBody = await recovered.json() as { redisConnected: boolean };
|
||||||
|
assert.equal(recovered.status, 200);
|
||||||
|
assert.equal(recoveredBody.redisConnected, true);
|
||||||
|
} finally {
|
||||||
|
serverMod.setRedisClientForTests(undefined);
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -1549,3 +1549,45 @@ test("R2 PHASE-03 BUG-15: GET /api/export rejects inverted date range with 400 a
|
|||||||
assert.match(serverIndexSrc, /const filterClientId =\s*rawClientId && rawClientId !== 'all' \? normalizeClientId\(rawClientId\) : ''/);
|
assert.match(serverIndexSrc, /const filterClientId =\s*rawClientId && rawClientId !== 'all' \? normalizeClientId\(rawClientId\) : ''/);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
|
test("P02 R-02 diagnostic: post-CSV Redis trim failure replays identical event on retry", async () => {
|
||||||
|
const redis = new FakeRedisClient();
|
||||||
|
const realTrim = redis.ltrim.bind(redis);
|
||||||
|
let injected = false;
|
||||||
|
redis.ltrim = async (key: string, start: number, stop: number): Promise<"OK"> => {
|
||||||
|
if (key === REDIS_BUFFER_KEY && !injected) {
|
||||||
|
injected = true;
|
||||||
|
throw new Error("P02 simulated crash boundary after CSV append, before Redis trim");
|
||||||
|
}
|
||||||
|
return realTrim(key, start, stop);
|
||||||
|
};
|
||||||
|
const event: DataEvent = {
|
||||||
|
clientId: "p02_csv_replay",
|
||||||
|
eventId: "p02-replay-001",
|
||||||
|
timestamp: "2026-11-22T01:00:00.000Z",
|
||||||
|
payload: JSON.stringify({ marker: "p02" }),
|
||||||
|
};
|
||||||
|
DataBufferEngine.setRedisClientForTests(redis);
|
||||||
|
const originalAppend = CSVStorageManager.appendBatch;
|
||||||
|
const writtenIds: string[] = [];
|
||||||
|
CSVStorageManager.appendBatch = async (events) => {
|
||||||
|
const result = await originalAppend.call(CSVStorageManager, events);
|
||||||
|
writtenIds.push(...events.map((row) => row.eventId));
|
||||||
|
return result;
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
await DataBufferEngine.pushBatchAsync([event]);
|
||||||
|
assert.equal(await redis.llen(REDIS_BUFFER_KEY), 1);
|
||||||
|
await assert.rejects(() => DataBufferEngine.flush(), /P02 simulated crash boundary/);
|
||||||
|
assert.equal(injected, true);
|
||||||
|
assert.equal(writtenIds.filter((id) => id === event.eventId).length, 1);
|
||||||
|
assert.equal(await redis.llen(REDIS_BUFFER_KEY), 1, "Redis still queues previously appended CSV event");
|
||||||
|
await DataBufferEngine.flush();
|
||||||
|
assert.equal(await redis.llen(REDIS_BUFFER_KEY), 0);
|
||||||
|
assert.equal(writtenIds.filter((id) => id === event.eventId).length, 2,
|
||||||
|
"A second flush appends the same eventId again after failed trim: at-least-once, not exactly-once");
|
||||||
|
} finally {
|
||||||
|
CSVStorageManager.appendBatch = originalAppend;
|
||||||
|
DataBufferEngine.setRedisClientForTests(undefined);
|
||||||
|
}
|
||||||
|
});
|
||||||
Reference in new issue
Block a user