docs: freeze DataStack production readiness forensic roadmap

This commit is contained in:
victorsteele428 committed 2026-11-22 07:13:44 +06:00
1 parent d725841ebc
commit d8ac96e39d
6 files changed
+436

No files matched your search

+66
View File
@@ -0,0 +1,66 @@
# DataStack — Production-Readiness Forensic Audit (Documentation Baseline)
Status: **DOCUMENTED / NO BUG FIX APPLIED**. Date: 2026-11-22 (Asia/Dhaka).
Frozen source HEAD: `d725841ebc828d58cbe90f517da9e600f23c4630`. Local, Gitea `vibtools/DataStack`, GitLab `ivibtools/datastack`, and GitHub `JILLU-Games/DataStack` confirmed at this exact HEAD before writing these notes.
Scope of this document: **evidence-backed CI and source review**, not proof that every production feature works. No Coolify live write, rollback, restart or deployment was conducted. Public GET `https://data.ivib.cc/api/health` returned `status: online`; full DB/Redis/MCP/S3 transaction health is not established.
## Findings classification
- **8 finding groups** from actual CI: **5 code/test/lockfile groups** (CI-01…05) and **3 CI infrastructure/security-review groups** (CI-06…08). **Not 43 separate bugs**: the 43 compiler errors belong to CI-05.
- **2 failing assertions, 4 timeout suites** (24 TypeScript suites in run #45: 18 clean, 6 fail/timeout).
- **3 extra source-observed reliability risks** (R-01…03) have NOT been reproduced as production failures; they require targeted fault injection before any fix or behavior change.
- **Do not equate successful test/source feature with verified production runtime.**
## Eight confirmed findings / blockers
| ID | Severity* | Evidence (run/job) | Exact affected area | Verified observation | Planned phase |
|---|---|---|---|---|---|
| CI-01 | High | Run #42 job 63; #45 job 71 | `package-lock.json`, `package.json`; `.gitea/workflows/datastack-ci.yml` Windows job | Original Windows `npm ci` reports missing platform dependency entries; regenerate lock **only in runner** -> SDK builds. Lock not fixed in repo. | P01 |
| CI-02 | Medium | Run #45 job 66 | `bun.lock` and `tests/phase07-deployment.test.ts:648-649` | Tracked `bun.lock` conflicts with assertion requiring its absence. Validate references before removing. | P01 |
| CI-03 | Medium | Run #45 job 66 | `tests/r3-phase04-ui-metrics-rbac.test.ts`, `tests/r4-phase03-sdk-deploy-auth.test.ts`, `tests/r5-phase01-mcp-foundation-logs.test.ts`, `tests/r5-phase03-mcp-ui.test.ts` | Each suite timed out with exit **124** despite reaching passing assertions; investigate open handles/teardown without changing application timers by default. | P01 |
| CI-04 | Medium | Run #45 job 66 | `tests/v2-phase01-storage-quotas.test.ts:788-804` vs `app/vcon/clients/page.tsx:15-20,70-95` and `lib/vconUiHelpers.ts:717-780` | Old regex expects `function isClientExpired(` in page; production page uses shared `classifyUiClientStatus` and `summarizeUiClientCounts`. Test contract is stale; not proof expiry feature fails. | P01 |
| CI-05 | Medium | Run #45 job 68 | Nine `tests/*.test.ts` files (listed below), `tsconfig.json:27` | `npx tsc --noEmit` produced **43 compiler errors**, despite Next.js production build PASS. Type errors in test sources require focused correction; do not disable strict TypeScript. | P01 |
| CI-06 | Low/CI | Run #45 job 67, run #48 job 74 | `.gitea/workflows/datastack-ci.yml` Python job | Linux Bullseye `apt-get` returned Debian-security 404; separate Windows Python job passed **14/14**. Runner provisioning defect, not SDK regression. | P03 |
| CI-07 | High/CI | Run #50 job 76 | `.gitea/workflows/datastack-docker-ci.yml`, Docker daemon; `docker/Dockerfile.web:13` | Compose validation and ingestion image PASS, then web image layer copy fails with containerd `lstat .../app/node_modules/@uiw: no such file or directory`. Root cause **unconfirmed** (runner snapshot/storage or reproducible build interaction); proxy never built. | P03 |
| CI-08 | Review | Run #50 job 76 | `package.json` / `package-lock.json` / resolved npm graph | npm reports **24 (23 moderate, 1 high)** with `--omit=dev`, **26 (25 moderate, 1 high)** with `--include=dev`. Counts overlap; no CVE list, reachable-exploit assessment or remediation done. | P03 |
*Severity is release-priority triage, NOT a CVSS rating and NOT proof of real-world exploitation.
### CI-05 exact file map (43 total, all inside tests)
| File | Error count | Families / main anchors |
|---|---:|---|
| `tests/phase01-auth.test.ts` | 4 | TS2540; 88,560,773,794 |
| `tests/phase03-storage-buffer.test.ts` | 2 | TS2349; 1158,1202 |
| `tests/r3-phase01-security-vault-webhooks.test.ts` | 6 | TS2353, TS2345; 185,191,202,210,295,430 |
| `tests/r3-phase02-storage-scheduler-gzn.test.ts` | 1 | TS2345; 49 |
| `tests/r4-phase01-security-boot.test.ts` | 3 | TS2345; 102,537,571 |
| `tests/r4-phase02-ingestion-audit-streaming.test.ts` | 9 | TS2345/TS2833; 87,332,335,338,407 |
| `tests/r5-phase02-mcp-server.test.ts` | 12 | TS18046; 510,524,538,549,560,570,581,604,613,635,644,657 |
| `tests/v2-phase01-storage-quotas.test.ts` | 2 | TS2554; 762,783 |
| `tests/v2-phase02-alerts-ip-prometheus.test.ts` | 4 | TS2339; 479,480,490,491 |
No source fix is authorized here. Use typed narrowing and correct test fixtures, not `skipLibCheck`, `any` everywhere, `@ts-ignore`, or changing real API responses merely to make tests compile.
## Three source-observed production reliability risks — CONDITIONAL GATES, not confirmed bugs
| Gate | Source proof | Why verification is required | Phase |
|---|---|---|---|
| R-01 Redis outage ACK durability | `server/index.js:165-193`, `server/socket/handlers.js:583-602` | `redis.rpush` failure falls back to RAM, and `onBufferPush()` can resolve, then Socket.IO ACK success; process crash before flush may lose acknowledged events. Fail-injection test and explicit durability contract required. | P02 |
| R-02 Batch replay duplication | `server/index.js:943-994`, `server/workers/csvWriter.js:265-323` | CSV write precedes Redis list trim. Crash/partial write between these operations can replay already written rows. Validate at-least-once vs exactly-once guarantee before mitigation. | P02 |
| R-03 Liveness ≠ readiness | `server/index.js:1939-1956`; `app/api/health/route.ts:95-111` | Health returns online without full PostgreSQL + Redis + disk transaction proof. Preserve existing `/health` API semantics; consider separate, internal readiness rather than breaking public health. | P02 |
Optional configuration review (not counted among eight): `server/index.js:1086-1089` Socket.IO uses CORS delegate; verify WebSocket origin/access and JWT guard in target deployments before recommending policy changes. `sdk/python/datastack/pyproject.toml:47-51` still references legacy `anik-d-art/datastack` URLs; metadata only, not current release blocker. Do not silently modify either under approved scope.
## CI evidence / original logs
- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/42` — initial Windows lockfile failure (job 63), cancelled after partial node evidence.
- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/45` — 24 suites = 18 clean / 6 fail-or-timeout, 43 TS diagnostics, Next.js production build PASS, Windows diagnostic SDK build PASS; Linux Python apt and Docker-CLI job blocked.
- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/48` — Python SDK 14/14 PASS.
- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/49` — full ESLint PASS.
- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/50` — Compose PASS, Docker ingestion image PASS, web image containerd failure.
- See `notes/006-BASELINE-AND-PATCH-EVIDENCE.md` for precise code/log excerpts and proposed patch anchors.
## Release acceptance rule
NOT production-update-ready until P01, conditional P02 gates, and P03 finish, 24/24 TypeScript suites exit 0, 14/14 Python tests pass, full strict `tsc --noEmit` and ESLint pass, Linux/Windows clean install is reproducible, standalone Next build passes, all 3 Docker images build, security advisories are triaged, and approved **non-mutating** live readiness checks pass. Production deploy is a separate operator-approved action; never automatic from this documentation phase.
+72
View File
@@ -0,0 +1,72 @@
# DataStack — Production Update Roadmap / Exact Scope Lock
Planning baseline: `d725841ebc828d58cbe90f517da9e600f23c4630` (frozen; documentation commits will advance Git HEAD but NOT this source baseline).
Current state: **3 implementation phases P01–P03 planned; 0 implemented; approvals not yet granted**.
Prerequisite for any phase: compare current HEAD and changed paths with this baseline and previously committed notes. For new commits since source baseline, inspect exact diff; no implicit rebase or silent discard.
## Global non-negotiable Scope Lock (applies to ALL phases)
1. Existing UI/UX, public API response contracts, authentication, RBAC, JWT/SDK interoperability, MCP permissions, data retention, CSV content, persistent databases, keys, Redis values, S3/R2 objects, Coolify configuration and production deployment: **unchanged unless an exact targeted fix is proven and separately approved**.
2. No live event sending, live database mutation, destructive test, production restart/deploy, automatic package vulnerability `--force`, tag or release without explicit approval.
3. Limit changes to exact allowlisted paths per phase. Any additional path, altered schema, migration, UI behavior, API code, or security default = STOP / document scope exception / obtain new approval. No broad refactoring.
4. One phase = targeted patch, **one focused CI verification**, exact diff/security review, log/register updates, one Git commit and 3-remote sync verification. No repetitive test loops or local production builds. If CI infra is broken, report a blocker rather than spoof PASS.
5. Phase close requires written evidence with exit codes, commit SHA, changed paths, feature preservation checklist, remaining issues and a next-phase approval command.
6. Reference test logs and source code, not README assertions alone. Code candidate risks require fault reproduction before production behavior changes.
7. Secrets, personal data, local `.env` and runtime tokens never copied into reports, logs or commits. Never perform unsafe `git reset --hard` / `git clean -fd`.
8. `AGENTS.md` cross-IDE memory requirement: update `MEMORY.md` only under explicit scope exception, otherwise provide exact handoff Markdown in `notes/003-PHASE-STATUS-LOG.md` for copy/sync.
## Phase P01 — Dependency, tests & type-contract reconciliation (CI-01…CI-05)
**Goal:** fix confirmed test/lockfile blockers without changing actual app/SDK runtime behavior.
**Allowed paths only:**
- `package-lock.json`; `bun.lock` (delete ONLY after verifying no build consumers; npm lock stays canonical).
- `tests/phase07-deployment.test.ts`; `tests/v2-phase01-storage-quotas.test.ts`;
- `tests/r3-phase04-ui-metrics-rbac.test.ts`; `tests/r4-phase03-sdk-deploy-auth.test.ts`; `tests/r5-phase01-mcp-foundation-logs.test.ts`; `tests/r5-phase03-mcp-ui.test.ts`;
- `tests/phase01-auth.test.ts`; `tests/phase03-storage-buffer.test.ts`; `tests/r3-phase01-security-vault-webhooks.test.ts`; `tests/r3-phase02-storage-scheduler-gzn.test.ts`;
- `tests/r4-phase01-security-boot.test.ts`; `tests/r4-phase02-ingestion-audit-streaming.test.ts`; `tests/r5-phase02-mcp-server.test.ts`; `tests/v2-phase02-alerts-ip-prometheus.test.ts`;
- `.gitea/workflows/datastack-ci.yml` (narrow clean-install/test harness only); `notes/001…006` (status/evidence only).
**Not allowed:** `app/**`, `components/**`, `lib/**`, `core/**`, `server/**`, `docker/**`, `tsconfig.json`, `package.json` or SDK runtime files. If open-handle leak proves core timer bug, STOP and request explicit additional exception for `core/data/buffer.ts` instead of altering it as a side effect.
**Patch tasks and references:**
- CI-01: deterministic regenerated `package-lock.json` with cross-OS optional entries, verify clean `npm ci` on Linux/Windows with same pinned npm; no dependency version bump unless separately approved. Reference `.gitea/workflows/datastack-ci.yml:91-120`.
- CI-02: check `bun.lock` consumers, then remove stale lock or resolve test contract with proof; `tests/phase07-deployment.test.ts:649`.
- CI-03: isolate 4 suites, enumerate open handles/intervals, add explicit test teardown / mocks; source `core/data/buffer.ts:347-391` is **read-only diagnostic reference**. Avoid `process.exit()` hacks and wholesale timeout increases.
- CI-04: replace obsolete regex tests in `tests/v2-phase01-storage-quotas.test.ts:788-804` with precise shared-helper/integration assertions matching `app/vcon/clients/page.tsx:70-95` and `lib/vconUiHelpers.ts:717-780`.
- CI-05: fix 43 test-code type errors across 9 listed files (see `001`). Keep strict TS compile; preserve production API signatures. `tsconfig.json:27` includes tests by glob.
- **P01 acceptance:** 24/24 test files exit 0 (no timeout), `tsc --noEmit` zero errors, `npm run lint` PASS, Python 14/14 PASS regression, Windows/Linux original `npm ci` PASS without diagnostic lock regeneration; one focused Gitea matrix. If network/runner blocker, mark BLOCKED not PASS.
- **Close:** update `notes/003` and `notes/005` only after verifying; commit and sync.
## Phase P02 — Ingestion reliability / error-handling acceptance gates (R-01…R-03)
**Goal:** reproduce or dismiss source-observed risk, preserve response semantics unless proven fix is necessary. **Risk ≠ bug until test fails.**
**Diagnostic-only allowlist:** `server/index.js`, `server/socket/handlers.js`, `server/workers/csvWriter.js`, `core/data/buffer.ts`, `app/api/health/route.ts`, `tests/phase02-sync.test.ts`, `tests/phase03-storage-buffer.test.ts`, `tests/r4-phase02-ingestion-audit-streaming.test.ts`, `notes/001…006`.
**Default write allowlist:** only the THREE listed test files plus `notes/001…006`. Runtime source files above are **READ ONLY until a reproducible failing test demonstrates the need**; then the P02 approval must name an exact method/line range and explicitly authorize the smallest required change. This is not preapproval for changing server source.
**Test matrix (disposable, never production):**
- R-01: fake Redis `rpush` rejection + ACK + simulated process exit/reopen; ensure API explicitly returns failed/pending unless durable, or document intended guarantee. Preserve existing duplicate/rate/size/auth error codes.
- R-02: injected CSV partial write, simulated failure before Redis `ltrim`, restart and compare event IDs/rows; decide documented at-least-once semantics vs dedupe strategy before altering storage.
- R-03: separate liveness/readiness checks with mocked Redis/PG errors; old public `/health` stays backwards compatible. Verify startup fail-closed when secrets or DB unavailable.
- Review source CORS/WebSocket origin and security access policy as **read-only**, no default policy change automatically.
- **P02 acceptance:** targeted fault-injection tests PASS, no acknowledged-event loss in the documented guarantee, no unbounded duplicate amplification, correct 503/507/429/401 semantics and recovery. If risk not reproduced, close as verified with **zero runtime source changes**; never invent a fix.
- **Close:** record precise outcome and next P03 approval.
## Phase P03 — CI runner, Docker images, dependency security, release evidence (CI-06…CI-08)
**Goal:** reproducible Gitea verification, safe dependency advisory triage, complete all three Docker builds, confirm non-mutating production health.
**Default write allowlist:** `.gitea/workflows/datastack-ci.yml`, `.gitea/workflows/datastack-python-ci.yml`, `.gitea/workflows/datastack-docker-ci.yml`, `.gitea/workflows/datastack-eslint-ci.yml`, `package.json`, `package-lock.json`, `notes/001…006`.
**Conditional, separate approval only:** `docker/Dockerfile.web` / `docker/Dockerfile.server` if isolated runner/containerd evidence establishes source-level Dockerfile defect. **Not allowed** to change Compose deployment services, secrets, volumes or Dockerfiles merely to work around a runner failure.
**Tasks:**
- CI-06: replace stale Debian Bullseye `apt-get` dependency step with preinstalled compatible Python image/toolchain OR canonical Windows Python job; avoid duplicate CI runs.
- CI-07: inspect runner daemon snapshotter/filesystem free space, cache/BuildKit and known `node_modules/@uiw` layer failure; use properly configured disposable BuildKit builder or another isolated runner. If reproducible Dockerfile defect is proven, obtain narrow scope amendment. Build images **without running containers**; no `docker system prune` or shared cache destruction.
- CI-08: run read-only detailed `npm audit --json` in isolated runner, map actual direct/transitive paths and runtime impact, document advisory IDs/severity, apply minimal compatible pin bumps only where necessary. Never `npm audit fix --force`.
- Verify canonical package metadata and Git remote SHA; GitHub login/account issue is **already resolved** and is not a current blocker.
- **P03 acceptance:** all 3 images build, Compose config PASS, Next standalone PASS, lint + strict TS PASS, 24/24 TS suites no hang, Python 14/14, regression error codes unchanged, advisory disposition documented, read-only public health PASS. Any real deployment/production migration is a **separate approval**.
- **Close:** exact diff, one CI acceptance run (plus only blocked job rerun where necessary), statuses/feature matrix update, one commit, synced remote HEADs, phase CLOSED.
## Sequence and approval gate
P01 -> P02 -> P03, strictly in this order. **No P01 implementation has begun** in this documentation-only update.
Approval command for next turn:
`APPROVED — DataStack Production Readiness P01. Baseline: d725841ebc828d58cbe90f517da9e600f23c4630. Implement ONLY notes/002 P01 exact Scope Lock; preserve all app, UI, SDK, backend, live data, deployment and unrelated files. One focused disposable Gitea CI, exact diff/security audit, notes/003–005 update, one commit, push GitHub/GitLab/Gitea and verify all HEADs. No live deployment, tags, releases or scope expansion.`
+52
View File
@@ -0,0 +1,52 @@
# DataStack — Phase Status & Change Log (Source of Progress Truth)
Frozen source baseline: `d725841ebc828d58cbe90f517da9e600f23c4630`. Generated: 2026-11-22 Asia/Dhaka.
Scope: documentation-only kickoff. Any future phase must update this log **after** verification, not on intention.
**Completed implementation phases: 0 / 3. Remaining: 3.** Documentation baseline preparation is a separate non-implementation milestone.
| Phase | Status | IDs / function | Changed source | Verified CI evidence | Commit / remote sync |
|---|---|---|---|---|---|
| P00 Notes / freeze | COMPLETED (documentation only) | `001–006` docs, source baseline and exact scope | `notes/**` only | 6 Markdown files; balanced fences; staged diff `--check` PASS | Documentation commit containing this file; remote SHA verification recorded in session handoff |
| P01 Tests & lockfile | NOT STARTED | CI-01,02,03,04,05 | None | Not run | — |
| P02 Ingestion reliability | NOT STARTED | Conditional R-01,02,03 | None | Not run | — |
| P03 CI/Docker/security | NOT STARTED | CI-06,07,08; release gates | None | Not run | — |
## Existing diagnostic CI snapshot (NOT implementation phase completions)
| Run | Baseline commit | Result | Meaning |
|---|---|---|---|
| #42 | `f7c1e7f` | Cancelled | Windows original lock fail logged; long combined regression stopped |
| #45 | `d3a93c6` | Failure | TS 18 clean/6 fail-or-timeout; 43 type errors; build and Windows SDK pass; Python apt / Docker CLI blockers |
| #48 | `ff1333d` | Success | Python SDK 14/14 |
| #49 | `559f180` | Success | ESLint |
| #50 | `d725841` | Failure | Compose + ingestion image pass; web image fails containerd snapshot copy; proxy unrun |
Historical baseline code stayed frozen through these CI workflow changes. Current HEAD before documentation changes = `d725841ebc828d58cbe90f517da9e600f23c4630` on local/GitHub/GitLab/Gitea.
## Per-phase closure record template — COPY for each P01/P02/P03
### P0X — [TITLE] (OPEN / BLOCKED / CLOSED)
- Authorized baseline / parent SHA:
- Approval command / exact scope document reference:
- Findings addressed + any conditional gates:
- Exact modified paths (one path per row):
- Applied code patch summary (function + lines):
- Existing feature/UI/API/auth/SDK/data preservation audit:
- Focused CI run URL / test counts / exit codes:
- Security check / handling of secrets, data and scope exceptions:
- Verification result: PASS / FAIL / BLOCKED, not assumed:
- Closure commit SHA + GitHub/GitLab/Gitea main SHAs:
- Completed implementation phases: X/3; remaining: Y.
- Next phase: P0Y; **specific approved features** / allowed files:
- New issues / deferred risks / notes evidence links:
- MEMORY.md cross-IDE handoff text (copyable):
`[DataStack P0X] Baseline=<SHA>; findings=<IDs>; scope=<exact paths>; CI=<run + exit>; commit=<SHA>; remote sync=<all 3>; phase=CLOSED; next=P0Y; production deployment=unchanged.`
## Status transition rules
- OPEN -> IN PROGRESS only with explicit approval and baseline SHA match.
- IN PROGRESS -> CLOSED only after one focused CI and exact diff audit, updated `003` + `005` + relevant `001/004/006`, Git commit, sync, and confirmed remote SHA.
- Failed test -> BLOCKED with log excerpt and scope decision; never log as complete.
- Unexpected file change -> STOP and revert only own documented changes safely; never touch pre-existing work.
- At any time, `actual production deployment status` is a separate tracked dimension (do not infer deployed SHA from Git HEAD).
+47
View File
@@ -0,0 +1,47 @@
# DataStack — Error Handling Inventory & Proposed Safety Gates
Source baseline: `d725841ebc828d58cbe90f517da9e600f23c4630`. This is an **implementation inventory**, NOT an independently executed production fault-injection certificate. Existing codes and APIs below are source-grounded.
## Existing handlers: observed code paths
| Failure / scenario | Exact path and lines | Current response / behavior | CI coverage/status |
|---|---|---|---|
| Missing/bad client ID | `app/api/events/route.ts:433-442` | `400 VALIDATION_ERROR` | TS suites; no live write |
| Deleted/missing/revoked client | `app/api/events/route.ts:445-474` | `403 CLIENT_REVOKED` | Auth regression passed in suite |
| Rotated token | `app/api/events/route.ts:476-487` | `401 TOKEN_ROTATED` | Auth/SDK regression coverage |
| Expired client token | `app/api/events/route.ts:489-504` | `401 CLIENT_TOKEN_EXPIRED` + webhook trigger | Python SDK tests pass |
| Disallowed IP | `app/api/events/route.ts:506-526`; `app/api/health/route.ts:82-91` | `403 CLIENT_IP_FORBIDDEN` | Source + TS regression |
| Disk low / client quota exhausted | `core/Storage/csvStorage.ts:1159-1193`; `app/api/events/route.ts:567-589` | `507 DISK_SPACE_LOW` or `STORAGE_QUOTA_EXCEEDED`; webhook | Python 14/14, TS related suite passed except stale UI assertion |
| Duplicate, payload oversize, rate limit | `app/api/events/route.ts:633-681` | dedupe; `413 PAYLOAD_TOO_LARGE`, `429 RATE_LIMIT_EXCEEDED`, `400 BATCH_REJECTED` | Regression suites |
| Socket buffer persistence exception | `server/socket/handlers.js:583-602` | `BUFFER_PERSIST_FAILED` callback **if onBufferPush throws**; success ACK after normal resolution | R-01 requires RAM-fallback failure injection |
| Redis distributed flush lock | `app/api/storage/route.ts:102-127` and :187-197 | `503 FLUSH_LOCK_TIMEOUT` on guarded flush | Existing TS test pass |
| Webhook disabled/blocked/redirect/timeout | `core/alerts/webhookNotifier.ts:94-176` | SSRF URL check, HMAC, timeout, manual redirect policy, per-client cooldown | Source + TS security tests |
| Admin cookie integrity/revocation | `core/auth/adminSession.ts:70-154`; `app/api/auth/**` | HMAC verified session with tokenVersion support; 401/403 boundaries | Auth TS regression passed |
| MCP permissions / filesystem sandbox | `server/mcp/mcpEngine.js:12-50,167-248` | 17 granular permissions; writes disabled default; path jail/symlink rejects | MCP server suite passed |
| Python SDK typed error mapping | `sdk/python/datastack/datastack/exceptions.py:8-84` | Auth/config/quota/payload/file errors mostly non-retryable, connection/rate retryable | Windows Python CI 14/14 |
| Public liveness health | `server/index.js:1939-1956`; `app/api/health/route.ts:95-111` | `online` reflects responder; not full transactional readiness | Public endpoint GET online only |
## Missing / not proven handling — proposals, NOT implemented
| Gate | Existing evidence | Narrow proposed addition | Risk / invariant |
|---|---|---|---|
| EH-01 Durable ACK under Redis loss | `server/index.js:165-193` RAM fallback; `server/socket/handlers.js:583-602` ACK | Test Redis RPUSH failure followed by crash. If confirmed, add reliable negative ACK/persist-queue contract with focused regression. | Preserve SDK fallback/spooling & existing auth/ack fields |
| EH-02 Partial CSV replay duplication | `server/index.js:981-994` write before trim | Crash/replay fault test; clarify at-least-once vs exactly-once and idempotent recovery strategy. | Do not silently drop events or rewrite existing CSV |
| EH-03 Degraded readiness | `/health` source | Keep existing public liveness backward compatible; optionally protected readiness with PG/Redis/disk dependency evidence. | No public status/availability semantic regression |
| EH-04 Docker runner filesystem | CI #50 containerd lstat error | Capture runner snapshotter/storage/logs and BuildKit proof, retry only affected job once on clean isolated builder. | No production Docker prune or volume change |
| EH-05 Linux Python provisioning | CI #45 apt 404 | Use stable Python runner or existing Windows 14/14 job in canonical CI matrix. | No change to Python SDK package semantics |
| EH-06 Supply-chain advisory details | CI #50 24/26 summaries | Generate advisory ID/package/version/production-path inventory, apply compatible pinned patches only when justified. | No blanket major upgrades, no `audit fix --force` |
| EH-07 WebSocket origin/defaults | `server/index.js:1086-1089` | Document access control and test raw WS handshake plus verified JWT behavior; adjust only with repro and approval. | Avoid breaking legitimate SDK cross-origin connection |
## Mandatory error-handling acceptance matrix (disposable only)
- 401 invalid/expired/rotated JWT, 403 revoked/IP forbidden, 400 bad client ID, 413 oversized event, 429 rate limit, 507 quota/disk low, 503 flush-lock timeout: contract and safe error payload tests.
- Verify duplicate rejection across HTTP + Socket.IO, Redis outage/resume, multi-client isolation, process crash and CSV replay boundaries.
- Verify Python/Node SDK retryability: auth and quota failures never endlessly retry/spool; temporary network errors backoff/spool safely.
- Webhooks: URL SSRF/private/cloud metadata reject, redirects rejected, signatures match, bounded timeout/cooldown.
- MCP: disabled by default, read vs write permissions, bearer hashing, IP allowlist, sandbox traversal and symlink reject, no secret emission.
- Do not run these failure injections against `https://data.ivib.cc/`. Use disposable CI fixtures/data directories/Redis containers only.
## Change control
Any new error code, status, SDK public exception or alert flow is **feature/API behavior change**: STOP and request explicit new scope approval. This register is documentation for deciding whether a new handler is warranted, not blanket authorization to add handlers.
+46
View File
@@ -0,0 +1,46 @@
# DataStack — Actual Implementation / Working-Feature Register
Frozen source HEAD: `d725841ebc828d58cbe90f517da9e600f23c4630`. Date: 2026-11-22.
**Evidence levels**: SOURCE = verified code present; CI PASS = focused automated tests/build passed; LIVE READ = public read-only endpoint observed; NOT VERIFIED = no independent safe live runtime evidence. These are deliberately different. Do NOT infer production DB/S3/MCP health from a public `online` response.
| Capability / feature | Exact representative path | Baseline evidence | Actual live conclusion | Changed by P01/P02/P03 |
|---|---|---|---|---|
| Next.js 15 public landing and app pages | `app/**`, `docker/Dockerfile.web` | Next.js CI build PASS and 22/22 static pages; public landing loads | LIVE READ for landing only | None |
| Console admin authentication/setup | `app/api/auth/**`, `core/auth/adminUsers.ts`, `core/auth/adminSession.ts` | SOURCE + auth regression suite PASS | Admin login NOT live-tested | None |
| Roles / owner-admin-operator-viewer | `core/auth/**`, `components/layout/VconShell.tsx` | SOURCE + RBAC TS tests PASS | NOT live-tested | None |
| SDK registered clients, token issuance/revoke/rotate | `core/sdk/sdkManager.ts`, `app/api/clients/**` | SOURCE + client/auth TS regression PASS | NOT live-tested | None |
| Next.js HTTP event ingestion | `app/api/events/route.ts` | SOURCE + TS tests PASS | No live writes performed | None |
| Express + Socket.IO ingestion | `server/index.js`, `server/socket/handlers.js` | SOURCE + sync/SDK tests PASS | Not live end-to-end verified | None |
| Rate limit, dedupe, payload validation | `server/socket/handlers.js`, `app/api/events/route.ts` | SOURCE + tests PASS | Not live verified | None |
| Redis ingestion queue/fallback | `server/index.js:165-193`, `core/data/buffer.ts` | SOURCE + TS storage tests; R-01 ACK durability risk | Not guaranteed lossless on outage | None |
| CSV partition/rotation/flush | `server/workers/csvWriter.js`, `core/Storage/csvStorage.ts` | SOURCE + regression; ingestion image CI build PASS | R-02 crash replay semantics NOT confirmed | None |
| Compression and S3 archival | `core/Storage/s3Archive.ts`, `core/Storage/csvStorage.ts` | SOURCE + storage tests | No live S3 credentials or archive tests | None |
| Data browser, export CSV/JSON/NDJSON | `app/api/export/route.ts`, `app/api/data/route.ts` | SOURCE + TS tests | NOT live-tested | None |
| Storage quota / disk admission | `core/Storage/csvStorage.ts` | SOURCE + tests; 507 codes | NOT live-tested | None |
| Dashboard, client status, counts | `lib/vconUiHelpers.ts`, `app/vcon/clients/page.tsx` | SOURCE; client status helper tests; stale old regex CI-04 | No live user session proof | None |
| Prometheus and metrics bridge | `app/api/prometheus/route.ts`, `server/index.js` | SOURCE + tests | NOT live authenticated | None |
| Webhook alerts and SSRF/HMAC | `core/alerts/webhookNotifier.ts` | SOURCE + tests | NOT live-tested | None |
| CloudCmd file browser | `server/filesManager.js`, `tests/files-manager.test.ts` | SOURCE; files-manager suite 4 tests PASS | NOT live authenticated | None |
| MCP engine (17 permissions) | `server/mcp/mcpEngine.js`, `app/api/mcp/route.ts` | SOURCE + MCP server suite 3 tests PASS; write flags default false | NOT live connected/authenticated | None |
| Python SDK v2.2 + compat package | `sdk/python/datastack/datastack/**`, `sdk-python/**` | Windows CI **14/14** PASS | No real external SDK session | None |
| Node SDK v2.2 + offline spool | `sdk/node/datastack/**` | Native Windows SDK compile PASS; SDK-specific TS regression PASS | No live external SDK session | None |
| PWA/offline shell | `app/**`, `public/**` | SOURCE + phase06 suite PASS | Browser offline not live-tested | None |
| Docker Compose stack | `docker/docker-compose.yml` | CI config PASS, ingestion image PASS | Deployed app already serves public health; deployed SHA unverified | None |
| Docker web/proxy image | `docker/Dockerfile.web`, `docker/Dockerfile.proxy` | Web layer fail on containerd; proxy unrun | Existing Coolify instance appears online but CI image parity unverified | None |
| Health endpoint | `app/api/health/route.ts` | LIVE READ `status:online` on 2026-11-22T00:01:19Z | Confirmed public API liveness; not end-to-end readiness | None |
## Mandatory post-phase update convention
After EACH P01/P02/P03 close, amend only affected table rows:
- `Feature/capability` (exact added/changed behavior or `UNCHANGED`)
- `Source file/line + new SHA`
- `CI evidence` (run URL, pass/failed count, build exit)
- `Live status` (READ ONLY, AUTHENTICATED VERIFIED, NOT VERIFIED, etc.; no fictitious claims)
- `Compatibility` (UI/auth/SDK/API/storage preserved? test proof)
- `Risk remaining` and next approval.
Do not mark "actual working" simply because a patch compiled. Distinguish source, CI, and actual deployment.
## Change log
- P00: No feature added/modified. `notes/**` documentation only. 0/3 implementation phases completed.
- P01: pending.
- P02: pending.
- P03: pending.
+153
View File
@@ -0,0 +1,153 @@
# DataStack — Baseline Freeze, Code Proof & Proposed Patch Index
Frozen source baseline SHA: `d725841ebc828d58cbe90f517da9e600f23c4630` (2026-11-22, Asia/Dhaka).
Source verification: exact GitHub/GitLab/Gitea remote `main` and local HEAD matched this SHA. Local working tree clean; local `origin` configured to `MdNurnobi-Dev/DataStack.git`, and its *local tracking ref* was stale (`725d3c5`), so `ahead 5` alone was NOT evidence GitHub `JILLU-Games` was behind. GitHub `JILLU-Games` read-only branch HEAD verified at `d725841ebc828d58cbe90f517da9e600f23c4630`. No force push, ref rewrite or upstream reset authorized.
## A. Proof excerpts (verbatim short source fragments, exact baseline files/lines)
### CI-02 — stale lockfile contract (P01)
`tests/phase07-deployment.test.ts:648-650`:
```ts
test("FINAL-DEPLOY-FIX-01: Single npm lockfile, Dockerfile .runtime-logs ownership parity, 8-table prisma sync, and cold-boot MCP bridge hydration", async () => {
assert.equal(existsSync(resolve(process.cwd(), "bun.lock")), false, "Stale bun.lock must stay removed");
```
`git ls-files bun.lock` listed the file on baseline. **Patch reference**: check if any build consumer references it; if none, remove exactly `bun.lock` while preserving `package-lock.json`. Re-run this specific test.
### CI-04 — stale UI test expectation (P01)
`tests/v2-phase01-storage-quotas.test.ts:788-804`:
```ts
assert.match(clientsPageSrc, /function isClientExpired\(/);
assert.match(clientsPageSrc, /const expired = !revoked && isClientExpired\(client\.expiresAt,\s*now\)/);
```
`app/vcon/clients/page.tsx:70-85`:
```ts
const counts = useMemo(
() => summarizeUiClientCounts(clients, now),
[clients, now],
);
const computedStatus = classifyUiClientStatus(client, now);
```
`lib/vconUiHelpers.ts:717,730,754` exports `isUiClientExpired`, `classifyUiClientStatus`, `summarizeUiClientCounts`.
**Proposed patch shape** (planning-only, not applied):
```diff
--- a/tests/v2-phase01-storage-quotas.test.ts
+++ b/tests/v2-phase01-storage-quotas.test.ts
@@ obsolete regex-based UI expectations, lines 788-804 @@
- assert.match(clientsPageSrc, /function isClientExpired\(/);
+ assert.match(clientsPageSrc, /summarizeUiClientCounts/);
+ assert.match(clientsPageSrc, /classifyUiClientStatus/);
```
Also **replace the other 3 obsolete implementation-detail regexes** with behavioral assertions against exported helper and page filtering/count data; do not just drop coverage. Exact final diff is generated only during approved P01.
### CI-03 — test-process hang anchor (P01)
`core/data/buffer.ts:347-352,379-391`:
```ts
state.flushTimer = setInterval(() => {
void this.flush().catch(() => undefined);
}, state.batchFlushIntervalMs);
```
Run #45 suite exit codes 124: `r3-phase04-ui-metrics-rbac`, `r4-phase03-sdk-deploy-auth`, `r5-phase01-mcp-foundation-logs`, `r5-phase03-mcp-ui`. Timer is a **suspect**, not a proven sole root cause. **Patch reference:** instrument open handles and add narrow teardown in four named tests; `core/data/buffer.ts` is **read-only** unless new approval.
### CI-05 — compiler-source proof (P01)
`tsconfig.json:27`:
```json
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"]
```
Run #45 job 68 examples:
```text
tests/phase01-auth.test.ts(88,17): error TS2540: Cannot assign to 'NODE_ENV' because it is a read-only property.
tests/r5-phase02-mcp-server.test.ts(510,8): error TS18046: 'deniedClientCreate.content' is of type 'unknown'.
tests/v2-phase01-storage-quotas.test.ts(762,64): error TS2554: Expected 0 arguments, but got 1.
```
**Patch reference:** exact 9 test files / line map in `001`. Use correct typings/type guards and current API signatures, not TS ignore or global tsconfig exclusion.
### CI-01 — Windows clean-install lockfile mismatch (P01)
Run #42 job 63:
```text
npm error Missing: @img/sharp-win32-x64@0.35.5 from lock file
npm error Missing: @unrs/resolver-binding-win32-x64-msvc@1.13.0 from lock file
```
Run #45 job 71 showed success **only after temporary runner** `npm install --package-lock-only`, followed by `npm ci` and Node SDK build. **Patch reference:** `package-lock.json` optional/native dependencies, retain pinned package graph; verify unchanged tracked lock under CI on Linux/Windows.
### CI-06 — Python runner packages (P03)
Run #45 job 67:
```text
E: Failed to fetch ... python3.9-venv_3.9.2-1+deb11u7_amd64.deb 404 Not Found
```
`.gitea/workflows/datastack-ci.yml` Python install step uses `apt-get update -qq; apt-get install -y python3 python3-venv python3-pip`. Run #48 job 74 executes 14 tests PASS on Windows. **Patch reference:** CI job Python provisioning only.
### CI-07 — Docker web layer failure (P03)
`docker/Dockerfile.web:13`:
```dockerfile
COPY --from=dependencies /app/node_modules ./node_modules
```
Run #50 job 76:
```text
failed to create diff tar stream: lstat .../app/node_modules/@uiw: no such file or directory
```
`docker/Dockerfile.server` produced tagged ingestion image; Docker Compose `config --no-interpolate --quiet` PASS. **Patch reference:** `.gitea/workflows/datastack-docker-ci.yml` runner/cache/BuildKit; Dockerfile modification conditional on independent reproduction and new approval.
### CI-08 — npm advisory summary (P03)
Run #50 job 76:
```text
24 vulnerabilities (23 moderate, 1 high)
26 vulnerabilities (25 moderate, 1 high)
```
**No evidence of specific advisory IDs yet.** Patch reference `package.json`/`package-lock.json` only after isolated `npm audit --json` with package/CVE/runtime attribution. Never add counts or claim 50 unique advisories.
### R-01 — Redis volatile fallback ACK risk (P02, conditional)
`server/index.js:165-193`:
```js
try {
bufferLen = await redis.rpush(REDIS_BUFFER_KEY, ...serialized);
redisConnected = true;
} catch {
redisConnected = false;
}
// ...
bufferQueue.push(...events);
return bufferQueue.length;
```
`server/socket/handlers.js:583-602`:
```js
bufferQueue.push(event);
try {
await onBufferPush();
} catch (persistErr) {
// callback BUFFER_PERSIST_FAILED
}
if (callback) callback({ success: true, eventId });
```
**Potential issue**: a successful ACK does not necessarily mean durable storage after Redis failure; test first. Proposed patch is NOT predetermined.
### R-02 — CSV before Redis trim (P02, conditional)
`server/index.js:981-994`:
```js
const res = await writer.writeBatch(parsedBatch);
// ...
await redis.ltrim(REDIS_BUFFER_KEY, rawItems.length, -1);
```
**Potential issue**: crash after partial write or before list trim may replay buffered rows. Test with injected partial failures and compare event IDs before prescribing exactly-once guarantee.
### R-03 — Health shows service liveness (P02, conditional)
`server/index.js:1939-1955`:
```js
const redisBufferSize = await getRedisBufferLen();
res.json({ status: 'online', /* ... */ redisConnected, /* ... */ });
```
`app/api/health/route.ts:95-105`:
```ts
const bufferStats = await DataBufferEngine.getBufferStatsAsync();
return NextResponse.json({ status: 'online', service: 'DataStack Web API', /* ... */ });
```
**Potential issue**: liveness is not transactional readiness. Preserve existing public endpoint shape and verify separately.
## B. Source-to-patch navigation
- P01: `package-lock.json`; `bun.lock`; `tests/phase07-deployment.test.ts:649`; `tests/v2-phase01-storage-quotas.test.ts:762,783,788-804`; 4 timeout tests; 9 TS test files; `.gitea/workflows/datastack-ci.yml`.
- P02 conditional: `server/index.js:165-193,943-994,1939-1956`, `server/socket/handlers.js:583-602`, `server/workers/csvWriter.js:265-323`, `app/api/health/route.ts:95-111`; associated test files. Read-only until fault reproduction.
- P03: `.gitea/workflows/datastack-ci.yml`, `datastack-python-ci.yml`, `datastack-docker-ci.yml`, `datastack-eslint-ci.yml`; `package.json` / `package-lock.json` only justified dependency changes. Source Dockerfile edits separately approved only if needed.
## C. Documentation baseline update record
This file **freezes the source baseline**, not the future documentation-commit SHA. After adding `notes/**`, record commit SHA and three remote SHAs in `notes/003` and the user-facing result. Subsequent P01 implementation should inspect `git diff d725841ebc828d58cbe90f517da9e600f23c4630..HEAD` and permit only these notes as intervening changes; any unexpected source edits require a new review.