diff --git a/notes/001-PRODUCTION-FORENSIC-AUDIT.md b/notes/001-PRODUCTION-FORENSIC-AUDIT.md new file mode 100644 index 0000000..a83cdac --- /dev/null +++ b/notes/001-PRODUCTION-FORENSIC-AUDIT.md @@ -0,0 +1,66 @@ +# DataStack — Production-Readiness Forensic Audit (Documentation Baseline) + +Status: **DOCUMENTED / NO BUG FIX APPLIED**. Date: 2026-11-22 (Asia/Dhaka). +Frozen source HEAD: `d725841ebc828d58cbe90f517da9e600f23c4630`. Local, Gitea `vibtools/DataStack`, GitLab `ivibtools/datastack`, and GitHub `JILLU-Games/DataStack` confirmed at this exact HEAD before writing these notes. +Scope of this document: **evidence-backed CI and source review**, not proof that every production feature works. No Coolify live write, rollback, restart or deployment was conducted. Public GET `https://data.ivib.cc/api/health` returned `status: online`; full DB/Redis/MCP/S3 transaction health is not established. + +## Findings classification + +- **8 finding groups** from actual CI: **5 code/test/lockfile groups** (CI-01…05) and **3 CI infrastructure/security-review groups** (CI-06…08). **Not 43 separate bugs**: the 43 compiler errors belong to CI-05. +- **2 failing assertions, 4 timeout suites** (24 TypeScript suites in run #45: 18 clean, 6 fail/timeout). +- **3 extra source-observed reliability risks** (R-01…03) have NOT been reproduced as production failures; they require targeted fault injection before any fix or behavior change. +- **Do not equate successful test/source feature with verified production runtime.** + +## Eight confirmed findings / blockers + +| ID | Severity* | Evidence (run/job) | Exact affected area | Verified observation | Planned phase | +|---|---|---|---|---|---| +| CI-01 | High | Run #42 job 63; #45 job 71 | `package-lock.json`, `package.json`; `.gitea/workflows/datastack-ci.yml` Windows job | Original Windows `npm ci` reports missing platform dependency entries; regenerate lock **only in runner** -> SDK builds. Lock not fixed in repo. | P01 | +| CI-02 | Medium | Run #45 job 66 | `bun.lock` and `tests/phase07-deployment.test.ts:648-649` | Tracked `bun.lock` conflicts with assertion requiring its absence. Validate references before removing. | P01 | +| CI-03 | Medium | Run #45 job 66 | `tests/r3-phase04-ui-metrics-rbac.test.ts`, `tests/r4-phase03-sdk-deploy-auth.test.ts`, `tests/r5-phase01-mcp-foundation-logs.test.ts`, `tests/r5-phase03-mcp-ui.test.ts` | Each suite timed out with exit **124** despite reaching passing assertions; investigate open handles/teardown without changing application timers by default. | P01 | +| CI-04 | Medium | Run #45 job 66 | `tests/v2-phase01-storage-quotas.test.ts:788-804` vs `app/vcon/clients/page.tsx:15-20,70-95` and `lib/vconUiHelpers.ts:717-780` | Old regex expects `function isClientExpired(` in page; production page uses shared `classifyUiClientStatus` and `summarizeUiClientCounts`. Test contract is stale; not proof expiry feature fails. | P01 | +| CI-05 | Medium | Run #45 job 68 | Nine `tests/*.test.ts` files (listed below), `tsconfig.json:27` | `npx tsc --noEmit` produced **43 compiler errors**, despite Next.js production build PASS. Type errors in test sources require focused correction; do not disable strict TypeScript. | P01 | +| CI-06 | Low/CI | Run #45 job 67, run #48 job 74 | `.gitea/workflows/datastack-ci.yml` Python job | Linux Bullseye `apt-get` returned Debian-security 404; separate Windows Python job passed **14/14**. Runner provisioning defect, not SDK regression. | P03 | +| CI-07 | High/CI | Run #50 job 76 | `.gitea/workflows/datastack-docker-ci.yml`, Docker daemon; `docker/Dockerfile.web:13` | Compose validation and ingestion image PASS, then web image layer copy fails with containerd `lstat .../app/node_modules/@uiw: no such file or directory`. Root cause **unconfirmed** (runner snapshot/storage or reproducible build interaction); proxy never built. | P03 | +| CI-08 | Review | Run #50 job 76 | `package.json` / `package-lock.json` / resolved npm graph | npm reports **24 (23 moderate, 1 high)** with `--omit=dev`, **26 (25 moderate, 1 high)** with `--include=dev`. Counts overlap; no CVE list, reachable-exploit assessment or remediation done. | P03 | + +*Severity is release-priority triage, NOT a CVSS rating and NOT proof of real-world exploitation. + +### CI-05 exact file map (43 total, all inside tests) + +| File | Error count | Families / main anchors | +|---|---:|---| +| `tests/phase01-auth.test.ts` | 4 | TS2540; 88,560,773,794 | +| `tests/phase03-storage-buffer.test.ts` | 2 | TS2349; 1158,1202 | +| `tests/r3-phase01-security-vault-webhooks.test.ts` | 6 | TS2353, TS2345; 185,191,202,210,295,430 | +| `tests/r3-phase02-storage-scheduler-gzn.test.ts` | 1 | TS2345; 49 | +| `tests/r4-phase01-security-boot.test.ts` | 3 | TS2345; 102,537,571 | +| `tests/r4-phase02-ingestion-audit-streaming.test.ts` | 9 | TS2345/TS2833; 87,332,335,338,407 | +| `tests/r5-phase02-mcp-server.test.ts` | 12 | TS18046; 510,524,538,549,560,570,581,604,613,635,644,657 | +| `tests/v2-phase01-storage-quotas.test.ts` | 2 | TS2554; 762,783 | +| `tests/v2-phase02-alerts-ip-prometheus.test.ts` | 4 | TS2339; 479,480,490,491 | + +No source fix is authorized here. Use typed narrowing and correct test fixtures, not `skipLibCheck`, `any` everywhere, `@ts-ignore`, or changing real API responses merely to make tests compile. + +## Three source-observed production reliability risks — CONDITIONAL GATES, not confirmed bugs + +| Gate | Source proof | Why verification is required | Phase | +|---|---|---|---| +| R-01 Redis outage ACK durability | `server/index.js:165-193`, `server/socket/handlers.js:583-602` | `redis.rpush` failure falls back to RAM, and `onBufferPush()` can resolve, then Socket.IO ACK success; process crash before flush may lose acknowledged events. Fail-injection test and explicit durability contract required. | P02 | +| R-02 Batch replay duplication | `server/index.js:943-994`, `server/workers/csvWriter.js:265-323` | CSV write precedes Redis list trim. Crash/partial write between these operations can replay already written rows. Validate at-least-once vs exactly-once guarantee before mitigation. | P02 | +| R-03 Liveness ≠ readiness | `server/index.js:1939-1956`; `app/api/health/route.ts:95-111` | Health returns online without full PostgreSQL + Redis + disk transaction proof. Preserve existing `/health` API semantics; consider separate, internal readiness rather than breaking public health. | P02 | + +Optional configuration review (not counted among eight): `server/index.js:1086-1089` Socket.IO uses CORS delegate; verify WebSocket origin/access and JWT guard in target deployments before recommending policy changes. `sdk/python/datastack/pyproject.toml:47-51` still references legacy `anik-d-art/datastack` URLs; metadata only, not current release blocker. Do not silently modify either under approved scope. + +## CI evidence / original logs + +- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/42` — initial Windows lockfile failure (job 63), cancelled after partial node evidence. +- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/45` — 24 suites = 18 clean / 6 fail-or-timeout, 43 TS diagnostics, Next.js production build PASS, Windows diagnostic SDK build PASS; Linux Python apt and Docker-CLI job blocked. +- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/48` — Python SDK 14/14 PASS. +- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/49` — full ESLint PASS. +- `https://dev.ivib.cc/vibtools/DataStack/actions/runs/50` — Compose PASS, Docker ingestion image PASS, web image containerd failure. +- See `notes/006-BASELINE-AND-PATCH-EVIDENCE.md` for precise code/log excerpts and proposed patch anchors. + +## Release acceptance rule + +NOT production-update-ready until P01, conditional P02 gates, and P03 finish, 24/24 TypeScript suites exit 0, 14/14 Python tests pass, full strict `tsc --noEmit` and ESLint pass, Linux/Windows clean install is reproducible, standalone Next build passes, all 3 Docker images build, security advisories are triaged, and approved **non-mutating** live readiness checks pass. Production deploy is a separate operator-approved action; never automatic from this documentation phase. diff --git a/notes/002-PRODUCTION-UPDATE-ROADMAP.md b/notes/002-PRODUCTION-UPDATE-ROADMAP.md new file mode 100644 index 0000000..b55abdc --- /dev/null +++ b/notes/002-PRODUCTION-UPDATE-ROADMAP.md @@ -0,0 +1,72 @@ +# DataStack — Production Update Roadmap / Exact Scope Lock + +Planning baseline: `d725841ebc828d58cbe90f517da9e600f23c4630` (frozen; documentation commits will advance Git HEAD but NOT this source baseline). +Current state: **3 implementation phases P01–P03 planned; 0 implemented; approvals not yet granted**. +Prerequisite for any phase: compare current HEAD and changed paths with this baseline and previously committed notes. For new commits since source baseline, inspect exact diff; no implicit rebase or silent discard. + +## Global non-negotiable Scope Lock (applies to ALL phases) + +1. Existing UI/UX, public API response contracts, authentication, RBAC, JWT/SDK interoperability, MCP permissions, data retention, CSV content, persistent databases, keys, Redis values, S3/R2 objects, Coolify configuration and production deployment: **unchanged unless an exact targeted fix is proven and separately approved**. +2. No live event sending, live database mutation, destructive test, production restart/deploy, automatic package vulnerability `--force`, tag or release without explicit approval. +3. Limit changes to exact allowlisted paths per phase. Any additional path, altered schema, migration, UI behavior, API code, or security default = STOP / document scope exception / obtain new approval. No broad refactoring. +4. One phase = targeted patch, **one focused CI verification**, exact diff/security review, log/register updates, one Git commit and 3-remote sync verification. No repetitive test loops or local production builds. If CI infra is broken, report a blocker rather than spoof PASS. +5. Phase close requires written evidence with exit codes, commit SHA, changed paths, feature preservation checklist, remaining issues and a next-phase approval command. +6. Reference test logs and source code, not README assertions alone. Code candidate risks require fault reproduction before production behavior changes. +7. Secrets, personal data, local `.env` and runtime tokens never copied into reports, logs or commits. Never perform unsafe `git reset --hard` / `git clean -fd`. +8. `AGENTS.md` cross-IDE memory requirement: update `MEMORY.md` only under explicit scope exception, otherwise provide exact handoff Markdown in `notes/003-PHASE-STATUS-LOG.md` for copy/sync. + +## Phase P01 — Dependency, tests & type-contract reconciliation (CI-01…CI-05) + +**Goal:** fix confirmed test/lockfile blockers without changing actual app/SDK runtime behavior. +**Allowed paths only:** +- `package-lock.json`; `bun.lock` (delete ONLY after verifying no build consumers; npm lock stays canonical). +- `tests/phase07-deployment.test.ts`; `tests/v2-phase01-storage-quotas.test.ts`; +- `tests/r3-phase04-ui-metrics-rbac.test.ts`; `tests/r4-phase03-sdk-deploy-auth.test.ts`; `tests/r5-phase01-mcp-foundation-logs.test.ts`; `tests/r5-phase03-mcp-ui.test.ts`; +- `tests/phase01-auth.test.ts`; `tests/phase03-storage-buffer.test.ts`; `tests/r3-phase01-security-vault-webhooks.test.ts`; `tests/r3-phase02-storage-scheduler-gzn.test.ts`; +- `tests/r4-phase01-security-boot.test.ts`; `tests/r4-phase02-ingestion-audit-streaming.test.ts`; `tests/r5-phase02-mcp-server.test.ts`; `tests/v2-phase02-alerts-ip-prometheus.test.ts`; +- `.gitea/workflows/datastack-ci.yml` (narrow clean-install/test harness only); `notes/001…006` (status/evidence only). + +**Not allowed:** `app/**`, `components/**`, `lib/**`, `core/**`, `server/**`, `docker/**`, `tsconfig.json`, `package.json` or SDK runtime files. If open-handle leak proves core timer bug, STOP and request explicit additional exception for `core/data/buffer.ts` instead of altering it as a side effect. + +**Patch tasks and references:** +- CI-01: deterministic regenerated `package-lock.json` with cross-OS optional entries, verify clean `npm ci` on Linux/Windows with same pinned npm; no dependency version bump unless separately approved. Reference `.gitea/workflows/datastack-ci.yml:91-120`. +- CI-02: check `bun.lock` consumers, then remove stale lock or resolve test contract with proof; `tests/phase07-deployment.test.ts:649`. +- CI-03: isolate 4 suites, enumerate open handles/intervals, add explicit test teardown / mocks; source `core/data/buffer.ts:347-391` is **read-only diagnostic reference**. Avoid `process.exit()` hacks and wholesale timeout increases. +- CI-04: replace obsolete regex tests in `tests/v2-phase01-storage-quotas.test.ts:788-804` with precise shared-helper/integration assertions matching `app/vcon/clients/page.tsx:70-95` and `lib/vconUiHelpers.ts:717-780`. +- CI-05: fix 43 test-code type errors across 9 listed files (see `001`). Keep strict TS compile; preserve production API signatures. `tsconfig.json:27` includes tests by glob. +- **P01 acceptance:** 24/24 test files exit 0 (no timeout), `tsc --noEmit` zero errors, `npm run lint` PASS, Python 14/14 PASS regression, Windows/Linux original `npm ci` PASS without diagnostic lock regeneration; one focused Gitea matrix. If network/runner blocker, mark BLOCKED not PASS. +- **Close:** update `notes/003` and `notes/005` only after verifying; commit and sync. + +## Phase P02 — Ingestion reliability / error-handling acceptance gates (R-01…R-03) + +**Goal:** reproduce or dismiss source-observed risk, preserve response semantics unless proven fix is necessary. **Risk ≠ bug until test fails.** +**Diagnostic-only allowlist:** `server/index.js`, `server/socket/handlers.js`, `server/workers/csvWriter.js`, `core/data/buffer.ts`, `app/api/health/route.ts`, `tests/phase02-sync.test.ts`, `tests/phase03-storage-buffer.test.ts`, `tests/r4-phase02-ingestion-audit-streaming.test.ts`, `notes/001…006`. +**Default write allowlist:** only the THREE listed test files plus `notes/001…006`. Runtime source files above are **READ ONLY until a reproducible failing test demonstrates the need**; then the P02 approval must name an exact method/line range and explicitly authorize the smallest required change. This is not preapproval for changing server source. +**Test matrix (disposable, never production):** +- R-01: fake Redis `rpush` rejection + ACK + simulated process exit/reopen; ensure API explicitly returns failed/pending unless durable, or document intended guarantee. Preserve existing duplicate/rate/size/auth error codes. +- R-02: injected CSV partial write, simulated failure before Redis `ltrim`, restart and compare event IDs/rows; decide documented at-least-once semantics vs dedupe strategy before altering storage. +- R-03: separate liveness/readiness checks with mocked Redis/PG errors; old public `/health` stays backwards compatible. Verify startup fail-closed when secrets or DB unavailable. +- Review source CORS/WebSocket origin and security access policy as **read-only**, no default policy change automatically. +- **P02 acceptance:** targeted fault-injection tests PASS, no acknowledged-event loss in the documented guarantee, no unbounded duplicate amplification, correct 503/507/429/401 semantics and recovery. If risk not reproduced, close as verified with **zero runtime source changes**; never invent a fix. +- **Close:** record precise outcome and next P03 approval. + +## Phase P03 — CI runner, Docker images, dependency security, release evidence (CI-06…CI-08) + +**Goal:** reproducible Gitea verification, safe dependency advisory triage, complete all three Docker builds, confirm non-mutating production health. +**Default write allowlist:** `.gitea/workflows/datastack-ci.yml`, `.gitea/workflows/datastack-python-ci.yml`, `.gitea/workflows/datastack-docker-ci.yml`, `.gitea/workflows/datastack-eslint-ci.yml`, `package.json`, `package-lock.json`, `notes/001…006`. +**Conditional, separate approval only:** `docker/Dockerfile.web` / `docker/Dockerfile.server` if isolated runner/containerd evidence establishes source-level Dockerfile defect. **Not allowed** to change Compose deployment services, secrets, volumes or Dockerfiles merely to work around a runner failure. +**Tasks:** +- CI-06: replace stale Debian Bullseye `apt-get` dependency step with preinstalled compatible Python image/toolchain OR canonical Windows Python job; avoid duplicate CI runs. +- CI-07: inspect runner daemon snapshotter/filesystem free space, cache/BuildKit and known `node_modules/@uiw` layer failure; use properly configured disposable BuildKit builder or another isolated runner. If reproducible Dockerfile defect is proven, obtain narrow scope amendment. Build images **without running containers**; no `docker system prune` or shared cache destruction. +- CI-08: run read-only detailed `npm audit --json` in isolated runner, map actual direct/transitive paths and runtime impact, document advisory IDs/severity, apply minimal compatible pin bumps only where necessary. Never `npm audit fix --force`. +- Verify canonical package metadata and Git remote SHA; GitHub login/account issue is **already resolved** and is not a current blocker. +- **P03 acceptance:** all 3 images build, Compose config PASS, Next standalone PASS, lint + strict TS PASS, 24/24 TS suites no hang, Python 14/14, regression error codes unchanged, advisory disposition documented, read-only public health PASS. Any real deployment/production migration is a **separate approval**. +- **Close:** exact diff, one CI acceptance run (plus only blocked job rerun where necessary), statuses/feature matrix update, one commit, synced remote HEADs, phase CLOSED. + +## Sequence and approval gate + +P01 -> P02 -> P03, strictly in this order. **No P01 implementation has begun** in this documentation-only update. + +Approval command for next turn: + +`APPROVED — DataStack Production Readiness P01. Baseline: d725841ebc828d58cbe90f517da9e600f23c4630. Implement ONLY notes/002 P01 exact Scope Lock; preserve all app, UI, SDK, backend, live data, deployment and unrelated files. One focused disposable Gitea CI, exact diff/security audit, notes/003–005 update, one commit, push GitHub/GitLab/Gitea and verify all HEADs. No live deployment, tags, releases or scope expansion.` diff --git a/notes/003-PHASE-STATUS-LOG.md b/notes/003-PHASE-STATUS-LOG.md new file mode 100644 index 0000000..df4b578 --- /dev/null +++ b/notes/003-PHASE-STATUS-LOG.md @@ -0,0 +1,52 @@ +# DataStack — Phase Status & Change Log (Source of Progress Truth) + +Frozen source baseline: `d725841ebc828d58cbe90f517da9e600f23c4630`. Generated: 2026-11-22 Asia/Dhaka. +Scope: documentation-only kickoff. Any future phase must update this log **after** verification, not on intention. +**Completed implementation phases: 0 / 3. Remaining: 3.** Documentation baseline preparation is a separate non-implementation milestone. + +| Phase | Status | IDs / function | Changed source | Verified CI evidence | Commit / remote sync | +|---|---|---|---|---|---| +| P00 Notes / freeze | COMPLETED (documentation only) | `001–006` docs, source baseline and exact scope | `notes/**` only | 6 Markdown files; balanced fences; staged diff `--check` PASS | Documentation commit containing this file; remote SHA verification recorded in session handoff | +| P01 Tests & lockfile | NOT STARTED | CI-01,02,03,04,05 | None | Not run | — | +| P02 Ingestion reliability | NOT STARTED | Conditional R-01,02,03 | None | Not run | — | +| P03 CI/Docker/security | NOT STARTED | CI-06,07,08; release gates | None | Not run | — | + +## Existing diagnostic CI snapshot (NOT implementation phase completions) + +| Run | Baseline commit | Result | Meaning | +|---|---|---|---| +| #42 | `f7c1e7f` | Cancelled | Windows original lock fail logged; long combined regression stopped | +| #45 | `d3a93c6` | Failure | TS 18 clean/6 fail-or-timeout; 43 type errors; build and Windows SDK pass; Python apt / Docker CLI blockers | +| #48 | `ff1333d` | Success | Python SDK 14/14 | +| #49 | `559f180` | Success | ESLint | +| #50 | `d725841` | Failure | Compose + ingestion image pass; web image fails containerd snapshot copy; proxy unrun | + +Historical baseline code stayed frozen through these CI workflow changes. Current HEAD before documentation changes = `d725841ebc828d58cbe90f517da9e600f23c4630` on local/GitHub/GitLab/Gitea. + +## Per-phase closure record template — COPY for each P01/P02/P03 + +### P0X — [TITLE] (OPEN / BLOCKED / CLOSED) + +- Authorized baseline / parent SHA: +- Approval command / exact scope document reference: +- Findings addressed + any conditional gates: +- Exact modified paths (one path per row): +- Applied code patch summary (function + lines): +- Existing feature/UI/API/auth/SDK/data preservation audit: +- Focused CI run URL / test counts / exit codes: +- Security check / handling of secrets, data and scope exceptions: +- Verification result: PASS / FAIL / BLOCKED, not assumed: +- Closure commit SHA + GitHub/GitLab/Gitea main SHAs: +- Completed implementation phases: X/3; remaining: Y. +- Next phase: P0Y; **specific approved features** / allowed files: +- New issues / deferred risks / notes evidence links: +- MEMORY.md cross-IDE handoff text (copyable): + `[DataStack P0X] Baseline=; findings=; scope=; CI=; commit=; remote sync=; phase=CLOSED; next=P0Y; production deployment=unchanged.` + +## Status transition rules + +- OPEN -> IN PROGRESS only with explicit approval and baseline SHA match. +- IN PROGRESS -> CLOSED only after one focused CI and exact diff audit, updated `003` + `005` + relevant `001/004/006`, Git commit, sync, and confirmed remote SHA. +- Failed test -> BLOCKED with log excerpt and scope decision; never log as complete. +- Unexpected file change -> STOP and revert only own documented changes safely; never touch pre-existing work. +- At any time, `actual production deployment status` is a separate tracked dimension (do not infer deployed SHA from Git HEAD). diff --git a/notes/004-ERROR-HANDLING-REGISTER.md b/notes/004-ERROR-HANDLING-REGISTER.md new file mode 100644 index 0000000..62975f3 --- /dev/null +++ b/notes/004-ERROR-HANDLING-REGISTER.md @@ -0,0 +1,47 @@ +# DataStack — Error Handling Inventory & Proposed Safety Gates + +Source baseline: `d725841ebc828d58cbe90f517da9e600f23c4630`. This is an **implementation inventory**, NOT an independently executed production fault-injection certificate. Existing codes and APIs below are source-grounded. + +## Existing handlers: observed code paths + +| Failure / scenario | Exact path and lines | Current response / behavior | CI coverage/status | +|---|---|---|---| +| Missing/bad client ID | `app/api/events/route.ts:433-442` | `400 VALIDATION_ERROR` | TS suites; no live write | +| Deleted/missing/revoked client | `app/api/events/route.ts:445-474` | `403 CLIENT_REVOKED` | Auth regression passed in suite | +| Rotated token | `app/api/events/route.ts:476-487` | `401 TOKEN_ROTATED` | Auth/SDK regression coverage | +| Expired client token | `app/api/events/route.ts:489-504` | `401 CLIENT_TOKEN_EXPIRED` + webhook trigger | Python SDK tests pass | +| Disallowed IP | `app/api/events/route.ts:506-526`; `app/api/health/route.ts:82-91` | `403 CLIENT_IP_FORBIDDEN` | Source + TS regression | +| Disk low / client quota exhausted | `core/Storage/csvStorage.ts:1159-1193`; `app/api/events/route.ts:567-589` | `507 DISK_SPACE_LOW` or `STORAGE_QUOTA_EXCEEDED`; webhook | Python 14/14, TS related suite passed except stale UI assertion | +| Duplicate, payload oversize, rate limit | `app/api/events/route.ts:633-681` | dedupe; `413 PAYLOAD_TOO_LARGE`, `429 RATE_LIMIT_EXCEEDED`, `400 BATCH_REJECTED` | Regression suites | +| Socket buffer persistence exception | `server/socket/handlers.js:583-602` | `BUFFER_PERSIST_FAILED` callback **if onBufferPush throws**; success ACK after normal resolution | R-01 requires RAM-fallback failure injection | +| Redis distributed flush lock | `app/api/storage/route.ts:102-127` and :187-197 | `503 FLUSH_LOCK_TIMEOUT` on guarded flush | Existing TS test pass | +| Webhook disabled/blocked/redirect/timeout | `core/alerts/webhookNotifier.ts:94-176` | SSRF URL check, HMAC, timeout, manual redirect policy, per-client cooldown | Source + TS security tests | +| Admin cookie integrity/revocation | `core/auth/adminSession.ts:70-154`; `app/api/auth/**` | HMAC verified session with tokenVersion support; 401/403 boundaries | Auth TS regression passed | +| MCP permissions / filesystem sandbox | `server/mcp/mcpEngine.js:12-50,167-248` | 17 granular permissions; writes disabled default; path jail/symlink rejects | MCP server suite passed | +| Python SDK typed error mapping | `sdk/python/datastack/datastack/exceptions.py:8-84` | Auth/config/quota/payload/file errors mostly non-retryable, connection/rate retryable | Windows Python CI 14/14 | +| Public liveness health | `server/index.js:1939-1956`; `app/api/health/route.ts:95-111` | `online` reflects responder; not full transactional readiness | Public endpoint GET online only | + +## Missing / not proven handling — proposals, NOT implemented + +| Gate | Existing evidence | Narrow proposed addition | Risk / invariant | +|---|---|---|---| +| EH-01 Durable ACK under Redis loss | `server/index.js:165-193` RAM fallback; `server/socket/handlers.js:583-602` ACK | Test Redis RPUSH failure followed by crash. If confirmed, add reliable negative ACK/persist-queue contract with focused regression. | Preserve SDK fallback/spooling & existing auth/ack fields | +| EH-02 Partial CSV replay duplication | `server/index.js:981-994` write before trim | Crash/replay fault test; clarify at-least-once vs exactly-once and idempotent recovery strategy. | Do not silently drop events or rewrite existing CSV | +| EH-03 Degraded readiness | `/health` source | Keep existing public liveness backward compatible; optionally protected readiness with PG/Redis/disk dependency evidence. | No public status/availability semantic regression | +| EH-04 Docker runner filesystem | CI #50 containerd lstat error | Capture runner snapshotter/storage/logs and BuildKit proof, retry only affected job once on clean isolated builder. | No production Docker prune or volume change | +| EH-05 Linux Python provisioning | CI #45 apt 404 | Use stable Python runner or existing Windows 14/14 job in canonical CI matrix. | No change to Python SDK package semantics | +| EH-06 Supply-chain advisory details | CI #50 24/26 summaries | Generate advisory ID/package/version/production-path inventory, apply compatible pinned patches only when justified. | No blanket major upgrades, no `audit fix --force` | +| EH-07 WebSocket origin/defaults | `server/index.js:1086-1089` | Document access control and test raw WS handshake plus verified JWT behavior; adjust only with repro and approval. | Avoid breaking legitimate SDK cross-origin connection | + +## Mandatory error-handling acceptance matrix (disposable only) + +- 401 invalid/expired/rotated JWT, 403 revoked/IP forbidden, 400 bad client ID, 413 oversized event, 429 rate limit, 507 quota/disk low, 503 flush-lock timeout: contract and safe error payload tests. +- Verify duplicate rejection across HTTP + Socket.IO, Redis outage/resume, multi-client isolation, process crash and CSV replay boundaries. +- Verify Python/Node SDK retryability: auth and quota failures never endlessly retry/spool; temporary network errors backoff/spool safely. +- Webhooks: URL SSRF/private/cloud metadata reject, redirects rejected, signatures match, bounded timeout/cooldown. +- MCP: disabled by default, read vs write permissions, bearer hashing, IP allowlist, sandbox traversal and symlink reject, no secret emission. +- Do not run these failure injections against `https://data.ivib.cc/`. Use disposable CI fixtures/data directories/Redis containers only. + +## Change control + +Any new error code, status, SDK public exception or alert flow is **feature/API behavior change**: STOP and request explicit new scope approval. This register is documentation for deciding whether a new handler is warranted, not blanket authorization to add handlers. diff --git a/notes/005-ACTUAL-FEATURE-REGISTER.md b/notes/005-ACTUAL-FEATURE-REGISTER.md new file mode 100644 index 0000000..288a636 --- /dev/null +++ b/notes/005-ACTUAL-FEATURE-REGISTER.md @@ -0,0 +1,46 @@ +# DataStack — Actual Implementation / Working-Feature Register + +Frozen source HEAD: `d725841ebc828d58cbe90f517da9e600f23c4630`. Date: 2026-11-22. +**Evidence levels**: SOURCE = verified code present; CI PASS = focused automated tests/build passed; LIVE READ = public read-only endpoint observed; NOT VERIFIED = no independent safe live runtime evidence. These are deliberately different. Do NOT infer production DB/S3/MCP health from a public `online` response. + +| Capability / feature | Exact representative path | Baseline evidence | Actual live conclusion | Changed by P01/P02/P03 | +|---|---|---|---|---| +| Next.js 15 public landing and app pages | `app/**`, `docker/Dockerfile.web` | Next.js CI build PASS and 22/22 static pages; public landing loads | LIVE READ for landing only | None | +| Console admin authentication/setup | `app/api/auth/**`, `core/auth/adminUsers.ts`, `core/auth/adminSession.ts` | SOURCE + auth regression suite PASS | Admin login NOT live-tested | None | +| Roles / owner-admin-operator-viewer | `core/auth/**`, `components/layout/VconShell.tsx` | SOURCE + RBAC TS tests PASS | NOT live-tested | None | +| SDK registered clients, token issuance/revoke/rotate | `core/sdk/sdkManager.ts`, `app/api/clients/**` | SOURCE + client/auth TS regression PASS | NOT live-tested | None | +| Next.js HTTP event ingestion | `app/api/events/route.ts` | SOURCE + TS tests PASS | No live writes performed | None | +| Express + Socket.IO ingestion | `server/index.js`, `server/socket/handlers.js` | SOURCE + sync/SDK tests PASS | Not live end-to-end verified | None | +| Rate limit, dedupe, payload validation | `server/socket/handlers.js`, `app/api/events/route.ts` | SOURCE + tests PASS | Not live verified | None | +| Redis ingestion queue/fallback | `server/index.js:165-193`, `core/data/buffer.ts` | SOURCE + TS storage tests; R-01 ACK durability risk | Not guaranteed lossless on outage | None | +| CSV partition/rotation/flush | `server/workers/csvWriter.js`, `core/Storage/csvStorage.ts` | SOURCE + regression; ingestion image CI build PASS | R-02 crash replay semantics NOT confirmed | None | +| Compression and S3 archival | `core/Storage/s3Archive.ts`, `core/Storage/csvStorage.ts` | SOURCE + storage tests | No live S3 credentials or archive tests | None | +| Data browser, export CSV/JSON/NDJSON | `app/api/export/route.ts`, `app/api/data/route.ts` | SOURCE + TS tests | NOT live-tested | None | +| Storage quota / disk admission | `core/Storage/csvStorage.ts` | SOURCE + tests; 507 codes | NOT live-tested | None | +| Dashboard, client status, counts | `lib/vconUiHelpers.ts`, `app/vcon/clients/page.tsx` | SOURCE; client status helper tests; stale old regex CI-04 | No live user session proof | None | +| Prometheus and metrics bridge | `app/api/prometheus/route.ts`, `server/index.js` | SOURCE + tests | NOT live authenticated | None | +| Webhook alerts and SSRF/HMAC | `core/alerts/webhookNotifier.ts` | SOURCE + tests | NOT live-tested | None | +| CloudCmd file browser | `server/filesManager.js`, `tests/files-manager.test.ts` | SOURCE; files-manager suite 4 tests PASS | NOT live authenticated | None | +| MCP engine (17 permissions) | `server/mcp/mcpEngine.js`, `app/api/mcp/route.ts` | SOURCE + MCP server suite 3 tests PASS; write flags default false | NOT live connected/authenticated | None | +| Python SDK v2.2 + compat package | `sdk/python/datastack/datastack/**`, `sdk-python/**` | Windows CI **14/14** PASS | No real external SDK session | None | +| Node SDK v2.2 + offline spool | `sdk/node/datastack/**` | Native Windows SDK compile PASS; SDK-specific TS regression PASS | No live external SDK session | None | +| PWA/offline shell | `app/**`, `public/**` | SOURCE + phase06 suite PASS | Browser offline not live-tested | None | +| Docker Compose stack | `docker/docker-compose.yml` | CI config PASS, ingestion image PASS | Deployed app already serves public health; deployed SHA unverified | None | +| Docker web/proxy image | `docker/Dockerfile.web`, `docker/Dockerfile.proxy` | Web layer fail on containerd; proxy unrun | Existing Coolify instance appears online but CI image parity unverified | None | +| Health endpoint | `app/api/health/route.ts` | LIVE READ `status:online` on 2026-11-22T00:01:19Z | Confirmed public API liveness; not end-to-end readiness | None | + +## Mandatory post-phase update convention + +After EACH P01/P02/P03 close, amend only affected table rows: +- `Feature/capability` (exact added/changed behavior or `UNCHANGED`) +- `Source file/line + new SHA` +- `CI evidence` (run URL, pass/failed count, build exit) +- `Live status` (READ ONLY, AUTHENTICATED VERIFIED, NOT VERIFIED, etc.; no fictitious claims) +- `Compatibility` (UI/auth/SDK/API/storage preserved? test proof) +- `Risk remaining` and next approval. +Do not mark "actual working" simply because a patch compiled. Distinguish source, CI, and actual deployment. +## Change log +- P00: No feature added/modified. `notes/**` documentation only. 0/3 implementation phases completed. +- P01: pending. +- P02: pending. +- P03: pending. diff --git a/notes/006-BASELINE-AND-PATCH-EVIDENCE.md b/notes/006-BASELINE-AND-PATCH-EVIDENCE.md new file mode 100644 index 0000000..405006f --- /dev/null +++ b/notes/006-BASELINE-AND-PATCH-EVIDENCE.md @@ -0,0 +1,153 @@ +# DataStack — Baseline Freeze, Code Proof & Proposed Patch Index + +Frozen source baseline SHA: `d725841ebc828d58cbe90f517da9e600f23c4630` (2026-11-22, Asia/Dhaka). +Source verification: exact GitHub/GitLab/Gitea remote `main` and local HEAD matched this SHA. Local working tree clean; local `origin` configured to `MdNurnobi-Dev/DataStack.git`, and its *local tracking ref* was stale (`725d3c5`), so `ahead 5` alone was NOT evidence GitHub `JILLU-Games` was behind. GitHub `JILLU-Games` read-only branch HEAD verified at `d725841ebc828d58cbe90f517da9e600f23c4630`. No force push, ref rewrite or upstream reset authorized. + +## A. Proof excerpts (verbatim short source fragments, exact baseline files/lines) + +### CI-02 — stale lockfile contract (P01) +`tests/phase07-deployment.test.ts:648-650`: +```ts +test("FINAL-DEPLOY-FIX-01: Single npm lockfile, Dockerfile .runtime-logs ownership parity, 8-table prisma sync, and cold-boot MCP bridge hydration", async () => { + assert.equal(existsSync(resolve(process.cwd(), "bun.lock")), false, "Stale bun.lock must stay removed"); +``` +`git ls-files bun.lock` listed the file on baseline. **Patch reference**: check if any build consumer references it; if none, remove exactly `bun.lock` while preserving `package-lock.json`. Re-run this specific test. + +### CI-04 — stale UI test expectation (P01) +`tests/v2-phase01-storage-quotas.test.ts:788-804`: +```ts +assert.match(clientsPageSrc, /function isClientExpired\(/); +assert.match(clientsPageSrc, /const expired = !revoked && isClientExpired\(client\.expiresAt,\s*now\)/); +``` +`app/vcon/clients/page.tsx:70-85`: +```ts +const counts = useMemo( + () => summarizeUiClientCounts(clients, now), + [clients, now], +); +const computedStatus = classifyUiClientStatus(client, now); +``` +`lib/vconUiHelpers.ts:717,730,754` exports `isUiClientExpired`, `classifyUiClientStatus`, `summarizeUiClientCounts`. +**Proposed patch shape** (planning-only, not applied): +```diff +--- a/tests/v2-phase01-storage-quotas.test.ts ++++ b/tests/v2-phase01-storage-quotas.test.ts +@@ obsolete regex-based UI expectations, lines 788-804 @@ +- assert.match(clientsPageSrc, /function isClientExpired\(/); ++ assert.match(clientsPageSrc, /summarizeUiClientCounts/); ++ assert.match(clientsPageSrc, /classifyUiClientStatus/); +``` +Also **replace the other 3 obsolete implementation-detail regexes** with behavioral assertions against exported helper and page filtering/count data; do not just drop coverage. Exact final diff is generated only during approved P01. + +### CI-03 — test-process hang anchor (P01) +`core/data/buffer.ts:347-352,379-391`: +```ts +state.flushTimer = setInterval(() => { + void this.flush().catch(() => undefined); +}, state.batchFlushIntervalMs); +``` +Run #45 suite exit codes 124: `r3-phase04-ui-metrics-rbac`, `r4-phase03-sdk-deploy-auth`, `r5-phase01-mcp-foundation-logs`, `r5-phase03-mcp-ui`. Timer is a **suspect**, not a proven sole root cause. **Patch reference:** instrument open handles and add narrow teardown in four named tests; `core/data/buffer.ts` is **read-only** unless new approval. + +### CI-05 — compiler-source proof (P01) +`tsconfig.json:27`: +```json +"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"] +``` +Run #45 job 68 examples: +```text +tests/phase01-auth.test.ts(88,17): error TS2540: Cannot assign to 'NODE_ENV' because it is a read-only property. +tests/r5-phase02-mcp-server.test.ts(510,8): error TS18046: 'deniedClientCreate.content' is of type 'unknown'. +tests/v2-phase01-storage-quotas.test.ts(762,64): error TS2554: Expected 0 arguments, but got 1. +``` +**Patch reference:** exact 9 test files / line map in `001`. Use correct typings/type guards and current API signatures, not TS ignore or global tsconfig exclusion. + +### CI-01 — Windows clean-install lockfile mismatch (P01) +Run #42 job 63: +```text +npm error Missing: @img/sharp-win32-x64@0.35.5 from lock file +npm error Missing: @unrs/resolver-binding-win32-x64-msvc@1.13.0 from lock file +``` +Run #45 job 71 showed success **only after temporary runner** `npm install --package-lock-only`, followed by `npm ci` and Node SDK build. **Patch reference:** `package-lock.json` optional/native dependencies, retain pinned package graph; verify unchanged tracked lock under CI on Linux/Windows. + +### CI-06 — Python runner packages (P03) +Run #45 job 67: +```text +E: Failed to fetch ... python3.9-venv_3.9.2-1+deb11u7_amd64.deb 404 Not Found +``` +`.gitea/workflows/datastack-ci.yml` Python install step uses `apt-get update -qq; apt-get install -y python3 python3-venv python3-pip`. Run #48 job 74 executes 14 tests PASS on Windows. **Patch reference:** CI job Python provisioning only. + +### CI-07 — Docker web layer failure (P03) +`docker/Dockerfile.web:13`: +```dockerfile +COPY --from=dependencies /app/node_modules ./node_modules +``` +Run #50 job 76: +```text +failed to create diff tar stream: lstat .../app/node_modules/@uiw: no such file or directory +``` +`docker/Dockerfile.server` produced tagged ingestion image; Docker Compose `config --no-interpolate --quiet` PASS. **Patch reference:** `.gitea/workflows/datastack-docker-ci.yml` runner/cache/BuildKit; Dockerfile modification conditional on independent reproduction and new approval. + +### CI-08 — npm advisory summary (P03) +Run #50 job 76: +```text +24 vulnerabilities (23 moderate, 1 high) +26 vulnerabilities (25 moderate, 1 high) +``` +**No evidence of specific advisory IDs yet.** Patch reference `package.json`/`package-lock.json` only after isolated `npm audit --json` with package/CVE/runtime attribution. Never add counts or claim 50 unique advisories. + +### R-01 — Redis volatile fallback ACK risk (P02, conditional) +`server/index.js:165-193`: +```js +try { + bufferLen = await redis.rpush(REDIS_BUFFER_KEY, ...serialized); + redisConnected = true; +} catch { + redisConnected = false; +} +// ... +bufferQueue.push(...events); +return bufferQueue.length; +``` +`server/socket/handlers.js:583-602`: +```js +bufferQueue.push(event); +try { + await onBufferPush(); +} catch (persistErr) { + // callback BUFFER_PERSIST_FAILED +} +if (callback) callback({ success: true, eventId }); +``` +**Potential issue**: a successful ACK does not necessarily mean durable storage after Redis failure; test first. Proposed patch is NOT predetermined. + +### R-02 — CSV before Redis trim (P02, conditional) +`server/index.js:981-994`: +```js +const res = await writer.writeBatch(parsedBatch); +// ... +await redis.ltrim(REDIS_BUFFER_KEY, rawItems.length, -1); +``` +**Potential issue**: crash after partial write or before list trim may replay buffered rows. Test with injected partial failures and compare event IDs before prescribing exactly-once guarantee. + +### R-03 — Health shows service liveness (P02, conditional) +`server/index.js:1939-1955`: +```js +const redisBufferSize = await getRedisBufferLen(); +res.json({ status: 'online', /* ... */ redisConnected, /* ... */ }); +``` +`app/api/health/route.ts:95-105`: +```ts +const bufferStats = await DataBufferEngine.getBufferStatsAsync(); +return NextResponse.json({ status: 'online', service: 'DataStack Web API', /* ... */ }); +``` +**Potential issue**: liveness is not transactional readiness. Preserve existing public endpoint shape and verify separately. + +## B. Source-to-patch navigation + +- P01: `package-lock.json`; `bun.lock`; `tests/phase07-deployment.test.ts:649`; `tests/v2-phase01-storage-quotas.test.ts:762,783,788-804`; 4 timeout tests; 9 TS test files; `.gitea/workflows/datastack-ci.yml`. +- P02 conditional: `server/index.js:165-193,943-994,1939-1956`, `server/socket/handlers.js:583-602`, `server/workers/csvWriter.js:265-323`, `app/api/health/route.ts:95-111`; associated test files. Read-only until fault reproduction. +- P03: `.gitea/workflows/datastack-ci.yml`, `datastack-python-ci.yml`, `datastack-docker-ci.yml`, `datastack-eslint-ci.yml`; `package.json` / `package-lock.json` only justified dependency changes. Source Dockerfile edits separately approved only if needed. + +## C. Documentation baseline update record + +This file **freezes the source baseline**, not the future documentation-commit SHA. After adding `notes/**`, record commit SHA and three remote SHAs in `notes/003` and the user-facing result. Subsequent P01 implementation should inspect `git diff d725841ebc828d58cbe90f517da9e600f23c4630..HEAD` and permit only these notes as intervening changes; any unexpected source edits require a new review.