10 KiB
DataStack — Production Update Roadmap / Exact Scope Lock
Planning baseline: d725841ebc828d58cbe90f517da9e600f23c4630 (frozen; documentation commits will advance Git HEAD but NOT this source baseline).
Current state: P01 CLOSED and CI-verified (1 of 3 phases complete); P02 and P03 NOT STARTED; P02 requires new explicit approval.
Prerequisite for any phase: compare current HEAD and changed paths with this baseline and previously committed notes. For new commits since source baseline, inspect exact diff; no implicit rebase or silent discard.
Global non-negotiable Scope Lock (applies to ALL phases)
- Existing UI/UX, public API response contracts, authentication, RBAC, JWT/SDK interoperability, MCP permissions, data retention, CSV content, persistent databases, keys, Redis values, S3/R2 objects, Coolify configuration and production deployment: unchanged unless an exact targeted fix is proven and separately approved.
- No live event sending, live database mutation, destructive test, production restart/deploy, automatic package vulnerability
--force, tag or release without explicit approval. - Limit changes to exact allowlisted paths per phase. Any additional path, altered schema, migration, UI behavior, API code, or security default = STOP / document scope exception / obtain new approval. No broad refactoring.
- One phase = targeted patch, one focused CI verification, exact diff/security review, log/register updates, one Git commit and 3-remote sync verification. No repetitive test loops or local production builds. If CI infra is broken, report a blocker rather than spoof PASS.
- Phase close requires written evidence with exit codes, commit SHA, changed paths, feature preservation checklist, remaining issues and a next-phase approval command.
- Reference test logs and source code, not README assertions alone. Code candidate risks require fault reproduction before production behavior changes.
- Secrets, personal data, local
.envand runtime tokens never copied into reports, logs or commits. Never perform unsafegit reset --hard/git clean -fd. AGENTS.mdcross-IDE memory requirement: updateMEMORY.mdonly under explicit scope exception, otherwise provide exact handoff Markdown innotes/003-PHASE-STATUS-LOG.mdfor copy/sync.
Phase P01 — Dependency, tests & type-contract reconciliation (CI-01…CI-05)
Goal: fix confirmed test/lockfile blockers without changing actual app/SDK runtime behavior. Allowed paths only:
package-lock.json;bun.lock(delete ONLY after verifying no build consumers; npm lock stays canonical).tests/phase07-deployment.test.ts;tests/v2-phase01-storage-quotas.test.ts;tests/r3-phase04-ui-metrics-rbac.test.ts;tests/r4-phase03-sdk-deploy-auth.test.ts;tests/r5-phase01-mcp-foundation-logs.test.ts;tests/r5-phase03-mcp-ui.test.ts;tests/phase01-auth.test.ts;tests/phase03-storage-buffer.test.ts;tests/r3-phase01-security-vault-webhooks.test.ts;tests/r3-phase02-storage-scheduler-gzn.test.ts;tests/r4-phase01-security-boot.test.ts;tests/r4-phase02-ingestion-audit-streaming.test.ts;tests/r5-phase02-mcp-server.test.ts;tests/v2-phase02-alerts-ip-prometheus.test.ts;.gitea/workflows/datastack-ci.yml(narrow clean-install/test harness only);notes/001…006(status/evidence only).
Not allowed: app/**, components/**, lib/**, core/**, server/**, docker/**, tsconfig.json, package.json or SDK runtime files. If open-handle leak proves core timer bug, STOP and request explicit additional exception for core/data/buffer.ts instead of altering it as a side effect.
Patch tasks and references:
- CI-01: deterministic regenerated
package-lock.jsonwith cross-OS optional entries, verify cleannpm cion Linux/Windows with same pinned npm; no dependency version bump unless separately approved. Reference.gitea/workflows/datastack-ci.yml:91-120. - CI-02: check
bun.lockconsumers, then remove stale lock or resolve test contract with proof;tests/phase07-deployment.test.ts:649. - CI-03: isolate 4 suites, enumerate open handles/intervals, add explicit test teardown / mocks; source
core/data/buffer.ts:347-391is read-only diagnostic reference. Avoidprocess.exit()hacks and wholesale timeout increases. - CI-04: replace obsolete regex tests in
tests/v2-phase01-storage-quotas.test.ts:788-804with precise shared-helper/integration assertions matchingapp/vcon/clients/page.tsx:70-95andlib/vconUiHelpers.ts:717-780. - CI-05: fix 43 test-code type errors across 9 listed files (see
001). Keep strict TS compile; preserve production API signatures.tsconfig.json:27includes tests by glob. - P01 acceptance: 24/24 test files exit 0 (no timeout),
tsc --noEmitzero errors,npm run lintPASS, Python 14/14 PASS regression, Windows/Linux originalnpm ciPASS without diagnostic lock regeneration; one focused Gitea matrix. If network/runner blocker, mark BLOCKED not PASS. - Close: update
notes/003andnotes/005only after verifying; commit and sync.
Phase P02 — Ingestion reliability / error-handling acceptance gates (R-01…R-03)
Goal: reproduce or dismiss source-observed risk, preserve response semantics unless proven fix is necessary. Risk ≠ bug until test fails.
Diagnostic-only allowlist: server/index.js, server/socket/handlers.js, server/workers/csvWriter.js, core/data/buffer.ts, app/api/health/route.ts, tests/phase02-sync.test.ts, tests/phase03-storage-buffer.test.ts, tests/r4-phase02-ingestion-audit-streaming.test.ts, notes/001…006.
Default write allowlist: only the THREE listed test files plus notes/001…006. Runtime source files above are READ ONLY until a reproducible failing test demonstrates the need; then the P02 approval must name an exact method/line range and explicitly authorize the smallest required change. This is not preapproval for changing server source.
Test matrix (disposable, never production):
- R-01: fake Redis
rpushrejection + ACK + simulated process exit/reopen; ensure API explicitly returns failed/pending unless durable, or document intended guarantee. Preserve existing duplicate/rate/size/auth error codes. - R-02: injected CSV partial write, simulated failure before Redis
ltrim, restart and compare event IDs/rows; decide documented at-least-once semantics vs dedupe strategy before altering storage. - R-03: separate liveness/readiness checks with mocked Redis/PG errors; old public
/healthstays backwards compatible. Verify startup fail-closed when secrets or DB unavailable. - Review source CORS/WebSocket origin and security access policy as read-only, no default policy change automatically.
- P02 acceptance: targeted fault-injection tests PASS, no acknowledged-event loss in the documented guarantee, no unbounded duplicate amplification, correct 503/507/429/401 semantics and recovery. If risk not reproduced, close as verified with zero runtime source changes; never invent a fix.
- Close: record precise outcome and next P03 approval.
Phase P03 — CI runner, Docker images, dependency security, release evidence (CI-06…CI-08)
Goal: reproducible Gitea verification, safe dependency advisory triage, complete all three Docker builds, confirm non-mutating production health.
Default write allowlist: .gitea/workflows/datastack-ci.yml, .gitea/workflows/datastack-python-ci.yml, .gitea/workflows/datastack-docker-ci.yml, .gitea/workflows/datastack-eslint-ci.yml, package.json, package-lock.json, notes/001…006.
Conditional, separate approval only: docker/Dockerfile.web / docker/Dockerfile.server if isolated runner/containerd evidence establishes source-level Dockerfile defect. Not allowed to change Compose deployment services, secrets, volumes or Dockerfiles merely to work around a runner failure.
Tasks:
- CI-06: replace stale Debian Bullseye
apt-getdependency step with preinstalled compatible Python image/toolchain OR canonical Windows Python job; avoid duplicate CI runs. - CI-07: inspect runner daemon snapshotter/filesystem free space, cache/BuildKit and known
node_modules/@uiwlayer failure; use properly configured disposable BuildKit builder or another isolated runner. If reproducible Dockerfile defect is proven, obtain narrow scope amendment. Build images without running containers; nodocker system pruneor shared cache destruction. - CI-08: run read-only detailed
npm audit --jsonin isolated runner, map actual direct/transitive paths and runtime impact, document advisory IDs/severity, apply minimal compatible pin bumps only where necessary. Nevernpm audit fix --force. - Verify canonical package metadata and Git remote SHA; GitHub login/account issue is already resolved and is not a current blocker.
- P03 acceptance: all 3 images build, Compose config PASS, Next standalone PASS, lint + strict TS PASS, 24/24 TS suites no hang, Python 14/14, regression error codes unchanged, advisory disposition documented, read-only public health PASS. Any real deployment/production migration is a separate approval.
- Close: exact diff, one CI acceptance run (plus only blocked job rerun where necessary), statuses/feature matrix update, one commit, synced remote HEADs, phase CLOSED.
Sequence and approval gate
P01 CLOSED -> P02 awaiting approval -> P03 awaiting P02 closure. P01 acceptance evidence: Gitea CI #59 (Windows + Linux clean install, 24/24 tests, Node SDK, Next standalone) and #60 (strict TypeScript + ESLint), with historical #59 single test-type error corrected before #60. No actual app/runtime/UI/SDK change or live deployment.
Approval command for next turn:
APPROVED — DataStack Production Readiness P02. Follow notes/002 P02 Exact Scope Lock. Verify R-01 Redis ACK durability, R-02 CSV crash/replay duplication, and R-03 liveness/readiness with disposable fault-injection tests. App/runtime source READ-ONLY unless a reproducible bug and separate exact patch approval. Preserve all existing UI, API, auth, SDK, data, and production. One focused CI, forensic diff, notes update, commit, three-remote sync. No deploy, releases, migrations or scope expansion.