chore: establish cross-IDE synchronization protocol
Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
This commit is contained in:
1 parent
05407fd338
commit
4279da45bb
14 files changed
+1162
No files matched your search
@@ -0,0 +1,17 @@
|
||||
# Universal Cross-IDE AI Rules (`.ai-rules`)
|
||||
|
||||
CRITICAL RULE: This is a multi-IDE project. Whenever any modification, update, or bug fix is completed in our session, you must automatically update the `MEMORY.md` file or provide the exact markdown blocks so the project context remains perfectly synchronized across all IDEs.
|
||||
|
||||
## Session Startup Protocol
|
||||
1. Read `MEMORY.md` and `AGENTS.md` before planning or editing code to restore accurate multi-IDE project context, runtime boundaries, and architecture constraints.
|
||||
2. Treat `MEMORY.md` as the single cross-IDE source of truth for current tech stack, directory structure, active features, and recent fixes.
|
||||
|
||||
## Architectural & Engineering Guardrails
|
||||
- **Dual-Runtime Boundary**: Keep Next.js 15 web/API routes (`app/`, Port `3000`) and Express 5 + Socket.IO ingestion (`server/index.js`, Port `4000`) cleanly decoupled. Never initialize Socket.IO servers inside Next.js API routes.
|
||||
- **Storage Boundary**: Persist telemetry events exclusively to partitioned CSVs (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`) via `DataBufferEngine` / `CSVStorageManager` / Redis queue. Use PostgreSQL (`pg` + `prisma/schema.prisma`) strictly for metadata (`admin_users`, `admin_login_attempts`, `client_sdks`, `deleted_clients`, `admin_settings`).
|
||||
- **UI Consistency**: Assemble pages from existing primitives in `components/ui/` and `components/layout/` adhering to design tokens in `app/globals.css` (`--radius: 0rem`, `--shadow: none`).
|
||||
- **Non-Destructive Persistence**: Never add destructive SQL (`DROP`, `TRUNCATE`) or automatic startup scripts that purge production CSV partitions without hydrated DB settings confirmation.
|
||||
|
||||
## Verification & Sync Before Finishing Any Change
|
||||
1. Run `npm run lint` and `npm run build` plus relevant test suites (`npx --yes tsx --test tests/phase*.test.ts` and `python3 -m unittest tests/phase05_sdk_test.py` when applicable).
|
||||
2. Update `MEMORY.md` with any structural, dependency, endpoint, environment variable, or status changes made during the session before finishing.
|
||||
@@ -24,3 +24,8 @@
|
||||
- **Font Stack**: `"Segoe UI", "Segoe UI Web", system-ui, -apple-system, "Helvetica Neue", sans-serif`.
|
||||
- **Border Radius**: `--radius: 0rem` across all UI components.
|
||||
- **Shadows**: None (`--shadow: none`). Crisp 1px borders with dark and light theme support.
|
||||
|
||||
## 5. Cross-IDE Persistent Memory (`MEMORY.md` & `.ai-rules`)
|
||||
|
||||
- CRITICAL RULE: This is a multi-IDE project. Whenever any modification, update, or bug fix is completed in our session, automatically update `MEMORY.md` (or provide exact markdown blocks) so project context remains synchronized across VS Code, Google Antigravity, Cursor, and other IDEs (see `.ai-rules`).
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
# MEMORY.md — DataStack Cross-IDE Persistent Project Memory
|
||||
|
||||
> **Last Synchronized**: 2026-11-17
|
||||
> **Status**: Production-Ready (`Phase-01` through `Phase-07` Completed & Verified)
|
||||
> **Scope Rule**: Maintain this file accurately across VS Code, Google Antigravity, Cursor, and any other IDE after every modification, feature update, or bug fix (see `.ai-rules` and `AGENTS.md`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Project Identity & Core Principles
|
||||
- **Project**: **DataStack** — Self-hosted, real-time telemetry ingestion, admin governance (`/vcon/*`), and partitioned CSV storage/export platform.
|
||||
- **Runtime Split (CRITICAL)**:
|
||||
- **Port `3000` (`web`)**: Next.js 15 App Router — public landing (`/`), PWA lifecycle, `/vcon/*` Admin Console, REST ingestion (`POST /api/events`), metrics/settings/client APIs, and streaming exports (`GET /api/export`).
|
||||
- **Port `4000` (`ingestion`)**: Dedicated Node.js Express 5 + Socket.IO server (`server/index.js`) — WebSocket handshakes, JWT client auth, rate limiting, idempotency deduplication, Redis/RAM batching, and CSV flush worker.
|
||||
- Socket.IO connections NEVER run inside Next.js serverless routes; they run strictly on Port `4000`.
|
||||
- **Storage & Database Boundary (CRITICAL)**:
|
||||
- **PostgreSQL (`pg` + `prisma/schema.prisma`)**: Stores strictly control-plane metadata (`admin_users`, `admin_login_attempts`, `client_sdks`, `deleted_clients`, `admin_settings`). **Never insert per-row telemetry events into PostgreSQL.**
|
||||
- **Partitioned CSV Storage (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`)**: All ingested telemetry events flush in batches to daily partitioned CSV files with cooperative file locking, configurable size rotation (`fileRotationMb`, e.g., `-part2.csv`), and retention pruning (`autoPruneDays`).
|
||||
- **Redis (`ioredis`)**: Shared cross-process buffer queue (`datastack:buffer:events`), recent event cache (`datastack:recent:events`), and distributed flush lock (`datastack:flush:lock`). Persisted in Docker via `--appendonly yes` (`redisdata:/data`). Falls back gracefully to local RAM when `REDIS_URL` is omitted.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tech Stack & Local Setup
|
||||
|
||||
### Stack
|
||||
- **Frontend / Web API**: Next.js `15.4.9` (App Router, `output: "standalone"`), React `19.2.1`, TypeScript `5.9.3`, Tailwind CSS v4 (`@tailwindcss/postcss`), `lucide-react`, `motion`.
|
||||
- **Ingestion Server**: Node.js `22` / Express `5.3.0`, Socket.IO `4.8.4`.
|
||||
- **Persistence & Caching**: `pg` `8.23.1` (PostgreSQL 16 in Docker), `ioredis` `6.0.0` (Redis 7 AOF in Docker), `csv-parse` `7.0.3`, `json2csv` `6.0.0-alpha.2`.
|
||||
- **Auth & Security**: `jsonwebtoken` `9.0.3`, Node `crypto.scrypt` password hashing, WebCrypto HMAC-SHA256 admin session cookies (`middleware.ts`), CSRF `Origin`/`Referer` vs `Host`/`X-Forwarded-Host` checks, `DATASTACK_INTERNAL_SECRET` timing-safe bridge auth between Port `3000` ↔ `4000`.
|
||||
- **Client SDKs**: Official pip-installable Python SDK in `sdk/python/datastack/` (`datastack`) + backward-compatible wrapper `sdk-python/` (`datastack-sdk`).
|
||||
|
||||
### Environment & Setup Commands
|
||||
1. **Configure Environment**: Copy `.env.example` to `.env` (in production `NODE_ENV=production`, all 4 secrets MUST be 32+ chars: `DATASTACK_ADMIN_SESSION_SECRET`, `DATASTACK_JWT_SECRET`, `DATASTACK_ADMIN_SETUP_KEY`, `DATASTACK_INTERNAL_SECRET`; optional `DATABASE_URL` and `REDIS_URL`).
|
||||
2. **Install Dependencies**: `npm ci` (or `npm install`) using root `package.json` + `package-lock.json`.
|
||||
3. **Run Locally (Two Terminals)**:
|
||||
- Terminal 1 (Port `3000`): `npm run dev`
|
||||
- Terminal 2 (Port `4000`): `npm run server`
|
||||
4. **Build, Lint & Test Verification**:
|
||||
- Build: `npm run build`
|
||||
- Lint: `npm run lint` (`eslint .`)
|
||||
- TS Test Suite (Phases 01–03, 06, 07): `npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts`
|
||||
- Python SDK Tests: `python3 -m unittest tests/phase05_sdk_test.py`
|
||||
|
||||
---
|
||||
|
||||
## 3. Folder Structure & File Architecture
|
||||
|
||||
```text
|
||||
datastack/
|
||||
├── .ai-rules # Universal cross-IDE rules (enforces MEMORY.md sync)
|
||||
├── AGENTS.md # Engineering rules, UI single-source-of-truth & memory sync
|
||||
├── MEMORY.md # Persistent cross-IDE project memory (this file)
|
||||
├── package.json / package-lock.json
|
||||
├── middleware.ts # /vcon/* auth guard & Origin/X-Forwarded-Host CSRF enforcement
|
||||
├── app/
|
||||
│ ├── (landing)/page.tsx # Public landing page
|
||||
│ ├── globals.css # Tailwind v4 + centralized CSS variables (--radius: 0rem, --shadow: none)
|
||||
│ ├── layout.tsx / not-found.tsx / error.tsx
|
||||
│ ├── api/
|
||||
│ │ ├── auth/ # Admin login, logout, setup (/vcon/setup), profile update
|
||||
│ │ ├── clients/ # CRUD clients, token regeneration, revocation
|
||||
│ │ ├── data/ # Paginated CSV search/query across partitions
|
||||
│ │ ├── events/ # Authenticated HTTP ingestion (single/batch/file records) & recent list
|
||||
│ │ ├── export/ # Streaming CSV / JSON filtered export (UTC dates, optional flush=false)
|
||||
│ │ ├── health/ # Web health + client Bearer token probe + async buffer depth
|
||||
│ │ ├── metrics/ # Aggregated Port 3000 + Port 4000 + DB + CSV metrics
|
||||
│ │ ├── settings/ # Allowlisted runtime config get/post + live Port 4000 sync
|
||||
│ │ └── storage/ # CSV partition listing & manual cross-process buffer flush
|
||||
│ └── vcon/ # Protected Admin Console pages:
|
||||
│ ├── dashboard/ clients/ data/ export/ login/ profile/ sdk/ settings/ setup/ simulator/ storage/
|
||||
├── components/
|
||||
│ ├── ui/ # Single-source-of-truth UI primitives (Button, Input, Table, Modal, Panel, etc.)
|
||||
│ ├── layout/ # MainLayout, Navbar, Sidebar, PageShell, AdminPageHeader, Footer
|
||||
│ ├── sections/ # Modular feature sections (DashboardOverview, ClientsTable, DataBrowser, ExportWorkspace, SettingsWorkspace, etc.)
|
||||
│ └── pwa/ # PwaLifecycle.tsx service worker registration
|
||||
├── core/
|
||||
│ ├── adminSettings/ # settingsManager.ts (DB hydration, Port 4000 sync, unhydrated prune guard)
|
||||
│ ├── apiEngine/ # apiService.ts (dashboard metrics & Port 4000 internal bridge queries)
|
||||
│ ├── auth/ # adminUsers.ts, adminSession.ts (scrypt, brute-force throttle, tokenVersion), jwt.ts
|
||||
│ ├── data/ # buffer.ts (Redis + RAM queue, distributed lock, graceful SIGTERM/SIGINT flush)
|
||||
│ ├── Database/ # database.ts (pg.Pool + idempotent DDL + additive ALTER TABLE guards)
|
||||
│ ├── sdk/ # sdkManager.ts (DB JWT hydration, client lifecycle, deleted_clients tombstone)
|
||||
│ ├── Security/ # security.ts (rate limit, maxPayloadSizeKb, idempotency, internal headers)
|
||||
│ └── Storage/ # csvStorage.ts (cooperative lock, rotation, cached streaming row count, prune)
|
||||
├── server/
|
||||
│ ├── index.js # Express 5 + Socket.IO (Port 4000), /health, /internal/* bridge, PG sync, graceful exit
|
||||
│ ├── socket/handlers.js # Handshake JWT auth, ping_verify, sync_data, batch_sync
|
||||
│ └── workers/csvWriter.js # Partitioned CSV batch writer & rotation/pruning worker
|
||||
├── sdk/python/datastack/ # Official pip-installable Python SDK (`datastack`)
|
||||
│ └── datastack/ # client.py, config.py, heartbeat.py, spool.py, exceptions.py
|
||||
├── sdk-python/ # Backward-compatible re-export package (`datastack-sdk`)
|
||||
├── docker/
|
||||
│ ├── docker-compose.yml # web, ingestion, proxy (Nginx), postgres (pgdata), redis (redisdata AOF)
|
||||
│ ├── Dockerfile.web # Multi-stage standalone Next.js build + safe /app/storage ownership check
|
||||
│ ├── Dockerfile.server # Multi-stage Port 4000 ingestion server + safe /app/storage ownership check
|
||||
│ ├── Dockerfile.proxy # Nginx reverse proxy container
|
||||
│ └── coolify-nginx.conf # Routes `/health` & `/socket.io/` -> `ingestion:4000`, `/` -> `web:3000`
|
||||
├── nginx/default.conf # Local Nginx proxy config matching coolify-nginx.conf
|
||||
├── prisma/schema.prisma # Reference PostgreSQL metadata schema
|
||||
├── public/ # PWA manifest.json, sw.js, icons (192, 512, maskable, apple-touch)
|
||||
├── storage/data/ # Runtime partitioned CSV root ({clientId}/{YYYY-MM-DD}.csv)
|
||||
├── docs/ # Official documentation suite:
|
||||
│ ├── MANIFEST.md # Documentation index, breakdown, and governance rules
|
||||
│ ├── users/ # End-user & operator guides (installation, features, tutorials, faq)
|
||||
│ └── dev/ # Contributor & developer guides (architecture, setup, build, contributing)
|
||||
├── notes/ # Audit logs & phase implementation plans (gitignored)
|
||||
└── tests/ # Automated verification suites:
|
||||
├── phase01-auth.test.ts # Auth fail-closed, tokenVersion, brute-force lockout, Bearer revocation, CSRF
|
||||
├── phase02-sync.test.ts # Internal bridge auth, settings sync, JWT rotation kick, rate/security metrics
|
||||
├── phase03-storage-buffer.test.ts # Redis durability, cooperative CSV locks, streaming row count, export filters
|
||||
├── phase05_sdk_test.py # Python SDK WebSocket/HTTP fallback, offline spool, file streaming
|
||||
├── phase06-pwa-landing.test.ts # PWA manifest/icons/sw + navigation routes
|
||||
└── phase07-deployment.test.ts # Compose/Redis AOF, Nginx /health, lockfile, prune guard, shutdown flush
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Active Features & Development Status
|
||||
|
||||
All planned hardening & deployment phases (**Phase-01 through Phase-07**) are **Complete & Verified**:
|
||||
|
||||
1. **Admin Authentication & CSRF Hardening (Completed — Phase-01)**
|
||||
- Fail-closed secret enforcement in `NODE_ENV=production` (`/vcon/setup` bootstrap via `DATASTACK_ADMIN_SETUP_KEY`).
|
||||
- Password hashing via `crypto.scrypt`, session invalidation via DB-backed `tokenVersion`, composite IP+email login lockout (`admin_login_attempts`), and strict `Origin`/`Referer` vs `Host`/`X-Forwarded-Host` CSRF verification.
|
||||
2. **Real-Time Dual-Runtime Sync (Completed — Phase-02)**
|
||||
- Authenticated `/internal/*` bridge between Port `3000` and Port `4000` (`DATASTACK_INTERNAL_SECRET`, backward-compatible Coolify fallback).
|
||||
- Live settings & client updates (create/rotate/revoke/delete) propagate immediately; JWT secret rotation re-signs stored client tokens and disconnects active sockets safely.
|
||||
3. **Redis Durability & Cooperative CSV Storage Engine (Completed — Phase-03)**
|
||||
- Shared Redis queue (`datastack:buffer:events`) with distributed flush lock (`NX PX`) where `LTRIM` executes strictly AFTER CSV write confirmation; automatic RAM fallback if Redis disconnects.
|
||||
- Cooperative partition file lockfiles (with stale-lock recovery), mtime/size-cached RFC 4180 streaming row counter, and filtered CSV/JSON exports (`GET /api/export`).
|
||||
4. **Admin Console UI & Data/Export Workspace (Completed — Phase-04)**
|
||||
- Built with shared `components/ui/` and `components/layout/` primitives (`0rem` radius, zero shadows, Fluent/Segoe dark & light tokens).
|
||||
5. **Production Python Client SDK (Completed — Phase-05)**
|
||||
- Supports single events (`send()`/`sync()`), auto-splitting/rate-limit-paced batches (`send_batch()`/`batch_sync()`), and direct file ingestion (`send_file()` for `.csv`, `.json`, `.jsonl`, `.ndjson` with SHA-256 deterministic `eventId`).
|
||||
- Automatic `config.datastack.py` discovery, WebSocket primary transport -> Heartbeat HTTP fallback -> `.datastack_spool/spool.jsonl` offline spool & auto-drain.
|
||||
6. **PWA & Landing Integrity (Completed — Phase-06)**
|
||||
- Complete `public/manifest.json`, icons, `public/sw.js`, and root-relative navigation links.
|
||||
7. **Docker / Coolify Deployment & Persistence Hardening (Completed — Phase-07)**
|
||||
- Redis AOF persistence (`redisdata:/data`), deterministic `package-lock.json` for `npm ci`, Nginx `location = /health` → `ingestion:4000`, `/api/health` container probes, non-blocking startup ownership check in Dockerfiles, unhydrated-DB `autoPruneDays` safety guard in `SettingsManager`, DB JWT hydration in `SDKManager`, graceful `SIGTERM`/`SIGINT` flush in `DataBufferEngine`, and non-destructive `ADD COLUMN IF NOT EXISTS` DDL guards.
|
||||
|
||||
---
|
||||
|
||||
## 5. Known Constraints & Operational Notes for Future Contributors
|
||||
- Never add per-row telemetry `INSERT` queries to PostgreSQL or add `DROP`/`TRUNCATE` auto-migrations.
|
||||
- Never move Socket.IO handlers into Next.js `app/api/*` routes.
|
||||
- When deploying updates to an existing Coolify VPS, always preserve existing `SERVICE_*` environment variables and avoid `docker compose down -v` so `pgdata`, `datastack-data`, and `redisdata` volumes remain intact.
|
||||
- After any code change, verify with `npm run build`, `npm run lint`, the TS `tests/phase*.test.ts` suite (`npx --yes tsx --test`), and `python3 -m unittest tests/phase05_sdk_test.py`, then synchronize this `MEMORY.md`.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Documentation Manifest (`docs/MANIFEST.md`)
|
||||
|
||||
> **Location**: `/docs/MANIFEST.md`
|
||||
> **Audience**: Contributors, Maintainers, and Documentation Authors
|
||||
> **Last Updated**: 2026-11-17
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory Purpose & Structure
|
||||
|
||||
The `docs/` directory contains the complete official documentation for **DataStack**, structured into distinct audiences to provide clarity, depth, and ease of navigation:
|
||||
|
||||
```text
|
||||
docs/
|
||||
├── MANIFEST.md # This document (directory index & governance)
|
||||
├── users/ # End-User & Operator Documentation
|
||||
│ ├── README.md # User documentation overview & quick links
|
||||
│ ├── installation.md # Self-hosting, Docker, and Coolify setup guides
|
||||
│ ├── features.md # Guide to /vcon/ console, client tokens, and telemetry
|
||||
│ ├── tutorials.md # Step-by-step user tutorials and integration walkthroughs
|
||||
│ └── faq.md # Frequently asked questions and troubleshooting
|
||||
└── dev/ # Developer & Contributor Documentation
|
||||
├── README.md # Developer guide overview & roadmap
|
||||
├── architecture.md # Deep dive into dual-runtime architecture and storage
|
||||
├── setup.md # Local development environment setup & configuration
|
||||
├── build.md # Build, compilation, Docker containerization & test suites
|
||||
└── contributing.md # Contribution rules, conventions, and workflow
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Directory Breakdown
|
||||
|
||||
### `docs/users/` (End-Users, System Operators & Administrators)
|
||||
Designed for non-technical users, platform administrators, and data engineers operating or integrating with a live DataStack deployment:
|
||||
|
||||
| File | Purpose | Key Topics Covered |
|
||||
| :--- | :--- | :--- |
|
||||
| `README.md` | Entry point for users | Documentation map, navigation guide, and core concepts overview. |
|
||||
| `installation.md` | Deployment guide | Production Docker Compose, Coolify 1-click deployment, environment variables, reverse proxy setup, and persistent volume management. |
|
||||
| `features.md` | Feature breakdown | Web console (`/vcon/`), client SDK tokens, real-time event simulation, daily partitioned CSV storage, streaming export, and platform settings. |
|
||||
| `tutorials.md` | Practical guides | Step-by-step tutorials: first telemetry event, connecting Python applications, viewing live events in the console, and exporting datasets. |
|
||||
| `faq.md` | Troubleshooting & FAQ | Common questions on data retention, token revocation, port configurations, CSV rotation, and connection diagnostics. |
|
||||
|
||||
### `docs/dev/` (Developers, Contributors & System Architects)
|
||||
Designed for engineers extending the codebase, fixing bugs, writing client SDKs, or contributing to the core platform:
|
||||
|
||||
| File | Purpose | Key Topics Covered |
|
||||
| :--- | :--- | :--- |
|
||||
| `README.md` | Entry point for devs | Contributor handbook, repository layout, and code navigation. |
|
||||
| `architecture.md` | Architectural blueprint | Dual-runtime model (Port 3000 Web vs Port 4000 Ingestion), Redis durability queue, partitioned CSV engine with cooperative locks, and PostgreSQL metadata store. |
|
||||
| `setup.md` | Local development | Prerequisites (Node.js 22, PostgreSQL, Redis), dual-terminal development, environment secrets, and sample configurations. |
|
||||
| `build.md` | Build & compilation | Next.js standalone compilation (`npm run build`), ESLint validation, Docker multi-stage builds, and running TypeScript (`npx tsx --test`) and Python unit test suites. |
|
||||
| `contributing.md` | Contribution standards | Multi-IDE memory synchronization (`MEMORY.md` & `.ai-rules`), UI component governance (`components/ui/`), code conventions, and Pull Request checklist. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Maintenance & Documentation Update Guidelines
|
||||
|
||||
Documentation accuracy is a core quality requirement in DataStack. Whenever modifying code or configurations, follow these update rules:
|
||||
|
||||
### When to Update `docs/users/`
|
||||
- **New User-Facing Features**: Whenever adding or altering pages in `/vcon/*` (e.g., changes to export options, client token regeneration, or UI settings), update `docs/users/features.md`.
|
||||
- **Installation / Deployment Changes**: Whenever modifying `docker/docker-compose.yml`, Dockerfiles, Nginx configurations, or `.env.example`, update `docs/users/installation.md`.
|
||||
- **API Endpoints & Integration**: If new parameters are added to `POST /api/events` or `GET /api/export`, update `docs/users/tutorials.md` and `docs/users/faq.md`.
|
||||
|
||||
### When to Update `docs/dev/`
|
||||
- **Architectural Changes**: Whenever altering the communication bridge between Port 3000 and Port 4000, changing Redis queues, or altering CSV storage mechanisms, update `docs/dev/architecture.md`.
|
||||
- **New Dependencies or Scripts**: Whenever adding npm dependencies, changing build targets, or adding test files, update `docs/dev/setup.md` and `docs/dev/build.md`.
|
||||
- **Contribution Standards**: If code standards, linting rules, or multi-IDE memory rules are revised, update `docs/dev/contributing.md` and sync with `AGENTS.md` and `.ai-rules`.
|
||||
|
||||
### Quality & Style Standards
|
||||
1. **Markdown Formatting**: Use clean GitHub-Flavored Markdown (GFM) with clear headings, bulleted lists, and language-tagged code blocks (e.g. ` ```bash `, ` ```python `).
|
||||
2. **Absolute vs Relative Links**: Inside markdown files, use relative links to reference other documentation files (e.g., `[Architecture Overview](../dev/architecture.md)`).
|
||||
3. **No Stale Placeholders**: Avoid unverified configuration keys or generic placeholder URLs; align variables with `.env.example`.
|
||||
4. **Cross-IDE Sync**: After any documentation update, update `MEMORY.md` in the root directory to maintain synchronization across all IDEs.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Developer & Contributor Guide
|
||||
|
||||
Welcome to the **DataStack Developer Documentation**. This section contains in-depth architectural specifications, development workflows, compilation instructions, and contribution conventions for engineers working on the DataStack codebase.
|
||||
|
||||
---
|
||||
|
||||
## Developer Documentation Index
|
||||
|
||||
Explore the core technical guides:
|
||||
|
||||
### 1. [Architecture Blueprint](./architecture.md)
|
||||
Detailed breakdown of DataStack's decoupled dual-runtime model:
|
||||
- **Port 3000 (`web`)**: Next.js 15 App Router, React 19, serverless REST API, streaming CSV/JSON export.
|
||||
- **Port 4000 (`ingestion`)**: Dedicated Express 5 + Socket.IO real-time ingestion server.
|
||||
- **Internal Service Bridge**: Authenticated `/internal/*` communication via `DATASTACK_INTERNAL_SECRET`.
|
||||
- **Storage Subsystem**: In-memory and Redis-backed buffering, cooperative partition file locking, and streaming row counters.
|
||||
- **PostgreSQL Control-Plane**: Schema provisioning and metadata storage rules (`pg.Pool` + `prisma/schema.prisma`).
|
||||
|
||||
### 2. [Local Development Setup](./setup.md)
|
||||
Instructions for configuring your local machine:
|
||||
- Prerequisites (Node.js 22, npm, PostgreSQL 16, Redis 7).
|
||||
- Environment variable configuration (`.env.example`).
|
||||
- Dual-terminal local workflow (`npm run dev` + `npm run server`).
|
||||
- Database schema initialization.
|
||||
|
||||
### 3. [Build, Compilation & Testing](./build.md)
|
||||
Standard build and testing commands:
|
||||
- Compiling Next.js standalone output (`npm run build`).
|
||||
- Code quality and linting (`npm run lint`).
|
||||
- Running the comprehensive TypeScript test suite with `node:test` and `tsx`.
|
||||
- Executing Python SDK unit tests.
|
||||
- Docker multi-stage container builds.
|
||||
|
||||
### 4. [Contribution Rules & Conventions](./contributing.md)
|
||||
Guidelines for submitting pull requests and modifying code:
|
||||
- **Universal Multi-IDE Memory Rule** (`MEMORY.md` & `.ai-rules`).
|
||||
- Single-source-of-truth UI primitives in `components/ui/` (`0rem` radius, zero shadows).
|
||||
- Architectural guardrails (zero telemetry rows in PostgreSQL, no Socket.IO in Next.js).
|
||||
- Pull Request and code review checklist.
|
||||
|
||||
---
|
||||
|
||||
## High-Level Repository Layout
|
||||
|
||||
```text
|
||||
datastack/
|
||||
├── app/ # Next.js 15 App Router (Frontend, /vcon/*, and /api/*)
|
||||
├── components/ # UI primitives (ui/), layout shells (layout/), and sections (sections/)
|
||||
├── core/ # Backend TypeScript modules (auth, Database, data, Storage, Security, sdk)
|
||||
├── server/ # Dedicated Port 4000 Express + Socket.IO service & CSV batch worker
|
||||
├── sdk/python/datastack/ # Official pip-installable Python SDK
|
||||
├── sdk-python/ # Backward-compatible Python SDK wrapper
|
||||
├── storage/data/ # Partitioned CSV telemetry storage root
|
||||
├── docker/ # Production Dockerfiles, Nginx proxy configs, and docker-compose.yml
|
||||
├── nginx/ # Local Nginx development configuration
|
||||
├── tests/ # Automated test suites (phase01 - phase07)
|
||||
└── docs/ # User and developer documentation
|
||||
```
|
||||
@@ -0,0 +1,105 @@
|
||||
# Architecture Blueprint
|
||||
|
||||
This document details the architectural principles, runtime boundaries, and subsystems powering DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Architectural Principles
|
||||
|
||||
DataStack is designed around three foundational engineering constraints:
|
||||
|
||||
1. **Decoupled Runtimes**: Long-lived WebSocket connections require stateful, persistent event loops, while modern administrative consoles benefit from serverless, SSR App Router paradigms. DataStack strictly decouples these into **Port 3000** (Web & REST API) and **Port 4000** (Real-Time Ingestion).
|
||||
2. **Zero Per-Row Database Writes**: Telemetry streams are never inserted per-row into relational tables. Events are buffered in memory and Redis, then flushed in sequential batches directly to daily partitioned CSV files on disk.
|
||||
3. **Control-Plane vs Data-Plane Isolation**: PostgreSQL is strictly reserved for the control plane (user authentication, client token registry, rate limits, and platform configuration). The data plane resides entirely on disk in partitioned CSV files.
|
||||
|
||||
---
|
||||
|
||||
## 2. Dual-Runtime Architecture
|
||||
|
||||
```text
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Nginx Reverse Proxy (Port 80) │
|
||||
└───────────────┬────────────────────────┬───────────────┘
|
||||
│ │
|
||||
/socket.io/ │ │ / & /api/*
|
||||
▼ ▼
|
||||
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
|
||||
│ Ingestion Service (Port 4000) │ │ Web & Admin Console (Port 3000) │
|
||||
│ Express 5 + Socket.IO 4 │ │ Next.js 15 App Router + React 19 │
|
||||
├───────────────────────────────────┤ ├───────────────────────────────────┤
|
||||
│ - WebSocket Handshake Auth (JWT) │ │ - Public Landing & PWA Lifecycle │
|
||||
│ - Real-time Rate Limiting │ │ - /vcon/* Admin Console │
|
||||
│ - 5-min Sliding Window Idempotency│ │ - Authenticated HTTP Ingestion │
|
||||
│ - Memory & Redis Batch Buffer │ │ - Streaming CSV / JSON Export │
|
||||
│ - CsvWriterWorker Flusher │ │ - Session Auth & Brute-Force Guard│
|
||||
└─────────────────┬─────────────────┘ └─────────────────┬─────────────────┘
|
||||
│ │
|
||||
│ Authenticated /internal/* Bridge │
|
||||
└────────────────◄═══════════════════════┘
|
||||
(DATASTACK_INTERNAL_SECRET)
|
||||
```
|
||||
|
||||
### Port 3000 (`web`): Next.js 15 App Router
|
||||
- **Role**: Serves the user interface and HTTP REST endpoints.
|
||||
- **Key Modules**:
|
||||
- `middleware.ts`: Authenticates `/vcon/*` routes using signed WebCrypto session cookies, and enforces CSRF protection by validating `Origin`/`Referer` headers against `Host` and `X-Forwarded-Host`.
|
||||
- `app/api/events/route.ts`: Accepts HTTP ingestion for single events, arrays (`{ batch: [...] }`), or structured file imports (`{ fileName, records: [...] }`).
|
||||
- `app/api/export/route.ts`: Streams filtered CSV or JSON files directly from disk partitions to the client browser without buffering entire datasets in server RAM.
|
||||
|
||||
### Port 4000 (`ingestion`): Dedicated Ingestion Server
|
||||
- **Role**: High-throughput Socket.IO WebSocket server (`server/index.js`).
|
||||
- **Key Modules**:
|
||||
- `server/socket/handlers.js`: Enforces client JWT authentication, sliding-window rate limits (per-client and global), payload size validation, and idempotency deduplication.
|
||||
- `server/workers/csvWriter.js`: Flushes buffered batches to daily partitioned CSV files with file rotation and cooperative locking.
|
||||
|
||||
---
|
||||
|
||||
## 3. Internal Service Bridge (`/internal/*`)
|
||||
|
||||
Because Port 3000 and Port 4000 operate as separate processes, administrative actions in `/vcon/` must synchronize with the real-time ingestion service instantly:
|
||||
|
||||
- **Security**: All `/internal/*` routes require a shared, timing-safe authentication header: `x-datastack-internal-secret`.
|
||||
- **Secret Derivation**: `DATASTACK_INTERNAL_SECRET` can be explicitly defined in `.env` or is automatically derived from the active JWT secret via SHA-256.
|
||||
|
||||
### Bridge Endpoints:
|
||||
1. `GET /internal/metrics`: Aggregates active Socket.IO connections, buffer depths, and security stats for the `/vcon/dashboard`.
|
||||
2. `POST /internal/settings-sync`: Pushes updated rate limits, payload sizes, flush thresholds, or rotated JWT secrets instantly to Port 4000.
|
||||
3. `POST /internal/client-sync`: Pushes registered, rotated, revoked, or deleted client updates. If a client is revoked, Port 4000 immediately kicks all active sockets for that client.
|
||||
4. `POST /internal/flush`: Triggers an immediate write of Port 4000's buffer to disk when an admin initiates a manual flush or export.
|
||||
5. `GET /internal/recent`: Fetches recently buffered events from Port 4000's queue so `/vcon/data` displays zero-latency activity.
|
||||
|
||||
---
|
||||
|
||||
## 4. Storage Subsystem & Durability
|
||||
|
||||
### Data Ingestion Flow:
|
||||
1. **Push**: Events enter via WebSocket (`sync_data`) or HTTP (`POST /api/events`).
|
||||
2. **Buffer Queue**: Events are written to the Redis queue (`datastack:buffer:events`) with local RAM fallback.
|
||||
3. **Batch Flush Condition**: A flush triggers when:
|
||||
- Buffer size reaches `batchFlushThreshold` (default: 500 events), OR
|
||||
- Time elapsed reaches `batchFlushIntervalMs` (default: 2000 ms), OR
|
||||
- A manual flush or export is requested.
|
||||
4. **Distributed Flush Lock**: In multi-process deployments, workers acquire `datastack:flush:lock` in Redis (`SET lockToken PX 15000 NX`) to prevent concurrent writes.
|
||||
5. **Safe Trim**: Redis queue items are trimmed (`LTRIM`) **strictly after** the batch append operation confirms success on disk.
|
||||
|
||||
### Cooperative Partition File Locking
|
||||
When writing to `/storage/data/{clientId}/{YYYY-MM-DD}.csv`:
|
||||
- The writer creates a cooperative `.lock` file alongside the partition (`{file}.lock`).
|
||||
- Stale locks (older than 10 seconds) created by crashed processes are automatically cleared.
|
||||
- This prevents race conditions when both Port 3000 and Port 4000 flush to the same client partition simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## 5. PostgreSQL Metadata Subsystem
|
||||
|
||||
PostgreSQL persistence is managed via raw `pg.Pool` connections (`core/Database/database.ts`), mirrored by the reference schema in `prisma/schema.prisma`.
|
||||
|
||||
### Managed Tables:
|
||||
1. `admin_users`: Stores admin credentials with `crypto.scrypt` password hashing and a `token_version` column for instant session invalidation.
|
||||
2. `admin_login_attempts`: Tracks failed login attempts per composite IP + Email hash with 15-minute brute-force lockout.
|
||||
3. `client_sdks`: Stores registered client metadata, hashed secrets, signed tokens, rate limits, status (`active`/`revoked`), and total event counters.
|
||||
4. `deleted_clients`: Tombstone table ensuring deleted client IDs and tokens can never be revived or accepted.
|
||||
5. `admin_settings`: Single-row configuration storing platform limits, ports, thresholds, and secrets.
|
||||
|
||||
### Non-Destructive Migrations
|
||||
On startup, `DatabaseEngine.ensureSchema()` runs non-destructive `CREATE TABLE IF NOT EXISTS` and additive `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` DDL statements, guaranteeing seamless upgrades on existing production databases without data loss.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Build, Compilation & Testing Guide
|
||||
|
||||
This guide outlines compilation commands, linter checks, test suite execution, and Docker container packaging.
|
||||
|
||||
---
|
||||
|
||||
## 1. Production Web Compilation
|
||||
|
||||
DataStack uses Next.js 15 App Router configured for **Standalone Output** (`output: "standalone"` in `next.config.ts`), which bundles only the required Node.js runtime files into `.next/standalone`.
|
||||
|
||||
### Compiling the Application:
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Build Verification Checklist:
|
||||
- Compiles TypeScript without type errors (`tsc --noEmit`).
|
||||
- Generates serverless routes and static assets under `.next/`.
|
||||
- Prepares standalone bundle under `.next/standalone/server.js`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Code Quality & Linting
|
||||
|
||||
DataStack uses ESLint 9 with `eslint-config-next`:
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
```
|
||||
|
||||
Ensure this command exits cleanly (`code: 0`) with zero warnings or errors prior to committing code or opening a pull request.
|
||||
|
||||
---
|
||||
|
||||
## 3. Automated Test Suites
|
||||
|
||||
DataStack includes comprehensive, fast-executing automated test suites verifying security, synchronization, storage durability, PWA compliance, and deployment health.
|
||||
|
||||
### Running TypeScript Integration & Unit Tests
|
||||
Run the test runner using `npx --yes tsx --test`:
|
||||
|
||||
```bash
|
||||
# Run all core test suites
|
||||
npx --yes tsx --test tests/phase01-auth.test.ts \
|
||||
tests/phase02-sync.test.ts \
|
||||
tests/phase03-storage-buffer.test.ts \
|
||||
tests/phase06-pwa-landing.test.ts \
|
||||
tests/phase07-deployment.test.ts
|
||||
```
|
||||
|
||||
#### What These Tests Verify:
|
||||
- **`phase01-auth.test.ts`**: Secret fail-closed checks, `tokenVersion` session invalidation, brute-force IP throttling, client token redaction, and CSRF Origin validation.
|
||||
- **`phase02-sync.test.ts`**: Port 3000 ↔ Port 4000 internal secret authentication, settings propagation, instant client revocation, and JWT rotation disconnection.
|
||||
- **`phase03-storage-buffer.test.ts`**: Redis AOF durability, distributed flush locking, cooperative partition `.lock` files, streaming row counting, and filtered CSV export.
|
||||
- **`phase06-pwa-landing.test.ts`**: Web App Manifest validity, Service Worker registration, and navigation integrity.
|
||||
- **`phase07-deployment.test.ts`**: Docker Compose configuration, Redis AOF persistence, Nginx `/health` proxying, deterministic lockfile parity, startup ownership checks, and unhydrated-DB auto-prune guards.
|
||||
|
||||
### Running Python SDK Unit Tests
|
||||
```bash
|
||||
python3 -m unittest tests/phase05_sdk_test.py
|
||||
```
|
||||
Verifies Python SDK WebSocket streaming, Heartbeat HTTP fallback, disk spooling (`.datastack_spool/`), and `.csv`/`.json` file streaming.
|
||||
|
||||
---
|
||||
|
||||
## 4. Docker Container Builds
|
||||
|
||||
To test production containerization locally:
|
||||
|
||||
### 1. Build the Web Container (Port 3000)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.web -t datastack-web:latest .
|
||||
```
|
||||
|
||||
### 2. Build the Ingestion Container (Port 4000)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.server -t datastack-ingestion:latest .
|
||||
```
|
||||
|
||||
### 3. Build the Nginx Proxy Container (Port 80)
|
||||
```bash
|
||||
docker build -f docker/Dockerfile.proxy -t datastack-proxy:latest .
|
||||
```
|
||||
|
||||
### 4. Build & Run the Entire Stack with Compose
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml build
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
```
|
||||
Verify container health with:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml ps
|
||||
```
|
||||
All containers (`web`, `ingestion`, `proxy`, `postgres`, `redis`) should display `(healthy)`.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Contribution Rules & Guidelines
|
||||
|
||||
Thank you for contributing to **DataStack**! To maintain architectural purity, production reliability, and seamless collaboration across multiple development environments, all contributors must adhere to these guidelines.
|
||||
|
||||
---
|
||||
|
||||
## 1. Universal Multi-IDE Persistent Memory (`MEMORY.md`)
|
||||
|
||||
> **CRITICAL RULE**: DataStack is developed across multiple IDEs and AI toolchains (such as VS Code, Google Antigravity, Cursor, and terminal environments). Whenever any modification, update, or bug fix is completed in your session, you must automatically update the `MEMORY.md` file (or provide the exact markdown blocks) so the project context remains perfectly synchronized across all IDEs (see `.ai-rules` and `AGENTS.md`).
|
||||
|
||||
When updating `MEMORY.md`:
|
||||
- Note any new files, directories, routes, or modules added to the repository.
|
||||
- Record any updated environment variables, runtime parameters, or architectural adjustments.
|
||||
- Update the status of active features or resolved bugs.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural Guardrails (Strict Enforcement)
|
||||
|
||||
1. **Dual-Runtime Separation**:
|
||||
- Next.js 15 App Router (`app/`) runs on **Port 3000**.
|
||||
- Express 5 + Socket.IO (`server/index.js`) runs on **Port 4000**.
|
||||
- **Never** initialize Socket.IO server instances inside Next.js serverless route handlers.
|
||||
2. **Zero Telemetry Writes to PostgreSQL**:
|
||||
- Incoming telemetry events **must never** be inserted as rows into PostgreSQL.
|
||||
- Events are buffered in memory and Redis, then appended to daily CSV partitions in `/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
- PostgreSQL is strictly reserved for `admin_users`, `admin_login_attempts`, `client_sdks`, `deleted_clients`, and `admin_settings`.
|
||||
3. **Non-Destructive Database Migrations**:
|
||||
- Never write code that executes `DROP TABLE`, `DROP COLUMN`, or `TRUNCATE` against production databases.
|
||||
- All schema upgrades must use defensive `CREATE TABLE IF NOT EXISTS` and `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` statements.
|
||||
4. **Cooperative CSV Locking**:
|
||||
- All disk write operations must acquire cooperative file locks (`.lock`) with automated stale-lock recovery to prevent dual-writer corruption.
|
||||
|
||||
---
|
||||
|
||||
## 3. UI Component & Design System Standards
|
||||
|
||||
- **Single Source of Truth**: All UI primitives must reside in `components/ui/` (`Button`, `Input`, `Modal`, `Table`, `Panel`, `StatCard`, etc.) and layout shells in `components/layout/`. Do not introduce ad-hoc isolated CSS or duplicate button components.
|
||||
- **Design Tokens**:
|
||||
- Border radius: `--radius: 0rem` across all components.
|
||||
- Shadows: None (`--shadow: none`). Crisp, clean 1px borders with dark and light mode support.
|
||||
- Font stack: `"Segoe UI", "Segoe UI Web", system-ui, -apple-system, "Helvetica Neue", sans-serif`.
|
||||
- **Icons**: Import all icons exclusively from `lucide-react`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Pull Request & Pre-Submission Checklist
|
||||
|
||||
Before submitting a Pull Request:
|
||||
1. **Lint Check**: Run `npm run lint` and verify zero errors and warnings.
|
||||
2. **Build Check**: Run `npm run build` and ensure the Next.js standalone application compiles cleanly.
|
||||
3. **Automated Tests**:
|
||||
- Run the TypeScript integration tests:
|
||||
```bash
|
||||
npx --yes tsx --test tests/phase01-auth.test.ts tests/phase02-sync.test.ts tests/phase03-storage-buffer.test.ts tests/phase06-pwa-landing.test.ts tests/phase07-deployment.test.ts
|
||||
```
|
||||
- Run the Python SDK tests if modifying SDK code:
|
||||
```bash
|
||||
python3 -m unittest tests/phase05_sdk_test.py
|
||||
```
|
||||
4. **Documentation**:
|
||||
- If adding or changing features, update the relevant files in `docs/users/` or `docs/dev/`.
|
||||
- Update `docs/MANIFEST.md` if new documentation files were introduced.
|
||||
5. **Memory Synchronization**: Ensure `MEMORY.md` reflects all changes made during your session.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Local Development Setup
|
||||
|
||||
This guide walks contributors through configuring and running DataStack locally on your development workstation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Ensure the following tools are installed:
|
||||
- **Node.js**: Version `22.x` (LTS recommended)
|
||||
- **Package Manager**: `npm` (v10+)
|
||||
- **Python**: Version `3.10+` (optional, for developing and testing the Python client SDK)
|
||||
- **Docker & Docker Compose**: Optional, useful for running local PostgreSQL and Redis instances.
|
||||
|
||||
---
|
||||
|
||||
## 2. Clone & Install Dependencies
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vibtoolshq/datastack.git
|
||||
cd datastack
|
||||
|
||||
# Install root dependencies deterministically using the committed lockfile
|
||||
npm ci
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment Configuration
|
||||
|
||||
Copy the example environment configuration:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
### Local Development Defaults:
|
||||
In local development (`NODE_ENV=development`), DataStack includes built-in fallbacks:
|
||||
- If `DATABASE_URL` is omitted, DataStack operates seamlessly in **In-Memory Mock Mode**, allowing you to test the full UI, client registration, and REST endpoints without running PostgreSQL.
|
||||
- If `REDIS_URL` is omitted, DataStack uses an **in-memory RAM buffer queue**, persisting directly to `./storage/data`.
|
||||
|
||||
For full production parity during development, you can spin up local PostgreSQL and Redis containers:
|
||||
```bash
|
||||
docker run --name datastack-postgres -e POSTGRES_USER=datastack -e POSTGRES_PASSWORD=password -e POSTGRES_DB=datastack_db -p 5432:5432 -d postgres:16-alpine
|
||||
docker run --name datastack-redis -p 6379:6379 -d redis:7-alpine redis-server --appendonly yes
|
||||
```
|
||||
|
||||
Then update your `.env`:
|
||||
```dotenv
|
||||
DATABASE_URL=postgresql://datastack:password@127.0.0.1:5432/datastack_db
|
||||
REDIS_URL=redis://127.0.0.1:6379
|
||||
PORT=3000
|
||||
PORT_SOCKET=4000
|
||||
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
|
||||
DATASTACK_ADMIN_EMAIL=admin@example.com
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Running the Development Servers
|
||||
|
||||
DataStack requires two concurrent runtime processes. Open two terminal tabs in the project root:
|
||||
|
||||
### Terminal 1: Web Console & REST API (Port 3000)
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
Starts the Next.js 15 development server with Fast Refresh at `http://localhost:3000`.
|
||||
|
||||
### Terminal 2: Ingestion Server (Port 4000)
|
||||
```bash
|
||||
npm run server
|
||||
```
|
||||
Starts the dedicated Express 5 + Socket.IO ingestion service at `http://localhost:4000`.
|
||||
|
||||
---
|
||||
|
||||
## 5. First-Time Setup in Development
|
||||
|
||||
1. Open `http://localhost:3000/vcon/setup` in your browser.
|
||||
2. Complete the setup wizard using your `DATASTACK_ADMIN_SETUP_KEY` from `.env`.
|
||||
3. In `/vcon/clients`, register a test client application to generate your first JWT token.
|
||||
4. In `/vcon/simulator`, test sending live events and observe the buffer counter in `/vcon/dashboard`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Python SDK Development
|
||||
|
||||
If you are modifying the Python SDK in `sdk/python/datastack/`:
|
||||
|
||||
```bash
|
||||
cd sdk/python/datastack
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
pip install -r ../../../tests/requirements.txt # If running SDK test suites
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
# DataStack User & Operator Guide
|
||||
|
||||
Welcome to the **DataStack User Documentation**. This guide is designed for operators, administrators, product teams, and engineers who wish to deploy, operate, and collect real-time telemetry using DataStack.
|
||||
|
||||
---
|
||||
|
||||
## What is DataStack?
|
||||
|
||||
**DataStack** is a self-hosted, lightweight telemetry and data collection platform. It enables your applications to stream events in real time, securely buffers them with Redis and in-memory batching, and persists them into clean, partitioned daily CSV files without overloading a relational database.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Navigation
|
||||
|
||||
Explore the following sections to get up and running:
|
||||
|
||||
### 1. [Installation & Deployment](./installation.md)
|
||||
Step-by-step instructions for deploying DataStack in production:
|
||||
- Running with **Docker Compose**
|
||||
- 1-Click deployment on **Coolify**
|
||||
- Configuring environment variables & persistent volumes
|
||||
- Reverse proxy and SSL configuration
|
||||
|
||||
### 2. [Feature Walkthrough](./features.md)
|
||||
Comprehensive overview of all platform capabilities:
|
||||
- **Admin Console (`/vcon/`)**: Dashboard overview, live activity preview, and storage monitors.
|
||||
- **Client SDK Management**: Registering client applications, issuing JWT tokens, and instant revocation.
|
||||
- **Data Simulator**: Interactive in-browser telemetry generation for testing without writing code.
|
||||
- **Partitioned Storage**: Size-based file rotation and automatic daily retention pruning.
|
||||
- **Streaming Export**: High-performance streaming of filtered CSV or JSON datasets.
|
||||
|
||||
### 3. [Usage Tutorials](./tutorials.md)
|
||||
Hands-on walkthroughs for common tasks:
|
||||
- Sending your first telemetry event via HTTP REST
|
||||
- Streaming real-time events using the official Python SDK
|
||||
- Streaming local `.csv`, `.json`, and `.jsonl` data files
|
||||
- Exporting datasets for analysis in Excel, pandas, or DuckDB
|
||||
|
||||
### 4. [Frequently Asked Questions & Troubleshooting](./faq.md)
|
||||
Answers to common operational questions:
|
||||
- How does DataStack prevent data loss during container restarts?
|
||||
- How to rotate client credentials safely without downtime
|
||||
- What to do if WebSocket handshakes fail or time out
|
||||
- How to adjust event rate limits and payload size thresholds
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Links
|
||||
- [Developer & Contributor Guide](../dev/README.md)
|
||||
- [Architecture Blueprint](../dev/architecture.md)
|
||||
- [Documentation Index](../MANIFEST.md)
|
||||
- [Official Website](https://vib.tools/)
|
||||
@@ -0,0 +1,83 @@
|
||||
# Frequently Asked Questions (FAQ) & Troubleshooting
|
||||
|
||||
This document answers common operational, configuration, and troubleshooting questions.
|
||||
|
||||
---
|
||||
|
||||
## General Architecture & Data Storage
|
||||
|
||||
### Why does DataStack store events in CSV files instead of PostgreSQL?
|
||||
Relational databases degrade significantly when handling thousands of concurrent writes per second due to WAL bottlenecks, indexing contention, and connection limits. DataStack separates concerns:
|
||||
- **PostgreSQL** is used exclusively for administrative accounts, client credentials, and platform configuration.
|
||||
- **Partitioned CSV Files** (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`) receive high-throughput batched writes sequentially on disk. This enables lightning-fast ingestion, predictable disk space usage, zero SQL table locking, and trivial backup and compression.
|
||||
|
||||
### What happens if the server crashes while events are buffered?
|
||||
DataStack implements multi-layer crash protection:
|
||||
1. **Redis AOF Durability**: In production with Redis enabled (`--appendonly yes`), incoming events are pushed immediately to a durable Redis queue (`datastack:buffer:events`).
|
||||
2. **Post-Write Trimming**: The Redis list is trimmed (`LTRIM`) **strictly after** the batch append to the CSV file succeeds on disk. If a crash occurs mid-write, Redis retains the items and flushes them upon reboot.
|
||||
3. **Graceful Signal Handling**: On `SIGTERM` or `SIGINT` (such as during container updates), DataStack halts new input, awaits the in-flight disk flush, and releases file locks before shutting down.
|
||||
|
||||
### How does file rotation and retention pruning work?
|
||||
- **Rotation (`fileRotationMb`)**: By default, when a daily CSV file reaches `100 MB`, DataStack safely rotates to `{YYYY-MM-DD}-part2.csv`.
|
||||
- **Retention Pruning (`autoPruneDays`)**: By default, partitions older than 30 days are automatically pruned. To retain all historical records indefinitely, set **Auto Prune Days** to `0` ("Keep all") in `/vcon/settings`.
|
||||
|
||||
---
|
||||
|
||||
## Networking & Ports
|
||||
|
||||
### What ports does DataStack use?
|
||||
In a standard Docker Compose deployment, the services communicate over an internal network:
|
||||
- **Port `80`**: Exposed by the Nginx reverse proxy (`proxy` container) to the public or host.
|
||||
- **Port `3000`**: Internal Next.js web application, Admin Console, and REST API (`web` container).
|
||||
- **Port `4000`**: Internal Express + Socket.IO real-time ingestion service (`ingestion` container).
|
||||
- **Port `5432`**: Internal PostgreSQL database (`postgres` container).
|
||||
- **Port `6379`**: Internal Redis buffer (`redis` container).
|
||||
|
||||
You only need to expose Port `80` (or `443` with SSL) through your firewall.
|
||||
|
||||
---
|
||||
|
||||
## Authentication & Client Security
|
||||
|
||||
### How does token rotation work? Does it disconnect active connections?
|
||||
When you click **Rotate Token** in `/vcon/clients`:
|
||||
1. A new HMAC-SHA256 token and secret are generated and stored in PostgreSQL.
|
||||
2. The change is instantly communicated to the Port 4000 ingestion server over the internal authenticated bridge.
|
||||
3. Any active WebSocket connections using the old token are immediately disconnected with `TOKEN_ROTATED`.
|
||||
4. Subsequent requests using the old token are rejected with `401 Unauthorized`.
|
||||
|
||||
### What does `CLIENT_REVOKED` mean?
|
||||
If a client registration is marked as `Revoked` or permanently deleted by an administrator, incoming requests are rejected with HTTP status `403 Forbidden` (`CLIENT_REVOKED`). If a WebSocket connection was open, it is kicked immediately.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Issue: "WebSocket ACK timeout" or Handshake Fails
|
||||
1. **Check Nginx Proxy Headers**: If using an external reverse proxy (like Cloudflare, Traefik, or Caddy) in front of DataStack, ensure `Upgrade` and `Connection: Upgrade` headers are passed through for WebSocket support:
|
||||
```nginx
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
```
|
||||
2. **Verify Port 4000**: In `/vcon/settings`, click **Connections -> Probe**. If the status reports offline, inspect the logs of the `ingestion` container:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml logs ingestion
|
||||
```
|
||||
|
||||
### Issue: "RATE_LIMIT_EXCEEDED" Errors
|
||||
Each client has a configurable rate limit (default: 100 events/sec) and a global platform limit. If your client bursts above this rate, the server returns status `429`.
|
||||
- **Solution**: In `/vcon/clients`, increase the client's rate limit, or in `/vcon/settings`, adjust the global maximum events per second.
|
||||
- **Python SDK**: The official Python SDK automatically handles rate limits by backing off and retrying remaining events without dropping data.
|
||||
|
||||
### Issue: "PAYLOAD_TOO_LARGE" Errors
|
||||
By default, individual events cannot exceed `100 KB` (`maxPayloadSizeKb`).
|
||||
- **Solution**: In `/vcon/settings`, increase the **Max Payload Size (KB)** setting to accommodate larger JSON objects.
|
||||
|
||||
### How do I back up my collected telemetry data?
|
||||
All partition files are located in the `datastack-data` volume (`/app/storage/data`). You can create a compressed backup at any time while the containers are running:
|
||||
```bash
|
||||
docker run --rm \
|
||||
--volumes-from datastack-web-1 \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar czf /backup/datastack_backup_$(date +%F).tar.gz /app/storage/data
|
||||
```
|
||||
@@ -0,0 +1,89 @@
|
||||
# Feature Guide
|
||||
|
||||
This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (`/vcon/`).
|
||||
|
||||
---
|
||||
|
||||
## 1. The Administration Console (`/vcon/`)
|
||||
|
||||
The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.
|
||||
|
||||
Access the console by signing in at:
|
||||
```text
|
||||
https://your-domain.example/vcon/login
|
||||
```
|
||||
|
||||
### Dashboard Overview (`/vcon/dashboard`)
|
||||
The primary operational cockpit displays:
|
||||
- **System Health**: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
|
||||
- **Storage Metrics**: Total CSV partition file count, active client directories, and total disk usage.
|
||||
- **Buffer Gauges**: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
|
||||
- **Live Activity Stream**: Real-time ticker showing recently ingested events with timestamps and payload previews.
|
||||
|
||||
---
|
||||
|
||||
## 2. Client SDK Registry & Security (`/vcon/clients`)
|
||||
|
||||
DataStack enforces strict authentication for all incoming telemetry streams.
|
||||
|
||||
### Registering a New Client
|
||||
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
|
||||
2. Provide:
|
||||
- **Client Identifier**: Unique lowercase alphanumeric slug (e.g., `pos_terminal_01`, `mobile_app_prod`).
|
||||
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
|
||||
- **Rate Limit**: Maximum allowed events per second for this client (default: `100 events/sec`).
|
||||
3. Click **Register Client**. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
|
||||
4. **Copy the Token**: Store the generated token securely in your client application's environment or configuration file.
|
||||
|
||||
### Credential Rotation & Access Control
|
||||
- **Rotate Token**: Click **Rotate Token** to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
|
||||
- **Revoke Access**: Set the client status to `Revoked`. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with `CLIENT_REVOKED (403)`.
|
||||
- **Permanent Deletion**: Deleting a client removes it from PostgreSQL and adds it to `deleted_clients` to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Simulator (`/vcon/simulator`)
|
||||
|
||||
The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:
|
||||
- **Single Event**: Send ad-hoc JSON payloads with custom event IDs.
|
||||
- **Batch Generator**: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
|
||||
- **Preset Schemas**: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.
|
||||
|
||||
---
|
||||
|
||||
## 4. Partitioned CSV Storage & Retention (`/vcon/storage`)
|
||||
|
||||
DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:
|
||||
- **Directory Layout**: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
||||
- **CSV Schema**: Each file contains 4 clean columns:
|
||||
```csv
|
||||
clientId,eventId,timestamp,payload
|
||||
pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
|
||||
```
|
||||
- **Size-Based Rotation**: When a partition reaches the configured threshold (default: `100 MB`), DataStack automatically closes the file and continues writing to `{YYYY-MM-DD}-part2.csv`.
|
||||
- **Automated Retention Pruning**: Configure `autoPruneDays` (e.g., `30` days). DataStack automatically removes expired partition files, or set to `0` ("Keep all") to retain historical archives indefinitely.
|
||||
- **Manual Buffer Flush**: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.
|
||||
|
||||
---
|
||||
|
||||
## 5. Streaming Data Export (`/vcon/export`)
|
||||
|
||||
The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:
|
||||
- **Client Filter**: Export all clients or filter by a specific client ID.
|
||||
- **Date Range**: Filter records within precise UTC start and end dates.
|
||||
- **Formats**:
|
||||
- **CSV**: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
|
||||
- **JSON**: Formatted JSON array stream.
|
||||
- **Non-Blocking Streaming**: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.
|
||||
|
||||
---
|
||||
|
||||
## 6. Platform Runtime Settings (`/vcon/settings`)
|
||||
|
||||
Fine-tune ingestion rules with instant cross-process propagation:
|
||||
- **Max Events Per Second**: Global ingestion rate ceiling.
|
||||
- **Max Payload Size (KB)**: Maximum permitted size per event payload (default: `100 KB`).
|
||||
- **Batch Flush Threshold**: Number of buffered events that triggers an immediate write to disk (default: `500 events`).
|
||||
- **Batch Flush Interval**: Maximum elapsed time before flushing pending events to disk (default: `2000 ms`).
|
||||
- **Idempotency Protection**: Deduplicates events with identical `eventId` values within a 5-minute sliding window.
|
||||
- **Connection Probing**: Test the live internal bridge between the Web console and the Port 4000 ingestion service.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Installation & Deployment Guide
|
||||
|
||||
This guide explains how to install and self-host DataStack on your own server or cloud VPS.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure your system meets the following minimum requirements:
|
||||
- **Operating System**: Linux (Ubuntu 22.04+, Debian 12+, Alpine), macOS, or Windows WSL2.
|
||||
- **Docker Engine**: Docker `v24.0+` and Docker Compose `v2.20+`.
|
||||
- **System Resources**:
|
||||
- Minimum: 1 vCPU, 1 GB RAM, 10 GB disk space.
|
||||
- Recommended: 2 vCPU, 2–4 GB RAM, SSD storage scaled to your expected data volume.
|
||||
- **Networking**: Ports `80` (HTTP) and `443` (HTTPS) accessible via your reverse proxy or cloud firewall.
|
||||
|
||||
---
|
||||
|
||||
## Method 1: Docker Compose (Recommended)
|
||||
|
||||
DataStack includes a complete, production-ready Docker Compose stack located in the [`docker/`](../../docker/) directory.
|
||||
|
||||
### 1. Clone the Repository
|
||||
```bash
|
||||
git clone https://github.com/vibtoolshq/datastack.git
|
||||
cd datastack
|
||||
```
|
||||
|
||||
### 2. Configure Environment Variables
|
||||
Copy the example environment configuration:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Generate secure, random 32+ character secrets for production:
|
||||
```bash
|
||||
# In Linux / macOS, generate 48-char random hex keys:
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Edit `.env` and fill in the required variables:
|
||||
```dotenv
|
||||
# Production Security Secrets (Minimum 32 characters each)
|
||||
DATASTACK_ADMIN_SESSION_SECRET=your-random-32-plus-char-session-secret-here
|
||||
DATASTACK_JWT_SECRET=your-random-32-plus-char-client-jwt-secret-here
|
||||
DATASTACK_ADMIN_SETUP_KEY=your-random-32-plus-char-admin-setup-key-here
|
||||
DATASTACK_INTERNAL_SECRET=your-random-32-plus-char-internal-service-secret-here
|
||||
|
||||
# Administrator Initial Email
|
||||
DATASTACK_ADMIN_EMAIL=admin@yourcompany.com
|
||||
```
|
||||
|
||||
### 3. Launch the Stack
|
||||
Run Docker Compose from the project root:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
The stack launches 5 containers:
|
||||
1. **`proxy`**: Nginx reverse proxy exposed on port `80`, routing `/socket.io/` to port `4000` and all other routes to port `3000`.
|
||||
2. **`web`**: Next.js 15 standalone server running on port `3000`.
|
||||
3. **`ingestion`**: Dedicated Express + Socket.IO ingestion server running on port `4000`.
|
||||
4. **`postgres`**: PostgreSQL 16 database storing administrative credentials, SDK client records, and settings.
|
||||
5. **`redis`**: Redis 7 container running with `--appendonly yes` for crash-resilient queue buffering.
|
||||
|
||||
---
|
||||
|
||||
## Method 2: Coolify Cloud / Self-Hosted PaaS
|
||||
|
||||
DataStack is optimized for deployment via [Coolify](https://coolify.io/):
|
||||
|
||||
1. **Create Resource**: In Coolify, click **+ Add Resource** → **Docker Compose**.
|
||||
2. **Repository Settings**:
|
||||
- Point to your DataStack Git repository.
|
||||
- Set **Base Directory** to `/docker`.
|
||||
- Set **Docker Compose Location** to `/docker/docker-compose.yml`.
|
||||
3. **Domain Assignment**:
|
||||
- Attach your custom domain (e.g. `https://datastack.yourdomain.com`) to the **`proxy`** service on internal port `80`.
|
||||
- Enable automatic SSL/TLS via Coolify's built-in Let's Encrypt integration.
|
||||
4. **Environment Variables**:
|
||||
- Coolify automatically injects generated `SERVICE_*` variables for database and JWT secrets.
|
||||
5. **Deploy**: Click **Deploy** and monitor the build logs until all 5 services are healthy.
|
||||
|
||||
---
|
||||
|
||||
## First-Run Administrator Setup (`/vcon/setup`)
|
||||
|
||||
Once the stack is running:
|
||||
1. Open your browser and navigate to:
|
||||
```text
|
||||
https://your-domain.example/vcon/setup
|
||||
```
|
||||
2. You will be prompted to enter:
|
||||
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` or your chosen admin email.
|
||||
- **Administrator Password**: Choose a strong password (minimum 8 characters).
|
||||
- **Setup Security Key**: Enter the exact value configured in `DATASTACK_ADMIN_SETUP_KEY`.
|
||||
3. Click **Complete Initial Setup**. You will be securely logged in and redirected to the `/vcon/dashboard` console.
|
||||
|
||||
---
|
||||
|
||||
## Persistent Storage & Volumes
|
||||
|
||||
DataStack uses three named volumes to safeguard your data across container updates:
|
||||
|
||||
| Volume Name | Container Path | Purpose |
|
||||
| :--- | :--- | :--- |
|
||||
| **`datastack-data`** | `/app/storage` | Contains `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. Shared by `web` and `ingestion`. |
|
||||
| **`pgdata`** | `/var/lib/postgresql/data` | Persists administrative users, client credentials, and settings. |
|
||||
| **`redisdata`** | `/data` | Persists Redis Append-Only File (`appendonly.aof`) for unflushed event buffers. |
|
||||
|
||||
> **IMPORTANT**: When upgrading or redeploying containers, never execute `docker compose down -v`, as the `-v` flag deletes named volumes. Use standard `docker compose down` followed by `docker compose up -d`.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Usage Tutorials
|
||||
|
||||
This document provides hands-on tutorials for sending data to DataStack, integrating client SDKs, and exporting datasets.
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 1: Sending Events via HTTP REST (`curl`)
|
||||
|
||||
The fastest way to send data to DataStack without writing application code is via the HTTP REST API.
|
||||
|
||||
### 1. Retrieve Your Client Token
|
||||
In `/vcon/clients`, register a client (e.g., `bash_script_01`) and copy the generated Bearer JWT token.
|
||||
|
||||
### 2. Send a Single Event
|
||||
```bash
|
||||
curl -X POST "https://your-domain.example/api/events" \
|
||||
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"eventId": "order_1001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"data": {
|
||||
"item": "Mechanical Keyboard",
|
||||
"price": 129.99,
|
||||
"status": "completed"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"eventId": "order_1001",
|
||||
"event": {
|
||||
"clientId": "bash_script_01",
|
||||
"eventId": "order_1001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"payload": "{\"item\":\"Mechanical Keyboard\",\"price\":129.99,\"status\":\"completed\"}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Send a Batch of Events
|
||||
```bash
|
||||
curl -X POST "https://your-domain.example/api/events" \
|
||||
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"batch": [
|
||||
{ "eventId": "evt_01", "data": { "metric": "cpu", "value": 45.2 } },
|
||||
{ "eventId": "evt_02", "data": { "metric": "ram", "value": 72.8 } }
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 2: Integrating the Python Client SDK
|
||||
|
||||
DataStack provides an official, pip-installable Python SDK featuring real-time Socket.IO streaming, automatic HTTP fallback, and offline disk spooling.
|
||||
|
||||
### 1. Install the SDK
|
||||
```bash
|
||||
pip install datastack
|
||||
# Or install backward-compatible package:
|
||||
pip install datastack-sdk
|
||||
```
|
||||
|
||||
### 2. Create Configuration (`config.datastack.py`)
|
||||
In the root of your Python project, create a `config.datastack.py` file:
|
||||
|
||||
```python
|
||||
DATASTACK_CONFIG = {
|
||||
"SERVER_URL": "https://your-domain.example", # WebSocket & proxy endpoint
|
||||
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Token from /vcon/clients
|
||||
"AUTO_CONNECT": True,
|
||||
"ENABLE_HTTP_FALLBACK": True,
|
||||
"OFFLINE_SPOOL_ENABLED": True,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Stream Telemetry from Python
|
||||
```python
|
||||
from datastack import DataStack
|
||||
import time
|
||||
|
||||
# Client automatically discovers and loads config.datastack.py
|
||||
client = DataStack()
|
||||
|
||||
print(f"Connected to DataStack as client: {client.client_id}")
|
||||
|
||||
for i in range(5):
|
||||
response = client.send(
|
||||
data={
|
||||
"sensor_id": "temp_probe_04",
|
||||
"temperature_celsius": 24.5 + i * 0.2,
|
||||
"humidity_percent": 55,
|
||||
},
|
||||
event_id=f"reading_{i}_{int(time.time())}"
|
||||
)
|
||||
print(f"Event sent: {response}")
|
||||
time.sleep(1)
|
||||
|
||||
client.disconnect()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 3: Streaming Local Files (`send_file`)
|
||||
|
||||
If you have offline log files or CSV datasets that need to be ingested into DataStack, use the SDK's built-in file streaming:
|
||||
|
||||
```python
|
||||
from datastack import DataStack
|
||||
|
||||
client = DataStack()
|
||||
|
||||
# Supports .csv, .json, .jsonl, and .ndjson
|
||||
result = client.send_file(
|
||||
file_path="./daily_orders.csv",
|
||||
chunk_size=250, # Stream in batches of 250 records
|
||||
extra_metadata={"imported_by": "nightly_cron", "env": "production"}
|
||||
)
|
||||
|
||||
print(f"File Ingestion Complete:")
|
||||
print(f" Processed rows: {result['processed']}")
|
||||
print(f" Duplicates skipped: {result['duplicates']}")
|
||||
print(f" Chunks sent: {result['chunksSent']}")
|
||||
```
|
||||
|
||||
- **Lossless Handling**: If your server rate limits the file, the SDK automatically pauses and retries.
|
||||
- **Deduplication**: Rows without an explicit `eventId` receive deterministic SHA-256 IDs, preventing duplicate rows if the script runs twice.
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 4: Exporting Data for Analysis
|
||||
|
||||
You can stream historical partitions for analysis in pandas, Python, or command-line scripts.
|
||||
|
||||
### 1. Stream CSV Export via `curl`
|
||||
```bash
|
||||
curl -X GET "https://your-domain.example/api/export?format=csv&clientId=bash_script_01" \
|
||||
-H "Cookie: datastack_admin_session=YOUR_ADMIN_SESSION_TOKEN" \
|
||||
-o export_bash_script.csv
|
||||
```
|
||||
|
||||
### 2. Load Directly into pandas
|
||||
```python
|
||||
import pandas as pd
|
||||
import requests
|
||||
import io
|
||||
|
||||
session_cookie = "YOUR_ADMIN_SESSION_COOKIE"
|
||||
export_url = "https://your-domain.example/api/export?format=csv&clientId=all"
|
||||
|
||||
response = requests.get(
|
||||
export_url,
|
||||
cookies={"datastack_admin_session": session_cookie},
|
||||
stream=True
|
||||
)
|
||||
|
||||
df = pd.read_csv(io.StringIO(response.text))
|
||||
print("Exported DataFrame:")
|
||||
print(df.head())
|
||||
```
|
||||
Reference in new issue
Block a user