feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
+112
-54
@@ -1,76 +1,134 @@
|
||||
# Documentation Manifest (`docs/MANIFEST.md`)
|
||||
|
||||
> **Location**: `/docs/MANIFEST.md`
|
||||
> **Audience**: Contributors, Maintainers, and Documentation Authors
|
||||
> **Last Updated**: 2026-11-17
|
||||
> **Audience**: Contributors, Maintainers, Technical Writers, and AI Engineering Agents
|
||||
> **Last Forensic Verification**: 2026-11-17
|
||||
> **Version**: 1.0.0 Production
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory Purpose & Structure
|
||||
## 1. Documentation Structure & Directory Mapping
|
||||
|
||||
The `docs/` directory contains the complete official documentation for **DataStack**, structured into distinct audiences to provide clarity, depth, and ease of navigation:
|
||||
The DataStack documentation suite is strictly organized into functional directories based on audience, abstraction level, and lifecycle stage:
|
||||
|
||||
```text
|
||||
docs/
|
||||
├── MANIFEST.md # This document (directory index & governance)
|
||||
├── users/ # End-User & Operator Documentation
|
||||
│ ├── README.md # User documentation overview & quick links
|
||||
│ ├── installation.md # Self-hosting, Docker, and Coolify setup guides
|
||||
│ ├── features.md # Guide to /vcon/ console, client tokens, and telemetry
|
||||
│ ├── tutorials.md # Step-by-step user tutorials and integration walkthroughs
|
||||
│ └── faq.md # Frequently asked questions and troubleshooting
|
||||
└── dev/ # Developer & Contributor Documentation
|
||||
├── README.md # Developer guide overview & roadmap
|
||||
├── architecture.md # Deep dive into dual-runtime architecture and storage
|
||||
├── setup.md # Local development environment setup & configuration
|
||||
├── build.md # Build, compilation, Docker containerization & test suites
|
||||
└── contributing.md # Contribution rules, conventions, and workflow
|
||||
├── index.md # Primary Documentation Hub & Portal Index
|
||||
├── MANIFEST.md # Master Documentation Inventory & Governance (This File)
|
||||
├── audit-coverage-report.md # Forensic Codebase Audit Matrix & Coverage Report
|
||||
├── users/ # End-User & System Operator Guides
|
||||
│ ├── index.md # User Guide Directory Index
|
||||
│ ├── getting-started.md # 5-minute fast-track operator onboarding guide
|
||||
│ ├── installation.md # Production Docker Compose, Coolify, and volume guide
|
||||
│ ├── configuration.md # Environment variables and platform runtime settings
|
||||
│ ├── features/ # In-depth feature guides
|
||||
│ │ ├── index.md # Feature guides index
|
||||
│ │ ├── admin-console.md # /vcon/ dashboard, monitors, and profile workspace
|
||||
│ │ ├── client-management.md # SDK client registration, JWT rotation, and revocation
|
||||
│ │ ├── data-simulator.md # Interactive browser telemetry simulator & presets
|
||||
│ │ ├── partitioned-storage.md # Daily CSV storage, file rotation, and auto-pruning
|
||||
│ │ └── data-export.md # Streaming CSV and JSON dataset export workspace
|
||||
│ ├── workflows.md # End-to-end operational workflows and user journeys
|
||||
│ ├── troubleshooting.md # Forensic error recovery, connection checks, and diagnostics
|
||||
│ └── faq.md # Comprehensive operational & architectural FAQ
|
||||
├── dev/ # Developer, Contributor & Architecture Reference
|
||||
│ ├── index.md # Developer Hub Directory Index
|
||||
│ ├── architecture.md # Dual-runtime blueprint, service bridge, Redis queue, and storage
|
||||
│ ├── codebase-structure.md # Complete file-by-file module, route, and component inventory
|
||||
│ ├── local-setup.md # Contributor workstation setup & dual-terminal development
|
||||
│ ├── api-reference.md # Exhaustive REST API & Socket.IO protocol specification
|
||||
│ ├── database-schema.md # PostgreSQL tables, columns, constraints, and DDL migrations
|
||||
│ ├── security.md # Security architecture, scrypt, sessions, CSRF, and JWTs
|
||||
│ ├── building-packaging.md # Next.js standalone build, Dockerfiles, and packaging
|
||||
│ ├── testing.md # Test suites guide (TypeScript node:test and Python unittest)
|
||||
│ └── contributing.md # Contribution rules, multi-IDE memory sync, and PR checklist
|
||||
└── reference/ # Authoritative Reference Tables & Glossary
|
||||
├── index.md # Reference Hub Directory Index
|
||||
├── configuration-reference.md # Exhaustive environment variable & setting bounds reference
|
||||
├── dependencies.md # Complete runtime & development dependencies inventory
|
||||
└── glossary.md # Domain and technical glossary
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Directory Breakdown
|
||||
## 2. Comprehensive File Inventory & Purpose Breakdown
|
||||
|
||||
### `docs/users/` (End-Users, System Operators & Administrators)
|
||||
Designed for non-technical users, platform administrators, and data engineers operating or integrating with a live DataStack deployment:
|
||||
|
||||
| File | Purpose | Key Topics Covered |
|
||||
| :--- | :--- | :--- |
|
||||
| `README.md` | Entry point for users | Documentation map, navigation guide, and core concepts overview. |
|
||||
| `installation.md` | Deployment guide | Production Docker Compose, Coolify 1-click deployment, environment variables, reverse proxy setup, and persistent volume management. |
|
||||
| `features.md` | Feature breakdown | Web console (`/vcon/`), client SDK tokens, real-time event simulation, daily partitioned CSV storage, streaming export, and platform settings. |
|
||||
| `tutorials.md` | Practical guides | Step-by-step tutorials: first telemetry event, connecting Python applications, viewing live events in the console, and exporting datasets. |
|
||||
| `faq.md` | Troubleshooting & FAQ | Common questions on data retention, token revocation, port configurations, CSV rotation, and connection diagnostics. |
|
||||
|
||||
### `docs/dev/` (Developers, Contributors & System Architects)
|
||||
Designed for engineers extending the codebase, fixing bugs, writing client SDKs, or contributing to the core platform:
|
||||
|
||||
| File | Purpose | Key Topics Covered |
|
||||
| :--- | :--- | :--- |
|
||||
| `README.md` | Entry point for devs | Contributor handbook, repository layout, and code navigation. |
|
||||
| `architecture.md` | Architectural blueprint | Dual-runtime model (Port 3000 Web vs Port 4000 Ingestion), Redis durability queue, partitioned CSV engine with cooperative locks, and PostgreSQL metadata store. |
|
||||
| `setup.md` | Local development | Prerequisites (Node.js 22, PostgreSQL, Redis), dual-terminal development, environment secrets, and sample configurations. |
|
||||
| `build.md` | Build & compilation | Next.js standalone compilation (`npm run build`), ESLint validation, Docker multi-stage builds, and running TypeScript (`npx tsx --test`) and Python unit test suites. |
|
||||
| `contributing.md` | Contribution standards | Multi-IDE memory synchronization (`MEMORY.md` & `.ai-rules`), UI component governance (`components/ui/`), code conventions, and Pull Request checklist. |
|
||||
### Root Directory
|
||||
- **`docs/index.md`**: The top-level documentation portal providing role-based navigation, a high-level system component table, and core links.
|
||||
- **`docs/MANIFEST.md`**: The authoritative index and governance standard for all documentation assets in this repository.
|
||||
- **`docs/audit-coverage-report.md`**: The forensic verification report documenting inspected modules, verified capabilities, test execution proofs, and zero-hallucination compliance.
|
||||
|
||||
---
|
||||
|
||||
## 3. Maintenance & Documentation Update Guidelines
|
||||
### User Portal (`docs/users/`)
|
||||
Audience: System administrators, DevOps operators, integration engineers, and non-technical managers.
|
||||
|
||||
Documentation accuracy is a core quality requirement in DataStack. Whenever modifying code or configurations, follow these update rules:
|
||||
| File Path | Document Title | Primary Content & Implementation Basis |
|
||||
| :--- | :--- | :--- |
|
||||
| `docs/users/index.md` | User Documentation Hub | Directory map, user role definitions, and core workflow summaries. |
|
||||
| `docs/users/getting-started.md` | Getting Started | First-time installation, `/vcon/setup` wizard, creating a client, and testing ingestion via `curl` and Python. |
|
||||
| `docs/users/installation.md` | Installation & Deployment | Multi-container Docker Compose (`docker/docker-compose.yml`), Coolify 1-click PaaS setup, persistent volume mappings (`datastack-data`, `pgdata`, `redisdata`), and reverse proxy requirements. |
|
||||
| `docs/users/configuration.md` | Configuration Guide | Comprehensive operator guide to configuring secrets in `.env`, initial admin credentials, and tuning runtime limits via `/vcon/settings`. |
|
||||
| `docs/users/features/index.md` | Features Index | High-level summary of all platform features with direct links to dedicated deep dives. |
|
||||
| `docs/users/features/admin-console.md` | Admin Console (`/vcon/`) | In-depth walkthrough of `/vcon/dashboard`, live activity preview, memory/buffer gauges, and administrator account management. |
|
||||
| `docs/users/features/client-management.md` | Client Token Registry | Managing SDK clients (`/vcon/clients`), HMAC-SHA256 JWT issuance, one-click credential rotation, instant socket revocation, and permanent client deletion. |
|
||||
| `docs/users/features/data-simulator.md` | Data Simulator | In-browser telemetry generation (`/vcon/simulator`), single vs batch dispatch, HTTP vs WebSocket transport, and preset sensor/e-commerce payloads. |
|
||||
| `docs/users/features/partitioned-storage.md`| Partitioned CSV Storage | Details the `/storage/data/{clientId}/{YYYY-MM-DD}.csv` structure, 100MB file rotation (`-part2.csv`), cooperative lock files, and automated retention pruning. |
|
||||
| `docs/users/features/data-export.md` | Streaming Data Export | Operational guide to `/vcon/export`, streaming filtered CSV and JSON files, UTC date filtering, and importing into pandas and Excel. |
|
||||
| `docs/users/workflows.md` | Operational Workflows | Complete step-by-step user journeys: onboarding an admin, onboarding an SDK client, rotating platform JWT secret, and running disaster recovery backups. |
|
||||
| `docs/users/troubleshooting.md` | Troubleshooting Guide | Diagnostic procedures for rate-limit violations (`429`), payload size rejections (`413`), WebSocket handshake timeouts, and lock cleanup. |
|
||||
| `docs/users/faq.md` | Frequently Asked Questions | Architectural rationale for CSV over SQL, Redis crash durability, container update rules, and backup commands. |
|
||||
|
||||
### When to Update `docs/users/`
|
||||
- **New User-Facing Features**: Whenever adding or altering pages in `/vcon/*` (e.g., changes to export options, client token regeneration, or UI settings), update `docs/users/features.md`.
|
||||
- **Installation / Deployment Changes**: Whenever modifying `docker/docker-compose.yml`, Dockerfiles, Nginx configurations, or `.env.example`, update `docs/users/installation.md`.
|
||||
- **API Endpoints & Integration**: If new parameters are added to `POST /api/events` or `GET /api/export`, update `docs/users/tutorials.md` and `docs/users/faq.md`.
|
||||
---
|
||||
|
||||
### When to Update `docs/dev/`
|
||||
- **Architectural Changes**: Whenever altering the communication bridge between Port 3000 and Port 4000, changing Redis queues, or altering CSV storage mechanisms, update `docs/dev/architecture.md`.
|
||||
- **New Dependencies or Scripts**: Whenever adding npm dependencies, changing build targets, or adding test files, update `docs/dev/setup.md` and `docs/dev/build.md`.
|
||||
- **Contribution Standards**: If code standards, linting rules, or multi-IDE memory rules are revised, update `docs/dev/contributing.md` and sync with `AGENTS.md` and `.ai-rules`.
|
||||
### Developer Portal (`docs/dev/`)
|
||||
Audience: Backend engineers, frontend developers, platform architects, and open-source contributors.
|
||||
|
||||
### Quality & Style Standards
|
||||
1. **Markdown Formatting**: Use clean GitHub-Flavored Markdown (GFM) with clear headings, bulleted lists, and language-tagged code blocks (e.g. ` ```bash `, ` ```python `).
|
||||
2. **Absolute vs Relative Links**: Inside markdown files, use relative links to reference other documentation files (e.g., `[Architecture Overview](../dev/architecture.md)`).
|
||||
3. **No Stale Placeholders**: Avoid unverified configuration keys or generic placeholder URLs; align variables with `.env.example`.
|
||||
4. **Cross-IDE Sync**: After any documentation update, update `MEMORY.md` in the root directory to maintain synchronization across all IDEs.
|
||||
| File Path | Document Title | Primary Content & Implementation Basis |
|
||||
| :--- | :--- | :--- |
|
||||
| `docs/dev/index.md` | Developer Hub | Contributor overview, architecture index, development philosophy, and repository roadmap. |
|
||||
| `docs/dev/architecture.md` | Architecture Blueprint | Decoupled runtime architecture (Port 3000 Web vs Port 4000 Ingestion), internal bridge protocol, Redis AOF durability, and storage subsystems. |
|
||||
| `docs/dev/codebase-structure.md` | Codebase Structure | Exhaustive file-by-file inventory of directories, source modules (`core/`, `app/`, `server/`, `components/`), and their relationships. |
|
||||
| `docs/dev/local-setup.md` | Local Development Setup | Step-by-step workstation setup, Node.js 22 requirements, dual-terminal local workflow, and in-memory vs Docker container modes. |
|
||||
| `docs/dev/api-reference.md` | API Reference Specification | Complete OpenAPI-style reference for all REST endpoints (`app/api/*`) and Socket.IO real-time events (`server/socket/handlers.js`). |
|
||||
| `docs/dev/database-schema.md` | Database Schema & Migrations | PostgreSQL control-plane tables (`admin_users`, `client_sdks`, `admin_settings`, etc.), `prisma/schema.prisma` reference, and additive DDL migrations. |
|
||||
| `docs/dev/security.md` | Security Architecture | In-depth breakdown of `crypto.scrypt` password hashing, WebCrypto HMAC session cookies, brute-force IP lockout, CSRF Origin validation, and JWT verification. |
|
||||
| `docs/dev/building-packaging.md`| Building & Packaging | Next.js standalone compilation (`.next/standalone`), multi-stage Dockerfiles (`web`, `server`, `proxy`), and production container assembly. |
|
||||
| `docs/dev/testing.md` | Testing Guide | Guide to running automated TypeScript integration tests (`npx tsx --test`) and Python SDK unit tests (`python3 -m unittest`). |
|
||||
| `docs/dev/contributing.md` | Contributing Guidelines | Architectural rules, zero-SQL-telemetry enforcement, single-source UI primitives, multi-IDE memory synchronization (`MEMORY.md`), and PR checklist. |
|
||||
|
||||
---
|
||||
|
||||
### Technical Reference Portal (`docs/reference/`)
|
||||
Audience: Technical teams requiring strict specifications, boundaries, and definitions.
|
||||
|
||||
| File Path | Document Title | Primary Content & Implementation Basis |
|
||||
| :--- | :--- | :--- |
|
||||
| `docs/reference/index.md` | Reference Index | Overview of platform specifications, configuration limits, and glossary. |
|
||||
| `docs/reference/configuration-reference.md` | Configuration Reference | Exhaustive reference table of all environment variables, default values, bounds, and `/vcon/settings` allowlisted keys. |
|
||||
| `docs/reference/dependencies.md` | Dependency Inventory | Audit of all runtime dependencies, dev tools, and Python client requirements with rationales. |
|
||||
| `docs/reference/glossary.md` | Technical Glossary | Clear definitions of platform concepts (cooperative locking, sliding-window rate limit, idempotency envelope, etc.). |
|
||||
|
||||
---
|
||||
|
||||
## 3. Maintenance & Documentation Governance Rules
|
||||
|
||||
To ensure documentation remains permanently synchronized with the actual implementation across future pull requests and releases:
|
||||
|
||||
### Mandatory Update Triggers
|
||||
1. **API & Route Changes**: If any endpoint under `app/api/` or Socket.IO event in `server/socket/handlers.js` is added, modified, or removed:
|
||||
- Update `docs/dev/api-reference.md`.
|
||||
- Update `docs/users/tutorials.md` and `docs/users/workflows.md` if user-facing.
|
||||
2. **Configuration & Environment Changes**: If any environment variable in `.env.example` or setting in `core/adminSettings/settingsManager.ts` is modified:
|
||||
- Update `docs/reference/configuration-reference.md`.
|
||||
- Update `docs/users/configuration.md`.
|
||||
3. **Database Schema Changes**: If any table, column, or constraint in `core/Database/database.ts` or `prisma/schema.prisma` is adjusted:
|
||||
- Update `docs/dev/database-schema.md`.
|
||||
4. **Storage & Ingestion Mechanics**: If batching, locking, rotation, or pruning algorithms in `core/Storage/` or `core/data/` change:
|
||||
- Update `docs/dev/architecture.md` and `docs/users/features/partitioned-storage.md`.
|
||||
5. **Cross-IDE Synchronization**: In compliance with `.ai-rules` and `AGENTS.md`, after every change session, `MEMORY.md` in the root directory must be updated to mirror documentation and code status.
|
||||
|
||||
### Zero-Hallucination Authoring Standards
|
||||
- **Implementation is Truth**: Never document planned or intended behavior as existing capability.
|
||||
- **Traceable Code References**: Every technical guide must reference verified file paths and actual function/class identifiers.
|
||||
- **Executable Examples**: All `curl`, Python, TypeScript, and Docker commands must be verified and syntactically valid.
|
||||
Reference in new issue
Block a user