- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
3.3 KiB
3.3 KiB
DataStack User & Operator Guide
Welcome to the DataStack User & Operator Documentation. This portal contains comprehensive guides for deploying, configuring, operating, and integrating applications with DataStack.
Guide Directory
docs/users/
├── index.md # This directory index
├── getting-started.md # 5-minute setup and quickstart tutorial
├── installation.md # Production Docker Compose & Coolify deployment
├── configuration.md # Secrets, environment variables & settings reference
├── features/ # Detailed functional walkthroughs
│ ├── index.md # Features directory index
│ ├── admin-console.md # /vcon/ dashboard, monitors, and profile workspace
│ ├── client-management.md # SDK client registration, JWT rotation, and revocation
│ ├── data-simulator.md # Interactive browser telemetry simulator & presets
│ ├── partitioned-storage.md # Daily CSV storage, file rotation, and auto-pruning
│ └── data-export.md # Streaming CSV and JSON dataset export workspace
├── workflows.md # Step-by-step user journeys & operations
├── troubleshooting.md # Forensic error recovery & diagnostics
└── faq.md # Frequently asked questions & storage architecture
Core Reading Pathways
Choose the pathway that matches your immediate goal:
Pathway 1: "I want to self-host DataStack on my VPS"
- Start with Installation & Deployment for Docker Compose or Coolify setup.
- Review Configuration Guide to generate 32+ character secrets.
- Follow Getting Started to run the
/vcon/setupwizard and log in.
Pathway 2: "I want to connect my Python application"
- Review Client Token Management to generate a client JWT token.
- Read Tutorial: Python SDK Integration to set up
config.datastack.py. - Check Troubleshooting if you encounter rate limits or connection drops.
Pathway 3: "I want to inspect or export collected data"
- Read Partitioned CSV Storage to understand directory and file layouts.
- Read Streaming Data Export to extract datasets via browser or
curl. - Follow the Data Analysis Workflow to load records into pandas or Excel.
Platform Philosophy
DataStack is built to solve the "High-Velocity Telemetry Dilemma":
- Why not PostgreSQL for everything? Inserting thousands of events per second into relational database tables causes index thrashing, lock contention, and WAL exhaustion.
- The DataStack Solution: PostgreSQL is used strictly for administrative accounts and client credentials. Ingested events flow through in-memory and Redis buffers directly into organized, daily partitioned CSV files (
/app/storage/data/{clientId}/{YYYY-MM-DD}.csv). - The Benefit: Virtually unlimited ingestion throughput, zero SQL lock contention, deterministic storage costs, and instant portability.