- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
86 lines
3.5 KiB
Markdown
86 lines
3.5 KiB
Markdown
# DataStack Official Python SDK (`datastack`)
|
|
|
|
[](https://pypi.org/project/datastack/)
|
|
[](https://www.python.org/downloads/)
|
|
[](../../../LICENSE)
|
|
[](https://vibtools.top/)
|
|
|
|
The official production-grade Python SDK for **[DataStack](https://vibtools.top/)** — high-throughput, self-hosted telemetry ingestion and partitioned CSV persistence platform.
|
|
|
|
---
|
|
|
|
## Key Capabilities
|
|
|
|
- **Zero-Boilerplate Auto-Discovery**: Automatically finds and loads `config.datastack.py` in your working directory.
|
|
- **Dual-Transport Resilience**: Real-time Socket.IO streaming on Port 4000 with automatic fallback to HTTP REST (`POST /api/events`) on Port 3000.
|
|
- **Zero-CPU Heartbeat Monitor**: Background recovery thread sleeps in 0-CPU `IDLE` mode when healthy; wakes only on network disruption.
|
|
- **Offline Disk Spooling**: Automatically appends unsent events to `.datastack_spool/spool.jsonl` when offline and replays them atomically upon reconnection.
|
|
- **Direct File Streaming**: Stream `.csv`, `.json`, `.jsonl`, and `.ndjson` files with deterministic SHA-256 deduplication and auto-halving on payload limits.
|
|
- **Backoff & Rate-Limit Pacing**: Lossless partial-batch retries on `429 RATE_LIMIT_EXCEEDED` with jitter.
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
# From PyPI
|
|
pip install datastack
|
|
|
|
# Or from local repository
|
|
pip install ./sdk/python/datastack
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Example
|
|
|
|
Create `config.datastack.py` in your project root:
|
|
```python
|
|
# config.datastack.py
|
|
DATASTACK_CONFIG = {
|
|
"SERVER_URL": "http://localhost:4000",
|
|
"API_URL": "http://localhost:3000",
|
|
"CLIENT_TOKEN": "YOUR_SIGNED_CLIENT_JWT",
|
|
}
|
|
```
|
|
|
|
Then stream telemetry in Python:
|
|
```python
|
|
from datastack import DataStack
|
|
|
|
with DataStack() as client:
|
|
# 1. Single Event
|
|
client.send({"service": "order_worker", "status": "completed", "amount": 99.50})
|
|
|
|
# 2. Batch Events
|
|
client.send_batch([
|
|
{"metric": "cpu", "value": 45.2},
|
|
{"metric": "ram_mb", "value": 2048},
|
|
])
|
|
|
|
# 3. Direct File Streaming
|
|
client.send_file("device_logs.csv", chunk_size=250)
|
|
```
|
|
|
|
---
|
|
|
|
## Documentation Suite
|
|
|
|
Exhaustive technical documentation is available in the **[`docs/`](./docs/README.md)** directory:
|
|
|
|
- **[Installation Guide (`docs/INSTALLATION.md`)](./docs/INSTALLATION.md)** — Runtimes, dependencies, and environment verification.
|
|
- **[Quickstart (`docs/QUICKSTART.md`)](./docs/QUICKSTART.md)** — 5-minute integration guide.
|
|
- **[API Reference (`docs/API_REFERENCE.md`)](./docs/API_REFERENCE.md)** — Complete method signatures, parameters, and return types.
|
|
- **[Configuration Reference (`docs/CONFIGURATION.md`)](./docs/CONFIGURATION.md)** — `config.datastack.py`, environment variables, and precedence order.
|
|
- **[Practical Examples (`docs/EXAMPLES.md`)](./docs/EXAMPLES.md)** — IoT streaming, FastAPI logging, batch transactions, and file ingestion.
|
|
- **[Error Handling (`docs/ERROR_HANDLING.md`)](./docs/ERROR_HANDLING.md)** — Exception hierarchy, status codes, and retry strategies.
|
|
- **[Lifecycle & Resilience (`docs/LIFECYCLE.md`)](./docs/LIFECYCLE.md)** — Connection state machine, Heartbeat Monitor, and offline disk spooler.
|
|
|
|
---
|
|
|
|
## Author & License
|
|
|
|
- **Organization**: [Vib Tools](https://vibtools.top/)
|
|
- **Author**: Md Anik (`ANIK.D.ART`) — `info@vibtools.top`
|
|
- **License**: MIT License
|