feat(sdk): refactor Python SDK documentation

- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
Team Dark Devil committed 2026-11-18 10:03:40 +06:00
1 parent 4279da45bb
commit 40a6855724
38 files changed
+3315 -141

No files matched your search

+48 -74
View File
@@ -1,110 +1,84 @@
# DataStack Official Python SDK (`datastack`)
> **Production-Grade Python Client for DataStack** — featuring automatic `config.datastack.py` configuration discovery, real-time Socket.IO streaming (`Port 4000`), automatic Heartbeat HTTP fallback (`Port 3000`), offline disk spooling, and direct `.csv`, `.json`, and `.jsonl` file ingestion.
[![PyPI version](https://img.shields.io/badge/pypi-v2.2.0-blue.svg)](https://pypi.org/project/datastack/)
[![Python: 3.8+](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](../../../LICENSE)
[![Maintained by Vib Tools](https://img.shields.io/badge/Maintained%20by-Vib%20Tools-0f172a)](https://vibtools.top/)
The official production-grade Python SDK for **[DataStack](https://vibtools.top/)** — high-throughput, self-hosted telemetry ingestion and partitioned CSV persistence platform.
---
## 1. Installation
## Key Capabilities
- **Zero-Boilerplate Auto-Discovery**: Automatically finds and loads `config.datastack.py` in your working directory.
- **Dual-Transport Resilience**: Real-time Socket.IO streaming on Port 4000 with automatic fallback to HTTP REST (`POST /api/events`) on Port 3000.
- **Zero-CPU Heartbeat Monitor**: Background recovery thread sleeps in 0-CPU `IDLE` mode when healthy; wakes only on network disruption.
- **Offline Disk Spooling**: Automatically appends unsent events to `.datastack_spool/spool.jsonl` when offline and replays them atomically upon reconnection.
- **Direct File Streaming**: Stream `.csv`, `.json`, `.jsonl`, and `.ndjson` files with deterministic SHA-256 deduplication and auto-halving on payload limits.
- **Backoff & Rate-Limit Pacing**: Lossless partial-batch retries on `429 RATE_LIMIT_EXCEEDED` with jitter.
---
## Installation
### Install from Local Repository
```bash
# From PyPI
pip install datastack
# Or from local repository
pip install ./sdk/python/datastack
```
### Editable Development Mode
```bash
pip install -e ./sdk/python/datastack
```
### PyPI Package Installation
```bash
pip install datastack
```
---
## 2. Zero-Boilerplate Setup with `config.datastack.py`
Create a file named **`config.datastack.py`** in your Python project's root directory (you can also copy a pre-filled version directly from the DataStack Admin Console at **`/vcon/sdk`**):
## Quick Example
Create `config.datastack.py` in your project root:
```python
# config.datastack.py
DATASTACK_CONFIG = {
"SERVER_URL": "http://localhost:4000", # Port 4000 Socket.IO Ingestion Service
"API_URL": "http://localhost:3000", # Port 3000 HTTP Fallback & File Ingestion API
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Provisioned in /vcon/clients
"CLIENT_ID": "", # Optional override (defaults to JWT sub/clientId)
"AUTO_CONNECT": True,
"ENABLE_HTTP_FALLBACK": True,
"HEARTBEAT_INTERVAL_SEC": 5.0,
"MAX_RETRIES": 3,
"RETRY_BACKOFF_BASE_SEC": 0.5,
"BATCH_CHUNK_SIZE": 250,
"OFFLINE_SPOOL_ENABLED": True,
"OFFLINE_SPOOL_DIR": ".datastack_spool",
"MAX_SPOOL_SIZE_MB": 50,
"SERVER_URL": "http://localhost:4000",
"API_URL": "http://localhost:3000",
"CLIENT_TOKEN": "YOUR_SIGNED_CLIENT_JWT",
}
```
---
## 3. Quickstart Usage
Then stream telemetry in Python:
```python
from datastack import DataStack
# Automatically discovers and loads config.datastack.py and connects!
client = DataStack()
with DataStack() as client:
# 1. Single Event
client.send({"service": "order_worker", "status": "completed", "amount": 99.50})
# 1. Push a Single Event
res = client.send({
"service": "checkout-worker",
"order_id": "ORD-9921",
"amount_usd": 149.99,
"status": "completed",
})
print("Single Event:", res)
# 2. Batch Events
client.send_batch([
{"metric": "cpu", "value": 45.2},
{"metric": "ram_mb", "value": 2048},
])
# 2. Push a Batch of Events
batch_res = client.send_batch([
{"data": {"metric": "cpu_load", "value": 0.42}},
{"data": {"metric": "mem_used_mb", "value": 2048}},
])
print("Batch Result:", batch_res)
# 3. Directly Stream .csv, .json, or .jsonl Files
file_res = client.send_file("monthly_records.csv", chunk_size=250)
print("File Ingestion:", file_res)
# 4. Inspect Connection & Heartbeat Status
print("SDK Status:", client.status())
client.disconnect()
# 3. Direct File Streaming
client.send_file("device_logs.csv", chunk_size=250)
```
---
## 4. Smart Connection State Machine & Auto-Recovery
## Documentation Suite
The DataStack Python SDK maintains a 4-state resilience lifecycle managed by `HeartbeatMonitor`:
Exhaustive technical documentation is available in the **[`docs/`](./docs/README.md)** directory:
1. **`CONNECTED_IDLE` (Primary WebSocket Active — Heartbeat IDLE)**:
- Events stream over Socket.IO (`Port 4000`) with sub-millisecond overhead.
- The Heartbeat Monitor sleeps in **IDLE mode** (`heartbeatMode: "IDLE"`), consuming zero network or CPU resources.
2. **`FALLBACK_HTTP` (WebSocket Disconnected — Auto HTTP Fallback Active)**:
- If the WebSocket connection drops or closes, `send()`, `send_batch()`, and `send_file()` automatically switch to `POST {API_URL}/api/events` so zero application calls fail.
- Simultaneously, the Heartbeat Monitor wakes up (`heartbeatMode: "ACTIVE_RECOVERY"`), probes and reconnects the WebSocket in the background, verifies the handshake (`ping_verify`), and transitions back to **`CONNECTED_IDLE` (`IDLE` mode)**.
3. **`OFFLINE_SPOOL` (Device Offline — Local Disk Spooling)**:
- If the client machine loses internet connectivity completely, events are safely appended to `.datastack_spool/spool.jsonl` with deterministic `eventId`s.
- When the device comes back online, the Heartbeat Monitor detects connectivity, reconnects the WebSocket, replays all spooled events without duplicates, and returns to **`CONNECTED_IDLE` (`IDLE` mode)**.
4. **Structured Error Handling & Auto-Chunking**:
- **`413 PAYLOAD_TOO_LARGE`**: Batch/file payloads automatically split in half (`chunk // 2`) recursively and retry.
- **`429 RATE_LIMIT_EXCEEDED`**: Automatically retries with exponential backoff and random jitter.
- **`401 / 403` (`CLIENT_REVOKED`, `TOKEN_ROTATED`)**: Immediately halts retries and raises `DataStackAuthError`.
- **[Installation Guide (`docs/INSTALLATION.md`)](./docs/INSTALLATION.md)** — Runtimes, dependencies, and environment verification.
- **[Quickstart (`docs/QUICKSTART.md`)](./docs/QUICKSTART.md)** — 5-minute integration guide.
- **[API Reference (`docs/API_REFERENCE.md`)](./docs/API_REFERENCE.md)** — Complete method signatures, parameters, and return types.
- **[Configuration Reference (`docs/CONFIGURATION.md`)](./docs/CONFIGURATION.md)** — `config.datastack.py`, environment variables, and precedence order.
- **[Practical Examples (`docs/EXAMPLES.md`)](./docs/EXAMPLES.md)** — IoT streaming, FastAPI logging, batch transactions, and file ingestion.
- **[Error Handling (`docs/ERROR_HANDLING.md`)](./docs/ERROR_HANDLING.md)** — Exception hierarchy, status codes, and retry strategies.
- **[Lifecycle & Resilience (`docs/LIFECYCLE.md`)](./docs/LIFECYCLE.md)** — Connection state machine, Heartbeat Monitor, and offline disk spooler.
---
## 5. Author & License
## Author & License
- **Organization**: [Vib Tools](https://vibtools.top/)
- **Author**: Md Anik (`ANIK.D.ART`) — `info@vibtools.top`
+208
View File
@@ -0,0 +1,208 @@
# API Reference Specification
This document provides the exhaustive specification for all public classes, methods, parameters, and return structures in the DataStack Python SDK.
---
## 1. Primary Client: `DataStack` / `DataStackClient`
Import path:
```python
from datastack import DataStack, DataStackClient
# Both identifiers point to the same underlying class.
```
### Constructor: `__init__(...)`
```python
DataStack(
server_url: Optional[str] = None,
token: Optional[str] = None,
api_url: Optional[str] = None,
config_path: Optional[str] = None,
auto_connect: Optional[bool] = None,
reconnection_attempts: Optional[int] = None,
reconnection_delay: Optional[float] = None,
client_id: Optional[str] = None,
**kwargs: Any,
)
```
| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `server_url` | `str` | `None` (loads from config or `http://localhost:4000`) | URL for Port 4000 Socket.IO ingestion service. |
| `token` | `str` | `None` (loads from config or env) | Signed HMAC-SHA256 client JWT token. |
| `api_url` | `str` | `None` (inferred from `server_url` or `http://localhost:3000`) | URL for Port 3000 REST API and automatic HTTP fallback. |
| `config_path` | `str` | `None` | Path to a specific Python configuration file. |
| `auto_connect` | `bool` | `None` (defaults to `True` if config loaded) | Whether to connect immediately during initialization. |
| `reconnection_attempts` | `int` | `None` (alias for `MAX_RETRIES`, default: `3`) | Maximum number of retry attempts for transient errors. |
| `reconnection_delay` | `float`| `None` (alias for `HEARTBEAT_INTERVAL_SEC`) | Delay in seconds between reconnection probes. |
| `client_id` | `str` | `None` | Explicit client identifier slug (must match JWT `sub`/`clientId`). |
| `**kwargs` | `Any` | — | Any configuration overrides passed to `load_config()`. |
---
### Core Public Methods
#### `connect(wait_timeout: Optional[float] = None) -> bool`
Establishes the primary WebSocket connection to Port 4000 and verifies the handshake via `ping_verify`.
- **Behavior**:
- If WebSocket succeeds: Drains any offline spooled events and transitions the Heartbeat Monitor to `CONNECTED_IDLE` (zero-CPU IDLE mode).
- If WebSocket fails and `enable_http_fallback` is `True`: Checks HTTP API health. If online, enters `FALLBACK_HTTP` and starts background recovery.
- If both are unreachable and `offline_spool_enabled` is `True`: Enters `OFFLINE_SPOOL`.
- **Returns**: `True` if connected to WebSocket or HTTP fallback is operational; `False` if offline.
- **Raises**: `DataStackAuthError` if credentials are rejected (`401`/`403`).
---
#### `disconnect() -> None`
Stops the background Heartbeat Monitor thread and gracefully closes any active Socket.IO connection.
---
#### `send(data: Dict[str, Any], event_id: Optional[str] = None, timestamp: Optional[str] = None, timeout: Optional[float] = None) -> Dict[str, Any]`
Sends a single telemetry event.
- **Parameters**:
- `data` (`Dict[str, Any]`): Arbitrary JSON-serializable dictionary containing the event payload.
- `event_id` (`str`, optional): Unique event identifier for idempotency. Auto-generated (`evt_<timestamp>_<uuid>`) if omitted.
- `timestamp` (`str`, optional): ISO 8601 UTC timestamp. Auto-generated (`_utc_now_iso()`) if omitted.
- `timeout` (`float`, optional): Timeout in seconds for waiting for server acknowledgment.
- **Returns**: Dictionary with delivery details:
```python
{
"status": "ok",
"eventId": "evt_1700000000000_a1b2c3d4",
"transport": "WEBSOCKET" # or "HTTP_FALLBACK" or "OFFLINE_SPOOL"
}
```
---
#### `sync(...) -> Dict[str, Any]`
Backward-compatible alias for `send()`.
```python
client.sync(data={"status": "active"}, timeout=5.0)
```
---
#### `send_batch(events: Optional[List[Dict[str, Any]]] = None, timeout: Optional[float] = None, file_name: Optional[str] = None, *, records: Optional[List[Dict[str, Any]]] = None) -> Dict[str, Any]`
Transmits a list of event dictionaries in a single network round-trip.
- **Parameters**:
- `events` (`List[Dict[str, Any]]`): List of event dictionaries (accepts either raw data dicts or wrapped envelopes `{eventId, timestamp, data}`).
- `records`: Keyword-only alias for `events`.
- `timeout`: Network ACK timeout.
- `file_name` (optional): Source filename if sending structured file records.
- **Built-in Auto-Recovery**:
- **Auto-Splitting on `413 PAYLOAD_TOO_LARGE`**: Recursively halves the batch (`chunk // 2`) and sends each half.
- **Lossless Partial Rate-Limit Retry**: If the server returns `rateLimitedEventIds`, the SDK automatically waits 1.05s for the 1-second sliding window to reset and retries only the throttled items.
- **Returns**:
```python
{
"success": True,
"processed": 500,
"rejected": 0,
"duplicates": 0,
"spooledCount": 0,
"transport": "WEBSOCKET"
}
```
---
#### `batch_sync(...) -> Dict[str, Any]`
Backward-compatible alias for `send_batch()`. Accepts either `events` or `records`.
---
#### `send_file(file_path: Union[str, Path], chunk_size: Optional[int] = None, extra_metadata: Optional[Dict[str, Any]] = None) -> Dict[str, Any]`
Streams a local file directly into DataStack in deterministic chunks.
- **Supported File Types**: `.csv`, `.json`, `.jsonl`, `.ndjson`.
- **Parameters**:
- `file_path`: Path to local data file.
- `chunk_size`: Rows per batch chunk (defaults to `BATCH_CHUNK_SIZE`, default: 250).
- `extra_metadata`: Optional key-value dictionary merged into every record.
- **Features**:
- Deterministic SHA-256 event ID generation (`file_{stem}_{row}_{hash}`) preventing duplicates on re-runs.
- UTF-8 with Byte Order Mark (`utf-8-sig`) support.
- **Returns**:
```python
{
"success": True,
"fileName": "transactions.csv",
"fileType": "csv",
"chunksSent": 4,
"processed": 1000,
"rejected": 0,
"duplicates": 0,
"spooledCount": 0,
"transport": "WEBSOCKET"
}
```
---
#### `status() -> Dict[str, Any]`
Returns a diagnostic snapshot of the client instance:
```python
{
"connectedWebSocket": True,
"clientId": "pos_terminal_01",
"serverUrl": "http://localhost:4000",
"apiUrl": "http://localhost:3000",
"configSource": "/path/to/config.datastack.py",
"connectionState": "CONNECTED_IDLE",
"heartbeatMode": "IDLE", # "IDLE" or "ACTIVE_RECOVERY"
"recoveryAttempts": 0,
"lastError": None,
"offlineSpoolPending": 0,
"offlineSpoolBytes": 0
}
```
---
## 2. Configuration Class: `DataStackConfig`
Dataclass storing resolved SDK settings:
```python
from datastack import DataStackConfig
@dataclass
class DataStackConfig:
server_url: str = "http://localhost:4000"
api_url: str = "http://localhost:3000"
client_token: str = ""
client_id: str = ""
auto_connect: bool = True
enable_http_fallback: bool = True
heartbeat_interval_sec: float = 5.0
request_timeout_sec: float = 8.0
max_retries: int = 3
retry_backoff_base_sec: float = 0.5
batch_chunk_size: int = 250
offline_spool_enabled: bool = True
offline_spool_dir: str = ".datastack_spool"
max_spool_size_mb: int = 50
config_source_path: Optional[str] = None
```
---
## 3. Configuration Loader: `load_config`
```python
from datastack import load_config
config: DataStackConfig = load_config(
config_path=None,
require_token=False,
**overrides
)
```
Discovers `config.datastack.py`, layers environment variables (`DATASTACK_*`), and merges keyword overrides.
@@ -0,0 +1,80 @@
# Configuration Guide
The DataStack Python SDK features a zero-boilerplate auto-discovery configuration system supporting Python configuration files (`config.datastack.py`), environment variables (`DATASTACK_*`), and explicit runtime overrides.
---
## 1. Configuration Auto-Discovery
When instantiating `DataStack()` without arguments, the SDK locates `config.datastack.py` using this search order:
1. **Explicit Path**: If `config_path="path/to/config.py"` is passed to the constructor.
2. **Environment Variable**: `DATASTACK_CONFIG_PATH`.
3. **Current Working Directory**: The directory from which the application is run, searching upwards through up to 3 parent directories.
4. **Entry Script Directory**: The directory containing `sys.argv[0]`, searching upwards through up to 2 parent directories.
---
## 2. Configuration Precedence
Settings are resolved using strict layered precedence:
```text
Explicit Keyword Arguments > Environment Variables > config.datastack.py > Built-in Defaults
```
If a setting is defined in multiple places, the higher precedence layer wins.
---
## 3. Configuration Reference Table
| Key in `config.datastack.py` | Environment Variable | Type | Default | Description |
| :--- | :--- | :--- | :--- | :--- |
| `SERVER_URL` | `DATASTACK_SERVER_URL` | `str` | `http://localhost:4000` | Port 4000 Socket.IO ingestion service URL. |
| `API_URL` | `DATASTACK_API_URL` | `str` | `http://localhost:3000` | Port 3000 REST API and automatic HTTP fallback URL. |
| `CLIENT_TOKEN` | `DATASTACK_CLIENT_TOKEN` | `str` | `""` | Signed HMAC-SHA256 client JWT token provisioned in `/vcon/clients`. |
| `CLIENT_ID` | `DATASTACK_CLIENT_ID` | `str` | `""` | Unique client identifier slug. If omitted, automatically extracted from JWT `clientId` or `sub` claim. |
| `AUTO_CONNECT` | `DATASTACK_AUTO_CONNECT` | `bool` | `True` | Automatically connect and verify WebSocket handshake during initialization. |
| `ENABLE_HTTP_FALLBACK` | `DATASTACK_ENABLE_HTTP_FALLBACK` | `bool` | `True` | Seamlessly route events via HTTP `POST /api/events` when WebSocket is disconnected. |
| `HEARTBEAT_INTERVAL_SEC` | `DATASTACK_HEARTBEAT_INTERVAL_SEC` | `float` | `5.0` | Probing frequency in seconds when recovering disconnected connections. |
| `REQUEST_TIMEOUT_SEC` | `DATASTACK_REQUEST_TIMEOUT_SEC` | `float` | `8.0` | Per-request timeout in seconds for server responses and handshakes. |
| `MAX_RETRIES` | `DATASTACK_MAX_RETRIES` | `int` | `3` | Maximum number of exponential backoff retry attempts for rate-limited or transient errors. |
| `RETRY_BACKOFF_BASE_SEC` | `DATASTACK_RETRY_BACKOFF_BASE_SEC` | `float` | `0.5` | Initial backoff delay in seconds for retry attempts (scaled exponentially with jitter). |
| `BATCH_CHUNK_SIZE` | `DATASTACK_BATCH_CHUNK_SIZE` | `int` | `250` | Default number of records per batch when streaming files via `send_file()`. |
| `OFFLINE_SPOOL_ENABLED` | `DATASTACK_OFFLINE_SPOOL_ENABLED` | `bool` | `True` | Enable saving events to local disk when the device loses network connectivity entirely. |
| `OFFLINE_SPOOL_DIR` | `DATASTACK_OFFLINE_SPOOL_DIR` | `str` | `.datastack_spool` | Local filesystem directory path where `spool.jsonl` is written. |
| `MAX_SPOOL_SIZE_MB` | `DATASTACK_MAX_SPOOL_SIZE_MB` | `int` | `50` | Maximum disk size in megabytes allocated for offline event buffering before rejecting new writes. |
---
## 4. Configuration Formats
### Standard Dictionary Format (`DATASTACK_CONFIG`)
```python
# config.datastack.py
DATASTACK_CONFIG = {
"SERVER_URL": "https://ingestion.datastack.example",
"API_URL": "https://api.datastack.example",
"CLIENT_TOKEN": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"BATCH_CHUNK_SIZE": 500,
"MAX_SPOOL_SIZE_MB": 100,
}
```
### Module-Level Variables Format
```python
# config.datastack.py
SERVER_URL = "https://ingestion.datastack.example"
API_URL = "https://api.datastack.example"
CLIENT_TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
```
---
## 5. Client ID Validation Rule
To prevent misattribution:
- If `CLIENT_ID` is explicitly specified in your configuration or constructor, it **must match** the `clientId` or `sub` claim inside your signed `CLIENT_TOKEN`.
- If there is a mismatch, the SDK raises `DataStackConfigError` immediately on initialization.
- If `CLIENT_ID` is omitted, the SDK automatically extracts the subject claim from the token.
+107
View File
@@ -0,0 +1,107 @@
# Error Handling & Exception Reference
The DataStack Python SDK implements a structured exception hierarchy distinguishing between fatal configuration/auth errors and transient, auto-recoverable issues.
---
## 1. Exception Hierarchy
All exceptions inherit from `DataStackError`:
```text
DataStackError (Base)
├── DataStackConfigError (Configuration missing, malformed, or CLIENT_ID mismatch)
├── DataStackAuthError (Token rejected, client revoked, or rotated)
├── DataStackConnectionError (Network unreachable, endpoints down, offline)
├── DataStackRateLimitError (Rate limit exceeded after all retry attempts)
├── DataStackPayloadTooLargeError (Payload exceeds maximum permitted size)
├── DataStackFileError (File missing, unsupported extension, or corrupted)
└── DataStackOfflineSpoolError (Disk spool directory unwriteable or disk quota full)
```
---
## 2. Exceptions Specification
### `DataStackConfigError`
- **Raised When**:
- `CLIENT_TOKEN` is missing or contains placeholder strings.
- An explicit `CLIENT_ID` does not match the subject claim inside `CLIENT_TOKEN`.
- `config.datastack.py` contains invalid Python syntax.
- **Retryable**: **No**. Requires modifying configuration files or environment variables.
---
### `DataStackAuthError`
- **Raised When**:
- The server returns HTTP `401 Unauthorized` or `403 Forbidden`.
- The WebSocket handshake is rejected with `AUTH_REQUIRED`, `AUTH_INVALID`, `CLIENT_REVOKED`, or `TOKEN_ROTATED`.
- **Special Behavior**: Immediately stops the background Heartbeat Monitor thread to prevent hammering the server with invalid credentials.
- **Retryable**: **No**. The operator must update the client token or re-activate the client in `/vcon/clients`.
---
### `DataStackRateLimitError`
- **Raised When**:
- The client exceeds its allocated events-per-second quota (`429 RATE_LIMIT_EXCEEDED`).
- **Retryable**: **Yes**. The SDK automatically executes exponential backoff retries with jitter and a minimum 1.05-second pause to allow the 1-second server sliding window to reset. This exception is only raised if all configured retries (`MAX_RETRIES`) are exhausted.
---
### `DataStackPayloadTooLargeError`
- **Raised When**:
- A single event payload exceeds `maxPayloadSizeKb` (HTTP `413 PAYLOAD_TOO_LARGE`).
- **Special Handling in Batches**: When calling `send_batch()`, the SDK catches this error and automatically divides the batch in half (`chunk // 2`) recursively, retrying each sub-chunk transparently.
---
### `DataStackConnectionError`
- **Raised When**:
- Both the primary WebSocket (`Port 4000`) and the HTTP fallback API (`Port 3000`) are unreachable, **AND** offline disk spooling is disabled (`OFFLINE_SPOOL_ENABLED=False`).
- **Retryable**: **Yes**.
---
### `DataStackFileError`
- **Raised When**:
- A file path passed to `send_file()` does not exist or is a directory.
- The file extension is unsupported (only `.csv`, `.json`, `.jsonl`, `.ndjson` allowed).
- A JSONL file contains corrupted or unparseable JSON on a specific line number.
---
### `DataStackOfflineSpoolError`
- **Raised When**:
- The `.datastack_spool/` directory cannot be created due to permission errors.
- The spool file size exceeds `MAX_SPOOL_SIZE_MB` (default: 50 MB).
---
## 3. Recommended Error Handling Pattern
```python
from datastack import (
DataStack,
DataStackAuthError,
DataStackConfigError,
DataStackConnectionError,
DataStackRateLimitError,
DataStackError,
)
try:
client = DataStack()
client.send({"temperature": 23.4})
except DataStackConfigError as exc:
print(f"Configuration defect: {exc}")
except DataStackAuthError as exc:
# Token expired, revoked, or rotated
print(f"Authentication failure: {exc.code} - {exc.message}")
# Alert operations team to rotate or update CLIENT_TOKEN
except DataStackRateLimitError as exc:
print(f"Rate limit exceeded after retries: {exc}")
except DataStackConnectionError as exc:
print(f"Network unreachable and spooling disabled: {exc}")
except DataStackError as exc:
print(f"Unexpected DataStack error: {exc}")
```
+179
View File
@@ -0,0 +1,179 @@
# Practical Usage Examples
This guide provides production-ready code examples covering common telemetry, logging, and data pipeline scenarios.
---
## 1. IoT Edge Sensor Streaming (with Offline Resilience)
Edge devices frequently face intermittent Wi-Fi or cellular drops. The SDK buffers events to local disk (`.datastack_spool/`) when the connection drops, and drains them automatically when restored:
```python
import time
import random
from datastack import DataStack
# Initialize client using config.datastack.py
client = DataStack()
try:
while True:
# Simulate reading from hardware sensors
sensor_reading = {
"sensor_id": "thermostat-zone-a",
"temperature_c": round(20.0 + random.uniform(-2.5, 3.0), 2),
"humidity_percent": round(45.0 + random.uniform(-5.0, 5.0), 1),
"battery_level": 94,
}
# Automatically routes via WebSocket, HTTP fallback, or local disk spool
res = client.send(sensor_reading)
print(f"Delivered via {res.get('transport')}: eventId={res.get('eventId')}")
time.sleep(2.0)
except KeyboardInterrupt:
print("Shutting down sensor stream...")
finally:
client.disconnect()
```
---
## 2. Web Application Audit Logging (FastAPI Middleware)
Record HTTP request and response metrics asynchronously from a Python web service:
```python
import time
from fastapi import FastAPI, Request
from datastack import DataStack
app = FastAPI()
datastack_client = DataStack()
@app.middleware("http")
async def datastack_telemetry_middleware(request: Request, call_next):
start_time = time.time()
response = await call_next(request)
duration_ms = round((time.time() - start_time) * 1000, 2)
# Dispatch audit record to DataStack
datastack_client.send({
"eventType": "http_request",
"method": request.method,
"path": request.url.path,
"statusCode": response.status_code,
"durationMs": duration_ms,
"clientIp": request.client.host if request.client else "unknown",
})
return response
@app.on_event("shutdown")
def on_shutdown():
datastack_client.disconnect()
```
---
## 3. High-Volume Batch Ingestion (Auto-Splitting on 413)
When emitting high-volume bursts, the SDK automatically splits batches in half if they exceed the server's maximum payload limit:
```python
from datastack import DataStack
client = DataStack()
# Construct a high-volume batch of 1,000 transaction events
transactions = [
{
"transactionId": f"tx_{idx:06d}",
"amount": round(idx * 1.25, 2),
"currency": "EUR",
"status": "settled",
}
for idx in range(1000)
]
# send_batch handles chunking, rate-limit backoff, and 413 halving automatically
result = client.send_batch(transactions)
print("Batch Ingestion Summary:")
print(f"- Processed Rows: {result['processed']}")
print(f"- Duplicate Rows: {result['duplicates']}")
print(f"- Rejected Rows: {result['rejected']}")
print(f"- Transport Used: {result['transport']}")
client.disconnect()
```
---
## 4. Bulk CSV File Ingestion with Metadata Tagging
Stream a monthly export or historical log file into DataStack in chunks of 500 rows, injecting tenant metadata:
```python
from datastack import DataStack
client = DataStack()
file_summary = client.send_file(
"store_sales_2026_11.csv",
chunk_size=500,
extra_metadata={
"ingestedBy": "etl_pipeline_v2",
"region": "us-east-1",
}
)
print(f"File {file_summary['fileName']} streamed in {file_summary['chunksSent']} chunks.")
print(f"Total rows successfully ingested: {file_summary['processed']}")
client.disconnect()
```
---
## 5. Explicit Initialization (Without Configuration File)
In dynamic serverless jobs, container environments, or tests where creating `config.datastack.py` is inconvenient:
```python
from datastack import DataStack
client = DataStack(
server_url="https://ingestion.datastack.example",
api_url="https://api.datastack.example",
token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
auto_connect=True,
max_retries=5,
offline_spool_enabled=False, # Ephemeral container: disable disk spooling
)
client.send({"job": "nightly_cleanup", "status": "success"})
client.disconnect()
```
---
## 6. Inspecting Client Health & Spool Status
Periodically check connection health, Heartbeat mode, and pending offline spool depth:
```python
from datastack import DataStack
client = DataStack()
status = client.status()
print("DataStack Client Health Snapshot:")
print(f"- WebSocket Connected: {status['connectedWebSocket']}")
print(f"- Connection State: {status['connectionState']}")
print(f"- Heartbeat Mode: {status['heartbeatMode']}")
print(f"- Pending Spool Count: {status['offlineSpoolPending']}")
print(f"- Spool Disk Bytes: {status['offlineSpoolBytes']} bytes")
client.disconnect()
```
+73
View File
@@ -0,0 +1,73 @@
# Installation & Environment Setup
This guide details installation options, supported runtimes, package requirements, and environment verification for the DataStack Python SDK.
---
## 1. Supported Environments & Runtime Requirements
- **Python Version**: `>= 3.8` (Tested on Python `3.8`, `3.9`, `3.10`, `3.11`, and `3.12`).
- **Operating Systems**: Linux, macOS, Windows (OS-independent).
- **Network Access**:
- Ingestion WebSocket: `Port 4000` (or reverse proxy `/socket.io/`).
- HTTP REST API & Fallback: `Port 3000` (or reverse proxy `/api/*`).
---
## 2. Package Installation
### Option A: From PyPI
```bash
pip install datastack
```
### Option B: From Local Repository Directory
Install directly from the root of the repository:
```bash
pip install ./sdk/python/datastack
```
### Option C: Editable Development Mode
Use this mode when contributing to or testing local modifications:
```bash
pip install -e ./sdk/python/datastack
```
### Option D: Compatibility Alias (`datastack-sdk`)
For existing codebases using the `datastack_sdk` namespace:
```bash
pip install datastack-sdk
# Or from local repository:
pip install ./sdk-python
```
Both `from datastack import DataStack` and `from datastack_sdk import DataStackClient` point to the identical underlying engine.
---
## 3. Dependencies
The SDK specifies the following dependencies in `pyproject.toml` and `setup.py`:
| Dependency | Minimum Version | Purpose |
| :--- | :--- | :--- |
| `python-socketio[client]` | `>= 5.8.0` | High-performance Socket.IO client for WebSocket communication. |
| `requests` | `>= 2.28.0` | Robust HTTP transport for REST ingestion and automatic fallback. |
| `websocket-client` | `>= 1.5.0` | Low-level WebSocket transport engine for Python Socket.IO. |
> **Minimal Environments**: If `requests` is absent in a stripped-down embedded container, the SDK automatically falls back to Python's built-in `urllib.request` standard library.
---
## 4. Verifying Installation
Verify that the package is correctly installed and accessible in your environment:
```bash
python3 -c "import datastack; print('DataStack SDK Version:', datastack.__version__)"
```
Expected output:
```text
DataStack SDK Version: 2.2.0
```
+90
View File
@@ -0,0 +1,90 @@
# Lifecycle, Resilience & Offline Spooling
This document explains the internal connection state machine, the zero-CPU Heartbeat Monitor, and the thread-safe offline disk spooler powering the DataStack Python SDK.
---
## 1. Connection State Machine (`ConnectionState`)
The client transitions automatically through five lifecycle states:
```text
┌────────────────┐ WebSocket Drop ┌────────────────┐
│ CONNECTED_IDLE ├────────────────────────────────►│ FALLBACK_HTTP │
│ (Heartbeat │◄────────────────────────────────┤ (Heartbeat │
│ IDLE Mode) │ Reconnected & Verified │ ACTIVE_REC) │
└───────┬────────┘ (via ping_verify) └───────┬────────┘
│ │ Both WS & HTTP
│ │ Unreachable
│ ▼
│ ┌────────────────┐
│ │ OFFLINE_SPOOL │
│ │ (Local Disk │
│ │ spool.jsonl) │
│ └───────┬────────┘
│ │
│ client.disconnect() or │ Network Returns
│ Fatal Auth Revocation │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ STOPPED │◄────────────────────────────────┤ RECONNECTING_ │
│ │ │ VERIFY │
└────────────────┘ └────────────────┘
```
### State Details
| State | Transport Used | Heartbeat Mode | Description |
| :--- | :--- | :--- | :--- |
| **`CONNECTED_IDLE`** | WebSocket (Port 4000) | `IDLE` (0% CPU) | Primary WebSocket connection is active and verified. The Heartbeat thread sleeps, consuming zero network or CPU resources. |
| **`FALLBACK_HTTP`** | HTTP REST (Port 3000) | `ACTIVE_RECOVERY` | WebSocket connection dropped. Outgoing events automatically route via HTTP `POST /api/events`. The Heartbeat thread wakes up and probes WebSocket reconnection in the background. |
| **`OFFLINE_SPOOL`** | Local Disk Queue | `ACTIVE_RECOVERY` | The client device has lost internet connectivity completely. Telemetry events are appended to `.datastack_spool/spool.jsonl`. |
| **`RECONNECTING_VERIFY`** | WebSocket | `ACTIVE_RECOVERY` | Reconnection handshake in progress. The client issues a `ping_verify` emission and replays spooled disk batches. |
| **`STOPPED`** | None | `STOPPED` | Client was closed via `disconnect()` or credentials were permanently revoked (`401`/`403`). All background threads terminate. |
---
## 2. Zero-CPU Heartbeat Monitor (`HeartbeatMonitor`)
Traditional SDKs constantly emit polling pings even when connections are healthy, wasting bandwidth and battery. DataStack implements an **Event-Driven Sleep Architecture**:
1. **When Healthy**: Once `connect()` succeeds and passes `ping_verify`, the monitor clears its internal wake event (`_wake_event.clear()`). The background thread remains completely idle.
2. **On Disconnect**: The Socket.IO `disconnect` event or an HTTP delivery trigger wakes the thread (`_wake_event.set()`).
3. **Background Reconnection**: Every `HEARTBEAT_INTERVAL_SEC` (default: 5.0s), the monitor attempts to reconnect and verify the WebSocket.
4. **Return to Idle**: Once verified, the thread drains any offline spooled events and immediately returns to sleep.
---
## 3. Offline Disk Spooling (`DiskSpooler`)
When edge devices, field laptops, or factory floor terminals lose internet connectivity:
### Storage Mechanics
- Events are appended to `.datastack_spool/spool.jsonl` using UTF-8 JSON lines.
- Each event preserves its deterministic `eventId`, `timestamp`, and `clientId`.
- Disk capacity is protected by `MAX_SPOOL_SIZE_MB` (default: 50 MB).
### Atomic Drain & Crash Recovery
- When connectivity returns, the Heartbeat Monitor reads spooled records in batches of `BATCH_CHUNK_SIZE` (default: 250).
- Batches are emitted to the server.
- The spool file is trimmed atomically using `ack_and_requeue_front()` and `os.replace()`, guaranteeing that if the process crashes mid-stream, unacknowledged records remain at the head of the file without duplication or data loss.
---
## 4. Resource Disposal & Cleanup
Always disconnect the client when your application shuts down to avoid dangling threads:
```python
client = DataStack()
try:
client.send({"metric": "active"})
finally:
client.disconnect()
```
Or use the context manager:
```python
with DataStack() as client:
client.send({"metric": "active"})
```
+112
View File
@@ -0,0 +1,112 @@
# Quickstart Guide
Get up and running with the DataStack Python SDK in under 5 minutes.
---
## 1. Prerequisites
1. A running DataStack instance (local dev or production server).
2. A signed client token obtained from the Admin Console at **`/vcon/clients`** or **`/vcon/sdk`**.
---
## 2. Step 1: Create `config.datastack.py`
Place a `config.datastack.py` file in your application root directory (next to your main entry script). The SDK automatically searches your current working directory and up to three parent levels.
```python
# config.datastack.py
DATASTACK_CONFIG = {
"SERVER_URL": "http://localhost:4000", # Port 4000 WebSocket Ingestion Server
"API_URL": "http://localhost:3000", # Port 3000 REST API & HTTP Fallback
"CLIENT_TOKEN": "YOUR_SIGNED_CLIENT_JWT", # Provisioned in /vcon/clients
"CLIENT_ID": "pos_terminal_01", # Optional (auto-derived from JWT if omitted)
"AUTO_CONNECT": True,
}
```
---
## 3. Step 2: Push Your First Telemetry Event
Create `main.py` and run it:
```python
from datastack import DataStack
# Automatically discovers config.datastack.py, connects, and verifies handshake
client = DataStack()
# Push a single event
response = client.send({
"service": "checkout_worker",
"orderId": "ORD-98214",
"amount": 49.95,
"currency": "USD",
"status": "completed",
})
print("Ingestion Result:", response)
# Output:
# {'status': 'ok', 'eventId': 'evt_1700000000000_a1b2c3d4', 'transport': 'WEBSOCKET'}
# Graceful cleanup
client.disconnect()
```
---
## 4. Step 3: High-Volume Batch Streaming
Send multiple records in a single network round-trip. If the batch exceeds the server's maximum payload limit (`413 PAYLOAD_TOO_LARGE`), the SDK automatically splits the batch in half recursively:
```python
from datastack import DataStack
client = DataStack()
metrics_batch = [
{"host": "worker-01", "cpu": 42.1, "mem_mb": 1024},
{"host": "worker-02", "cpu": 68.4, "mem_mb": 2048},
{"host": "worker-03", "cpu": 15.2, "mem_mb": 512},
]
result = client.send_batch(metrics_batch)
print(f"Processed: {result['processed']}, Duplicates: {result['duplicates']}")
client.disconnect()
```
---
## 5. Step 4: Direct File Ingestion
Stream an entire `.csv`, `.json`, `.jsonl`, or `.ndjson` dataset directly from disk. Records are assigned deterministic SHA-256 event IDs to guarantee idempotent delivery without memory bloat:
```python
from datastack import DataStack
client = DataStack()
# Stream records in chunks of 250 rows
summary = client.send_file("device_logs.csv", chunk_size=250)
print(f"File: {summary['fileName']}, Processed {summary['processed']} rows.")
client.disconnect()
```
---
## 6. Context Manager (Auto-Cleanup)
Use Python's `with` statement to guarantee clean disconnection when execution completes:
```python
from datastack import DataStack
with DataStack() as client:
client.send({"action": "system_startup", "timestamp_utc": "2026-11-17T12:00:00Z"})
print("Status:", client.status()["connectionState"])
# Client disconnects cleanly here automatically
```
+51
View File
@@ -0,0 +1,51 @@
# DataStack Python SDK Documentation
Welcome to the official developer documentation for the **DataStack Python SDK** (`datastack`).
This documentation suite provides a complete, compact technical reference for installing, configuring, operating, and troubleshooting the SDK in production environments.
---
## 1. Documentation Index
| Guide | Description | Target Audience |
| :--- | :--- | :--- |
| **[Installation (`INSTALLATION.md`)](./INSTALLATION.md)** | Supported Python runtimes, package installation, virtual environments, and dependency requirements. | All Developers |
| **[Quickstart (`QUICKSTART.md`)](./QUICKSTART.md)** | 5-minute zero-boilerplate integration guide using `config.datastack.py`. | Getting Started |
| **[API Reference (`API_REFERENCE.md`)](./API_REFERENCE.md)** | Full specification of `DataStack`, public methods, parameter types, defaults, and return schemas. | Integration Engineers |
| **[Configuration Guide (`CONFIGURATION.md`)](./CONFIGURATION.md)** | Auto-discovery rules, configuration keys, environment variables (`DATASTACK_*`), and precedence order. | DevOps & Systems Engineers |
| **[Practical Examples (`EXAMPLES.md`)](./EXAMPLES.md)** | Real-world patterns: IoT sensor streaming, batch transactions, CSV/JSON file ingestion, and context managers. | Application Developers |
| **[Error Handling (`ERROR_HANDLING.md`)](./ERROR_HANDLING.md)** | Exception hierarchy, error codes (`401`, `403`, `413`, `429`), retry backoff, and non-retryable errors. | Production Support & QA |
| **[Lifecycle & Resilience (`LIFECYCLE.md`)](./LIFECYCLE.md)** | Smart Heartbeat Monitor, connection state machine, zero-CPU IDLE mode, and offline disk spooling. | Systems Architects |
---
## 2. Architecture at a Glance
The DataStack Python SDK is built around a resilient dual-transport and auto-recovery architecture:
```text
┌────────────────────────────┐
│ DataStack Python SDK │
└──────────────┬─────────────┘
│
┌─────────────────────────┴─────────────────────────┐
│ Primary Transport │ Auto Fallback
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Socket.IO Ingestion │ │ HTTP REST Fallback │
│ Port 4000 (WebSocket) │ │ Port 3000 (/api/events│
└───────────┬───────────┘ └───────────┬───────────┘
│ │
└─────────────────────────┬─────────────────────────┘
│ Both Unreachable (Offline)
▼
┌───────────────────────────┐
│ Offline Disk Spooler │
│ (.datastack_spool/*.jsonl)│
└───────────────────────────┘
```
1. **Primary Real-Time Streaming (Port 4000)**: Events emit via Socket.IO WebSocket (`sync_data`, `batch_sync`). Once connected and verified (`ping_verify`), the background Heartbeat Monitor transitions to **zero-CPU IDLE mode**.
2. **Automatic HTTP Fallback (Port 3000)**: If the WebSocket disconnects, all calls seamlessly route through `POST /api/events` while the Heartbeat Monitor probes background reconnection.
3. **Offline Disk Spooling**: If the device loses internet entirely, events are safely persisted to `.datastack_spool/spool.jsonl` and automatically replayed in order once connectivity returns.