- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
209 lines
7.4 KiB
Markdown
209 lines
7.4 KiB
Markdown
# API Reference Specification
|
|
|
|
This document provides the exhaustive specification for all public classes, methods, parameters, and return structures in the DataStack Python SDK.
|
|
|
|
---
|
|
|
|
## 1. Primary Client: `DataStack` / `DataStackClient`
|
|
|
|
Import path:
|
|
```python
|
|
from datastack import DataStack, DataStackClient
|
|
# Both identifiers point to the same underlying class.
|
|
```
|
|
|
|
### Constructor: `__init__(...)`
|
|
|
|
```python
|
|
DataStack(
|
|
server_url: Optional[str] = None,
|
|
token: Optional[str] = None,
|
|
api_url: Optional[str] = None,
|
|
config_path: Optional[str] = None,
|
|
auto_connect: Optional[bool] = None,
|
|
reconnection_attempts: Optional[int] = None,
|
|
reconnection_delay: Optional[float] = None,
|
|
client_id: Optional[str] = None,
|
|
**kwargs: Any,
|
|
)
|
|
```
|
|
|
|
| Parameter | Type | Default | Description |
|
|
| :--- | :--- | :--- | :--- |
|
|
| `server_url` | `str` | `None` (loads from config or `http://localhost:4000`) | URL for Port 4000 Socket.IO ingestion service. |
|
|
| `token` | `str` | `None` (loads from config or env) | Signed HMAC-SHA256 client JWT token. |
|
|
| `api_url` | `str` | `None` (inferred from `server_url` or `http://localhost:3000`) | URL for Port 3000 REST API and automatic HTTP fallback. |
|
|
| `config_path` | `str` | `None` | Path to a specific Python configuration file. |
|
|
| `auto_connect` | `bool` | `None` (defaults to `True` if config loaded) | Whether to connect immediately during initialization. |
|
|
| `reconnection_attempts` | `int` | `None` (alias for `MAX_RETRIES`, default: `3`) | Maximum number of retry attempts for transient errors. |
|
|
| `reconnection_delay` | `float`| `None` (alias for `HEARTBEAT_INTERVAL_SEC`) | Delay in seconds between reconnection probes. |
|
|
| `client_id` | `str` | `None` | Explicit client identifier slug (must match JWT `sub`/`clientId`). |
|
|
| `**kwargs` | `Any` | — | Any configuration overrides passed to `load_config()`. |
|
|
|
|
---
|
|
|
|
### Core Public Methods
|
|
|
|
#### `connect(wait_timeout: Optional[float] = None) -> bool`
|
|
Establishes the primary WebSocket connection to Port 4000 and verifies the handshake via `ping_verify`.
|
|
- **Behavior**:
|
|
- If WebSocket succeeds: Drains any offline spooled events and transitions the Heartbeat Monitor to `CONNECTED_IDLE` (zero-CPU IDLE mode).
|
|
- If WebSocket fails and `enable_http_fallback` is `True`: Checks HTTP API health. If online, enters `FALLBACK_HTTP` and starts background recovery.
|
|
- If both are unreachable and `offline_spool_enabled` is `True`: Enters `OFFLINE_SPOOL`.
|
|
- **Returns**: `True` if connected to WebSocket or HTTP fallback is operational; `False` if offline.
|
|
- **Raises**: `DataStackAuthError` if credentials are rejected (`401`/`403`).
|
|
|
|
---
|
|
|
|
#### `disconnect() -> None`
|
|
Stops the background Heartbeat Monitor thread and gracefully closes any active Socket.IO connection.
|
|
|
|
---
|
|
|
|
#### `send(data: Dict[str, Any], event_id: Optional[str] = None, timestamp: Optional[str] = None, timeout: Optional[float] = None) -> Dict[str, Any]`
|
|
Sends a single telemetry event.
|
|
|
|
- **Parameters**:
|
|
- `data` (`Dict[str, Any]`): Arbitrary JSON-serializable dictionary containing the event payload.
|
|
- `event_id` (`str`, optional): Unique event identifier for idempotency. Auto-generated (`evt_<timestamp>_<uuid>`) if omitted.
|
|
- `timestamp` (`str`, optional): ISO 8601 UTC timestamp. Auto-generated (`_utc_now_iso()`) if omitted.
|
|
- `timeout` (`float`, optional): Timeout in seconds for waiting for server acknowledgment.
|
|
- **Returns**: Dictionary with delivery details:
|
|
```python
|
|
{
|
|
"status": "ok",
|
|
"eventId": "evt_1700000000000_a1b2c3d4",
|
|
"transport": "WEBSOCKET" # or "HTTP_FALLBACK" or "OFFLINE_SPOOL"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
#### `sync(...) -> Dict[str, Any]`
|
|
Backward-compatible alias for `send()`.
|
|
```python
|
|
client.sync(data={"status": "active"}, timeout=5.0)
|
|
```
|
|
|
|
---
|
|
|
|
#### `send_batch(events: Optional[List[Dict[str, Any]]] = None, timeout: Optional[float] = None, file_name: Optional[str] = None, *, records: Optional[List[Dict[str, Any]]] = None) -> Dict[str, Any]`
|
|
Transmits a list of event dictionaries in a single network round-trip.
|
|
|
|
- **Parameters**:
|
|
- `events` (`List[Dict[str, Any]]`): List of event dictionaries (accepts either raw data dicts or wrapped envelopes `{eventId, timestamp, data}`).
|
|
- `records`: Keyword-only alias for `events`.
|
|
- `timeout`: Network ACK timeout.
|
|
- `file_name` (optional): Source filename if sending structured file records.
|
|
- **Built-in Auto-Recovery**:
|
|
- **Auto-Splitting on `413 PAYLOAD_TOO_LARGE`**: Recursively halves the batch (`chunk // 2`) and sends each half.
|
|
- **Lossless Partial Rate-Limit Retry**: If the server returns `rateLimitedEventIds`, the SDK automatically waits 1.05s for the 1-second sliding window to reset and retries only the throttled items.
|
|
- **Returns**:
|
|
```python
|
|
{
|
|
"success": True,
|
|
"processed": 500,
|
|
"rejected": 0,
|
|
"duplicates": 0,
|
|
"spooledCount": 0,
|
|
"transport": "WEBSOCKET"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
#### `batch_sync(...) -> Dict[str, Any]`
|
|
Backward-compatible alias for `send_batch()`. Accepts either `events` or `records`.
|
|
|
|
---
|
|
|
|
#### `send_file(file_path: Union[str, Path], chunk_size: Optional[int] = None, extra_metadata: Optional[Dict[str, Any]] = None) -> Dict[str, Any]`
|
|
Streams a local file directly into DataStack in deterministic chunks.
|
|
|
|
- **Supported File Types**: `.csv`, `.json`, `.jsonl`, `.ndjson`.
|
|
- **Parameters**:
|
|
- `file_path`: Path to local data file.
|
|
- `chunk_size`: Rows per batch chunk (defaults to `BATCH_CHUNK_SIZE`, default: 250).
|
|
- `extra_metadata`: Optional key-value dictionary merged into every record.
|
|
- **Features**:
|
|
- Deterministic SHA-256 event ID generation (`file_{stem}_{row}_{hash}`) preventing duplicates on re-runs.
|
|
- UTF-8 with Byte Order Mark (`utf-8-sig`) support.
|
|
- **Returns**:
|
|
```python
|
|
{
|
|
"success": True,
|
|
"fileName": "transactions.csv",
|
|
"fileType": "csv",
|
|
"chunksSent": 4,
|
|
"processed": 1000,
|
|
"rejected": 0,
|
|
"duplicates": 0,
|
|
"spooledCount": 0,
|
|
"transport": "WEBSOCKET"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
#### `status() -> Dict[str, Any]`
|
|
Returns a diagnostic snapshot of the client instance:
|
|
|
|
```python
|
|
{
|
|
"connectedWebSocket": True,
|
|
"clientId": "pos_terminal_01",
|
|
"serverUrl": "http://localhost:4000",
|
|
"apiUrl": "http://localhost:3000",
|
|
"configSource": "/path/to/config.datastack.py",
|
|
"connectionState": "CONNECTED_IDLE",
|
|
"heartbeatMode": "IDLE", # "IDLE" or "ACTIVE_RECOVERY"
|
|
"recoveryAttempts": 0,
|
|
"lastError": None,
|
|
"offlineSpoolPending": 0,
|
|
"offlineSpoolBytes": 0
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Configuration Class: `DataStackConfig`
|
|
|
|
Dataclass storing resolved SDK settings:
|
|
|
|
```python
|
|
from datastack import DataStackConfig
|
|
|
|
@dataclass
|
|
class DataStackConfig:
|
|
server_url: str = "http://localhost:4000"
|
|
api_url: str = "http://localhost:3000"
|
|
client_token: str = ""
|
|
client_id: str = ""
|
|
auto_connect: bool = True
|
|
enable_http_fallback: bool = True
|
|
heartbeat_interval_sec: float = 5.0
|
|
request_timeout_sec: float = 8.0
|
|
max_retries: int = 3
|
|
retry_backoff_base_sec: float = 0.5
|
|
batch_chunk_size: int = 250
|
|
offline_spool_enabled: bool = True
|
|
offline_spool_dir: str = ".datastack_spool"
|
|
max_spool_size_mb: int = 50
|
|
config_source_path: Optional[str] = None
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Configuration Loader: `load_config`
|
|
|
|
```python
|
|
from datastack import load_config
|
|
|
|
config: DataStackConfig = load_config(
|
|
config_path=None,
|
|
require_token=False,
|
|
**overrides
|
|
)
|
|
```
|
|
Discovers `config.datastack.py`, layers environment variables (`DATASTACK_*`), and merges keyword overrides.
|