Files
DataStack/sdk/python/datastack/docs/LIFECYCLE.md
T
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

91 lines
5.4 KiB
Markdown

# Lifecycle, Resilience & Offline Spooling
This document explains the internal connection state machine, the zero-CPU Heartbeat Monitor, and the thread-safe offline disk spooler powering the DataStack Python SDK.
---
## 1. Connection State Machine (`ConnectionState`)
The client transitions automatically through five lifecycle states:
```text
┌────────────────┐ WebSocket Drop ┌────────────────┐
│ CONNECTED_IDLE ├────────────────────────────────►│ FALLBACK_HTTP │
│ (Heartbeat │◄────────────────────────────────┤ (Heartbeat │
│ IDLE Mode) │ Reconnected & Verified │ ACTIVE_REC) │
└───────┬────────┘ (via ping_verify) └───────┬────────┘
│ │ Both WS & HTTP
│ │ Unreachable
│ ▼
│ ┌────────────────┐
│ │ OFFLINE_SPOOL │
│ │ (Local Disk │
│ │ spool.jsonl) │
│ └───────┬────────┘
│ │
│ client.disconnect() or │ Network Returns
│ Fatal Auth Revocation │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ STOPPED │◄────────────────────────────────┤ RECONNECTING_ │
│ │ │ VERIFY │
└────────────────┘ └────────────────┘
```
### State Details
| State | Transport Used | Heartbeat Mode | Description |
| :--- | :--- | :--- | :--- |
| **`CONNECTED_IDLE`** | WebSocket (Port 4000) | `IDLE` (0% CPU) | Primary WebSocket connection is active and verified. The Heartbeat thread sleeps, consuming zero network or CPU resources. |
| **`FALLBACK_HTTP`** | HTTP REST (Port 3000) | `ACTIVE_RECOVERY` | WebSocket connection dropped. Outgoing events automatically route via HTTP `POST /api/events`. The Heartbeat thread wakes up and probes WebSocket reconnection in the background. |
| **`OFFLINE_SPOOL`** | Local Disk Queue | `ACTIVE_RECOVERY` | The client device has lost internet connectivity completely. Telemetry events are appended to `.datastack_spool/spool.jsonl`. |
| **`RECONNECTING_VERIFY`** | WebSocket | `ACTIVE_RECOVERY` | Reconnection handshake in progress. The client issues a `ping_verify` emission and replays spooled disk batches. |
| **`STOPPED`** | None | `STOPPED` | Client was closed via `disconnect()` or credentials were permanently revoked (`401`/`403`). All background threads terminate. |
---
## 2. Zero-CPU Heartbeat Monitor (`HeartbeatMonitor`)
Traditional SDKs constantly emit polling pings even when connections are healthy, wasting bandwidth and battery. DataStack implements an **Event-Driven Sleep Architecture**:
1. **When Healthy**: Once `connect()` succeeds and passes `ping_verify`, the monitor clears its internal wake event (`_wake_event.clear()`). The background thread remains completely idle.
2. **On Disconnect**: The Socket.IO `disconnect` event or an HTTP delivery trigger wakes the thread (`_wake_event.set()`).
3. **Background Reconnection**: Every `HEARTBEAT_INTERVAL_SEC` (default: 5.0s), the monitor attempts to reconnect and verify the WebSocket.
4. **Return to Idle**: Once verified, the thread drains any offline spooled events and immediately returns to sleep.
---
## 3. Offline Disk Spooling (`DiskSpooler`)
When edge devices, field laptops, or factory floor terminals lose internet connectivity:
### Storage Mechanics
- Events are appended to `.datastack_spool/spool.jsonl` using UTF-8 JSON lines.
- Each event preserves its deterministic `eventId`, `timestamp`, and `clientId`.
- Disk capacity is protected by `MAX_SPOOL_SIZE_MB` (default: 50 MB).
### Atomic Drain & Crash Recovery
- When connectivity returns, the Heartbeat Monitor reads spooled records in batches of `BATCH_CHUNK_SIZE` (default: 250).
- Batches are emitted to the server.
- The spool file is trimmed atomically using `ack_and_requeue_front()` and `os.replace()`, guaranteeing that if the process crashes mid-stream, unacknowledged records remain at the head of the file without duplication or data loss.
---
## 4. Resource Disposal & Cleanup
Always disconnect the client when your application shuts down to avoid dangling threads:
```python
client = DataStack()
try:
client.send({"metric": "active"})
finally:
client.disconnect()
```
Or use the context manager:
```python
with DataStack() as client:
client.send({"metric": "active"})
```