- Fetch events from external ingestion service in API routes - Decouple idempotency check from registration in Security utility - Refine CSV storage logic to verify file content for headers - Replace Python SDK with comprehensive, feature-rich implementation
DataStack Official Python SDK (datastack)
Production-Grade Python Client for DataStack — featuring automatic
config.datastack.pyconfiguration discovery, real-time Socket.IO streaming (Port 4000), automatic Heartbeat HTTP fallback (Port 3000), offline disk spooling, and direct.csv,.json, and.jsonlfile ingestion.
1. Installation
Install from Local Repository
pip install ./sdk/python/datastack
Editable Development Mode
pip install -e ./sdk/python/datastack
PyPI Package Installation
pip install datastack
2. Zero-Boilerplate Setup with config.datastack.py
Create a file named config.datastack.py in your Python project's root directory (you can also copy a pre-filled version directly from the DataStack Admin Console at /vcon/sdk):
# config.datastack.py
DATASTACK_CONFIG = {
"SERVER_URL": "http://localhost:4000", # Port 4000 Socket.IO Ingestion Service
"API_URL": "http://localhost:3000", # Port 3000 HTTP Fallback & File Ingestion API
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Provisioned in /vcon/clients
"AUTO_CONNECT": True,
"ENABLE_HTTP_FALLBACK": True,
"HEARTBEAT_INTERVAL_SEC": 5.0,
"MAX_RETRIES": 3,
"RETRY_BACKOFF_BASE_SEC": 0.5,
"BATCH_CHUNK_SIZE": 250,
"OFFLINE_SPOOL_ENABLED": True,
"OFFLINE_SPOOL_DIR": ".datastack_spool",
"MAX_SPOOL_SIZE_MB": 50,
}
3. Quickstart Usage
from datastack import DataStack
# Automatically discovers and loads config.datastack.py and connects!
client = DataStack()
# 1. Push a Single Event
res = client.send({
"service": "checkout-worker",
"order_id": "ORD-9921",
"amount_usd": 149.99,
"status": "completed",
})
print("Single Event:", res)
# 2. Push a Batch of Events
batch_res = client.send_batch([
{"data": {"metric": "cpu_load", "value": 0.42}},
{"data": {"metric": "mem_used_mb", "value": 2048}},
])
print("Batch Result:", batch_res)
# 3. Directly Stream .csv, .json, or .jsonl Files
file_res = client.send_file("monthly_records.csv", chunk_size=250)
print("File Ingestion:", file_res)
# 4. Inspect Connection & Heartbeat Status
print("SDK Status:", client.status())
client.disconnect()
4. Smart Connection State Machine & Auto-Recovery
The DataStack Python SDK maintains a 4-state resilience lifecycle managed by HeartbeatMonitor:
CONNECTED_IDLE(Primary WebSocket Active — Heartbeat IDLE):- Events stream over Socket.IO (
Port 4000) with sub-millisecond overhead. - The Heartbeat Monitor sleeps in IDLE mode (
heartbeatMode: "IDLE"), consuming zero network or CPU resources.
- Events stream over Socket.IO (
FALLBACK_HTTP(WebSocket Disconnected — Auto HTTP Fallback Active):- If the WebSocket connection drops or closes,
send(),send_batch(), andsend_file()automatically switch toPOST {API_URL}/api/eventsso zero application calls fail. - Simultaneously, the Heartbeat Monitor wakes up (
heartbeatMode: "ACTIVE_RECOVERY"), probes and reconnects the WebSocket in the background, verifies the handshake (ping_verify), and transitions back toCONNECTED_IDLE(IDLEmode).
- If the WebSocket connection drops or closes,
OFFLINE_SPOOL(Device Offline — Local Disk Spooling):- If the client machine loses internet connectivity completely, events are safely appended to
.datastack_spool/spool.jsonlwith deterministiceventIds. - When the device comes back online, the Heartbeat Monitor detects connectivity, reconnects the WebSocket, replays all spooled events without duplicates, and returns to
CONNECTED_IDLE(IDLEmode).
- If the client machine loses internet connectivity completely, events are safely appended to
- Structured Error Handling & Auto-Chunking:
413 PAYLOAD_TOO_LARGE: Batch/file payloads automatically split in half (chunk // 2) recursively and retry.429 RATE_LIMIT_EXCEEDED: Automatically retries with exponential backoff and random jitter.401 / 403(CLIENT_REVOKED,TOKEN_ROTATED): Immediately halts retries and raisesDataStackAuthError.
5. Author & License
- Organization: Vib Tools
- Author: Md Anik (
ANIK.D.ART) —info@vibtools.top - License: MIT License