Files
DataStack/sdk/python/datastack/docs/API_REFERENCE.md
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

7.4 KiB

API Reference Specification

This document provides the exhaustive specification for all public classes, methods, parameters, and return structures in the DataStack Python SDK.


1. Primary Client: DataStack / DataStackClient

Import path:

from datastack import DataStack, DataStackClient
# Both identifiers point to the same underlying class.

Constructor: __init__(...)

DataStack(
    server_url: Optional[str] = None,
    token: Optional[str] = None,
    api_url: Optional[str] = None,
    config_path: Optional[str] = None,
    auto_connect: Optional[bool] = None,
    reconnection_attempts: Optional[int] = None,
    reconnection_delay: Optional[float] = None,
    client_id: Optional[str] = None,
    **kwargs: Any,
)
Parameter Type Default Description
server_url str None (loads from config or http://localhost:4000) URL for Port 4000 Socket.IO ingestion service.
token str None (loads from config or env) Signed HMAC-SHA256 client JWT token.
api_url str None (inferred from server_url or http://localhost:3000) URL for Port 3000 REST API and automatic HTTP fallback.
config_path str None Path to a specific Python configuration file.
auto_connect bool None (defaults to True if config loaded) Whether to connect immediately during initialization.
reconnection_attempts int None (alias for MAX_RETRIES, default: 3) Maximum number of retry attempts for transient errors.
reconnection_delay float None (alias for HEARTBEAT_INTERVAL_SEC) Delay in seconds between reconnection probes.
client_id str None Explicit client identifier slug (must match JWT sub/clientId).
**kwargs Any — Any configuration overrides passed to load_config().

Core Public Methods

connect(wait_timeout: Optional[float] = None) -> bool

Establishes the primary WebSocket connection to Port 4000 and verifies the handshake via ping_verify.

  • Behavior:
    • If WebSocket succeeds: Drains any offline spooled events and transitions the Heartbeat Monitor to CONNECTED_IDLE (zero-CPU IDLE mode).
    • If WebSocket fails and enable_http_fallback is True: Checks HTTP API health. If online, enters FALLBACK_HTTP and starts background recovery.
    • If both are unreachable and offline_spool_enabled is True: Enters OFFLINE_SPOOL.
  • Returns: True if connected to WebSocket or HTTP fallback is operational; False if offline.
  • Raises: DataStackAuthError if credentials are rejected (401/403).

disconnect() -> None

Stops the background Heartbeat Monitor thread and gracefully closes any active Socket.IO connection.


send(data: Dict[str, Any], event_id: Optional[str] = None, timestamp: Optional[str] = None, timeout: Optional[float] = None) -> Dict[str, Any]

Sends a single telemetry event.

  • Parameters:
    • data (Dict[str, Any]): Arbitrary JSON-serializable dictionary containing the event payload.
    • event_id (str, optional): Unique event identifier for idempotency. Auto-generated (evt_<timestamp>_<uuid>) if omitted.
    • timestamp (str, optional): ISO 8601 UTC timestamp. Auto-generated (_utc_now_iso()) if omitted.
    • timeout (float, optional): Timeout in seconds for waiting for server acknowledgment.
  • Returns: Dictionary with delivery details:
    {
        "status": "ok",
        "eventId": "evt_1700000000000_a1b2c3d4",
        "transport": "WEBSOCKET"  # or "HTTP_FALLBACK" or "OFFLINE_SPOOL"
    }
    

sync(...) -> Dict[str, Any]

Backward-compatible alias for send().

client.sync(data={"status": "active"}, timeout=5.0)

send_batch(events: Optional[List[Dict[str, Any]]] = None, timeout: Optional[float] = None, file_name: Optional[str] = None, *, records: Optional[List[Dict[str, Any]]] = None) -> Dict[str, Any]

Transmits a list of event dictionaries in a single network round-trip.

  • Parameters:
    • events (List[Dict[str, Any]]): List of event dictionaries (accepts either raw data dicts or wrapped envelopes {eventId, timestamp, data}).
    • records: Keyword-only alias for events.
    • timeout: Network ACK timeout.
    • file_name (optional): Source filename if sending structured file records.
  • Built-in Auto-Recovery:
    • Auto-Splitting on 413 PAYLOAD_TOO_LARGE: Recursively halves the batch (chunk // 2) and sends each half.
    • Lossless Partial Rate-Limit Retry: If the server returns rateLimitedEventIds, the SDK automatically waits 1.05s for the 1-second sliding window to reset and retries only the throttled items.
  • Returns:
    {
        "success": True,
        "processed": 500,
        "rejected": 0,
        "duplicates": 0,
        "spooledCount": 0,
        "transport": "WEBSOCKET"
    }
    

batch_sync(...) -> Dict[str, Any]

Backward-compatible alias for send_batch(). Accepts either events or records.


send_file(file_path: Union[str, Path], chunk_size: Optional[int] = None, extra_metadata: Optional[Dict[str, Any]] = None) -> Dict[str, Any]

Streams a local file directly into DataStack in deterministic chunks.

  • Supported File Types: .csv, .json, .jsonl, .ndjson.
  • Parameters:
    • file_path: Path to local data file.
    • chunk_size: Rows per batch chunk (defaults to BATCH_CHUNK_SIZE, default: 250).
    • extra_metadata: Optional key-value dictionary merged into every record.
  • Features:
    • Deterministic SHA-256 event ID generation (file_{stem}_{row}_{hash}) preventing duplicates on re-runs.
    • UTF-8 with Byte Order Mark (utf-8-sig) support.
  • Returns:
    {
        "success": True,
        "fileName": "transactions.csv",
        "fileType": "csv",
        "chunksSent": 4,
        "processed": 1000,
        "rejected": 0,
        "duplicates": 0,
        "spooledCount": 0,
        "transport": "WEBSOCKET"
    }
    

status() -> Dict[str, Any]

Returns a diagnostic snapshot of the client instance:

{
    "connectedWebSocket": True,
    "clientId": "pos_terminal_01",
    "serverUrl": "http://localhost:4000",
    "apiUrl": "http://localhost:3000",
    "configSource": "/path/to/config.datastack.py",
    "connectionState": "CONNECTED_IDLE",
    "heartbeatMode": "IDLE",  # "IDLE" or "ACTIVE_RECOVERY"
    "recoveryAttempts": 0,
    "lastError": None,
    "offlineSpoolPending": 0,
    "offlineSpoolBytes": 0
}

2. Configuration Class: DataStackConfig

Dataclass storing resolved SDK settings:

from datastack import DataStackConfig

@dataclass
class DataStackConfig:
    server_url: str = "http://localhost:4000"
    api_url: str = "http://localhost:3000"
    client_token: str = ""
    client_id: str = ""
    auto_connect: bool = True
    enable_http_fallback: bool = True
    heartbeat_interval_sec: float = 5.0
    request_timeout_sec: float = 8.0
    max_retries: int = 3
    retry_backoff_base_sec: float = 0.5
    batch_chunk_size: int = 250
    offline_spool_enabled: bool = True
    offline_spool_dir: str = ".datastack_spool"
    max_spool_size_mb: int = 50
    config_source_path: Optional[str] = None

3. Configuration Loader: load_config

from datastack import load_config

config: DataStackConfig = load_config(
    config_path=None,
    require_token=False,
    **overrides
)

Discovers config.datastack.py, layers environment variables (DATASTACK_*), and merges keyword overrides.