Files
DataStack/docs/users/features/data-export.md
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

2.9 KiB

Streaming Data Export (/vcon/export)

The DataStack Export Workspace (/vcon/export) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.


1. Streaming Architecture

Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses Streaming HTTP Responses:

  • Zero-RAM Buffering: Partition files are opened as read streams (fs.createReadStream) and piped directly through Node.js transform streams into the response socket.
  • Constant Memory Footprint: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
  • Client Backpressure: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.

2. Export Configuration & Options

The export workspace provides targeted filtering:

Option Values Description
Client Filter All Clients or specific clientId Filter by an individual registered application (e.g., pos_terminal_01) or export across the entire tenant.
Start Date Date Picker (YYYY-MM-DD) Earliest UTC date to include in the exported records.
End Date Date Picker (YYYY-MM-DD) Latest UTC date to include in the exported records.
Format CSV or JSON Output format for the downloaded file.

3. Supported Formats & Tool Compatibility

CSV Format (text/csv)

  • UTF-8 Byte Order Mark (BOM): DataStack prepends the \uFEFF marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
  • RFC 4180 Escaping: Embedded JSON payloads are escaped according to standard CSV rules.

Importing CSV into Python (pandas)

import pandas as pd
import json

# Load exported CSV
df = pd.read_csv("datastack_export_pos_terminal_01.csv")

# Parse JSON payloads into a structured DataFrame
df["parsed_payload"] = df["payload"].apply(json.loads)
print(df.head())

Importing CSV into DuckDB

-- Query directly from the exported CSV without importing
SELECT 
    clientId, 
    timestamp, 
    json_extract_string(payload, '$.temperature') AS temperature
FROM read_csv_auto('datastack_export.csv')
WHERE temperature > 25.0;

JSON Format (application/json)

  • Streams a valid JSON array ([ ... ]) of parsed event objects.
  • Each event includes clientId, eventId, timestamp, and the native parsed JSON payload object.

4. Programmatic Export via API

You can automate data pipeline extraction via curl or cron scripts using your administrator session cookie:

curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
     -H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
     --output export.csv