- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2.9 KiB
2.9 KiB
Streaming Data Export (/vcon/export)
The DataStack Export Workspace (/vcon/export) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
1. Streaming Architecture
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses Streaming HTTP Responses:
- Zero-RAM Buffering: Partition files are opened as read streams (
fs.createReadStream) and piped directly through Node.js transform streams into the response socket. - Constant Memory Footprint: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
- Client Backpressure: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
2. Export Configuration & Options
The export workspace provides targeted filtering:
| Option | Values | Description |
|---|---|---|
| Client Filter | All Clients or specific clientId |
Filter by an individual registered application (e.g., pos_terminal_01) or export across the entire tenant. |
| Start Date | Date Picker (YYYY-MM-DD) |
Earliest UTC date to include in the exported records. |
| End Date | Date Picker (YYYY-MM-DD) |
Latest UTC date to include in the exported records. |
| Format | CSV or JSON |
Output format for the downloaded file. |
3. Supported Formats & Tool Compatibility
CSV Format (text/csv)
- UTF-8 Byte Order Mark (BOM): DataStack prepends the
\uFEFFmarker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking. - RFC 4180 Escaping: Embedded JSON payloads are escaped according to standard CSV rules.
Importing CSV into Python (pandas)
import pandas as pd
import json
# Load exported CSV
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
# Parse JSON payloads into a structured DataFrame
df["parsed_payload"] = df["payload"].apply(json.loads)
print(df.head())
Importing CSV into DuckDB
-- Query directly from the exported CSV without importing
SELECT
clientId,
timestamp,
json_extract_string(payload, '$.temperature') AS temperature
FROM read_csv_auto('datastack_export.csv')
WHERE temperature > 25.0;
JSON Format (application/json)
- Streams a valid JSON array (
[ ... ]) of parsed event objects. - Each event includes
clientId,eventId,timestamp, and the native parsed JSONpayloadobject.
4. Programmatic Export via API
You can automate data pipeline extraction via curl or cron scripts using your administrator session cookie:
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
--output export.csv