Files
DataStack/docs/users/features/data-export.md
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

75 lines
2.9 KiB
Markdown

# Streaming Data Export (`/vcon/export`)
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
---
## 1. Streaming Architecture
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
---
## 2. Export Configuration & Options
The export workspace provides targeted filtering:
| Option | Values | Description |
| :--- | :--- | :--- |
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
---
## 3. Supported Formats & Tool Compatibility
### CSV Format (`text/csv`)
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
#### Importing CSV into Python (pandas)
```python
import pandas as pd
import json
# Load exported CSV
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
# Parse JSON payloads into a structured DataFrame
df["parsed_payload"] = df["payload"].apply(json.loads)
print(df.head())
```
#### Importing CSV into DuckDB
```sql
-- Query directly from the exported CSV without importing
SELECT
clientId,
timestamp,
json_extract_string(payload, '$.temperature') AS temperature
FROM read_csv_auto('datastack_export.csv')
WHERE temperature > 25.0;
```
### JSON Format (`application/json`)
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
---
## 4. Programmatic Export via API
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
```bash
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
--output export.csv
```