feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
@@ -0,0 +1,74 @@
|
||||
# Streaming Data Export (`/vcon/export`)
|
||||
|
||||
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Streaming Architecture
|
||||
|
||||
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
|
||||
|
||||
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
|
||||
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
|
||||
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
|
||||
|
||||
---
|
||||
|
||||
## 2. Export Configuration & Options
|
||||
|
||||
The export workspace provides targeted filtering:
|
||||
|
||||
| Option | Values | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
|
||||
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
|
||||
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
|
||||
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported Formats & Tool Compatibility
|
||||
|
||||
### CSV Format (`text/csv`)
|
||||
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
|
||||
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
|
||||
|
||||
#### Importing CSV into Python (pandas)
|
||||
```python
|
||||
import pandas as pd
|
||||
import json
|
||||
|
||||
# Load exported CSV
|
||||
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
|
||||
|
||||
# Parse JSON payloads into a structured DataFrame
|
||||
df["parsed_payload"] = df["payload"].apply(json.loads)
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
#### Importing CSV into DuckDB
|
||||
```sql
|
||||
-- Query directly from the exported CSV without importing
|
||||
SELECT
|
||||
clientId,
|
||||
timestamp,
|
||||
json_extract_string(payload, '$.temperature') AS temperature
|
||||
FROM read_csv_auto('datastack_export.csv')
|
||||
WHERE temperature > 25.0;
|
||||
```
|
||||
|
||||
### JSON Format (`application/json`)
|
||||
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
|
||||
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
|
||||
|
||||
---
|
||||
|
||||
## 4. Programmatic Export via API
|
||||
|
||||
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
|
||||
|
||||
```bash
|
||||
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
|
||||
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
|
||||
--output export.csv
|
||||
```
|
||||
Reference in new issue
Block a user