chore: establish cross-IDE synchronization protocol
Implement `.ai-rules`, formalize `MEMORY.md` as the source of truth, and initialize `docs/` structure to ensure consistent architectural context across development environments.
This commit is contained in:
1 parent
05407fd338
commit
4279da45bb
14 files changed
+1162
No files matched your search
@@ -0,0 +1,52 @@
|
||||
# DataStack User & Operator Guide
|
||||
|
||||
Welcome to the **DataStack User Documentation**. This guide is designed for operators, administrators, product teams, and engineers who wish to deploy, operate, and collect real-time telemetry using DataStack.
|
||||
|
||||
---
|
||||
|
||||
## What is DataStack?
|
||||
|
||||
**DataStack** is a self-hosted, lightweight telemetry and data collection platform. It enables your applications to stream events in real time, securely buffers them with Redis and in-memory batching, and persists them into clean, partitioned daily CSV files without overloading a relational database.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Navigation
|
||||
|
||||
Explore the following sections to get up and running:
|
||||
|
||||
### 1. [Installation & Deployment](./installation.md)
|
||||
Step-by-step instructions for deploying DataStack in production:
|
||||
- Running with **Docker Compose**
|
||||
- 1-Click deployment on **Coolify**
|
||||
- Configuring environment variables & persistent volumes
|
||||
- Reverse proxy and SSL configuration
|
||||
|
||||
### 2. [Feature Walkthrough](./features.md)
|
||||
Comprehensive overview of all platform capabilities:
|
||||
- **Admin Console (`/vcon/`)**: Dashboard overview, live activity preview, and storage monitors.
|
||||
- **Client SDK Management**: Registering client applications, issuing JWT tokens, and instant revocation.
|
||||
- **Data Simulator**: Interactive in-browser telemetry generation for testing without writing code.
|
||||
- **Partitioned Storage**: Size-based file rotation and automatic daily retention pruning.
|
||||
- **Streaming Export**: High-performance streaming of filtered CSV or JSON datasets.
|
||||
|
||||
### 3. [Usage Tutorials](./tutorials.md)
|
||||
Hands-on walkthroughs for common tasks:
|
||||
- Sending your first telemetry event via HTTP REST
|
||||
- Streaming real-time events using the official Python SDK
|
||||
- Streaming local `.csv`, `.json`, and `.jsonl` data files
|
||||
- Exporting datasets for analysis in Excel, pandas, or DuckDB
|
||||
|
||||
### 4. [Frequently Asked Questions & Troubleshooting](./faq.md)
|
||||
Answers to common operational questions:
|
||||
- How does DataStack prevent data loss during container restarts?
|
||||
- How to rotate client credentials safely without downtime
|
||||
- What to do if WebSocket handshakes fail or time out
|
||||
- How to adjust event rate limits and payload size thresholds
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Links
|
||||
- [Developer & Contributor Guide](../dev/README.md)
|
||||
- [Architecture Blueprint](../dev/architecture.md)
|
||||
- [Documentation Index](../MANIFEST.md)
|
||||
- [Official Website](https://vib.tools/)
|
||||
@@ -0,0 +1,83 @@
|
||||
# Frequently Asked Questions (FAQ) & Troubleshooting
|
||||
|
||||
This document answers common operational, configuration, and troubleshooting questions.
|
||||
|
||||
---
|
||||
|
||||
## General Architecture & Data Storage
|
||||
|
||||
### Why does DataStack store events in CSV files instead of PostgreSQL?
|
||||
Relational databases degrade significantly when handling thousands of concurrent writes per second due to WAL bottlenecks, indexing contention, and connection limits. DataStack separates concerns:
|
||||
- **PostgreSQL** is used exclusively for administrative accounts, client credentials, and platform configuration.
|
||||
- **Partitioned CSV Files** (`/storage/data/{clientId}/{YYYY-MM-DD}.csv`) receive high-throughput batched writes sequentially on disk. This enables lightning-fast ingestion, predictable disk space usage, zero SQL table locking, and trivial backup and compression.
|
||||
|
||||
### What happens if the server crashes while events are buffered?
|
||||
DataStack implements multi-layer crash protection:
|
||||
1. **Redis AOF Durability**: In production with Redis enabled (`--appendonly yes`), incoming events are pushed immediately to a durable Redis queue (`datastack:buffer:events`).
|
||||
2. **Post-Write Trimming**: The Redis list is trimmed (`LTRIM`) **strictly after** the batch append to the CSV file succeeds on disk. If a crash occurs mid-write, Redis retains the items and flushes them upon reboot.
|
||||
3. **Graceful Signal Handling**: On `SIGTERM` or `SIGINT` (such as during container updates), DataStack halts new input, awaits the in-flight disk flush, and releases file locks before shutting down.
|
||||
|
||||
### How does file rotation and retention pruning work?
|
||||
- **Rotation (`fileRotationMb`)**: By default, when a daily CSV file reaches `100 MB`, DataStack safely rotates to `{YYYY-MM-DD}-part2.csv`.
|
||||
- **Retention Pruning (`autoPruneDays`)**: By default, partitions older than 30 days are automatically pruned. To retain all historical records indefinitely, set **Auto Prune Days** to `0` ("Keep all") in `/vcon/settings`.
|
||||
|
||||
---
|
||||
|
||||
## Networking & Ports
|
||||
|
||||
### What ports does DataStack use?
|
||||
In a standard Docker Compose deployment, the services communicate over an internal network:
|
||||
- **Port `80`**: Exposed by the Nginx reverse proxy (`proxy` container) to the public or host.
|
||||
- **Port `3000`**: Internal Next.js web application, Admin Console, and REST API (`web` container).
|
||||
- **Port `4000`**: Internal Express + Socket.IO real-time ingestion service (`ingestion` container).
|
||||
- **Port `5432`**: Internal PostgreSQL database (`postgres` container).
|
||||
- **Port `6379`**: Internal Redis buffer (`redis` container).
|
||||
|
||||
You only need to expose Port `80` (or `443` with SSL) through your firewall.
|
||||
|
||||
---
|
||||
|
||||
## Authentication & Client Security
|
||||
|
||||
### How does token rotation work? Does it disconnect active connections?
|
||||
When you click **Rotate Token** in `/vcon/clients`:
|
||||
1. A new HMAC-SHA256 token and secret are generated and stored in PostgreSQL.
|
||||
2. The change is instantly communicated to the Port 4000 ingestion server over the internal authenticated bridge.
|
||||
3. Any active WebSocket connections using the old token are immediately disconnected with `TOKEN_ROTATED`.
|
||||
4. Subsequent requests using the old token are rejected with `401 Unauthorized`.
|
||||
|
||||
### What does `CLIENT_REVOKED` mean?
|
||||
If a client registration is marked as `Revoked` or permanently deleted by an administrator, incoming requests are rejected with HTTP status `403 Forbidden` (`CLIENT_REVOKED`). If a WebSocket connection was open, it is kicked immediately.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Issue: "WebSocket ACK timeout" or Handshake Fails
|
||||
1. **Check Nginx Proxy Headers**: If using an external reverse proxy (like Cloudflare, Traefik, or Caddy) in front of DataStack, ensure `Upgrade` and `Connection: Upgrade` headers are passed through for WebSocket support:
|
||||
```nginx
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
```
|
||||
2. **Verify Port 4000**: In `/vcon/settings`, click **Connections -> Probe**. If the status reports offline, inspect the logs of the `ingestion` container:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml logs ingestion
|
||||
```
|
||||
|
||||
### Issue: "RATE_LIMIT_EXCEEDED" Errors
|
||||
Each client has a configurable rate limit (default: 100 events/sec) and a global platform limit. If your client bursts above this rate, the server returns status `429`.
|
||||
- **Solution**: In `/vcon/clients`, increase the client's rate limit, or in `/vcon/settings`, adjust the global maximum events per second.
|
||||
- **Python SDK**: The official Python SDK automatically handles rate limits by backing off and retrying remaining events without dropping data.
|
||||
|
||||
### Issue: "PAYLOAD_TOO_LARGE" Errors
|
||||
By default, individual events cannot exceed `100 KB` (`maxPayloadSizeKb`).
|
||||
- **Solution**: In `/vcon/settings`, increase the **Max Payload Size (KB)** setting to accommodate larger JSON objects.
|
||||
|
||||
### How do I back up my collected telemetry data?
|
||||
All partition files are located in the `datastack-data` volume (`/app/storage/data`). You can create a compressed backup at any time while the containers are running:
|
||||
```bash
|
||||
docker run --rm \
|
||||
--volumes-from datastack-web-1 \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar czf /backup/datastack_backup_$(date +%F).tar.gz /app/storage/data
|
||||
```
|
||||
@@ -0,0 +1,89 @@
|
||||
# Feature Guide
|
||||
|
||||
This document provides a comprehensive tour of DataStack's features and capabilities available in the administration console (`/vcon/`).
|
||||
|
||||
---
|
||||
|
||||
## 1. The Administration Console (`/vcon/`)
|
||||
|
||||
The DataStack Admin Console provides a clean, zero-clutter workspace for monitoring telemetry, registering client applications, inspecting stored partitions, and configuring platform limits.
|
||||
|
||||
Access the console by signing in at:
|
||||
```text
|
||||
https://your-domain.example/vcon/login
|
||||
```
|
||||
|
||||
### Dashboard Overview (`/vcon/dashboard`)
|
||||
The primary operational cockpit displays:
|
||||
- **System Health**: Dual-runtime connectivity indicators for both Port 3000 (Web API) and Port 4000 (Express/Socket.IO ingestion service).
|
||||
- **Storage Metrics**: Total CSV partition file count, active client directories, and total disk usage.
|
||||
- **Buffer Gauges**: Active unflushed events currently buffered in memory and Redis, alongside flush intervals.
|
||||
- **Live Activity Stream**: Real-time ticker showing recently ingested events with timestamps and payload previews.
|
||||
|
||||
---
|
||||
|
||||
## 2. Client SDK Registry & Security (`/vcon/clients`)
|
||||
|
||||
DataStack enforces strict authentication for all incoming telemetry streams.
|
||||
|
||||
### Registering a New Client
|
||||
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
|
||||
2. Provide:
|
||||
- **Client Identifier**: Unique lowercase alphanumeric slug (e.g., `pos_terminal_01`, `mobile_app_prod`).
|
||||
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
|
||||
- **Rate Limit**: Maximum allowed events per second for this client (default: `100 events/sec`).
|
||||
3. Click **Register Client**. DataStack instantly signs a dedicated HMAC-SHA256 JWT token.
|
||||
4. **Copy the Token**: Store the generated token securely in your client application's environment or configuration file.
|
||||
|
||||
### Credential Rotation & Access Control
|
||||
- **Rotate Token**: Click **Rotate Token** to generate a new secret and token. The old token is immediately invalidated, and active WebSocket connections using the old credential are disconnected.
|
||||
- **Revoke Access**: Set the client status to `Revoked`. Incoming HTTP requests and WebSocket handshakes will immediately be blocked with `CLIENT_REVOKED (403)`.
|
||||
- **Permanent Deletion**: Deleting a client removes it from PostgreSQL and adds it to `deleted_clients` to ensure stale tokens can never be reused. You can optionally purge historical CSV archives.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Simulator (`/vcon/simulator`)
|
||||
|
||||
The built-in Data Simulator allows operators to test telemetry ingestion, rate limits, and buffer flushes without writing code:
|
||||
- **Single Event**: Send ad-hoc JSON payloads with custom event IDs.
|
||||
- **Batch Generator**: Trigger bursts of 10 to 500 events to observe real-time buffer threshold flushing.
|
||||
- **Preset Schemas**: Pre-configured mock payloads for IoT temperature sensors, e-commerce transactions, server metrics, and user journey events.
|
||||
|
||||
---
|
||||
|
||||
## 4. Partitioned CSV Storage & Retention (`/vcon/storage`)
|
||||
|
||||
DataStack eliminates the performance overhead of per-row database inserts by streaming events directly into structured, partitioned CSV files on disk:
|
||||
- **Directory Layout**: `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`
|
||||
- **CSV Schema**: Each file contains 4 clean columns:
|
||||
```csv
|
||||
clientId,eventId,timestamp,payload
|
||||
pos_terminal_01,evt_1700000000_a1b2,2026-11-17T12:00:00Z,"{""status"":""ok""}"
|
||||
```
|
||||
- **Size-Based Rotation**: When a partition reaches the configured threshold (default: `100 MB`), DataStack automatically closes the file and continues writing to `{YYYY-MM-DD}-part2.csv`.
|
||||
- **Automated Retention Pruning**: Configure `autoPruneDays` (e.g., `30` days). DataStack automatically removes expired partition files, or set to `0` ("Keep all") to retain historical archives indefinitely.
|
||||
- **Manual Buffer Flush**: Force an immediate flush of both in-memory and Redis buffers to disk with a single click.
|
||||
|
||||
---
|
||||
|
||||
## 5. Streaming Data Export (`/vcon/export`)
|
||||
|
||||
The Export Workspace provides fast, low-memory streaming extraction of telemetry data for data science and analytics:
|
||||
- **Client Filter**: Export all clients or filter by a specific client ID.
|
||||
- **Date Range**: Filter records within precise UTC start and end dates.
|
||||
- **Formats**:
|
||||
- **CSV**: Clean RFC 4180 streaming CSV output with BOM support for direct import into Microsoft Excel, pandas, or DuckDB.
|
||||
- **JSON**: Formatted JSON array stream.
|
||||
- **Non-Blocking Streaming**: Large multi-gigabyte partitions stream directly from disk to the client browser without buffering the full dataset in server RAM.
|
||||
|
||||
---
|
||||
|
||||
## 6. Platform Runtime Settings (`/vcon/settings`)
|
||||
|
||||
Fine-tune ingestion rules with instant cross-process propagation:
|
||||
- **Max Events Per Second**: Global ingestion rate ceiling.
|
||||
- **Max Payload Size (KB)**: Maximum permitted size per event payload (default: `100 KB`).
|
||||
- **Batch Flush Threshold**: Number of buffered events that triggers an immediate write to disk (default: `500 events`).
|
||||
- **Batch Flush Interval**: Maximum elapsed time before flushing pending events to disk (default: `2000 ms`).
|
||||
- **Idempotency Protection**: Deduplicates events with identical `eventId` values within a 5-minute sliding window.
|
||||
- **Connection Probing**: Test the live internal bridge between the Web console and the Port 4000 ingestion service.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Installation & Deployment Guide
|
||||
|
||||
This guide explains how to install and self-host DataStack on your own server or cloud VPS.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure your system meets the following minimum requirements:
|
||||
- **Operating System**: Linux (Ubuntu 22.04+, Debian 12+, Alpine), macOS, or Windows WSL2.
|
||||
- **Docker Engine**: Docker `v24.0+` and Docker Compose `v2.20+`.
|
||||
- **System Resources**:
|
||||
- Minimum: 1 vCPU, 1 GB RAM, 10 GB disk space.
|
||||
- Recommended: 2 vCPU, 2–4 GB RAM, SSD storage scaled to your expected data volume.
|
||||
- **Networking**: Ports `80` (HTTP) and `443` (HTTPS) accessible via your reverse proxy or cloud firewall.
|
||||
|
||||
---
|
||||
|
||||
## Method 1: Docker Compose (Recommended)
|
||||
|
||||
DataStack includes a complete, production-ready Docker Compose stack located in the [`docker/`](../../docker/) directory.
|
||||
|
||||
### 1. Clone the Repository
|
||||
```bash
|
||||
git clone https://github.com/vibtoolshq/datastack.git
|
||||
cd datastack
|
||||
```
|
||||
|
||||
### 2. Configure Environment Variables
|
||||
Copy the example environment configuration:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Generate secure, random 32+ character secrets for production:
|
||||
```bash
|
||||
# In Linux / macOS, generate 48-char random hex keys:
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Edit `.env` and fill in the required variables:
|
||||
```dotenv
|
||||
# Production Security Secrets (Minimum 32 characters each)
|
||||
DATASTACK_ADMIN_SESSION_SECRET=your-random-32-plus-char-session-secret-here
|
||||
DATASTACK_JWT_SECRET=your-random-32-plus-char-client-jwt-secret-here
|
||||
DATASTACK_ADMIN_SETUP_KEY=your-random-32-plus-char-admin-setup-key-here
|
||||
DATASTACK_INTERNAL_SECRET=your-random-32-plus-char-internal-service-secret-here
|
||||
|
||||
# Administrator Initial Email
|
||||
DATASTACK_ADMIN_EMAIL=admin@yourcompany.com
|
||||
```
|
||||
|
||||
### 3. Launch the Stack
|
||||
Run Docker Compose from the project root:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
The stack launches 5 containers:
|
||||
1. **`proxy`**: Nginx reverse proxy exposed on port `80`, routing `/socket.io/` to port `4000` and all other routes to port `3000`.
|
||||
2. **`web`**: Next.js 15 standalone server running on port `3000`.
|
||||
3. **`ingestion`**: Dedicated Express + Socket.IO ingestion server running on port `4000`.
|
||||
4. **`postgres`**: PostgreSQL 16 database storing administrative credentials, SDK client records, and settings.
|
||||
5. **`redis`**: Redis 7 container running with `--appendonly yes` for crash-resilient queue buffering.
|
||||
|
||||
---
|
||||
|
||||
## Method 2: Coolify Cloud / Self-Hosted PaaS
|
||||
|
||||
DataStack is optimized for deployment via [Coolify](https://coolify.io/):
|
||||
|
||||
1. **Create Resource**: In Coolify, click **+ Add Resource** → **Docker Compose**.
|
||||
2. **Repository Settings**:
|
||||
- Point to your DataStack Git repository.
|
||||
- Set **Base Directory** to `/docker`.
|
||||
- Set **Docker Compose Location** to `/docker/docker-compose.yml`.
|
||||
3. **Domain Assignment**:
|
||||
- Attach your custom domain (e.g. `https://datastack.yourdomain.com`) to the **`proxy`** service on internal port `80`.
|
||||
- Enable automatic SSL/TLS via Coolify's built-in Let's Encrypt integration.
|
||||
4. **Environment Variables**:
|
||||
- Coolify automatically injects generated `SERVICE_*` variables for database and JWT secrets.
|
||||
5. **Deploy**: Click **Deploy** and monitor the build logs until all 5 services are healthy.
|
||||
|
||||
---
|
||||
|
||||
## First-Run Administrator Setup (`/vcon/setup`)
|
||||
|
||||
Once the stack is running:
|
||||
1. Open your browser and navigate to:
|
||||
```text
|
||||
https://your-domain.example/vcon/setup
|
||||
```
|
||||
2. You will be prompted to enter:
|
||||
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` or your chosen admin email.
|
||||
- **Administrator Password**: Choose a strong password (minimum 8 characters).
|
||||
- **Setup Security Key**: Enter the exact value configured in `DATASTACK_ADMIN_SETUP_KEY`.
|
||||
3. Click **Complete Initial Setup**. You will be securely logged in and redirected to the `/vcon/dashboard` console.
|
||||
|
||||
---
|
||||
|
||||
## Persistent Storage & Volumes
|
||||
|
||||
DataStack uses three named volumes to safeguard your data across container updates:
|
||||
|
||||
| Volume Name | Container Path | Purpose |
|
||||
| :--- | :--- | :--- |
|
||||
| **`datastack-data`** | `/app/storage` | Contains `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`. Shared by `web` and `ingestion`. |
|
||||
| **`pgdata`** | `/var/lib/postgresql/data` | Persists administrative users, client credentials, and settings. |
|
||||
| **`redisdata`** | `/data` | Persists Redis Append-Only File (`appendonly.aof`) for unflushed event buffers. |
|
||||
|
||||
> **IMPORTANT**: When upgrading or redeploying containers, never execute `docker compose down -v`, as the `-v` flag deletes named volumes. Use standard `docker compose down` followed by `docker compose up -d`.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Usage Tutorials
|
||||
|
||||
This document provides hands-on tutorials for sending data to DataStack, integrating client SDKs, and exporting datasets.
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 1: Sending Events via HTTP REST (`curl`)
|
||||
|
||||
The fastest way to send data to DataStack without writing application code is via the HTTP REST API.
|
||||
|
||||
### 1. Retrieve Your Client Token
|
||||
In `/vcon/clients`, register a client (e.g., `bash_script_01`) and copy the generated Bearer JWT token.
|
||||
|
||||
### 2. Send a Single Event
|
||||
```bash
|
||||
curl -X POST "https://your-domain.example/api/events" \
|
||||
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"eventId": "order_1001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"data": {
|
||||
"item": "Mechanical Keyboard",
|
||||
"price": 129.99,
|
||||
"status": "completed"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"eventId": "order_1001",
|
||||
"event": {
|
||||
"clientId": "bash_script_01",
|
||||
"eventId": "order_1001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"payload": "{\"item\":\"Mechanical Keyboard\",\"price\":129.99,\"status\":\"completed\"}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Send a Batch of Events
|
||||
```bash
|
||||
curl -X POST "https://your-domain.example/api/events" \
|
||||
-H "Authorization: Bearer YOUR_CLIENT_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"batch": [
|
||||
{ "eventId": "evt_01", "data": { "metric": "cpu", "value": 45.2 } },
|
||||
{ "eventId": "evt_02", "data": { "metric": "ram", "value": 72.8 } }
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 2: Integrating the Python Client SDK
|
||||
|
||||
DataStack provides an official, pip-installable Python SDK featuring real-time Socket.IO streaming, automatic HTTP fallback, and offline disk spooling.
|
||||
|
||||
### 1. Install the SDK
|
||||
```bash
|
||||
pip install datastack
|
||||
# Or install backward-compatible package:
|
||||
pip install datastack-sdk
|
||||
```
|
||||
|
||||
### 2. Create Configuration (`config.datastack.py`)
|
||||
In the root of your Python project, create a `config.datastack.py` file:
|
||||
|
||||
```python
|
||||
DATASTACK_CONFIG = {
|
||||
"SERVER_URL": "https://your-domain.example", # WebSocket & proxy endpoint
|
||||
"CLIENT_TOKEN": "YOUR_CLIENT_JWT_TOKEN", # Token from /vcon/clients
|
||||
"AUTO_CONNECT": True,
|
||||
"ENABLE_HTTP_FALLBACK": True,
|
||||
"OFFLINE_SPOOL_ENABLED": True,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Stream Telemetry from Python
|
||||
```python
|
||||
from datastack import DataStack
|
||||
import time
|
||||
|
||||
# Client automatically discovers and loads config.datastack.py
|
||||
client = DataStack()
|
||||
|
||||
print(f"Connected to DataStack as client: {client.client_id}")
|
||||
|
||||
for i in range(5):
|
||||
response = client.send(
|
||||
data={
|
||||
"sensor_id": "temp_probe_04",
|
||||
"temperature_celsius": 24.5 + i * 0.2,
|
||||
"humidity_percent": 55,
|
||||
},
|
||||
event_id=f"reading_{i}_{int(time.time())}"
|
||||
)
|
||||
print(f"Event sent: {response}")
|
||||
time.sleep(1)
|
||||
|
||||
client.disconnect()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 3: Streaming Local Files (`send_file`)
|
||||
|
||||
If you have offline log files or CSV datasets that need to be ingested into DataStack, use the SDK's built-in file streaming:
|
||||
|
||||
```python
|
||||
from datastack import DataStack
|
||||
|
||||
client = DataStack()
|
||||
|
||||
# Supports .csv, .json, .jsonl, and .ndjson
|
||||
result = client.send_file(
|
||||
file_path="./daily_orders.csv",
|
||||
chunk_size=250, # Stream in batches of 250 records
|
||||
extra_metadata={"imported_by": "nightly_cron", "env": "production"}
|
||||
)
|
||||
|
||||
print(f"File Ingestion Complete:")
|
||||
print(f" Processed rows: {result['processed']}")
|
||||
print(f" Duplicates skipped: {result['duplicates']}")
|
||||
print(f" Chunks sent: {result['chunksSent']}")
|
||||
```
|
||||
|
||||
- **Lossless Handling**: If your server rate limits the file, the SDK automatically pauses and retries.
|
||||
- **Deduplication**: Rows without an explicit `eventId` receive deterministic SHA-256 IDs, preventing duplicate rows if the script runs twice.
|
||||
|
||||
---
|
||||
|
||||
## Tutorial 4: Exporting Data for Analysis
|
||||
|
||||
You can stream historical partitions for analysis in pandas, Python, or command-line scripts.
|
||||
|
||||
### 1. Stream CSV Export via `curl`
|
||||
```bash
|
||||
curl -X GET "https://your-domain.example/api/export?format=csv&clientId=bash_script_01" \
|
||||
-H "Cookie: datastack_admin_session=YOUR_ADMIN_SESSION_TOKEN" \
|
||||
-o export_bash_script.csv
|
||||
```
|
||||
|
||||
### 2. Load Directly into pandas
|
||||
```python
|
||||
import pandas as pd
|
||||
import requests
|
||||
import io
|
||||
|
||||
session_cookie = "YOUR_ADMIN_SESSION_COOKIE"
|
||||
export_url = "https://your-domain.example/api/export?format=csv&clientId=all"
|
||||
|
||||
response = requests.get(
|
||||
export_url,
|
||||
cookies={"datastack_admin_session": session_cookie},
|
||||
stream=True
|
||||
)
|
||||
|
||||
df = pd.read_csv(io.StringIO(response.text))
|
||||
print("Exported DataFrame:")
|
||||
print(df.head())
|
||||
```
|
||||
Reference in new issue
Block a user