feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
This commit is contained in:
1 parent
4279da45bb
commit
40a6855724
38 files changed
+3315
-141
No files matched your search
@@ -0,0 +1,95 @@
|
||||
# Platform Configuration Guide
|
||||
|
||||
This guide covers all environment variables, initial administrator bootstrap secrets, runtime tuning parameters, and synchronization behaviors across the DataStack architecture.
|
||||
|
||||
---
|
||||
|
||||
## 1. Environment Variables Overview
|
||||
|
||||
DataStack reads runtime configuration from system environment variables (or a root `.env` file). In development, safe fallback values are used. In production (`NODE_ENV=production`), all security keys must be explicitly configured with at least 32 cryptographically secure characters.
|
||||
|
||||
```dotenv
|
||||
# Security & Session Secrets (REQUIRED in production, min 32 characters)
|
||||
DATASTACK_ADMIN_SESSION_SECRET=your-32-plus-char-session-secret-here
|
||||
DATASTACK_JWT_SECRET=your-32-plus-char-client-jwt-secret-here
|
||||
DATASTACK_ADMIN_SETUP_KEY=your-32-plus-char-bootstrap-setup-key-here
|
||||
DATASTACK_INTERNAL_SECRET=your-32-plus-char-internal-service-secret
|
||||
|
||||
# Administrator Account Initial Bootstrap
|
||||
DATASTACK_ADMIN_EMAIL=admin@example.com
|
||||
|
||||
# PostgreSQL Persistence (recommended for production metadata)
|
||||
DATABASE_URL=postgresql://datastack:password@postgres:5432/datastack_db
|
||||
|
||||
# Shared Redis Buffer (optional; enables cluster buffer queue & distributed locks)
|
||||
REDIS_URL=redis://redis:6379
|
||||
|
||||
# Service Networking & Ports
|
||||
PORT=3000
|
||||
PORT_SOCKET=4000
|
||||
INGESTION_INTERNAL_URL=http://127.0.0.1:4000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment Variables Specification
|
||||
|
||||
### Security & Authentication
|
||||
|
||||
| Variable | Required in Prod | Default (Dev) | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `DATASTACK_ADMIN_SESSION_SECRET` | **Yes** | Built-in fallback | Used by WebCrypto HMAC-SHA256 to sign and verify operator session cookies (`datastack_admin_session`). Must be ≥ 32 characters. |
|
||||
| `DATASTACK_JWT_SECRET` | **Yes** | Built-in fallback | Master signing secret used to generate and verify HMAC-SHA256 JWT tokens for registered SDK clients. |
|
||||
| `DATASTACK_ADMIN_SETUP_KEY` | **Yes** | Built-in fallback | Secret key required to complete the initial admin onboarding wizard at `/vcon/setup`. |
|
||||
| `DATASTACK_INTERNAL_SECRET` | Recommended | Derived from JWT | Secret used for authenticated HTTP communication between Port 3000 and the Port 4000 ingestion server via `x-datastack-internal-secret`. If omitted, automatically derived via SHA-256 of `DATASTACK_JWT_SECRET`. |
|
||||
| `DATASTACK_ALLOW_DEMO_ADMIN` | No | `false` | When set to `true`, enables quick demo credentials on the login screen (`admin@example.com`). Strictly prohibited in production. |
|
||||
|
||||
### PostgreSQL Database
|
||||
|
||||
| Variable | Required in Prod | Default (Dev) | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `DATABASE_URL` | Recommended | `undefined` | PostgreSQL connection URI (`postgresql://user:pass@host:port/dbname`). If omitted, DataStack operates in an in-memory mock mode. |
|
||||
|
||||
### Redis Shared Buffer
|
||||
|
||||
| Variable | Required in Prod | Default (Dev) | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `REDIS_URL` | Optional | `undefined` | Redis connection URI (`redis://host:6379`). When present, events are queued in Redis list `datastack:buffer:events`, surviving container restarts and enabling multi-worker flush locking. |
|
||||
|
||||
### Service Networking
|
||||
|
||||
| Variable | Required in Prod | Default | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `PORT` | No | `3000` | Port for the Next.js 15 Web Application and REST API. |
|
||||
| `PORT_SOCKET` | No | `4000` | Port for the Express 5 + Socket.IO ingestion service. |
|
||||
| `INGESTION_INTERNAL_URL` | No | `http://127.0.0.1:4000` | Internal URL used by Port 3000 to send bridge notifications to Port 4000 (`/internal/*`). |
|
||||
|
||||
---
|
||||
|
||||
## 3. Platform Runtime Settings (`/vcon/settings`)
|
||||
|
||||
In addition to environment variables, operators can adjust platform operational thresholds at runtime through the **Settings Workspace** (`/vcon/settings`).
|
||||
|
||||
These settings are stored in PostgreSQL (`admin_settings` table) and broadcast in real time to the Port 4000 ingestion service via `/internal/settings-sync`.
|
||||
|
||||
| Parameter | Type | Default | Permitted Range | Purpose |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| `maxEventsPerSec` | Number | `100` | `1` – `10,000` | Ingestion rate ceiling. Events exceeding this limit receive `RATE_LIMIT_EXCEEDED (429)`. |
|
||||
| `maxPayloadSizeKb` | Number | `100` | `1` – `10,240` (10 MB) | Maximum permitted size in kilobytes for any single event payload. Oversized payloads receive `PAYLOAD_TOO_LARGE (413)`. |
|
||||
| `batchFlushThreshold` | Number | `500` | `10` – `50,000` | Queue depth threshold that triggers an immediate write of buffered events to CSV files on disk. |
|
||||
| `batchFlushIntervalMs` | Number | `2000` | `200` – `60,000` (60s) | Maximum duration pending events can remain buffered in memory/Redis before being flushed to disk. |
|
||||
| `fileRotationMb` | Number | `100` | `5` – `1,024` (1 GB) | File size threshold per daily CSV partition. Once reached, DataStack writes to `{YYYY-MM-DD}-part2.csv`. |
|
||||
| `enableIdempotency` | Boolean | `true` | `true` / `false` | Enables 5-minute sliding-window deduplication based on client-provided `eventId`. |
|
||||
| `autoPruneDays` | Number | `30` | `0` – `3,650` | Automatically purges daily CSV partition files older than this value. Setting to `0` disables pruning and keeps all archives indefinitely. |
|
||||
| `jwtSecret` | String | `.env` value | ≥ 16 characters | Master JWT signing secret. Rotating this value automatically re-signs all active SDK client credentials and notifies Port 4000. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Cross-Process Synchronization Rules
|
||||
|
||||
When settings are modified in `/vcon/settings`:
|
||||
|
||||
1. **Database Persistence**: If PostgreSQL is configured, settings are written to `admin_settings WHERE id = 'default'`.
|
||||
2. **Web Process Hydration**: Port 3000 updates its local singleton instances (`SecurityEngine`, `DataBufferEngine`, `CSVStorageManager`).
|
||||
3. **Bridge Notification**: Port 3000 issues an authenticated HTTP `POST` to `${INGESTION_INTERNAL_URL}/internal/settings-sync` carrying the `x-datastack-internal-secret` header.
|
||||
4. **Ingestion Reload**: Port 4000 receives the updated thresholds and immediately applies them to its socket handlers and background flusher worker without requiring a process restart.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Administration Console (`/vcon/`)
|
||||
|
||||
The DataStack Administration Console (`/vcon/`) provides operators and systems engineers with a central interface for observing telemetry ingestion, tracking disk utilization, managing administrator accounts, and tuning ingestion limits.
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication & Access
|
||||
|
||||
Access the console by visiting:
|
||||
```text
|
||||
https://your-domain.example/vcon/login
|
||||
```
|
||||
|
||||
- **Authentication Method**: Signed HTTP-only session cookies (`datastack_admin_session`) using WebCrypto HMAC-SHA256.
|
||||
- **Session Security**: Session tokens encode the administrator's ID and current `tokenVersion`. Changing a password or clicking **Sign Out** immediately invalidates all active sessions globally.
|
||||
- **Brute-Force Protection**: IP + Email failed attempt tracking locks out callers for 15 minutes after 5 consecutive failures.
|
||||
|
||||
---
|
||||
|
||||
## 2. Dashboard Workspace (`/vcon/dashboard`)
|
||||
|
||||
The dashboard provides real-time operational visibility:
|
||||
|
||||
### Dual-Runtime Health Indicators
|
||||
- **Web App (Port 3000)**: Displays status, uptime, and database connection state.
|
||||
- **Ingestion Service (Port 4000)**: Verifies the real-time bridge via `GET /internal/metrics`. Displays active Socket.IO connection counts and server status.
|
||||
|
||||
### Metric Gauges & Telemetry Cards
|
||||
- **Total Ingested Events**: Lifetime count of telemetry events recorded across all registered client partitions.
|
||||
- **Active Clients**: Number of SDK client credentials currently active and authorized to send data.
|
||||
- **Total Storage Used**: Total disk footprint of all CSV partitions stored under `/app/storage/data`.
|
||||
- **Partition Files**: Count of daily partition files currently retained on disk.
|
||||
- **Unflushed Buffer Depth**: Number of events currently awaiting disk write in local RAM or Redis.
|
||||
- **Flush Threshold / Interval**: Configured batch write limits (e.g., 500 events or 2000 ms).
|
||||
|
||||
### Live Activity Ticker
|
||||
- Displays a real-time list of recently ingested events across all connected clients.
|
||||
- Shows timestamp (UTC), client identifier, event ID, and an expandable preview of the JSON payload.
|
||||
- Auto-refreshes periodically or on-demand.
|
||||
|
||||
### Manual Buffer Flush
|
||||
- Clicking **Flush Buffer** dispatches a request to `/api/events/flush` which synchronously notifies Port 4000 (`POST /internal/flush`) and flushes Port 3000's RAM queue to disk.
|
||||
|
||||
---
|
||||
|
||||
## 3. Profile & Security Workspace (`/vcon/profile`)
|
||||
|
||||
Manage administrator credentials and session tokens:
|
||||
|
||||
- **Current Operator Info**: Displays email address and creation timestamp.
|
||||
- **Password Modification**: Update account passwords with mandatory current password verification. Passwords are encrypted using `crypto.scrypt` (64-byte derived key, 16-byte cryptographically secure random salt).
|
||||
- **Session Revocation**: Clicking **Log Out Other Sessions** increments the user's `tokenVersion` in PostgreSQL, instantly invalidating any other browsers or cookies.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Client SDK Registry & Management (`/vcon/clients`)
|
||||
|
||||
DataStack enforces strict authentication for all telemetry sources. The **Client Management** workspace (`/vcon/clients`) allows operators to register client applications, generate and rotate JWT credentials, adjust per-client rate limits, and revoke access instantly.
|
||||
|
||||
---
|
||||
|
||||
## 1. Registering an SDK Client
|
||||
|
||||
To onboard a new application or telemetry producer:
|
||||
|
||||
1. Navigate to **Clients** (`/vcon/clients`) and click **+ Register Client**.
|
||||
2. Complete the form:
|
||||
- **Client Identifier (`clientId`)**: A unique lowercase slug (e.g., `pos_terminal_01`, `mobile_app_prod`, `fleet_vehicle_42`). Allowed characters: `a-z`, `0-9`, `_`, `-`.
|
||||
- **Client Name**: Human-readable label (e.g., `Point of Sale Terminal #1`).
|
||||
- **Rate Limit (Events/sec)**: Maximum allowed throughput for this specific client (default: `100`).
|
||||
3. Click **Register Client**.
|
||||
4. **Copy the Generated Token**:
|
||||
- DataStack generates a cryptographically random 32-character client secret.
|
||||
- Using `DATASTACK_JWT_SECRET`, DataStack signs an HMAC-SHA256 JWT payload containing:
|
||||
```json
|
||||
{
|
||||
"clientId": "pos_terminal_01",
|
||||
"name": "Point of Sale Terminal #1",
|
||||
"rateLimit": 100,
|
||||
"iat": 1700000000
|
||||
}
|
||||
```
|
||||
- Store this token in your client's environment or configuration.
|
||||
|
||||
---
|
||||
|
||||
## 2. Managing Client Credentials & Lifecycle
|
||||
|
||||
### Rotating Client Tokens
|
||||
If a client secret or JWT token is compromised or needs routine rotation:
|
||||
1. Locate the client in the table and click **Rotate Token**.
|
||||
2. Confirm the prompt.
|
||||
3. DataStack generates a fresh client secret, signs a new JWT, and updates PostgreSQL.
|
||||
4. **Instant Disconnection**: Port 3000 calls `POST /internal/client-sync`. The Port 4000 ingestion server immediately severs all active WebSocket connections holding the old token. The client must reconnect using the newly generated token.
|
||||
|
||||
### Revoking Access
|
||||
To temporarily pause ingestion from a specific client without deleting its history:
|
||||
1. Click the status dropdown or action button and select **Revoke**.
|
||||
2. The client status changes to `revoked`.
|
||||
3. Port 4000 immediately rejects new handshakes and terminates existing connections with `CLIENT_REVOKED (403)`.
|
||||
4. HTTP requests to `POST /api/events` with this token are rejected with `403 Forbidden`.
|
||||
|
||||
### Restoring Access
|
||||
1. Select a revoked client and click **Activate**.
|
||||
2. Handshakes and HTTP ingestion are instantly permitted again.
|
||||
|
||||
### Deleting a Client
|
||||
When an application is permanently decommissioned:
|
||||
1. Click **Delete Client**.
|
||||
2. Choose whether to:
|
||||
- **Keep Historical Data**: Retains `/app/storage/data/{clientId}/` on disk for future analytics and export.
|
||||
- **Purge Historical Data**: Deletes the partition folder and all CSV files for this client permanently.
|
||||
3. **Tombstone Protection**: DataStack records the client ID and secret in the `deleted_clients` PostgreSQL table. Even if someone attempts to reuse an old signed JWT, the ingestion server checks the tombstone list and rejects the request.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Streaming Data Export (`/vcon/export`)
|
||||
|
||||
The DataStack Export Workspace (`/vcon/export`) allows operators, data scientists, and business analysts to extract collected telemetry data into standardized CSV or JSON formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Streaming Architecture
|
||||
|
||||
Traditional analytics exports often fail when datasets grow large, triggering Out-Of-Memory (OOM) crashes on the server. DataStack uses **Streaming HTTP Responses**:
|
||||
|
||||
- **Zero-RAM Buffering**: Partition files are opened as read streams (`fs.createReadStream`) and piped directly through Node.js transform streams into the response socket.
|
||||
- **Constant Memory Footprint**: Exporting 50 megabytes or 50 gigabytes consumes the same nominal RAM footprint (~15 MB) on the server.
|
||||
- **Client Backpressure**: If the downloading client or network connection is slow, reading from disk pauses automatically to avoid memory saturation.
|
||||
|
||||
---
|
||||
|
||||
## 2. Export Configuration & Options
|
||||
|
||||
The export workspace provides targeted filtering:
|
||||
|
||||
| Option | Values | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| **Client Filter** | `All Clients` or specific `clientId` | Filter by an individual registered application (e.g., `pos_terminal_01`) or export across the entire tenant. |
|
||||
| **Start Date** | Date Picker (`YYYY-MM-DD`) | Earliest UTC date to include in the exported records. |
|
||||
| **End Date** | Date Picker (`YYYY-MM-DD`) | Latest UTC date to include in the exported records. |
|
||||
| **Format** | `CSV` or `JSON` | Output format for the downloaded file. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported Formats & Tool Compatibility
|
||||
|
||||
### CSV Format (`text/csv`)
|
||||
- **UTF-8 Byte Order Mark (BOM)**: DataStack prepends the `\uFEFF` marker to ensure Microsoft Excel correctly recognizes special characters and UTF-8 encodings when opened by double-clicking.
|
||||
- **RFC 4180 Escaping**: Embedded JSON payloads are escaped according to standard CSV rules.
|
||||
|
||||
#### Importing CSV into Python (pandas)
|
||||
```python
|
||||
import pandas as pd
|
||||
import json
|
||||
|
||||
# Load exported CSV
|
||||
df = pd.read_csv("datastack_export_pos_terminal_01.csv")
|
||||
|
||||
# Parse JSON payloads into a structured DataFrame
|
||||
df["parsed_payload"] = df["payload"].apply(json.loads)
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
#### Importing CSV into DuckDB
|
||||
```sql
|
||||
-- Query directly from the exported CSV without importing
|
||||
SELECT
|
||||
clientId,
|
||||
timestamp,
|
||||
json_extract_string(payload, '$.temperature') AS temperature
|
||||
FROM read_csv_auto('datastack_export.csv')
|
||||
WHERE temperature > 25.0;
|
||||
```
|
||||
|
||||
### JSON Format (`application/json`)
|
||||
- Streams a valid JSON array (`[ ... ]`) of parsed event objects.
|
||||
- Each event includes `clientId`, `eventId`, `timestamp`, and the native parsed JSON `payload` object.
|
||||
|
||||
---
|
||||
|
||||
## 4. Programmatic Export via API
|
||||
|
||||
You can automate data pipeline extraction via `curl` or cron scripts using your administrator session cookie:
|
||||
|
||||
```bash
|
||||
curl -X GET "https://your-domain.example/api/export?clientId=pos_terminal_01&from=2026-11-01&to=2026-11-17&format=csv" \
|
||||
-H "Cookie: datastack_admin_session=YOUR_SESSION_COOKIE" \
|
||||
--output export.csv
|
||||
```
|
||||
@@ -0,0 +1,83 @@
|
||||
# Data Simulator (`/vcon/simulator`)
|
||||
|
||||
The DataStack Data Simulator (`/vcon/simulator`) enables operators and developers to validate ingestion pipelines, test rate limits, and inspect buffer flushes directly from the browser without setting up external producer scripts.
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview & Capabilities
|
||||
|
||||
The Simulator allows you to:
|
||||
- Test ingestion using either **HTTP REST** (`POST /api/events`) or **Socket.IO WebSocket** (`sync_data`).
|
||||
- Select any active client from your registered client list, or simulate unauthenticated/custom payloads.
|
||||
- Dispatch single events or high-volume bursts (from 10 to 500 events).
|
||||
- Select from realistic schema presets or construct custom JSON objects.
|
||||
- Observe response status, payload transmission size, and round-trip latency in real time.
|
||||
|
||||
---
|
||||
|
||||
## 2. Telemetry Presets
|
||||
|
||||
The simulator includes four industry-standard telemetry presets:
|
||||
|
||||
### 1. IoT Temperature Sensor
|
||||
```json
|
||||
{
|
||||
"sensorId": "temp-north-01",
|
||||
"temperature": 23.4,
|
||||
"humidity": 45.2,
|
||||
"battery": 98,
|
||||
"unit": "celsius"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. E-Commerce Order Checkout
|
||||
```json
|
||||
{
|
||||
"orderId": "ord_9872134",
|
||||
"customerId": "cust_11029",
|
||||
"currency": "USD",
|
||||
"totalAmount": 149.50,
|
||||
"itemCount": 3,
|
||||
"paymentMethod": "credit_card"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Server System Resource Metrics
|
||||
```json
|
||||
{
|
||||
"host": "prod-api-worker-03",
|
||||
"cpuUsagePercent": 34.2,
|
||||
"memoryUsedMb": 1024,
|
||||
"memoryTotalMb": 4096,
|
||||
"loadAverage": [0.45, 0.62, 0.58]
|
||||
}
|
||||
```
|
||||
|
||||
### 4. User Audit Trail & Security Event
|
||||
```json
|
||||
{
|
||||
"actorId": "usr_9912",
|
||||
"action": "client_token_rotated",
|
||||
"ipAddress": "192.168.1.100",
|
||||
"userAgent": "Mozilla/5.0",
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Simulator Workflows
|
||||
|
||||
### Testing Single Event Ingestion
|
||||
1. Select your target **Client** from the dropdown.
|
||||
2. Select **Single Event** mode.
|
||||
3. Choose a preset or customize the JSON editor.
|
||||
4. Click **Send Event**.
|
||||
5. The output console displays the assigned `eventId`, server response code (`200 OK`), and processing time.
|
||||
|
||||
### Testing High-Throughput Batch Flushing
|
||||
1. Select **Batch Dispatch** mode.
|
||||
2. Set the burst count (e.g., `500` events).
|
||||
3. Click **Start Batch**.
|
||||
4. Observe the batch progress bar.
|
||||
5. Navigate to `/vcon/dashboard` to verify that the buffer threshold flush automatically triggered, writing the batch to the client's daily CSV partition file on disk.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Features Guide
|
||||
|
||||
Welcome to the DataStack Feature Documentation. This section details the complete feature set of the DataStack high-throughput telemetry platform.
|
||||
|
||||
---
|
||||
|
||||
## Feature Index
|
||||
|
||||
| Feature Document | Purpose & Capabilities | Primary Navigation Path |
|
||||
| :--- | :--- | :--- |
|
||||
| [Admin Console (`admin-console.md`)](./admin-console.md) | System health monitoring, dual-runtime status, memory/buffer gauges, and administrator account management. | `/vcon/dashboard`, `/vcon/profile` |
|
||||
| [Client Management (`client-management.md`)](./client-management.md) | Client SDK registration, HMAC-SHA256 JWT generation, credential rotation, socket revocation, and tombstone deletion. | `/vcon/clients` |
|
||||
| [Data Simulator (`data-simulator.md`)](./data-simulator.md) | Browser-based event generation, single vs. burst testing, WebSocket/HTTP transport modes, and telemetry payload presets. | `/vcon/simulator` |
|
||||
| [Partitioned CSV Storage (`partitioned-storage.md`)](./partitioned-storage.md) | Daily client directory partitioning, 100MB rotation (`-part2.csv`), cooperative lock files, and automated retention pruning. | `/vcon/storage` |
|
||||
| [Data Export (`data-export.md`)](./data-export.md) | Streaming CSV/JSON extraction, date range filters, client filtering, Excel/pandas compatibility, and non-blocking reads. | `/vcon/export` |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
Every feature in DataStack follows a strict separation of concerns:
|
||||
- **Control Plane**: Handled on Port 3000 via Next.js 15 App Router and PostgreSQL metadata tables (`admin_users`, `client_sdks`, `admin_settings`).
|
||||
- **Data Plane**: Handled on Port 4000 via Express 5, Socket.IO 4, and persistent append-only CSV partitions under `/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`.
|
||||
- **Zero SQL Insertions**: Telemetry events never create rows in relational tables, ensuring predictable query performance and infinite scale-to-zero storage efficiency.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Partitioned CSV Storage (`/vcon/storage`)
|
||||
|
||||
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
|
||||
|
||||
---
|
||||
|
||||
## 1. Storage Architecture & Directory Layout
|
||||
|
||||
Telemetry files are partitioned by client identifier and UTC date:
|
||||
|
||||
```text
|
||||
storage/
|
||||
└── data/
|
||||
├── pos_terminal_01/
|
||||
│ ├── 2026-11-16.csv
|
||||
│ ├── 2026-11-17.csv
|
||||
│ └── 2026-11-17-part2.csv
|
||||
└── mobile_app_prod/
|
||||
├── 2026-11-17.csv
|
||||
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
|
||||
```
|
||||
|
||||
### Partition File Format
|
||||
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
|
||||
|
||||
```csv
|
||||
clientId,eventId,timestamp,payload
|
||||
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
|
||||
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
|
||||
```
|
||||
|
||||
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
|
||||
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
|
||||
- **Newline**: Strict Unix standard `\n` line delimiters.
|
||||
|
||||
---
|
||||
|
||||
## 2. File Rotation Mechanism
|
||||
|
||||
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
|
||||
|
||||
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
|
||||
- **Rotation Behavior**:
|
||||
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
|
||||
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
|
||||
- Part 1: `2026-11-17.csv`
|
||||
- Part 2: `2026-11-17-part2.csv`
|
||||
- Part 3: `2026-11-17-part3.csv`
|
||||
3. Subsequent writes immediately target the active part file.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cooperative Locking Protocol
|
||||
|
||||
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
|
||||
|
||||
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
|
||||
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
|
||||
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
|
||||
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
|
||||
|
||||
---
|
||||
|
||||
## 4. Automated Retention Pruning
|
||||
|
||||
DataStack manages disk usage through an automated pruning engine:
|
||||
|
||||
- **Setting**: `autoPruneDays` (default: `30` days).
|
||||
- **Pruning Logic**:
|
||||
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
|
||||
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
|
||||
- If a client's folder becomes empty, the folder is safely removed.
|
||||
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
|
||||
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Getting Started with DataStack
|
||||
|
||||
This fast-track tutorial guides you from a fresh server to a fully running DataStack instance collecting live telemetry in under 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Deploy with Docker Compose
|
||||
|
||||
Clone the repository and copy the environment template:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vibtoolshq/datastack.git
|
||||
cd datastack
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Generate 32+ character secrets for production:
|
||||
```bash
|
||||
# Generate random keys in Linux or macOS:
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
In `.env`, set your secrets and admin email:
|
||||
```dotenv
|
||||
DATASTACK_ADMIN_SESSION_SECRET=your-32-char-session-secret-here
|
||||
DATASTACK_JWT_SECRET=your-32-char-jwt-secret-here
|
||||
DATASTACK_ADMIN_SETUP_KEY=your-32-char-setup-key-here
|
||||
DATASTACK_INTERNAL_SECRET=your-32-char-internal-secret-here
|
||||
DATASTACK_ADMIN_EMAIL=admin@example.com
|
||||
```
|
||||
|
||||
Launch the stack:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
Verify that all 5 services are healthy:
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml ps
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Complete Initial Administrator Setup
|
||||
|
||||
1. Open your browser and navigate to:
|
||||
```text
|
||||
http://localhost/vcon/setup
|
||||
# Or https://your-domain.example/vcon/setup
|
||||
```
|
||||
2. Enter:
|
||||
- **Administrator Email**: Must match `DATASTACK_ADMIN_EMAIL` (or your chosen admin email).
|
||||
- **Administrator Password**: Minimum 8 characters.
|
||||
- **Setup Security Key**: Enter the exact string from `DATASTACK_ADMIN_SETUP_KEY`.
|
||||
3. Click **Complete Initial Setup**. You are automatically logged in and redirected to `/vcon/dashboard`.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Register Your First SDK Client
|
||||
|
||||
1. In the sidebar, click **Clients** (`/vcon/clients`).
|
||||
2. Click **+ Register Client**.
|
||||
3. Fill in the fields:
|
||||
- **Client Identifier**: `sensor_gateway_01` (lowercase alphanumeric and underscores).
|
||||
- **Client Name**: `Primary Sensor Gateway`.
|
||||
- **Rate Limit**: `100` (max events per second).
|
||||
4. Click **Register Client**.
|
||||
5. Copy the generated **Bearer JWT Token**. Store it securely.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Dispatch Your First Event via `curl`
|
||||
|
||||
Send a test telemetry event to the REST ingestion API:
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost/api/events" \
|
||||
-H "Authorization: Bearer YOUR_GENERATED_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"eventId": "test_event_001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"data": {
|
||||
"temperature": 23.4,
|
||||
"humidity": 45,
|
||||
"status": "ONLINE"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"eventId": "test_event_001",
|
||||
"event": {
|
||||
"clientId": "sensor_gateway_01",
|
||||
"eventId": "test_event_001",
|
||||
"timestamp": "2026-11-17T12:00:00Z",
|
||||
"payload": "{\"temperature\":23.4,\"humidity\":45,\"status\":\"ONLINE\"}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Connect with the Python Client SDK
|
||||
|
||||
Install the official Python SDK:
|
||||
```bash
|
||||
pip install datastack
|
||||
```
|
||||
|
||||
In your Python project directory, create `config.datastack.py`:
|
||||
```python
|
||||
DATASTACK_CONFIG = {
|
||||
"SERVER_URL": "http://localhost",
|
||||
"CLIENT_TOKEN": "YOUR_GENERATED_JWT_TOKEN",
|
||||
"AUTO_CONNECT": True,
|
||||
}
|
||||
```
|
||||
|
||||
Create `main.py` and run it:
|
||||
```python
|
||||
from datastack import DataStack
|
||||
import time
|
||||
|
||||
# Auto-discovers config.datastack.py in the current directory
|
||||
client = DataStack()
|
||||
|
||||
print(f"Connected to DataStack as client: {client.client_id}")
|
||||
|
||||
# Stream 5 test events
|
||||
for i in range(5):
|
||||
res = client.send(
|
||||
data={"reading_index": i, "voltage": 3.3 + (i * 0.05)},
|
||||
event_id=f"reading_{i}_{int(time.time())}"
|
||||
)
|
||||
print(f"Dispatched event {i}: {res.get('eventId')} via {res.get('transport')}")
|
||||
time.sleep(1)
|
||||
|
||||
client.disconnect()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Verify Data in the Admin Console
|
||||
|
||||
1. **Dashboard Overview (`/vcon/dashboard`)**:
|
||||
- Check the **Active Buffer** gauge — incoming events are queued in Redis / memory.
|
||||
- Observe the **Live Activity Stream** updating in real time.
|
||||
2. **Data Browser (`/vcon/data`)**:
|
||||
- Filter by client `sensor_gateway_01`.
|
||||
- View your ingested events with exact timestamps and formatted JSON payloads.
|
||||
3. **Data Export (`/vcon/export`)**:
|
||||
- Click **Export Dataset** to download a clean, consolidated CSV file ready for Excel or Python analysis.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
- Consult the complete **[Python SDK Documentation](../../sdk/python/datastack/docs/README.md)** for API reference, offline spooling, and file ingestion.
|
||||
- Learn how to stream files (`.csv`, `.json`, `.jsonl`) with **[Tutorials](./tutorials.md)**.
|
||||
- Understand storage rotation and auto-pruning in **[Partitioned CSV Storage](./features/partitioned-storage.md)**.
|
||||
- Explore production deployment and backups in **[Installation Guide](./installation.md)**.
|
||||
@@ -0,0 +1,55 @@
|
||||
# DataStack User & Operator Guide
|
||||
|
||||
Welcome to the **DataStack User & Operator Documentation**. This portal contains comprehensive guides for deploying, configuring, operating, and integrating applications with DataStack.
|
||||
|
||||
---
|
||||
|
||||
## Guide Directory
|
||||
|
||||
```text
|
||||
docs/users/
|
||||
├── index.md # This directory index
|
||||
├── getting-started.md # 5-minute setup and quickstart tutorial
|
||||
├── installation.md # Production Docker Compose & Coolify deployment
|
||||
├── configuration.md # Secrets, environment variables & settings reference
|
||||
├── features/ # Detailed functional walkthroughs
|
||||
│ ├── index.md # Features directory index
|
||||
│ ├── admin-console.md # /vcon/ dashboard, monitors, and profile workspace
|
||||
│ ├── client-management.md # SDK client registration, JWT rotation, and revocation
|
||||
│ ├── data-simulator.md # Interactive browser telemetry simulator & presets
|
||||
│ ├── partitioned-storage.md # Daily CSV storage, file rotation, and auto-pruning
|
||||
│ └── data-export.md # Streaming CSV and JSON dataset export workspace
|
||||
├── workflows.md # Step-by-step user journeys & operations
|
||||
├── troubleshooting.md # Forensic error recovery & diagnostics
|
||||
└── faq.md # Frequently asked questions & storage architecture
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Reading Pathways
|
||||
|
||||
Choose the pathway that matches your immediate goal:
|
||||
|
||||
### Pathway 1: "I want to self-host DataStack on my VPS"
|
||||
1. Start with **[Installation & Deployment](./installation.md)** for Docker Compose or Coolify setup.
|
||||
2. Review **[Configuration Guide](./configuration.md)** to generate 32+ character secrets.
|
||||
3. Follow **[Getting Started](./getting-started.md)** to run the `/vcon/setup` wizard and log in.
|
||||
|
||||
### Pathway 2: "I want to connect my Python application"
|
||||
1. Review **[Client Token Management](./features/client-management.md)** to generate a client JWT token.
|
||||
2. Read **[Tutorial: Python SDK Integration](./getting-started.md#step-4-connect-with-the-python-sdk)** to set up `config.datastack.py`.
|
||||
3. Check **[Troubleshooting](./troubleshooting.md)** if you encounter rate limits or connection drops.
|
||||
|
||||
### Pathway 3: "I want to inspect or export collected data"
|
||||
1. Read **[Partitioned CSV Storage](./features/partitioned-storage.md)** to understand directory and file layouts.
|
||||
2. Read **[Streaming Data Export](./features/data-export.md)** to extract datasets via browser or `curl`.
|
||||
3. Follow the **[Data Analysis Workflow](./workflows.md#workflow-3-exporting-and-analyzing-data)** to load records into pandas or Excel.
|
||||
|
||||
---
|
||||
|
||||
## Platform Philosophy
|
||||
|
||||
DataStack is built to solve the **"High-Velocity Telemetry Dilemma"**:
|
||||
- **Why not PostgreSQL for everything?** Inserting thousands of events per second into relational database tables causes index thrashing, lock contention, and WAL exhaustion.
|
||||
- **The DataStack Solution**: PostgreSQL is used strictly for administrative accounts and client credentials. Ingested events flow through in-memory and Redis buffers directly into organized, daily partitioned CSV files (`/app/storage/data/{clientId}/{YYYY-MM-DD}.csv`).
|
||||
- **The Benefit**: Virtually unlimited ingestion throughput, zero SQL lock contention, deterministic storage costs, and instant portability.
|
||||
@@ -0,0 +1,114 @@
|
||||
# Troubleshooting & Forensic Diagnostics
|
||||
|
||||
This guide provides systematic troubleshooting instructions for diagnosing and resolving common operational issues, connection failures, and error codes in DataStack.
|
||||
|
||||
---
|
||||
|
||||
## 1. Quick Diagnostic Checklist
|
||||
|
||||
When investigating an issue, verify the following baseline checks:
|
||||
|
||||
1. **Check Container Status**:
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
All five services (`datastack-web`, `datastack-ingestion`, `datastack-proxy`, `datastack-postgres`, `datastack-redis`) should display `Up` and `(healthy)`.
|
||||
|
||||
2. **Check Nginx Proxy Health Endpoint**:
|
||||
```bash
|
||||
curl -i http://localhost/health
|
||||
# Expected response: HTTP/1.1 200 OK
|
||||
# Body: OK
|
||||
```
|
||||
|
||||
3. **Check Web Application Status**:
|
||||
```bash
|
||||
curl -i http://localhost:3000/api/health
|
||||
# Expected response: {"status":"ok","uptime":...}
|
||||
```
|
||||
|
||||
4. **Inspect Live Logs**:
|
||||
```bash
|
||||
docker compose logs -f --tail=100 datastack-web datastack-ingestion
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Ingestion & API Errors
|
||||
|
||||
### HTTP 401 Unauthorized (`INVALID_TOKEN`)
|
||||
- **Cause**: The client request did not supply an `Authorization` header, or the provided JWT signature is invalid.
|
||||
- **Resolution**:
|
||||
- Ensure the client request header contains: `Authorization: Bearer <TOKEN>`.
|
||||
- Check whether the platform JWT secret was recently rotated in `/vcon/settings`. If rotated, copy the newly signed token from `/vcon/clients`.
|
||||
|
||||
### HTTP 403 Forbidden (`CLIENT_REVOKED` or `CLIENT_DELETED`)
|
||||
- **Cause**: The client was marked as `revoked` in `/vcon/clients`, or the client ID exists in the `deleted_clients` tombstone registry.
|
||||
- **Resolution**:
|
||||
- In `/vcon/clients`, verify the client's status. If revoked, click **Activate**.
|
||||
- If the client was permanently deleted, register a new client slug.
|
||||
|
||||
### HTTP 403 Forbidden (`CSRF_FAILED`)
|
||||
- **Cause**: State-changing POST/PUT/DELETE requests to administrative routes did not match the required `Origin` or `Referer` headers.
|
||||
- **Resolution**:
|
||||
- When accessing behind a reverse proxy (e.g., Cloudflare, Traefik, Coolify), ensure the proxy forwards the original host header:
|
||||
```nginx
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
```
|
||||
|
||||
### HTTP 413 Payload Too Large (`PAYLOAD_TOO_LARGE`)
|
||||
- **Cause**: An incoming event payload exceeds `maxPayloadSizeKb` (default: 100 KB).
|
||||
- **Resolution**:
|
||||
- Split large batches into smaller chunks, or increase the payload limit in `/vcon/settings` (up to 10 MB).
|
||||
|
||||
### HTTP 429 Too Many Requests (`RATE_LIMIT_EXCEEDED`)
|
||||
- **Cause**: The client has exceeded its configured rate limit (events/second) or the platform-wide limit.
|
||||
- **Resolution**:
|
||||
- Navigate to `/vcon/clients`, click **Edit Client**, and increase the per-client rate limit.
|
||||
- Or in `/vcon/settings`, raise `maxEventsPerSec`.
|
||||
|
||||
---
|
||||
|
||||
## 3. WebSocket & Socket.IO Connection Issues
|
||||
|
||||
### Handshake Fails or Drops Back to Long-Polling
|
||||
- **Symptoms**: Browser or Python client reports continuous reconnect attempts; delays in data synchronization.
|
||||
- **Cause**: The upstream reverse proxy is not passing the WebSocket upgrade headers.
|
||||
- **Resolution**:
|
||||
- Verify your reverse proxy configuration includes:
|
||||
```nginx
|
||||
location /socket.io/ {
|
||||
proxy_pass http://datastack-ingestion:4000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "Upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_send_timeout 86400s;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Storage & Filesystem Issues
|
||||
|
||||
### Write Delays or Stale Lock Warning in Logs
|
||||
- **Symptoms**: Log message shows: `Acquiring lock for partition ... timeout`.
|
||||
- **Cause**: A previous worker process was abruptly terminated (e.g., `kill -9` or OOM) while writing to disk, leaving a `.lock` file behind.
|
||||
- **Resolution**:
|
||||
- DataStack automatically cleans up locks older than 30 seconds.
|
||||
- If an immediate manual clear is needed, check for `.lock` files in the partition directory:
|
||||
```bash
|
||||
find storage/data -name "*.lock" -type f
|
||||
```
|
||||
- Unlink locks older than 60 seconds:
|
||||
```bash
|
||||
find storage/data -name "*.lock" -type f -mmin +1 -delete
|
||||
```
|
||||
|
||||
### Missing Historical Partitions
|
||||
- **Cause**: Automated retention pruning (`autoPruneDays`) purges partition files older than the retention threshold.
|
||||
- **Resolution**:
|
||||
- Check the configured value in `/vcon/settings`. Set to `0` if you want all partition files retained indefinitely.
|
||||
@@ -0,0 +1,125 @@
|
||||
# Operational Workflows & User Journeys
|
||||
|
||||
This document provides step-by-step procedures for common operational workflows, from first-time administrator onboarding to credential rotation and disaster recovery.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 1: Initial Administrator Onboarding
|
||||
|
||||
When setting up a newly deployed instance:
|
||||
|
||||
```text
|
||||
Deploy Stack ──► Open /vcon/setup ──► Enter Setup Key & Admin Email ──► Create Password ──► Redirect to /vcon/dashboard
|
||||
```
|
||||
|
||||
1. Visit your instance URL at `/vcon/setup` (e.g., `https://your-domain.example/vcon/setup`).
|
||||
2. If an administrator already exists, DataStack redirects to `/vcon/login`.
|
||||
3. Provide the required credentials:
|
||||
- **Setup Key**: Enter the `DATASTACK_ADMIN_SETUP_KEY` configured in your `.env`.
|
||||
- **Admin Email**: Enter the primary operational contact email.
|
||||
- **Password**: Enter a strong password (minimum 8 characters, checked against common passwords).
|
||||
4. Click **Complete Setup**.
|
||||
5. DataStack hashes the password using `crypto.scrypt`, creates the first row in `admin_users`, establishes a signed session cookie, and redirects to `/vcon/dashboard`.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 2: Onboarding an External Client Application
|
||||
|
||||
Connecting a new producer (Python app, IoT gateway, mobile app, microservice):
|
||||
|
||||
```text
|
||||
/vcon/clients ──► Register Client ──► Copy JWT Token ──► Configure Client App ──► Stream Events
|
||||
```
|
||||
|
||||
1. Navigate to **Clients** (`/vcon/clients`).
|
||||
2. Click **+ Register Client**.
|
||||
3. Specify:
|
||||
- `Client Identifier`: e.g., `checkout_service_prod`
|
||||
- `Client Name`: e.g., `Checkout Service Production`
|
||||
- `Rate Limit`: e.g., `250` events/second
|
||||
4. Click **Register Client**.
|
||||
5. A modal appears displaying the newly signed HMAC-SHA256 JWT token.
|
||||
6. **Copy the token** and configure your application environment:
|
||||
```dotenv
|
||||
DATASTACK_CLIENT_ID=checkout_service_prod
|
||||
DATASTACK_AUTH_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
|
||||
DATASTACK_SERVER_URL=https://your-domain.example
|
||||
```
|
||||
7. Start your producer. Events will immediately stream into the system and appear in the live dashboard activity feed.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 3: Rotating a Client Token
|
||||
|
||||
If a client token has leaked or requires routine security rotation:
|
||||
|
||||
```text
|
||||
/vcon/clients ──► Rotate Token ──► Copy New Token ──► Update Client Config ──► Reconnect
|
||||
```
|
||||
|
||||
1. Open `/vcon/clients`.
|
||||
2. Locate the client row and click **Rotate Token**.
|
||||
3. Confirm the action in the confirmation modal.
|
||||
4. DataStack generates a new client secret, signs a fresh JWT, updates PostgreSQL, and notifies Port 4000 via `/internal/client-sync`.
|
||||
5. Active WebSocket connections using the prior token are disconnected immediately.
|
||||
6. Copy the new token and update your producer application's environment. Once restarted, the client reconnects with the new token.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 4: Rotating the Platform JWT Secret
|
||||
|
||||
In the event of an infrastructure-wide security review or platform secret compromise:
|
||||
|
||||
1. Open `/vcon/settings`.
|
||||
2. In the **Security & Secrets** card, click **Rotate Platform Secret**.
|
||||
3. Confirm the security alert.
|
||||
4. DataStack:
|
||||
- Generates a new cryptographically secure 48-character platform JWT secret.
|
||||
- Saves the secret to `admin_settings`.
|
||||
- Iterates through all registered clients in PostgreSQL and re-signs their tokens using the new master secret (`SDKManager.resignAllClientTokensAsync()`).
|
||||
- Pushes the new secret and re-signed tokens to Port 4000 via `/internal/settings-sync`.
|
||||
5. Navigate to `/vcon/clients` to export or view the refreshed client tokens to distribute to client application teams.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 5: Backup & Disaster Recovery
|
||||
|
||||
DataStack stores state in two distinct layers: PostgreSQL (metadata) and filesystem volumes (telemetry data).
|
||||
|
||||
### Backing Up the PostgreSQL Control Plane
|
||||
```bash
|
||||
docker exec -t datastack-postgres pg_dump -U datastack datastack_db > datastack_meta_$(date +%F).sql
|
||||
```
|
||||
|
||||
### Backing Up Telemetry Partitions
|
||||
Telemetry files reside in the named Docker volume `datastack-data` (or mounted host path `./storage/data`).
|
||||
|
||||
```bash
|
||||
# Archive all client partition directories
|
||||
docker run --rm \
|
||||
-v datastack-data:/data:ro \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar czf /backup/datastack_partitions_$(date +%F).tar.gz -C /data .
|
||||
```
|
||||
|
||||
### Restoring from Backup
|
||||
1. Stop the ingestion and web containers:
|
||||
```bash
|
||||
docker compose down
|
||||
```
|
||||
2. Restore the database:
|
||||
```bash
|
||||
docker compose up -d postgres
|
||||
cat datastack_meta_2026-11-17.sql | docker exec -i datastack-postgres psql -U datastack datastack_db
|
||||
```
|
||||
3. Restore the partition volume:
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v datastack-data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar xzf /backup/datastack_partitions_2026-11-17.tar.gz -C /data
|
||||
```
|
||||
4. Restart the full stack:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
Reference in new issue
Block a user