Files
DataStack/docs/users/features/partitioned-storage.md
T
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

75 lines
3.4 KiB
Markdown

# Partitioned CSV Storage (`/vcon/storage`)
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
---
## 1. Storage Architecture & Directory Layout
Telemetry files are partitioned by client identifier and UTC date:
```text
storage/
└── data/
├── pos_terminal_01/
│ ├── 2026-11-16.csv
│ ├── 2026-11-17.csv
│ └── 2026-11-17-part2.csv
└── mobile_app_prod/
├── 2026-11-17.csv
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
```
### Partition File Format
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
```csv
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
```
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
- **Newline**: Strict Unix standard `\n` line delimiters.
---
## 2. File Rotation Mechanism
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
- **Rotation Behavior**:
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
- Part 1: `2026-11-17.csv`
- Part 2: `2026-11-17-part2.csv`
- Part 3: `2026-11-17-part3.csv`
3. Subsequent writes immediately target the active part file.
---
## 3. Cooperative Locking Protocol
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
---
## 4. Automated Retention Pruning
DataStack manages disk usage through an automated pruning engine:
- **Setting**: `autoPruneDays` (default: `30` days).
- **Pruning Logic**:
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
- If a client's folder becomes empty, the folder is safely removed.
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.