74 lines
3.4 KiB
Markdown
74 lines
3.4 KiB
Markdown
# Partitioned CSV Storage (`/vcon/storage`)
|
|||
|
|
|
||
|
|
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 1. Storage Architecture & Directory Layout
|
||
|
|
|
||
|
|
Telemetry files are partitioned by client identifier and UTC date:
|
||
|
|
|
||
|
|
```text
|
||
|
|
storage/
|
||
|
|
└── data/
|
||
|
|
├── pos_terminal_01/
|
||
|
|
│ ├── 2026-11-16.csv
|
||
|
|
│ ├── 2026-11-17.csv
|
||
|
|
│ └── 2026-11-17-part2.csv
|
||
|
|
└── mobile_app_prod/
|
||
|
|
├── 2026-11-17.csv
|
||
|
|
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
|
||
|
|
```
|
||
|
|
|
||
|
|
### Partition File Format
|
||
|
|
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
|
||
|
|
|
||
|
|
```csv
|
||
|
|
clientId,eventId,timestamp,payload
|
||
|
|
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
|
||
|
|
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
|
||
|
|
```
|
||
|
|
|
||
|
|
- **Header**: Created automatically if the file does not exist when the first event of the day arrives.
|
||
|
|
- **Payload Field**: JSON strings are enclosed in double quotes with internal quotes escaped as `""`.
|
||
|
|
- **Newline**: Strict Unix standard `\n` line delimiters.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. File Rotation Mechanism
|
||
|
|
|
||
|
|
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
|
||
|
|
|
||
|
|
- **Configurable Limit**: Set via `fileRotationMb` in `/vcon/settings` (default: `100 MB`).
|
||
|
|
- **Rotation Behavior**:
|
||
|
|
1. Before appending a batch, the storage engine checks the current file size on disk (`fs.statSync`).
|
||
|
|
2. If appending the batch would exceed `fileRotationMb`, DataStack automatically creates the next increment:
|
||
|
|
- Part 1: `2026-11-17.csv`
|
||
|
|
- Part 2: `2026-11-17-part2.csv`
|
||
|
|
- Part 3: `2026-11-17-part3.csv`
|
||
|
|
3. Subsequent writes immediately target the active part file.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. Cooperative Locking Protocol
|
||
|
|
|
||
|
|
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
|
||
|
|
|
||
|
|
1. **Lock Creation**: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: `{filePath}.lock`.
|
||
|
|
2. **Lock Timeout**: Writers poll every 50ms up to a 10,000ms timeout.
|
||
|
|
3. **Stale Lock Recovery**: If a process crashes while holding a lock, subsequent processes inspect the lock's `mtime`. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
|
||
|
|
4. **Guaranteed Release**: Lock release is wrapped in `finally` blocks to guarantee cleanup after success or error.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 4. Automated Retention Pruning
|
||
|
|
|
||
|
|
DataStack manages disk usage through an automated pruning engine:
|
||
|
|
|
||
|
|
- **Setting**: `autoPruneDays` (default: `30` days).
|
||
|
|
- **Pruning Logic**:
|
||
|
|
- The engine parses date strings from filenames (`YYYY-MM-DD.csv`).
|
||
|
|
- Any partition whose date is older than `Date.now() - (autoPruneDays * 86400000)` is deleted.
|
||
|
|
- If a client's folder becomes empty, the folder is safely removed.
|
||
|
|
- **Disabling Pruning**: Setting `autoPruneDays: 0` in `/vcon/settings` disables automated pruning, retaining all partition history permanently.
|
||
|
|
- **Safety Guard**: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.
|