- Standardize Python SDK (`datastack`) documentation with a new multi-file suite. - Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package. - Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
3.4 KiB
Partitioned CSV Storage (/vcon/storage)
DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.
1. Storage Architecture & Directory Layout
Telemetry files are partitioned by client identifier and UTC date:
storage/
└── data/
├── pos_terminal_01/
│ ├── 2026-11-16.csv
│ ├── 2026-11-17.csv
│ └── 2026-11-17-part2.csv
└── mobile_app_prod/
├── 2026-11-17.csv
└── 2026-11-17.csv.lock <-- Temporary cooperative write lock
Partition File Format
Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:
clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
- Header: Created automatically if the file does not exist when the first event of the day arrives.
- Payload Field: JSON strings are enclosed in double quotes with internal quotes escaped as
"". - Newline: Strict Unix standard
\nline delimiters.
2. File Rotation Mechanism
To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:
- Configurable Limit: Set via
fileRotationMbin/vcon/settings(default:100 MB). - Rotation Behavior:
- Before appending a batch, the storage engine checks the current file size on disk (
fs.statSync). - If appending the batch would exceed
fileRotationMb, DataStack automatically creates the next increment:- Part 1:
2026-11-17.csv - Part 2:
2026-11-17-part2.csv - Part 3:
2026-11-17-part3.csv
- Part 1:
- Subsequent writes immediately target the active part file.
- Before appending a batch, the storage engine checks the current file size on disk (
3. Cooperative Locking Protocol
Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:
- Lock Creation: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV:
{filePath}.lock. - Lock Timeout: Writers poll every 50ms up to a 10,000ms timeout.
- Stale Lock Recovery: If a process crashes while holding a lock, subsequent processes inspect the lock's
mtime. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed. - Guaranteed Release: Lock release is wrapped in
finallyblocks to guarantee cleanup after success or error.
4. Automated Retention Pruning
DataStack manages disk usage through an automated pruning engine:
- Setting:
autoPruneDays(default:30days). - Pruning Logic:
- The engine parses date strings from filenames (
YYYY-MM-DD.csv). - Any partition whose date is older than
Date.now() - (autoPruneDays * 86400000)is deleted. - If a client's folder becomes empty, the folder is safely removed.
- The engine parses date strings from filenames (
- Disabling Pruning: Setting
autoPruneDays: 0in/vcon/settingsdisables automated pruning, retaining all partition history permanently. - Safety Guard: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.