Files
Team Dark Devil 40a6855724 feat(sdk): refactor Python SDK documentation
- Standardize Python SDK (`datastack`) documentation with a new multi-file suite.
- Update `datastack-sdk` to serve as a backward-compatible shim for the canonical `datastack` package.
- Update project manifest and memory map to reflect current SDK structure and v2.2.0 features.
2026-11-18 10:03:40 +06:00

3.4 KiB

Partitioned CSV Storage (/vcon/storage)

DataStack achieves sustained high throughput and scale-to-zero operational efficiency by bypassing per-row database inserts. All incoming telemetry streams are appended directly into partitioned CSV files on the local or mounted filesystem.


1. Storage Architecture & Directory Layout

Telemetry files are partitioned by client identifier and UTC date:

storage/
└── data/
    ├── pos_terminal_01/
    │   ├── 2026-11-16.csv
    │   ├── 2026-11-17.csv
    │   └── 2026-11-17-part2.csv
    └── mobile_app_prod/
        ├── 2026-11-17.csv
        └── 2026-11-17.csv.lock   <-- Temporary cooperative write lock

Partition File Format

Every CSV partition strictly adheres to RFC 4180 standard formatting with UTF-8 encoding:

clientId,eventId,timestamp,payload
pos_terminal_01,evt_1700000000000_a1b2,2026-11-17T12:00:00.000Z,"{""status"":""ok"",""temp"":23.4}"
pos_terminal_01,evt_1700000001000_c3d4,2026-11-17T12:00:01.000Z,"{""status"":""ok"",""temp"":23.6}"
  • Header: Created automatically if the file does not exist when the first event of the day arrives.
  • Payload Field: JSON strings are enclosed in double quotes with internal quotes escaped as "".
  • Newline: Strict Unix standard \n line delimiters.

2. File Rotation Mechanism

To ensure partition files remain easy to compress, transfer, and load into memory-constrained analytics tools:

  • Configurable Limit: Set via fileRotationMb in /vcon/settings (default: 100 MB).
  • Rotation Behavior:
    1. Before appending a batch, the storage engine checks the current file size on disk (fs.statSync).
    2. If appending the batch would exceed fileRotationMb, DataStack automatically creates the next increment:
      • Part 1: 2026-11-17.csv
      • Part 2: 2026-11-17-part2.csv
      • Part 3: 2026-11-17-part3.csv
    3. Subsequent writes immediately target the active part file.

3. Cooperative Locking Protocol

Because both the Port 3000 Web server (handling HTTP ingestion) and the Port 4000 Ingestion server (handling WebSocket flushes) access the same storage volume, DataStack enforces a cooperative file locking protocol:

  1. Lock Creation: Before reading or writing to a partition, the process attempts to create an atomic lock file alongside the CSV: {filePath}.lock.
  2. Lock Timeout: Writers poll every 50ms up to a 10,000ms timeout.
  3. Stale Lock Recovery: If a process crashes while holding a lock, subsequent processes inspect the lock's mtime. Any lock older than 30 seconds is treated as stale, forcefully unlinked, and reclaimed.
  4. Guaranteed Release: Lock release is wrapped in finally blocks to guarantee cleanup after success or error.

4. Automated Retention Pruning

DataStack manages disk usage through an automated pruning engine:

  • Setting: autoPruneDays (default: 30 days).
  • Pruning Logic:
    • The engine parses date strings from filenames (YYYY-MM-DD.csv).
    • Any partition whose date is older than Date.now() - (autoPruneDays * 86400000) is deleted.
    • If a client's folder becomes empty, the folder is safely removed.
  • Disabling Pruning: Setting autoPruneDays: 0 in /vcon/settings disables automated pruning, retaining all partition history permanently.
  • Safety Guard: Pruning runs only after database settings are fully hydrated, preventing accidental deletions during container bootstrap.