114 lines
4.4 KiB
Markdown
114 lines
4.4 KiB
Markdown
# Troubleshooting & Forensic Diagnostics
|
|||
|
|
|
||
|
|
This guide provides systematic troubleshooting instructions for diagnosing and resolving common operational issues, connection failures, and error codes in DataStack.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 1. Quick Diagnostic Checklist
|
||
|
|
|
||
|
|
When investigating an issue, verify the following baseline checks:
|
||
|
|
|
||
|
|
1. **Check Container Status**:
|
||
|
|
```bash
|
||
|
|
docker compose ps
|
||
|
|
```
|
||
|
|
All five services (`datastack-web`, `datastack-ingestion`, `datastack-proxy`, `datastack-postgres`, `datastack-redis`) should display `Up` and `(healthy)`.
|
||
|
|
|
||
|
|
2. **Check Nginx Proxy Health Endpoint**:
|
||
|
|
```bash
|
||
|
|
curl -i http://localhost/health
|
||
|
|
# Expected response: HTTP/1.1 200 OK
|
||
|
|
# Body: OK
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Check Web Application Status**:
|
||
|
|
```bash
|
||
|
|
curl -i http://localhost:3000/api/health
|
||
|
|
# Expected response: {"status":"ok","uptime":...}
|
||
|
|
```
|
||
|
|
|
||
|
|
4. **Inspect Live Logs**:
|
||
|
|
```bash
|
||
|
|
docker compose logs -f --tail=100 datastack-web datastack-ingestion
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. Ingestion & API Errors
|
||
|
|
|
||
|
|
### HTTP 401 Unauthorized (`INVALID_TOKEN`)
|
||
|
|
- **Cause**: The client request did not supply an `Authorization` header, or the provided JWT signature is invalid.
|
||
|
|
- **Resolution**:
|
||
|
|
- Ensure the client request header contains: `Authorization: Bearer <TOKEN>`.
|
||
|
|
- Check whether the platform JWT secret was recently rotated in `/vcon/settings`. If rotated, copy the newly signed token from `/vcon/clients`.
|
||
|
|
|
||
|
|
### HTTP 403 Forbidden (`CLIENT_REVOKED` or `CLIENT_DELETED`)
|
||
|
|
- **Cause**: The client was marked as `revoked` in `/vcon/clients`, or the client ID exists in the `deleted_clients` tombstone registry.
|
||
|
|
- **Resolution**:
|
||
|
|
- In `/vcon/clients`, verify the client's status. If revoked, click **Activate**.
|
||
|
|
- If the client was permanently deleted, register a new client slug.
|
||
|
|
|
||
|
|
### HTTP 403 Forbidden (`CSRF_FAILED`)
|
||
|
|
- **Cause**: State-changing POST/PUT/DELETE requests to administrative routes did not match the required `Origin` or `Referer` headers.
|
||
|
|
- **Resolution**:
|
||
|
|
- When accessing behind a reverse proxy (e.g., Cloudflare, Traefik, Coolify), ensure the proxy forwards the original host header:
|
||
|
|
```nginx
|
||
|
|
proxy_set_header Host $host;
|
||
|
|
proxy_set_header X-Forwarded-Host $host;
|
||
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||
|
|
```
|
||
|
|
|
||
|
|
### HTTP 413 Payload Too Large (`PAYLOAD_TOO_LARGE`)
|
||
|
|
- **Cause**: An incoming event payload exceeds `maxPayloadSizeKb` (default: 100 KB).
|
||
|
|
- **Resolution**:
|
||
|
|
- Split large batches into smaller chunks, or increase the payload limit in `/vcon/settings` (up to 10 MB).
|
||
|
|
|
||
|
|
### HTTP 429 Too Many Requests (`RATE_LIMIT_EXCEEDED`)
|
||
|
|
- **Cause**: The client has exceeded its configured rate limit (events/second) or the platform-wide limit.
|
||
|
|
- **Resolution**:
|
||
|
|
- Navigate to `/vcon/clients`, click **Edit Client**, and increase the per-client rate limit.
|
||
|
|
- Or in `/vcon/settings`, raise `maxEventsPerSec`.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. WebSocket & Socket.IO Connection Issues
|
||
|
|
|
||
|
|
### Handshake Fails or Drops Back to Long-Polling
|
||
|
|
- **Symptoms**: Browser or Python client reports continuous reconnect attempts; delays in data synchronization.
|
||
|
|
- **Cause**: The upstream reverse proxy is not passing the WebSocket upgrade headers.
|
||
|
|
- **Resolution**:
|
||
|
|
- Verify your reverse proxy configuration includes:
|
||
|
|
```nginx
|
||
|
|
location /socket.io/ {
|
||
|
|
proxy_pass http://datastack-ingestion:4000;
|
||
|
|
proxy_http_version 1.1;
|
||
|
|
proxy_set_header Upgrade $http_upgrade;
|
||
|
|
proxy_set_header Connection "Upgrade";
|
||
|
|
proxy_set_header Host $host;
|
||
|
|
proxy_read_timeout 86400s;
|
||
|
|
proxy_send_timeout 86400s;
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 4. Storage & Filesystem Issues
|
||
|
|
|
||
|
|
### Write Delays or Stale Lock Warning in Logs
|
||
|
|
- **Symptoms**: Log message shows: `Acquiring lock for partition ... timeout`.
|
||
|
|
- **Cause**: A previous worker process was abruptly terminated (e.g., `kill -9` or OOM) while writing to disk, leaving a `.lock` file behind.
|
||
|
|
- **Resolution**:
|
||
|
|
- DataStack automatically cleans up locks older than 30 seconds.
|
||
|
|
- If an immediate manual clear is needed, check for `.lock` files in the partition directory:
|
||
|
|
```bash
|
||
|
|
find storage/data -name "*.lock" -type f
|
||
|
|
```
|
||
|
|
- Unlink locks older than 60 seconds:
|
||
|
|
```bash
|
||
|
|
find storage/data -name "*.lock" -type f -mmin +1 -delete
|
||
|
|
```
|
||
|
|
|
||
|
|
### Missing Historical Partitions
|
||
|
|
- **Cause**: Automated retention pruning (`autoPruneDays`) purges partition files older than the retention threshold.
|
||
|
|
- **Resolution**:
|
||
|
|
- Check the configured value in `/vcon/settings`. Set to `0` if you want all partition files retained indefinitely.
|