Weather Station — Architecture Guide

Simulator: Ingest (WAL) + Read (Aggregated)

A live model of write-path congestion and read-path console load. In Production, the read endpoints are aggregated in ClickHouse, cutting the fleet view from O(N) round-trips to O(1).

Demand controls (N)

Fleet sensors (N) 50

Ingest interval ms (I) 1000 ms

Demand R:50.0 req/s

Capacity C:28 req/s

Write Path (ingest)

24 ms

Event loop

100%

Accepted (202)

Shed (503)

Timeout (504)

Read API Path (fleet view load)

Network requests
(round-trips N)

2.4 MB

Raw download
(total payload)

5.0s

Render time
(frontend wall-time)

Pipeline transit (ingest)

Mode: Production

Sensors Gateway Ingestor + WAL Loop blocked ClickHouse

Received: 0 OK (DB): 0 Shed (503): 0 Dropped (504): 0

System Architecture Models

The physical data flow, end to end. The inherited prototype vs. the production layout (load shedding, WAL, worker pool, aggregated reads).

Inherited Architecture (Spike)

Synchronous & blocking

IoT client & frontend

Sensors POST /ingest. The console does one fetch per device — O(N) — flooding the network.

Nginx gateway

1.0s timeout

Cuts the connection with a 504 if the ingestor takes too long.

Express ingestor

Loop frozen

~50,000 SHA-256 rounds block the thread. A shared mutable object races; a restart loses in-memory state.

ClickHouse DB

Unitary inserts

Row-by-row INSERT. Reads run a giant SELECT *, shipping MBs to the frontend.

Operational impact:

Write: violent collapse and loss of volatile in-memory data.
Read: React stalls iterating heavy raw O(N) endpoints.

Target Architecture (Production)

Decoupled & aggregated

IoT client & fleet console

Heavy write bursts are absorbed. The console fetches aggregated endpoints — O(1), lightweight.

Healthy gateway

Instant ACK

Passes traffic through quickly without holding sockets or timing out.

Express API (write + read)

202 / WAL

Validated ingest with shedding (503) and an append-ahead log (WAL). Reads expose clean endpoints via axios.

Worker pool

Background signing

Runs the signature off the main thread, keeping the event loop free.

ClickHouse (analytical)

Batch & aggregate

Absorbs efficient batch writes. Reads run summary queries (LIMIT 1 BY device_id) returning KBs.

Operational impact:

Write scalability: tolerant to local restarts (WAL) with protected rejection (503).
Read scalability: frontend downloads drop from MBs (5s) to KBs (~200ms).

Implementation Roadmap & Technical Milestones

Development plan organized by Git branch. Pick a milestone to inspect its diagnosis, solution, and trade-off. Canonical detail lives in DECISIONS.md.

Completed: 7

1. Quality

chore/restore-linter-and-tests

Override AGENTS.md's "don't run the linter" note and put the CI gate (lint + tests) back on.

2. Critical

fix/shared-reading-race-condition

Fix the data race and the recurring crash-loop coming from the glitch-mode simulator.

3. Critical

feat/decouple-ingestion-batch-insert

Async decoupling, load shedding and a worker pool to erase the ~15k 504 losses.

4. High

feat/ingestor-restart-resilience

A write-ahead log (WAL) to survive the deployer's 30s restart.

5. High

feat/read-api-aggregated-endpoints

ClickHouse-native aggregate endpoints cutting fleet wall-time ~25x.

6. Frontend

feat/fleet-console-realtime

React 18 console on TanStack Query + virtualized lists, no DOM freeze.

7. Hardening

chore/final-review-and-compaction

Supertest route tests, configurable API base, reviewer orientation and UX polish.

1. Quality

chore/restore-linter-and-tests

Git branch

Diagnosis (problem)

Engineering solution

Accepted trade-off

Production performance (measured, not estimated)

Fixed contracts safeguarded

Deferred / backlog

Scientific Metric Validator

The ingestor computes derived thermodynamic metrics. Use this interactive calculator to check the exact algorithms by entering physical inputs (type a precise value, e.g. 25.4, into the number fields).

IoT measurement variables

Air temperature (T)

°C

Relative humidity (H)

Atmospheric pressure (P)

Wind speed (V)

m/s

Dew Point

$$\alpha = \frac{17.27 \cdot T}{237.7 + T} + \ln\left(\frac{H}{100}\right)$$

16.7 °CAir saturation

Air Density

$$\rho = \frac{P}{287.05 \cdot (T + 273.15)}$$

1.18 kg/m³Mass per volume

Wind Chill

Non-linear adjustment when V > 4.8 m/s

25.0 °CConvective cooling

Heat Index

NWS Rothfusz regression, computed in °F

25.0 °CApparent heat