Skip to content
PhramePhrame

The node controller

The node controller

Section titled “The node controller”

One agent per host, with one job: make the containers running on this machine match the ones declared for it — then say what it actually has.

Orchestration — desired state, actual state, and the loop between DESIRED — what should be running Architect /Operatorauthor intent GitLabphrame-configversioned config-syncvalidate →retained MQTT MQTT brokerdesired topics(retained) commit webhook retained orchestration-reconcilerchooses a node · commits the placement writes node.jsononly to a branch,never to a tag ACTUAL — what is really running node-controlleron every host Dockerinput-proc · mixersTAMS · egress actual stateheartbeat ·host stats mirrors →web-ui converge publish aggregate desired state, per node Git is the source of truth for desired state — a commit is a deployment, and the running system is reported back, not assumed

  • Project: orchestration/node-orchestrator (Rust cargo workspace)
  • Runs: one node-controller per host, plus shared broker-side services
  • Talks to: the MQTT broker, and the local Docker socket

What it does

Section titled “What it does”

The controller subscribes to the desired container and network state for its own node, diffs that against what the local Docker daemon is really running, and converges the difference — creating, starting, stopping and removing containers and networks. It then publishes the actual state back, with a heartbeat and host statistics.

It is a loop, not a command. Nothing tells the controller to “deploy”. It is continuously comparing intent against reality, which is why a container that exits comes back, why a host that reboots restores itself without being prompted, and why an interrupted change is just a difference that has not been closed yet rather than a system in an unknown state.

Its scope is deliberately narrow: it decides nothing about where a container ought to run. That belongs to the reconciler. The controller only ever answers “what should be on this host, and is it?” — which is what lets a node be lost and rejoined without the rest of the cluster being told anything.

The workspace

Section titled “The workspace”
CrateKindWhat it is
node-controllerbinaryThe reconcile loop. Reads desired state from MQTT, converges the local Docker daemon, publishes actual state.
docker-managerlibraryA Bollard wrapper for container, network and image lifecycle — the only thing that touches the Docker socket.
mqtt-clientlibraryAuthenticated MQTT over WebSocket-over-TLS, with OAuth2 client-credentials token fetch and refresh, built on rumqttc, reqwest and rustls.
mqtt-mirrorbinaryWrites a debounced, aggregated JSON snapshot of every retained topic. Two instances run by default — one for the desired plane, one for actual.
topic-apibinaryAn Axum REST + Swagger API in front of the broker: KV access, container operations, environment import, wildcard queries, and Docker-Compose export.

Why mirrors exist

Section titled “Why mirrors exist”

Every browser holding a live MQTT subscription in order to render a dashboard is a poor trade: the broker carries the fan-out, and each client re-implements the same aggregation. mqtt-mirror subscribes once and writes a debounced aggregated snapshot of the retained topics, which a UI can simply read.

Debounced matters during a large change: a redeploy touches many topics in quick succession, and a naive mirror would emit a snapshot per message, most of them describing a system mid-transition. One mirror runs per plane, so “what was asked for” and “what is actually there” stay separately readable rather than being merged into a single blurred view.

Configuration

Section titled “Configuration”

Environment-driven, PHRAME_-prefixed: the desired and actual MQTT URLs, the OAuth2 endpoint and client credentials, status and heartbeat cadences, registry credentials, the Docker socket path, and TLS certificate paths.

The heartbeat cadence is worth setting deliberately — it is what the scheduler uses to decide whether this node is alive enough to be given work.

Development stack

Section titled “Development stack”

The project ships a docker-compose.yml bringing up the full local stack — the broker, both mirrors, the topic API and a controller — so the loop can be run end to end on one machine without a cluster.