How I manage my ESP32s declaratively with Nix

You can find every file mentioned in this post under Files at the end.

For the last 3-4 months I have been falling down the nix/NixOS rabbit hole. I started with the laptop I use every day and moved it from Ubuntu to NixOS. I had wanted to switch for a long time but I was too settled into what I already knew, and what finally pushed me was some decisions the Ubuntu project made that I think are just wrong. One of them is the snap package manager (which in my opinion adds nothing to the ecosystem except one more thing to deal with). The other was rewriting the standard linux tools in rust and shipping them as the default before they were ready, which caused compatibility problems for me. For example I used to manage my Ubuntu system imperatively with ansible, and the new rust sudo just broke it, no idea why. Sure, I could have found a workaround, most likely a duct taped one, but the real problem is that I think Ubuntu has spent the last few years on pointless innovation rather than on user experience.

Okay, enough pointless complaining about Ubuntu, let's get to the actual topic, NixOS.

A week or two after running it on my laptop I could not help myself and moved my cloud server and my home server over too. That might be one of the best technical decisions I have made for my own setup. I could write for hours about how happy declarative management makes me and how safe NixOS snapshots make me feel, but for this post I want to focus on how I manage my ESP32s declaratively. Maybe I will write about my general laptop and server setup another time.

So I started out driving the ESP32s through ESPHome's web ui. If you have not used ESPHome before, definitely check it out, it is a great project. You can think of it as a web ui for ESP-IDF, you write a yaml config and it turns that into firmware for you with ESP-IDF, there are tons of ready made modules, and on top of that it can push firmware to the ESP32 remotely over wifi, which is just lovely)))

But there is a problem. My whole system lives in a single flake in git. Three hosts, every package, every service, every secret:

nix-config
├── flake.nix
└── hosts
    ├── 314
    ├── main-server
    └── home-server
        ├── configuration.nix
        └── secrets.yaml

Everything except the ESP32s. Their config lived in /var/lib/esphome on the home server and got edited from the web ui. So part of my system was still manually managed state. I could not follow changes in git history, and when a change caused trouble I could not just git revert and rebuild. Basically I wanted my ESP32s to be like VMs managed from my nix config, same as my other hosts. On top of that those yamls only existed there, so I had to back them up with restic, and in my opinion backing up small text files with restic is pointless busywork when you can just put them in git.

My goal was simple put the yaml files in my nix config, have ESPHome read them from there, and only open the web ui to click the install button (you could actually bypass even that and trigger the install from a nix update, but honestly I do not want that).

I tried two ways to get ESPHome's yaml config files onto the server first, and neither of them worked.

First attempt: symlink from the store

I run ESPHome on NixOS through the module. But the module has no option for passing in device configs. ESPHome just reads configs from the folder it has decided on, /var/lib/esphome/. So I figured linking them in there from the store would be enough. (In case you do not use Nix, /nix/store is where Nix keeps every package and every file it builds, each in its own hash named directory, immutable. My yaml lands there on rebuild, and the ${./esphome/device-1.yaml} below returns that store path.)

# hosts/home-server/esphome.nix
systemd.tmpfiles.rules = [
  "L+ /var/lib/esphome/device-1.yaml - - - - ${./esphome/device-1.yaml}"
];

This pattern gets used everywhere in the NixOS world. After the rebuild both devices were still listed in the ESPHome dashboard and both showed up as Online. Then I clicked on one of them.

The configuration could not be loaded. Check your connection and retry.

That is what ESPHome gave me.

I went digging in the journal and found the reason.

The dashboard takes the file as a URL parameter:

http://home-server:6052/json-config?configuration=device-1.yaml

And for security, so that nobody can read other files on the server through that parameter, ESPHome checks where the file actually is before opening it. If the path points outside its own config folder (/var/lib/esphome/), or the file sitting there is a symlink that leads outside it, the dashboard refuses to open it.

My file was exactly the second case. It was sitting in /var/lib/esphome/ but it was a symlink pointing at /nix/store/..., and since that is not ESPHome's own folder it refused to read anything from there.

That check makes perfect sense for security. It just kills our symlink approach entirely.

Second attempt: copying with tmpfiles

If linking is out, then the file has to be copied into the ESPHome folder instead. tmpfiles has a copy rule for that, so I used C+:

# hosts/home-server/esphome.nix
"C+ /var/lib/esphome/device-1.yaml 0444 esphome esphome - ${./esphome/device-1.yaml}"

Rebuild, reload, the same 403.

When I looked at the file I saw that no copying had happened at all, the old symlink was still sitting there.

Turns out the tmpfiles copy rule only copies once, and only if there is nothing at the destination. If something is already there it leaves it alone. I was thinking the + meant "overwrite it if it exists", but it means something else. If the destination is a non empty folder, it keeps descending into it instead of skipping the copy. It has nothing to do with overwriting a file.

So this rule was the wrong tool. I needed something that refreshes the file on every rebuild.

What worked: preStart and restartTriggers

The one thing that definitely runs on every rebuild is the ESPHome service itself. So I dropped tmpfiles and handed the copying over to the service, as a script that runs right before it starts:

# hosts/home-server/esphome.nix
systemd.services.esphome = {
  restartTriggers = [ ./esphome ];
  preStart = ''
    src=${./esphome}
    find "$src" -mindepth 1 -type d -printf '%P\n' | while read -r dir; do
      mkdir -p "/var/lib/esphome/$dir"
    done
    find "$src" -type f -name '*.yaml' -printf '%P\n' | while read -r file; do
      install -m 0444 -T "$src/$file" "/var/lib/esphome/$file.new"
      mv -f "/var/lib/esphome/$file.new" "/var/lib/esphome/$file"
    done
  '';
};

It is a dead simple pre script that copies the files. src is the folder it copies from, which in my case is ./esphome:

nix-config
├── flake.nix
└── hosts
    └── home-server
        ├── configuration.nix
        ├── esphome.nix
        └── esphome
            ├── common
            │   └── base.yaml
            ├── device-1.yaml
            └── dht11-1.yaml

The preStart script looks at the subdirectories first and creates the ones that do not exist yet under /var/lib/esphome/. Then it copies each file under a .new name and swaps it over the real one.

The two steps are there for two reasons. First, I set the file to mode 0444 so that nothing can write to it. I do not want this config living in two places at once, so editing from the web ui has to be out of the question, and this way a save there fails right away instead of silently getting overwritten on the next restart. The side effect is that my own copy cannot write over the existing file either, but replacing it works, because that permission comes from the folder rather than from the file, and the folder belongs to the esphome user. Second, mv is atomic within the same filesystem, so either the old file is visible or the new one, and a half written yaml never sits there in between. Even if the dashboard reads the file at that exact moment it sees a complete config.

One more thing, restartTriggers = [ ./esphome ]. Without it there is nothing to make systemd restart the service when only a yaml changed, so preStart never runs. With it, the service restarts as soon as anything in the folder changes.

Secrets

ESPHome has a secrets section where you can put things like your wifi password or ssid, because normal device config yaml files are in the open and readable by anyone on the machine, and you do not want things like a wifi password readable by everyone. Once you have created those secrets you can reference them from your device yaml files.

In practice it looks like this, no value anywhere, just the name:

# esphome/device-1.yaml
api:
  encryption:
    key: !secret device_1_api_key

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

The values themselves live in ESPHome's own secrets.yaml. That file is exactly the part I wanted to make declarative, but there are two places I do not want it to end up in. First the git repo I push my nix config to, since anything in there stays in the history. Second /nix/store, because the store is readable by every user on the machine (in this case my home-server), so unlike the device yamls I cannot simply copy it in.

This is where sops-nix comes in. It keeps the values encrypted in the repo, so only the encrypted form ever reaches git. It does not write the decrypted form into /nix/store either, it decrypts on the machine at the end of nixos-rebuild switch and puts the result under /run/secrets, which lives in RAM and is gone after a reboot. The file gets assembled like this:

# hosts/home-server/esphome.nix
sops.secrets = {
  esphome_wifi_ssid = { };
  esphome_wifi_password = { };
  esphome_device_1_api_key = { };
};

sops.templates."esphome-secrets.yaml" = {
  owner = "esphome";
  group = "esphome";
  mode = "0400";
  restartUnits = [ "esphome.service" ];
  content = ''
    wifi_ssid: "${config.sops.placeholder.esphome_wifi_ssid}"
    wifi_password: "${config.sops.placeholder.esphome_wifi_password}"
    device_1_api_key: "${config.sops.placeholder.esphome_device_1_api_key}"
  '';
};

The assembled file ends up at /run/secrets/rendered/esphome-secrets.yaml. The dashboard expects it in its own folder under the name secrets.yaml, so I point a symlink at it:

# hosts/home-server/esphome.nix
"L+ /var/lib/esphome/secrets.yaml - - - - ${config.sops.templates."esphome-secrets.yaml".path}"

Yes, a symlink. The thing that broke everything a moment ago. It works here because that check was really guarding the file name coming in from the URL, which is user input. Nobody sends the name of secrets.yaml, ESPHome opens it itself under a fixed name in its own folder, so there is no user input left to validate.

The only casualty is the ESPHome Secrets screen, which does go through that check and now shows an error. I decided to leave it broken. /run/secrets is tmpfs, so with the symlink the decrypted secrets never touch the disk. To get that screen working I would have to copy them in like the device yamls, and then they would sit on the server's disk permanently. Not being able to open that screen costs me nothing, the real copy is in sops and sops -d shows it whenever I need it.

Shared config between devices

At this point every device's yaml lives in the repo, but the files I had were full of repetition. The wifi block, the logger settings and so on were identical in all of them, and coming from a backend background I took that duplication as a personal insult.

ESPHome has a packages: feature for exactly this, it merges another yaml file into the device's config. So I collected the repeated part into common/base.yaml:

nix-config
├── flake.nix
└── hosts
    └── home-server
        ├── configuration.nix
        ├── esphome.nix
        └── esphome
            ├── common
            │   └── base.yaml
            ├── device-1.yaml
            └── dht11-1.yaml

# esphome/common/base.yaml
logger:
  level: DEBUG

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  ap:
    ssid: ${device_name} Fallback
    password: !secret fallback_ap_password

captive_portal:

The file uses ${device_name}, but its value is not in common/base.yaml. It has to be unique per device, so it comes from the device's own yaml:

# esphome/device-1.yaml
substitutions:
  device_name: device-1

esphome:
  name: ${device_name}
  friendly_name: Device 1

packages:
  base: !include common/base.yaml

Two things happen in that file. substitutions sets the value, so every ${device_name} in the shared file (base.yaml) becomes device-1. And packages: is what pulls common/base.yaml in and merges it into this config (device-1.yaml).

So the fallback access point ends up being called "device-1 Fallback" for device-1 and "dht11-1 Fallback" for dht11-1, while the shared file stays a single copy.

Wrapping up

The ESPHome device yamls used to live only on the disk, which is why I had to back that folder up with restic. Now everything is in the git repo and I do not need a restic backup for ESPHome anymore. The rest of the benefits are just the usual benefits of declarative management, except they now apply to the ESP32s too. I can look at the flake and see what is on which device, every change is in git history, and when something goes wrong I can git revert and rebuild.

The workflow now is that I edit the yaml in the repo, run nixos-rebuild switch, then hit Install in the web ui and the new firmware goes to the ESP32 over wifi. I could automate the Install step too, but the home server holds a lot more than ESPHome. If I change something else on it and need to deploy that, I do not want a device config I am still working on to get flashed along with it. So keeping Install separate suits me.

Files

The nix and yaml files from this post are attached below, so you can take them as a starting point and build your own setup on top.

{ config, ... }:
{
  sops.secrets = {
    esphome_wifi_ssid = { };
    esphome_wifi_password = { };
    esphome_fallback_ap_password = { };
    esphome_device_1_api_key = { };
  };

  sops.templates."esphome-secrets.yaml" = {
    owner = "esphome";
    group = "esphome";
    mode = "0400";
    restartUnits = [ "esphome.service" ];
    content = ''
      wifi_ssid: "${config.sops.placeholder.esphome_wifi_ssid}"
      wifi_password: "${config.sops.placeholder.esphome_wifi_password}"
      fallback_ap_password: "${config.sops.placeholder.esphome_fallback_ap_password}"
      device_1_api_key: "${config.sops.placeholder.esphome_device_1_api_key}"
    '';
  };

  systemd.services.esphome = {
    restartTriggers = [ ./esphome ];
    preStart = ''
      src=${./esphome}
      find "$src" -mindepth 1 -type d -printf '%P\n' | while read -r dir; do
        mkdir -p "/var/lib/esphome/$dir"
      done
      find "$src" -type f -name '*.yaml' -printf '%P\n' | while read -r file; do
        install -m 0444 -T "$src/$file" "/var/lib/esphome/$file.new"
        mv -f "/var/lib/esphome/$file.new" "/var/lib/esphome/$file"
      done
    '';
  };

  systemd.tmpfiles.rules = [
    "d /var/lib/esphome 0750 esphome esphome - -"
    "L+ /var/lib/esphome/secrets.yaml - - - - ${config.sops.templates."esphome-secrets.yaml".path}"
  ];
}

logger:
  level: DEBUG

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  ap:
    ssid: ${device_name} Fallback
    password: !secret fallback_ap_password

captive_portal:

substitutions:
  device_name: device-1

esphome:
  name: ${device_name}
  friendly_name: Device 1

esp32:
  board: esp32-c3-devkitm-1
  framework:
    type: esp-idf

packages:
  base: !include common/base.yaml

api:
  encryption:
    key: !secret device_1_api_key

ota:
  - platform: esphome
    encryption: