ESPHome packages and substitutions tutorial: managing complex configurations effectively

Over the course of this year I've been getting closer to my goal of regulating temperature in each room of my house individually. I made small ESPHome devices to measure ambient and floor temperatures and placed them in every room. I also installed my smart thermostat at my relatives' house and flashed a couple of Shelly relays with ESPHome. Needless to say, my ESPHome configurations quickly got out of hand, so it was time to introduce some unification and refactor tons of copy-pasted yaml. In this blog post I will describe the system that I'm currently using. You can use my base configurations with your own devices by referencing them from my repo:

GitHub - olegtarasov/esphome-common

Contribute to olegtarasov/esphome-common development by creating an account on GitHub.

GitHubolegtarasov

How packages and substitutions work

ESPHome has two main tools for configuration composition: packages and substitutions. When I first read the documentation on these features, I struggled to understand how are they interconnected and what approach would be optimal for me. Here I will try to explain these features together and build a composition system on top.

Substitutions

Substitutions are quite simple: they are just shell-style variables that can be overriden in config file hierarchy. Let's consider the simplest compilable config file for ESPHome:

substitutions:
  foo: This is default value from innermost substitutions section

esphome:
  name: substest
  friendly_name: ${foo}

esp32:
  board: lolin_s2_mini

If you run esphome config inner.yaml, you will see that frinedly_name gets substituted with default value specified in substitutions section. This is already useful for cases when you need to use the same value in different places in your config file.

Includes

Substitutions become even more powerful when used with !include statement. In ESPHome you can include arbitrary config snippets almost anywhere, while specifying values for substitutions that are used in the file you are including. There is a good example in ESPHome documentation, which I will not repeat here for brevity.

The important thing here is that you can omit substitutions section with all the default values if you are not going to compile this file directly. Just be sure to provide substitution values for each file you are including.

Packages

The most interesting feature that allows you to build flexible composition architecture is Packges. They are different from !include statement in their ability to intelligently merge configurations. Consider the following configuration:

sensor:
  - platform: template
    id: sensor1
    name: "Sensor 1"

sensor:
  - platform: template
    id: sensor2
    name: "Sensor 2"

esphome:
  name: test

esp32:
  board: lolin_s2_mini

<<: !include sensor1.yaml
<<: !include sensor2.yaml

If you run esphome config sensors.yaml, you will notice that while configuration is considered valid, contents from sensor2.yaml is silently dropped, because YAML doesn't allow the duplicate sensor: section. But what happens if we import sensor1.yaml and sensor2.yaml as packages?

esphome:
  name: substest

esp32:
  board: lolin_s2_mini

packages:
  sensor1: !include sensor1.yaml
  sensor2: !include sensor2.yaml

Now when you run esphome config sensors.yaml, you will see that both sensors were included! This happens because package component intelligently merges config sections based on node names and component ids.

Substitutions in packages

Now we can go even further and eliminate repetition by extracting a generic sensor package and substituting the number in our main file:

sensor:
  - platform: template
    id: sensor${number}
    name: "Sensor ${number}"

esphome:
  name: substest

esp32:
  board: lolin_s2_mini

packages:
  sensor1: !include
    file: sensor_generic.yaml
    vars:
      number: 1
  sensor2: !include
    file: sensor_generic.yaml
    vars:
      number: 2

After checking the result with esphome config sensors.yaml, you will see that we achieved the same result as before, but reduced copy-pasted config even further. As you can see, package variables and substitutions are in fact the same thing, and this is what confused me a lot when I read the official docs. For a while I thought they are different, but as it turns out, you don't need to change anything about your inner config files, whether you include them with a simple !include or as a package.

Package hierarchy and substitutions

The last piece of the puzzle is the relation between packages, substitutions and defaults sections. Official docs say that a package can have a defaults section where default values for that package's substitutions can be specified. I'm not clear why did ESPHome creators decided to use two separate ways of doing basically the same thing — maybe defaults section came first, and when substitutions section was introduced, they decided to keep backwards compatibility.

Let's consider two levels of configuration:

substitutions:
  foo: This is default value from inner substitutions section

defaults:
  foo: This is default value from inner defaults section

esphome:
  name: substest
  friendly_name: ${foo}

esp32:
  board: lolin_s2_mini

substitutions:
  foo: This is default value from top-level substitutions section

packages:
  base: !include inner.yaml

After running esphome config top-level.yaml you will see that frinedly_name got replaced with value from defaults: section in inner.yaml, ignoring substitutions section from both files. But if we comment out the defaults: section, things change.

substitutions:
  foo: This is default value from inner substitutions section

# defaults:
#   foo: This is default value from inner defaults section

esphome:
  name: substest
  friendly_name: ${foo}

esp32:
  board: lolin_s2_mini

Now friendly_name got its value from top-level.yaml substitutions: section. So we can deduce the following rule:

This still means that we can specify substitution value directly in package → vars: section and it will override values from substitutions at any depth. To prove this, let's test three layers of packages:

substitutions:
  foo: This is default value from inner substitutions section

esphome:
  name: substest
  friendly_name: ${foo}

esp32:
  board: lolin_s2_mini

substitutions:
  foo: This is default value from middle substitutions section

packages:
  base: !include inner.yaml

substitutions:
  foo: This is default value from top-level substitutions section

packages:
  base: !include
    file: middle.yaml
    vars:
      foo: "Overriden!"

No matter how deep in include hierarchy the substitution foo is used, it still receives the Overriden! value explicitly defined in top-level.yaml.

How I manage my configuration

I hope that my examples help you better understand how packages and substitutions work. There are still more cool package features that I didn't cover here, like Extend and Remove directives, but my goal was to explain the main rules of package composition and not copy the official docs 😄 Now I will try to explain how I manage my own ESPHome configurations.

Composition over inheritance

My first stab at refactoring my configurations was an idea to use some sort of inheritance, where there is one main device template that can be configured in different ways to do different things. I quickly abandoned this idea because it proved difficult to create such a universal configuration.

Instead I opted for small, composable yaml snippets that can be included as packages so their contents is intelligently merged by ESPHome. Each package is focused on a single task. For example, specify device board and framework, configure Wi-Fi, logging, etc.

For each ESPHome device I then create a separate yaml file, most of which consists of package include statements and some configuration using substitutions. Let's look at some of the base files that I use.

Wi-Fi, OTA, and Home Assistant API

All of my devices that connect to Wi-Fi, should also connect to Home Assistant and have an OTA update capability. I decided to lump all these sections in a single file, but you may split them in different files depending on your requirements.

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  manual_ip:
    static_ip: ${wifi_ip}
    gateway: ${wifi_gateway}
    subnet: 255.255.255.0

# Enable Home Assistant API
api:
  encryption:
    key: !secret api_key

ota:
  - platform: esphome
    password: !secret ota_password

sensor:
  - platform: wifi_signal
    name: "RSSI"
    id: sensor_rssi
    update_interval: 60s
    entity_category: "diagnostic"

text_sensor:
  - platform: wifi_info
    ip_address:
      id: wifi_ip_address
      name: "IP Address"
      icon: "mdi:wan"
      entity_category: "diagnostic"
    bssid:
      id: wifi_bssid
      name: "BSSID"
      icon: "mdi:wan"
      entity_category: "diagnostic"

There are two substitutions, wifi_ip and wifi_gateway that are configured in top-level files. I configure wifi_gateway as a substitution because I have two homes that reside in different subnets with one main router in each subnet.

Base device: Wi-Fi, logging and restart

Then comes the base device package that combines previously defined Wi-Fi package with logging code that calls Home Assistant and a restart button:

packages:
  wifi: !include .wifi.yaml

esphome:
  name: ${device_name}
  friendly_name: ${device_friendly_name}

# Enable logging
logger:
  level: INFO
  logs:
    component: ERROR
    sensor: WARN
  on_message:
    level: DEBUG
    then:
      - homeassistant.action:
          action: system_log.write
          data:
            level: !lambda |-
              switch (level) {
                case 0:
                  return "notset";
                case 1:
                  return "error";
                case 2:
                  return "warning";
                case 3:
                  return "info";
              }
              return "debug";
            logger: !lambda |-
              return "esphome.${site}.${device_name}";
            message: !lambda |-
              return message;

button:
  - platform: restart
    id: restart_device
    entity_category: config
    name: "Restart"

I was looking for a solution that would allow me to continuously ship device logs to Loki. I decided against making an external HTTP call on each logging operation (my internet might be down), and opted to call a Home Assistant service that writes logs from all my devices to its system log, which is later shipped to Loki with Promtail add-on.

Boards

I use several boards in my projects, so I created a package for each combination of a board and target framework. Here are some of them:

esp32:
  board: mhetesp32minikit
  framework:
    type: esp-idf
    version: recommended

wifi:
  enable_btm: true
  enable_rrm: true

esp32:
  board: esp32-s3-devkitc-1
  variant: ESP32S3
  framework:
    type: esp-idf
    version: 5.1.2
    platform_version: 6.5.0

wifi:
  enable_btm: true
  enable_rrm: true

Debugging

There are also a couple of packages that can help debug devices. I don't usually include them in device config, but sometimes they are useful. Here is a package that configures debug sensors:

debug:
  update_interval: 5s

sensor:
  # Debug sensors
  - platform: debug
    free:
      id: debug_heap_free
      name: "Heap Free"
      entity_category: diagnostic
    block:
      id: debug_heam_max_block
      name: "Heap Max Block"
      entity_category: diagnostic
    loop_time:
      id: debug_loop_time
      name: "Loop Time"
      entity_category: diagnostic

text_sensor:
  - platform: wifi_info
    scan_results:
      id: wifi_scan_results
      name: "Wi-Fi scan results"
      icon: "mdi:wan"
      entity_category: "diagnostic"

And debug logging package. There are a lot of entries that are commented out — I just tune them on the spot when there is a need to add debug logs to a device.

logger:
  #level: INFO
  level: DEBUG
  #level: VERY_VERBOSE
  logs:
    component: ERROR
    ## Enable the following when base level is DEBUG
    #opentherm: WARN
    # sensor: WARN
    # opentherm.output: INFO
    ## Enable the following when base level is VERY_VERBOSE
    #esp32.preferences: WARN
    #dallas.sensor: WARN
    #scheduler: WARN
    #sensor.filter: WARN
    #api.service: WARN

Putting it all together

Now let's stitch all these files together. In each room in my house I have devices that measure room and heated floor temperatures. In order to manage them all, I first create a base configuration.

Base device config

packages:
  base: !include common/.base_device.yaml
  board: !include common/.board.s3_micro.yaml

sensor:
  - platform: dallas_temp
    id: floor_temp
    address: ${temp_floor_address}
    name: "Floor Temperature"
    one_wire_id: dallas_floor
    accuracy_decimals: 2
    update_interval: 30s
    filters:
      - sliding_window_moving_average:
          window_size: 5
          send_every: 2
  - platform: dallas_temp
    id: room_temp
    address: ${temp_room_address}
    name: "Temperature"
    one_wire_id: dallas_room
    accuracy_decimals: 2
    update_interval: 30s
    filters:
      - sliding_window_moving_average:
          window_size: 5
          send_every: 2

one_wire:
  - platform: gpio
    pin: GPIO4
    id: dallas_floor
  - platform: gpio
    pin: GPIO2
    id: dallas_room

# Turn off the on-board LED
light:
  - platform: esp32_rmt_led_strip
    id: status_light
    rgb_order: GRB
    pin: GPIO21
    num_leds: 1
    rmt_channel: 0
    chipset: ws2812
    name: "Status light"
    internal: True

esphome:
  on_boot:
    then:
      - light.turn_off:
          id: status_light
          transition_length: 0s

As you can see, this package uses pre-configured .base_device.yaml and .board.s3_micro.yaml, as well as defines two substitutions of its own: temp_floor_address and temp_room_address. These are the addresses of Dallas temperature sensors that I will configure for each end device.

End device config

And finally let's configure an example end device.

substitutions:
  device_name: sensotron-bedroom
  device_friendly_name: Sensotron Bedroom
  wifi_ip: 192.168.3.242
  site: estate
  wifi_gateway: 192.168.3.1
  temp_room_address: 0xc4012273a0e2d528
  temp_floor_address: 0x13062391ca90d928

packages:
  base: !include common/.sensotron.base.yaml

As you can see, there is not much that goes into end device config. I set all the substitutions that are scattered among files in this one place — it's easier to track them this way. I also set device-specific sensor addresses and include one base device package. And we are done! 🎉

Conclusion

Here I describe a heavily opinionated approach to managing ESPHome configuration files. It's still evolving and you shouldn't adopt it blindly. Look at your requirements and make a system that works for you. I hope this tutorial is helpful for people who, like me, struggled to understand how both packages and substitutions work in tandem.