Easier Home Assistant Zigbee2MQTT light integration - listen to specific topic and not just state

[htvekov]

EDIT: Updated HA example with needed mode when groups of lights are turned on/off simultanously.
EDIT: Revised setTopic for JSON example

Hi' @sieren

First of all: Great piece of software ! 👍🎉😎

I've had Homepoint in the test pipeline for quite a few months and finally I got time to test with my set up.
My initial struggle was not with Homepoint, but rather my newly bought, but semi defective, M5 stack.
Dead CP210X chip 😖, which took me half a day to realise before I finally programmed the ESP32 directly without further hazzle.

My initial impression of Homepoint - I love it ! 🤗🤗

I searched a lot, but found next to nothing regarding Z2M/MQTT Home Assistant integration - use examples/experiences etc.
So I had to move forward by trial and error 😉

It would really be helpful if users, with similar setup like me, could define a specific JSON payload key for Homepoint to read the light status (getTopic)

Example - (for type: Light as well):
Define that jsondata should be used for getTopic and the key to be used in firstKey

{
      "name": "Bed lamp",
      "setTopic": "zigbee2mqtt/Office bed lamp",
      "getTopic": "zigbee2mqtt/Office bed lamp",
      "jsondata": true,
      "firstKey": "state",
      "onValue": "ON",
      "offValue": "OFF"
      }]

Then Homepoint could read state directly from HA Z2M integrated lights via standard MQTT messages like this below:

Zigbee2MQTT:info 2021-01-04 23:57:45: MQTT publish: topic 'zigbee2mqtt/Office bed lamp', payload '{"brightness":251,"color":{"x":0.491,"y":0.39},"linkquality":105,"state":"ON","update":{"state":"idle"},"update_available":false}'

That would be awesome and much easier without the need for extra automation ! 😎

For now, I use following Home Assistant automation to push all states changes (for a list of MQTT light entities) to Homepoint on topic zigbee2mqtt/LIGHTS_FRIENDLY_NAME/get with payload state

- id: Homepoint push light state via MQTT
  alias: Homepoint push light state via MQTT
  mode: parallel # Parallel mode needed as several lights in group can be turned on/off simultanously
  trigger:
    platform: state
    entity_id:
    - light.0x680ae2fffe924460_light # Define your Z2M light entities here
    - light.0x90fd9ffffe838cb3_light # Define your Z2M light entities here
  condition:
    condition: template
    value_template: >
      {{ trigger.to_state.attributes.state !=
          trigger.from_state.attributes.state }}
  action:
  - service: mqtt.publish
    data_template:
      topic: 'zigbee2mqtt/{{trigger.to_state.attributes.friendly_name}}/get'
      payload: "{{ trigger.to_state.attributes.state }}"  

Happy New Year ! 🎉🚀

Ciao !

[sieren]

Awesome! This is super helpful for me, as some people asked fo Zigbee2MQTT Integration in the past (#71) but sadly my knowledge wasnt sufficient. Your post is a great intro.

One thing though, I suppose jsondata should be used consistently for both the Set and Get topics?
At least looking at the Zigbee2MQTT Documentation theres two ways to set a topic, either as json to zigbee2mqtt/FRIENDLY_NAME/set or directly to zigbee2mqtt/[FRIENDLY_NAME]/set/state.

[htvekov]

Hi' @sieren

No worries ! 😎
I'll gladly help and test for you if needed.
I would also love to see and enjoy using Homepoint with improved Z2M integration 😉

And, as you write, I would also suggest consistant use of JSON messages for Z2M integration as this is the 'standard'.
For most devices you can get/set the device state as/with 'raw' payloads. But using JSON messages there's no limitation.
Then it will also be much easier, if you're in future would like to expand and improve further on Homepoint Z2M integration.
You can set/get all allowed device attributes using JSON key/payload. I believe that Z2M is payload upper/lowercase tolerant. Any combination will work AFAIK.

One final suggestion to this Z2M integration. On startup, Homepoint should cycle through all Z2M devices and sync device status by issuing a get command with an empty payload for the device specified key.

Get specified key payload (well, in fact all payloads are returned):

topic: zigbee2mqtt/[FRIENDLY_NAME]/get
payload: {"state": ""}

Set specific attribute:

topic: zigbee2mqtt/[FRIENDLY_NAME]/set
payload: {"state": "[ATTRIBUTE]"}

Ciao ! 🙋‍♂️

[htvekov]

EDIT: Revised HA light automations to handle switches as well as light entities.

Hi' @sieren.

I've mocked up some Home Assistant automations that via MQTT will provide basic HomePoint integration with Home Assistant entities - Lights, Switches (turn on/off).
All other HA entities (sensors, binary sensors, zones, persons etc.) can also be pushed to HomePoint to be displayed.

Also made the one HA automation for Z2M light needed, to provide full integration for these as well.
This Z2M automation will be obsolete, when you close this issue 😉😁

Didn't really known where and if you would want such examples on your Homepoint Github page ?
So instead of a PR I've added my setup examples and HA automations below.
If you wish, you're more than welcome to copy and paste from this (I have both Z2M and Hue light entities in my Home Assistant setup 😉)

Wish you a lovely weekend! 🍻😎

Homepoint example with Z2M & Home Assistant entities:

Please note that Home Assistant entities are entered with [entity_id] in topic NOT [friendly_name] !
Entity payloads are in lower case ('on' or 'off').
Entity topics used:
homepoint/[entity_id]/set
(valid for switch and light domain entities)
&
homepoint/[entity_id]/get
(valid for all entities with a state that makes sense to display 😉 No domain checks are done in automations)

{
  "wifi": "my_ssid",
  "password": "my_password",
  "hostname": "HomePoint ILI9341",
  "login": "my_login",
  "webpass": "my_password",
  "mqttbroker": "mqtt://192.168.X.XX",
  "mqttusername": "my_username",
  "mqttpasswd": "my_password",
  "timezone": "CET-1CEST-2,M3.5.0/02:00:00,M10.5.0/03:00:00",
  "screenSaverMinutes": 10,
  "displayColorInverted": false,
  "ledPinPullup": true,
  "screenRotationAngle": 1,
  "scenes": [{
    "name": "Office",
    "type": "Light",
    "icon": "livingroom",
    "devices": [{
      "name": "Ceiling",
      "setTopic": "homepoint/light.kontor_loft/set",
      "getTopic": "homepoint/light.kontor_loft/get",
      "onValue": "on",
      "offValue": "off"
    },
    {
      "name": "Desk",
      "setTopic": "zigbee2mqtt/Office desk lamp/set",
      "getTopic": "zigbee2mqtt/Office desk lamp/get",
      "onValue": "ON",
      "offValue": "OFF"
    },
    {
      "name": "Bed lamp",
      "setTopic": "zigbee2mqtt/Office bed lamp/set",
      "getTopic": "zigbee2mqtt/Office bed lamp/get",
      "onValue": "ON",
      "offValue": "OFF"
    }]
  },
  {
    "name": "Kitchen",
    "type": "Light",
    "icon": "kitchen",
    "devices": [{
      "name": "Kitchen",
      "setTopic": "homepoint/light.kokken/set",
      "getTopic": "homepoint/light.kokken/get",
      "onValue": "on",
      "offValue": "off"
    }]  
  },
  {
    "name": "Terrasse HA",
    "type": "Switch",
    "icon": "garden",
    "devices": [{
      "name": "Terrasse",
      "setTopic": "homepoint/switch.terrasse_lys/set",
      "getTopic": "homepoint/switch.terrasse_lys/get",
      "onValue": "on",
      "offValue": "off"
    }]
  },
  {
    "name": "Terrasse MQTT",
    "type": "Switch",
    "icon": "garden",
    "devices": [{
      "name": "Terrasse",
      "type": "singleValue",
      "setTopic": "terrasse_lys/cmnd/POWER",
      "getTopic": "terrasse_lys/cmnd/POWER",
      "onValue": "ON",
      "offValue": "OFF"
    }]
  },
  {
    "name": "Office lux",
    "type": "Sensor",
    "icon": "power",
    "devices": [{
      "name": "Kontor lux",
      "type": "singleValue",
      "firstIcon":"wattage_small",
      "jsondata": true,
      "firstKey": "illuminance_lux",
      "getTopic": "zigbee2mqtt/Office lux"
    }]  
  },
  {
    "name": "Greenhouse",
    "type": "Sensor",
    "icon": "door",
    "devices": [{
      "name": "Greenhouse",
      "type": "combinedValues",
      "firstIcon": "temperature_small",
      "secondIcon": "humidity_small",
      "jsondata": true,
      "firstKey": "Temperature",
      "secondKey": "Humidity",
      "getTopic": "tasmota_46FEAB/tele/SENSOR"
    }]  
  },
  {
    "name": "Outdoor",
    "type": "Sensor",
    "icon": "door",
    "devices": [{
      "name": "Outdoor",
      "type": "combinedValues",
      "firstIcon": "temperature_small",
      "secondIcon": "humidity_small",
      "jsondata": true,
      "firstKey": "temperature",
      "secondKey": "humidity",
      "getTopic": "zigbee2mqtt/Outdoor"
    }]  
  }]
}

First Home Assistant automation is needed to complement HomePoint <--> Z2M integration.

Last two automations handles Homepoint integration with Home Assistant entities.

In both 'publish HA state' automations you need to enter all your light/switch entities/groups that you use in HomePoint.

# We only need to publish HA state changes to HomePoint, as HomePoint publish changes to Z2M bulbs directly 
- id: HomePoint - Publish HA Z2M light state change to HomePoint via MQTT
  alias: HomePoint - Publish HA Z2M light state change to HomePoint via MQTT
  mode: parallel                      # Use parallel as numerous automations could be running simultanously
  trigger:
    platform: state
    entity_id:
    - light.0x680ae2fffe924460_light  # Enter you Z2M light entities that are used in HomePoint
    - light.0x90fd9ffffe838cb3_light  # Homepoint topic for Z2M entities is [friendly_name] NOT [entity_id] !
  condition:
    condition: template
    value_template: "{{ trigger.to_state.attributes.state !=\n    trigger.from_state.attributes.state\
      \ }}\n"
  action:
  - service: mqtt.publish
    data_template:
      topic: zigbee2mqtt/{{trigger.to_state.attributes.friendly_name}}/get  # topic: zigbee2mqtt/[entity_id]/get
      payload: '{{ trigger.to_state.attributes.state }}'  # payload is UPPER case ('ON' or 'OFF')

# Supports all Home Assistant light & switch entities
# Also supports to publish other Home Asssistant entity states to HomePoint (sensors/binary_sensors/zones etc.)
- id: HomePoint - Publish HA light/switch state change to HomePoint via MQTT
  alias: HomePoint - Publish HA light/switch state change to HomePoint via MQTT
  mode: parallel        # Use parallel as numerous automations could be running simultanously
  trigger:
    entity_id:
    - light.kontor_loft # Enter your HA entities/groups that are used in HomePoint
    - light.kokken     # Homepoint topic for HA entities is [entity_id] NOT [friendly_name] !
    - switch.terrasse_lys
    - sensor.media_source
    platform: state
  condition:
    condition: template
    value_template: "{{ trigger.to_state.state !=\n    trigger.from_state.state }}\n"
  action:
  - service: mqtt.publish
    data_template:
      topic: homepoint/{{trigger.to_state.entity_id}}/get  # topic: homepoint/[entity_id]/get
      payload: '{{states(trigger.entity_id)}}'  # payload is lower case ('on' or 'off')

# Supports all Home Assistant light & switch entities
- id: Homepoint - Push state change from HomePoint device to light/switch via HA service call
  alias: Homepoint - Push state change from HomePoint device to light/switch via HA service call
  mode: parallel                # Use parallel as numerous automations could be running simultanously
  trigger:
    platform: mqtt
    topic: homepoint/+/set   # Note: Will trigger on ALL entities used in topic: homepoint/*/set !!
  action:                       # Turn light on/off depending on payload received
    service_template: "{% if trigger.payload == 'on' %}\n  homeassistant.turn_on\n{% else\
      \ %}\n  homeassistant.turn_off\n{% endif %}\n"
    data_template:
      entity_id: '{{ trigger.topic.split(''/'')[1] }}'  # Get entity_id from topic

[fastender]

EDIT: Revised HA light automations to handle switches as well as light entities.

Can I understand this in such a way that, for example, an alarm panel can be designed that passes the commands to HA, e.g. via M5Stack Core2 or better M5Paper?

[htvekov]

Hi' @fastender.

Well, Matts HomePoint user interface has clearly been developed for a specific use purpose - To handle simple entities as lights/switches/sensors and display their status. And does actually all that in quite a convincing, no nonsense way 🙂

HomePoint could (hopefully) develop into a more complex userinterface. It certainly has the potential.
But it would require further development involving more configurable tile/button sizes and types, text formatting/text size etc. to really expand the user possibilities. But the basic framework is already in place. So who knows... 😉

Homepoint could, as is, act as a simple alarm panel. But not with a full keypad for entering numbers and fancy graphics etc.
If you search for a stand alone alarm panel for M5Stack, I would look into this specific RFID alarm panel development. There are other similar alarm panel project 'out there'. Perhaps you'll even find a project for the new core2 or Epaper version. But I haven't found any - yet.

Ciao !

[sieren]

@htvekov I'm confused by your latest comment.
So there are two ways of accessing exposed MQTT Topics through HA, depending on if they're defined as friendly or as an entity and each one of them has the same payload structure but one with all caps the other one without?

The reason im asking is, this is so much custom stuff, I'm thinking of just adding a "Z2M": true property to each device that would send the payload accordingly (kind of a "Zigbee2MQTT" Mode)
Seeing most of the documentation I read comes from Zigbee2MQTT I'm confused about HAs role in all of this.

Update: Looking at the documentation again, doesnt using the "friendly_name" cover most of the cases and reduce complexity? While I want to support Zigbee2MQTT and HA, I have to keep some parts of HomePoint fairly generic to make it work with other tools, which would mean additional configuration on the user side.

[htvekov]

Hi' @sieren

Well, I'll admit it's a little confusing.

First of all: Adding a specific integration property for each device, will most likely solve many (unforeseen) future issues.
I'll definitely vote for that 👍🙂

Z2M uses Z2M device friendly_name as identifier for all commands. And AFAIK doesn't use the device ieee_address at all.
Issue herin lies in the fact that if no Z2M friendly_name has been specified in Z2M configuration file devices.yaml, then friendly_name equals device ieee_address. Z2m friendly_name can at all times be changed in numerous ways. Through commands or editing devices.yaml file directly.

Z2M devices.yaml example:

# Lights

'0x680ae2fffe924460':
  friendly_name: Office desk lamp

Depending on settings and at what point Home Assistant autodiscovers the exposed Z2M devices, HA's entity_id could be domain.ieee_address with a possible preceding domain indication at the end (_light) or Z2M friendly_name if entity is autodiscovered by HA after Z2M has initialized the Z2M friendly_name.

Same Z2M device as entity in HA:

But as both HA's entity_id and HA's friendly_name can be altered at any time, this can end up being extremely confusing. Especially for HA/Z2M newcomers !! 🙄 It's quite hard for new HA users to grasp, that these two different friendly_names could be identical, but not neccesarily are identical...

Hope all above makes any sense at all, Matt ?
I'm just a 'tinkerer' and not even near any kind of serious level when it comes to HA and/or Z2M.
So for a non HA user, I imagine that it can be somewhat difficult to understand my attempt to describe the difference.

So to wrap this up I would suggest to keep this as simple as possible.

• Z2M commands only addresses Z2M friendly_name and can't be altered.

• I could rewrite my two HA automations (that exposes state change of HA entities to HomePoint via MQTT and vice versa) to use HA entity friendly_name and not the HA device_id when communicating with HomePoint. But the automation trigger entities in HA can't be altered. It has to be the HA entity_id. That's why I kept it at entity_id as identifier for HA entities at both HA and Homepoint's end.

Suggestions, Matt ? 😉

Ciao !

[sieren]

Picked up works on this.

Here's my config idea.
Since the Payload coming from Homepoint is entirely different to its prior primitive values and needs to be in a constructed JSON Format I might as well introduce a new Mode for these devices.
In this particular case I'd just call it "Z2M" as it's catered towards Zigbee2MQTT, using the Friendly Name.

I suppose while I am doing so I can add a second service called "HAMQTT" that is specific to how HomeAssistant Publishes it's own automations through the entity system (if I understood correctly). This would require the lower-case state updates as opposed to the uppercase ones of Z2M.

Would that make sense?

Zigbee2MQTT

  {
    "name": "Some Device",
    "type": "Switch",
    "devices": [{
      "name": "Lightbulb",
      "mode": "Z2M",
      "topic": "zigbee2mqtt/[FRIENDLY_NAME]/",
    }]
  },

HomeAssistant

  {
    "name": "Some Automation Thing",
    "type": "Switch",
    "devices": [{
      "name": "Automation",
      "mode": "HAMQTT",
      "topic": "homeassistant/[ENTITY_ID]/",
    }]
  },

[htvekov]

Hi' Matt.

I think it's a sensible idea to separate the different 'integrations' into different modes.
To keep it simple and understandable for all 👍

It's not really a Z2M/HA integration more likely a 'representation' of the device/entity on the HomePoint screen.
But dealing with simple binary devices can easily be acieved this way and need not much else.
Directly with z2m via set/get command topics and with small template automation in HA.
Upper/lower case is really not an issue. Z2M accepts both upper and lower case. Even mixed upper/lower case.
HA will with my automation push the HA entity state, which AFAIK always is lower case. So it merely out of simplicity that I made automation that way.

If you at some point have the urge to develop 'real' HA integration, the easiest method imo is through HA MQTT autodiscovery.
I actually don't think that it's especially difficult.
As a start you could make the actual HomePoint device HA autodiscoverable and present device, screen, connection etc. status.

The full blown HA integration would be somewhat more of a challenge.
Here you would have to expose the defined buttons as such as a form of object with specific characteristics. Could be button type toggle, press type (short, long or hold), button identifier (entity, device name), object type (group or single device), object state etc.

With this sort of implementation, the individual buttons would become entities in HA. And you cold tie state changes for these buttons up in automations to do whatever. But some of the simplicity would be gone doing just that.
If demand is only there for simple tasks with binary devices, current metod is the easiest to set up.
But if you plan on evolving to different object types with sliders for brightness and color temp, color pickers different sized buttons then 'real' HA implementation would probably be needed.
I'm afraid my knowledge on the subject is far from enough to advise on how this should be approached.

Ciao !

[cbrightly]

This sounds like what HASwitchPlate accomplished with their code. https://github.com/aderusha/HASwitchPlate. Is this the type of thing that you are talking about?

[sieren]

Yeah, @cbrightly is right. Maybe that's the better fit?
I'll continue working on the minimal solution proposed above though. After all this is a general purpose project, not just something for HA exclusively, which makes it a bit harder to find the right fit for everyone.

[htvekov]

Hi' @sieren

I totally agree. Keep HomePoint as the simple solution for basic tasks.
The best thing about this approach is that setup will remain dead easy - even with HA via simple template automations.

@fvanroie has practically rewritten aderusha's entire original hasp code and made a fully object oriented touchscreen display UI - hasp-lvgl
This utilized MQTT as well and is based on the lvgl graphics library. With this the need for special Nextion display hardware is gone. Many standard 'of the shelf' display can be used.

So no need to go in the same direction and do exactly the same as aderusha and fvanroie has done.

Ciao !

[sieren]

Ahh, thanks for the pointer!
I've been looking at LVGL for ages for a Homepoint rewrite. Maybe it's time soon.

[htvekov]

Hi' Matt.

That library is simply awesome...
Really fast on an ESP32 and has many (30+) predefined widgets that can be tweaked in numerous ways.
It's hard to believe how much actually can be achieved using this library with only a 10-15 Euro investment in a ESP32 board and a touch screen.

This library will most likely push most future display projects towards touch screens.

I'll definitely vote for that library if you're to rewrite HomePoint for a graphics library update 👍

Ciao !

[htvekov]

👍🎉⚡🚀😎🙋‍♂️