[hue] Improve support for third party lights

[andrewfg]

This PR resolves many issues with "exotic" light models. In this context "exotic" mostly means lights made by third parties other than Philips / Signify ("Friends of Hue") which are therefore not certified for full compliance with the Hue API v2. Albeit it covers a special case with the Philips / Signify "Signe" model as well.

For testing purposes the JAR is here:

• org.openhab.binding.hue-5.3.0-SNAPSHOT.zip

Specifically this PR does the following:

1. In the prior version of the binding code I had wrongly assumed that the min .. max range of the dimming channels were "minimum dimming level .. 100%). This PR refactors the code so the min .. max range of the dimming channels are now "0% .. 100%" regardless of the actual minimum dimming level.

2. Therefore, this PR changes the threshold for a light being deemed ON from brightness >= minimum dimming level to brightness > 0.0.

3. Some non certified lights fail to provide a minimum dimming level anyway - see [hue] IKEA Trådfri bulb turns off when setting low brightness #21057 So with 2. above, this PR resolves this issue.

4. Some certified lights do provide a minimum dimming level of 0.0 (e.g. Philips Signe) and currently therefore sending 0.0 to the bulb fails to turn it off. So with 2. above (changing b >= 0.0 check to bri > 0.0) this PR ensures that a precise 0.0 command will turn the lamp off.

5. Some non certified lights fail to provide minimum and maximum supported Mirek values. So analogous to the case with dimming level above, this PR defaults to the min .. max range of 153Mk .. 500Mk.

6. Previously the binding did not use the Gamut provided by certified lights. It always used the default Gamut instead. As a general rule Philips / Signify lights do provide a Gamut, whereas non certified products generally do not. With this PR now we use the Gamut if provided, and only use the default if none is provided.

7. Some non certified lights provide invalid minimum and maximum supported Mirek values. So this PR checks those values and if necessary defaults to the min .. max range of 153Mk .. 500Mk.

8. As a consequence of the above changes, the thing handler calls a new extra added method that ensures the light state cache contains all the required parameters for mirek schema, and gamut -- be they provided by the device itself in its initial GET response, or otherwise using the system defaults.

9. Often non certified lights will respond to a PUT request with an HTTP 207 error rather than HTTP 200. In the past, when the bridge returned an HTTP 207 response "soft off" the binding would produce an INFO log output. This PR reduces that to DEBUG level.

10. On some non certified lights, when the bridge returned an HTTP 207 response, the devices would not send an SSE event update with the new state. This PR adds a loopback whereby the JSON of the PUT request that caused the 207 response is looped back to the binding as a dummy event. This prevents the states in the UI from "flapping".

11. As a consequence of my misunderstanding about the min .. max range of the dimming channels, the binding was incorrectly calculating the threshold to be used when determining the changeover point between sending a hard on or a hard off command. This applied in particular to non certified products that do not provide their own minimum dimming level. This PR adds an extra Setters method that fixes the threshold and implements the hard on/off logic.

12. The prior version of the binding code would wrongly produce Channel State values of UNDEF in some cases when certain DTO fields were missing. Again in particular it occurred for non certified products. In this PR some UNDEF cases are eliminated, and instead an exception may be thrown at runtime (see next point).

13. .. therefore, in this PR now when a device delivers a DTO with incomplete data, rather than producing the opaque UNDEF state, the binding now logs the detailed error with a full stack trace instead. The extra transparency should help fixing future non certified light models.

14. In order to test the above changes I purchased both a Lidl Livarno TS0502A E27 CT bulb that should be similar to the TS0505A of @kaikreuzer and also an Ikea Tradfri 1055 lumen E27 bulb that should be similar to the Tradfri lamp of @jlaur, for testing the code. And as far as I can tell the errors reported in the below mentioned issues have now been resolved.

15. It seems that the (also non certified) "Ikea Tradfri" bulbs suffer the same bug as the "LK Wiser Dimmer". So this PR extends the same fix to those Tradfri bulbs.

16. Furthermore, if a room or zone contains such a Tradfri or LK Wiser light, then the bug also occurs on the respective room or zone. So this PR applies the same fix to such things. It also applies the fix to the "all lights zone".

17. Some older (legacy) lights operate in a dual mode manner. Either they are in 'CT' mode, or 'XY' mode. By contrast the API v2 is designed to operate in a mode-less manner. In such mode-less operation the bridge and respective lights provide notifications containing both colour temperature and color xy values which are simultaneously valid. Whereas older legacy lights provide notifications where EITHER the colour temperature OR the color xy value is valid. And such notifications may be interleaved, thus causing flapping between channel states. So this PR introduces a new LegacyLightState class that implements a state machine to model the the operating mode of such lights, and apply a work-around to processes the notifications to only present values that belong to the correct legacy mode.

18. The Junit tests have been adapted to the above. And a new test class has been introduced to test the LegacyLightState state machine class.

Resolves #21057
Resolves #21044
Resolves #21045
Resolves #21043
Resolves #21011

Signed-off-by: Andrew Fiddian-Green software@whitebear.ch

[jlaur]

Do you happen to have any Hue lights with minimum dimming level >0.01?

I already found that I have one. I will verify this with API v2 later.

[jlaur]

I already found that I have one. I will verify this with API v2 later.

Initial GET when on at lowest brightness:

      "dimming": {
        "brightness": 0.0,
        "min_dim_level": 5.0
      },

PUT:

{
    "on": {
        "on": true
     },
    "dimming": {
      "brightness": 0.19
    }
}

Same result (still 0.0), so rounded down.

PUT:

{
    "on": {
        "on": true
     },
    "dimming": {
      "brightness": 0.20
    }
}

Now GET:

      "dimming": {
        "brightness": 0.4,
        "min_dim_level": 5.0
      },

So it looks like it supports the full range 0-100, rather than 5-100. And the same should be true in openHAB modelling, so actually it looks like it can be mapped directly without using the minimum dimming level as any kind of brightness-related threshold. The only exception is that 0 equals OFF in openHAB, so this particular value must be converted to e.g. 0.01.

[andrewfg]

@jlaur I think my only resistance against using on = brightness > 0.0 is the potential risk of rounding artifacts for brightness values close to zero; that is the reason why I prefer on = brightness > X where X is a number slightly above 0.0

[jlaur]

So it looks like it supports the full range 0-100, rather than 5-100. And the same should be true in openHAB modelling, so actually it looks like it can be mapped directly without using the minimum dimming level as any kind of brightness-related threshold. The only exception is that 0 equals OFF in openHAB, so this particular value must be converted to e.g. 0.01.

I just want to add that I did one more test with the current PR code with this Hue bulb. In the UI, I controlled brightness with a slider, and as soon as I got below 5%, it turned off. I expected this to happen, but it shouldn't.

@jlaur I think my only resistance against using on = brightness > 0.0 is the potential risk of rounding artifacts for brightness values close to zero; that is the reason why I prefer on = brightness > X where X is a number slightly above 0.0

Understood. But perhaps you can create an example? I'm not sure where this rounding would occur. Out of interest I tried to find a few other implementations:

Hue API v1:

openhab-addons/bundles/org.openhab.binding.hue/src/main/java/org/openhab/binding/hue/internal/handler/LightStateConverter.java

Lines 114 to 130 in edcadfc

	/**
	* Transforms the given {@link PercentType} into a light state containing
	* the brightness and the 'on' value represented by {@link PercentType}.
	*
	* @param percentType brightness represented as {@link PercentType}
	* @return light state containing the brightness and the 'on' value
	*/
	public static StateUpdate toBrightnessLightState(PercentType percentType) {
	boolean on = !percentType.equals(PercentType.ZERO);
	final StateUpdate stateUpdate = new StateUpdate().setOn(on);
	int brightness = (int) Math.floor(percentType.doubleValue() * BRIGHTNESS_FACTOR);
	if (brightness > 0) {
	stateUpdate.setBrightness(brightness);
	}
	return stateUpdate;
	}

deCONZ:

openhab-addons/bundles/org.openhab.binding.deconz/src/main/java/org/openhab/binding/deconz/internal/handler/LightThingHandler.java

Lines 232 to 236 in edcadfc

	// send on/off state together with brightness if not already set or unknown
	Integer newBri = newLightState.bri;
	if (newBri != null) {
	newLightState.on = (newBri > 0);
	}

Govee:

openhab-addons/bundles/org.openhab.binding.govee/src/main/java/org/openhab/binding/govee/internal/GoveeHandler.java

Lines 225 to 228 in edcadfc

	if (command instanceof PercentType percent) {
	taskQueue.add(() -> sendBrightness(percent));
	command = OnOffType.from(percent.intValue() > 0); // fall through
	}

Maybe it's as simple as that?

[jlaur]

I still have issues with my Light Solutions Dimmer-Switch-ZB3.0 that I'll need to dig further into.

No issue after all. I just forgot that my Philips Master Dimtone bulb supports insanely low dimming, and it works perfectly well with this dimmer. So I was not able to see it turning on at lowest brightness when having other light sources in the room. Even at night with all other lamps off, I still had to look directly at the bulb, as even then it was hard to notice the lamp being on. 🙂

Shot with exposure time 1/3 s:

[andrewfg]

@jlaur in today's commit I have completely removed minimum dimming level. Except as a thing property (information only) for the dimming range.

[andrewfg]

@jlaur for info: I am also changing brightness >= 0 to brightness > 0 in the OH Core LightModel

[andrewfg]

@kaikreuzer / @jlaur gentle reminder: From my side the code is stable, and I think it addresses all your exotica. So I am just awaiting your tests results to confirm that.

[wborn]

This is an additional AI review.

The PR addresses a substantial number of compatibility issues with third-party Hue lights, and most of the earlier review feedback appears to have been addressed.

AI found two remaining issues in the new IncreaseDecreaseType handling:

1. The HTTP 207 loopback does not apply dimming_delta or color_temperature_delta to the cached absolute state. If the bridge returns 207 and does not send the expected SSE update, the openHAB channel therefore remains at its previous value.
2. The new delta handling changes the existing user-visible step size. Brightness INCREASE/DECREASE previously changed by 10 percentage points, but now uses a delta of 20. Color-temperature-percent commands also previously changed by 10 percentage points of the individual light's mirek range, while the new implementation uses a fixed 70 mirek delta. Unless this behavior change is intentional, the existing command semantics should be preserved.

There is also an existing unresolved review thread concerning the TRÅDFRI off-transition workaround and its effect on the normal Hue transition time, which should be resolved before merge.

The PR description should also be updated where it still describes minimum-dimming behavior that no longer matches the current implementation.

[wborn]

AI re-review

AI re-reviewed the current head 5bf1fffe387802a8dcd7e36964545db8c906d744 after the latest delta-command revert. The previous INCREASE / DECREASE findings have been addressed and are no longer applicable.

Changes are still requested for the following regression risks:

• The optimistic loopback is currently triggered for every successful response containing an errors entry, rather than only for the known benign attribute_may_have_no_effect case. This can publish a requested state even when an attribute was actually rejected.
• The recursive room/zone workaround detection adds per-child HTTP GETs during initialization and can repeatedly fetch the same devices through different room/zone/home hierarchies.
• The existing unresolved transition-duration concern still applies: #21062 (comment). The new room/zone propagation makes its scope broader because a single affected child can cause grouped OFF commands to include dynamics.duration = 0, changing the transition behavior of unaffected lights in the same group.

The updateLightCacheRequiredFields() documentation and corresponding PR description are also stale after the recent refactoring. The remaining delta DTO/classes and setters appear unused after the revert and could be removed as non-blocking cleanup.

The LegacyLightState implementation looks internally consistent, but targeted hardware regression testing is still advisable for CT → color, color → CT, repeated XY notifications, restart/power-cycle behavior, and commands while soft-off.

A human maintainer review is still needed after the issues above are addressed.

[jlaur]

A few very minor comments/questions in addition to the AI comments.

I have verified that #21057 is fixed and that the LK Wiser work-around still works.

I have not tested anything related to color temperature, i.e. that the other linked issues are fixed. Hopefully @kaikreuzer can confirm that the scenarios he experienced are now as expected.

[wborn]

AI re-review

AI re-reviewed the current head 388f53b63aa602fe86aebf9d3eeb912cf2ac8599.

The latest changes address the previous repeated per-child HTTP lookup concern and substantially improve the 207 error handling. The earlier INCREASE / DECREASE findings also remain addressed after those changes were reverted.

Changes are still requested for the three issues noted inline: mixed 207 errors can still trigger an optimistic loopback, fixed-width Mirek schemas can cause a zero-denominator percentage calculation, and the off-transition work-around ID set is concurrently rebuilt and read without safe publication.

The existing unresolved TRÅDFRI transition-duration concern still applies, and point 8 of the PR description should still be updated because updateLightCacheRequiredFields() now fills the Mirek schema and gamut, but not a minimum dimming level.

A human maintainer review is still needed after these issues are addressed.

[andrewfg]

The existing unresolved TRÅDFRI transition-duration concern still applies

I have no idea what it is talking about. So ignoring it.

[wborn]

I have no idea what it is talking about.

The existing unresolved TRÅDFRI transition-duration concern still applies

I have no idea what it is talking about. So ignoring it.

Now you are ignoring @jlaur 😉

#21062 (review)

[andrewfg]

Now you are ignoring @jlaur 😉
#21062 (review)

Well that is AFAIK resolved, so the AI was wrong.

[wborn]

AI re-review

The latest changes address the previously raised 207 handling, startup lookup, and workaround-set concurrency concerns.

One remaining correctness issue was found in the refresh of the OFF-transition workaround IDs: an unsuccessful refresh can still publish a partial result and discard the previous known-good state.

There is also one narrower question about the TRÅDFRI 1055l workaround. The device testing indicates that dynamics is required for combined on:false + dimming requests, while a plain on:false succeeds without it. The current implementation also adds dynamics to plain OFF requests. This may be intentional for grouped resources, so this is raised as a question rather than as a confirmed regression.