Help Please! ZWave very weird behavior.

[GrizzlyAK]

I noticed today in Home Assistant that my Zooz ZSE19 Siren was unavailable so I changed the batteries, still dead/unavailable. Then I noticed that three other devices (two critical and one important), all powered, were also showing Dead. These latter three were ALL Fortrezz products, two MIMOLites and one Water Valve. Two control my high water alarm and my water main valve, and the third controls the garage door. The first two are pretty critical. Did something change with the treatment of Fortrezz in ZWave-JS? I can't imagine why all three of these devices (my only Fortrezz devices) are suddenly reporting dead. They've been working perfectly since mid-2020. Clicking on the water valve device produced the following error:

Why would this device no longer be provided?

I was on 2202.9.0, so I upgraded to 2022.9.5 and OS 9.0, but that didn't solve the issues. So I completely shutdown my host (Pi4) and after a reboot, these 4 devices are still dead. They show no configuration in the ZW2M Control Panel, almost like they've been wiped from ZW completely. Interviews just spin on ProtocolInfo, and all four are now listed as Unknown Manufacture and Unknown Product with "undefined" FW. I don't know if this is a related issue, but I also recently noticed my two Zooz ZSE20 Outdoor Motion Sensors were no longer reporting Battery Level (both showed 0% even after installing new batteries in both), although both report motion and Lux correctly.

Finally, another device, a Zoz ZEN25 Double Plug ALSO now shows dead. It has been work perfectly until today, yesterday, idk. Heal Node failed and Reinterview spins on ProtocolInfo just like the rest. It's configuration is missing as well.

In the matter of days, my ZWave network seems to have gone berserk. Any advice would be appreciated as to how to troubleshoot and/or resolve.

zwave-js: 10.1.0
zw2m: 7.2.0

[AlCalzone]

Any advice would be appreciated as to how to troubleshoot and/or resolve.

https://zwave-js.github.io/node-zwave-js/#/troubleshooting/index?id=rule-1-use-the-driver-log-luke

[GrizzlyAK]

It appears the problem with one of the MIMOLites (high water alarm) became an issue, went offline, immediately following an upgrade from 2022.8.6 to 2022.9.0, based upon my history. I don't have ZWave node history for the other devices. So the other two Fortrezz devices may have gone missing at the same time, I can't be sure at this point. I'll take a look at the zwave logs. Has anything changed that would have affected Fortrezz devices specifically?

[GrizzlyAK]

Also, history for the Zooz ZSE19 Siren shows that Temp and Humidity data flat lined on Sep 5, following the update to 2022.9.0, and which the change log shows that there was a config file change for the ZSE19 at that time (#5008). I know the Fortrezz devices are old zwave and have not been updated by the manufacturer, so some of these recent changes to zwave may have made them inoperable?

That's a lot of coincidence.

[AlCalzone]

Again: Speculation doesn't help troubleshoot. Logs do. Please re-interview problematic devices and make a driver log (loglevel "debug") of the failed re-interviews.

[GrizzlyAK]

I've restored back to a full backup of 2022.8.6 made on Sept 4 at 11:37PM, just before upgrading to 2022.9.0 on Sept 5, when the outages shown in the history reports above began. Although a re-interview of the three Fortrezz devices failed, the Manufacturer, Device names and FW version were now correct (instead of unknown as in the latest version). The Configuration is still empty, I assume because the interview failed. I think that was server version 10.0.3.

I restored back further to a stable full backup of 2022.08.1 and OS 8.4 taken on Aug 6. The behavior was the same as the previous restore where the interview failed, but the device name info was correct. Also of note is that there is a grey ? in the Zwave+ column for the three Fortrezz devices, which are the only devices that don't show either a 1 or a 2. (This is in the ZW2M Control Panel).

[GrizzlyAK]

Again: Speculation doesn't help troubleshoot. Logs do. Please re-interview problematic devices and make a driver log (loglevel "debug") of the failed re-interviews.

Yeah, I know, I'm trying to establish a timeline. These three have been working for two years, and it appears that with an update, they all three stop working? I've just restored back to the current version and will capture some logs.

[AlCalzone]

Its very likely the upgrade -> #4994

[GrizzlyAK]

Here is the entries following a re-interview request Node 13:

2022-09-20T11:45:13.805Z CNTRLR   [Node 013] [+] [Node Naming and Location] name: "Fortrezz MIMOLite Garage Cont
                                  rol"
2022-09-20T11:45:13.809Z CNTRLR   [Node 013] [+] [Node Naming and Location] location: "Garage"
2022-09-20T11:45:13.813Z CNTRLR   [Node 013] Beginning interview - last completed stage: None
2022-09-20T11:45:13.814Z CNTRLR   [Node 013] new node, doing a full interview...
2022-09-20T11:45:13.818Z CNTRLR » [Node 013] querying protocol info...
2022-09-20T11:45:13.841Z SERIAL » 0x010400410db7                                                       (6 bytes)
2022-09-20T11:45:13.843Z DRIVER » [REQ] [GetNodeProtocolInfo]
                                    payload: 0x0d
2022-09-20T11:45:13.848Z SERIAL « [ACK]                                                                   (0x06)
2022-09-20T11:45:13.851Z SERIAL « 0x01090141d39c00041000ed                                            (11 bytes)
2022-09-20T11:45:13.855Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:13.856Z DRIVER « [RES] [GetNodeProtocolInfo]
                                    payload: 0xd39c00041000
2022-09-20T11:45:13.867Z CNTRLR « [Node 013] received response for protocol info:
                                  basic device class:    Routing Slave
                                  generic device class:  Binary Switch
                                  specific device class: Unused
                                  node type:             End Node
                                  is always listening:   true
                                  is frequent listening: false
                                  can route messages:    true
                                  supports security:     false
                                  supports beaming:      true
                                  maximum data rate:     40000 kbps
                                  protocol version:      3
2022-09-20T11:45:13.873Z CNTRLR   [Node 013] Interview stage completed: ProtocolInfo
2022-09-20T11:45:13.873Z CNTRLR » [Node 013] pinging the node...
2022-09-20T11:45:13.883Z SERIAL » 0x010800130d0100258449                                              (10 bytes)
2022-09-20T11:45:13.885Z DRIVER » [Node 013] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      132
                                  └─[NoOperationCC]
2022-09-20T11:45:13.893Z SERIAL « [ACK]                                                                   (0x06)
2022-09-20T11:45:13.896Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2022-09-20T11:45:13.897Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:13.898Z DRIVER « [RES] [SendData]
                                    was sent: true
2022-09-20T11:45:16.177Z SERIAL « 0x011a0004005f129f039b00c1f93d87ba3c2b32f67e0d3d06e0be0056          (28 bytes)
2022-09-20T11:45:16.186Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:21.968Z SERIAL « 0x011a0004005f129f039b00c1f93d87ba3c2b32f67e0d3d06e0c90021          (28 bytes)
2022-09-20T11:45:21.973Z DRIVER   Dropping message with invalid payload
2022-09-20T11:45:21.974Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:25.954Z SERIAL « 0x01180013840104b6007f7f7f7f7f0000070000000002080000b1              (26 bytes)
2022-09-20T11:45:25.967Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:25.969Z DRIVER « [REQ] [SendData]
                                    callback id:            132
                                    transmit status:        NoAck, took 12060 ms
                                    routing attempts:       8
                                    protocol & route speed: Z-Wave, 40 kbit/s
                                    TX channel no.:         0
2022-09-20T11:45:26.008Z CNTRLR   [Node 013] The node did not respond after 1 attempts, it is presumed dead
2022-09-20T11:45:26.026Z CNTRLR   [Node 013] ping failed: Failed to send the command after 1 attempts (Status No
                                  Ack) (ZW0204)
2022-09-20T11:45:26.027Z CNTRLR » [Node 013] querying node info...
2022-09-20T11:45:26.029Z CNTRLR » [Node 013] pinging the node...
2022-09-20T11:45:26.046Z SERIAL » 0x010800130d0100258548                                              (10 bytes)
2022-09-20T11:45:26.048Z DRIVER » [Node 013] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      133
                                  └─[NoOperationCC]
2022-09-20T11:45:26.051Z SERIAL « [ACK]                                                                   (0x06)
2022-09-20T11:45:26.059Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2022-09-20T11:45:26.060Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:26.061Z DRIVER « [RES] [SendData]
                                    was sent: true
2022-09-20T11:45:37.980Z SERIAL « 0x01180013850104a8007f7f7f7f7f0000070000000002080000ae              (26 bytes)
2022-09-20T11:45:37.986Z SERIAL » [ACK]                                                                   (0x06)
2022-09-20T11:45:37.990Z DRIVER « [REQ] [SendData]
                                    callback id:            133
                                    transmit status:        NoAck, took 11920 ms
                                    routing attempts:       8
                                    protocol & route speed: Z-Wave, 40 kbit/s
                                    TX channel no.:         0
2022-09-20T11:45:37.998Z CNTRLR   [Node 013] The node did not respond after 1 attempts, it is presumed dead
2022-09-20T11:45:38.003Z CNTRLR   [Node 013] ping failed: Failed to send the command after 1 attempts (Status No
                                  Ack) (ZW0204)
2022-09-20T11:45:38.004Z CNTRLR   [Node 013] Interview attempt (1/5) failed, node is dead.

[AlCalzone]

I'll chalk it up to magic.

You'll still want to make sure the voodoo radio signals are strong enough, i.e. do the health check to avoid such surprises in the future.

[GrizzlyAK]

You'll still want to make sure the voodoo radio signals are strong enough...

You'll have a hard time convincing this physicist that this was a radio signal issue. More likely an update that left my three least capable ZWave devices (not even ZWave+) stranded in lala land, until a power cycle or two, or perhaps new CC comms handling that is more strict than before since, I suspect, in a multicast environment, if a device hears something it thinks is not meant for it, it won't respond. Or maybe it was related to recent power outages. But, radio signal strength... nah. Zero evidence supports that theory.

But of course, we can agree to disagree on that. 😉

[AlCalzone]

But, radio signal strength... nah. Zero evidence supports that theory

Well you posted a log with evidence. Do you want to read the specs where the transmit statuses are explained?

CCs are simply payload for a lower transport layer.

Of course a device may have been in a weird state due to power outages. But from the controller perspective it's the same - no acknowledgement from the device came back. In 99% of the cases I read, that's an issue with the radio signal.

[GrizzlyAK]

Well you posted a log with evidence. Do you want to read the specs where the transmit statuses are explained?

The logs showed an absence of acknowledgement, and I proposed two potential reasons for that. Plus, a lack of Ack in no way confirms that the signal didn't reach the device, it simply did not respond. Period. There is no, well, I kinda heard a reply, but I can't be sure kind of response, especially if the controller tried several different node paths as you indicated it did. That last bit alone, to me, is more of an indication of a device that is really dead, or really confused as to its own identity, which I believe was the case here.

In 99% of the cases I read

Maybe so, but 99% isn't 100%, especially given (1) that all these devices have worked flawlessly for two years prior and, (2) I've made zero mods to anything, anywhere, that would impact voodoo (except update HA) and, (3) they are again working flawlessly in the same locations as before. Logic tells me that this was not a radio signal strength issue. And it's not like these devices only work half the time, either.

Also, just as an FYI, since I began using HA over two years ago, I regularly have to reboot my Pi at least twice following some upgrades in order for things to come back up working correctly. Not sure who is at fault there, but it has become SOP for me if any one device is not reporting in. That, or cycling the power of the device, usually takes care of it.

Look, mate, I'm not blaming you for anything here. It appears that v10 was a pretty major upgrade, so some glitches are inevitable given the wide array of devices being supported. ZWave-JS is awesome and absolutely critical to my HA install. I just think there are potential insights to be gained here other than just chalking it up to a little rogue sheetrock and wiring. Cheers.

[AlCalzone]

since I began using HA over two years ago, I regularly have to reboot my Pi at least twice following some upgrades in order for things to come back up working correctly

Have you tried pulling and re-plugging the stick instead? Not sure what happens with them when HA updates, but it seems to be a fairly frequent complaint and I never had to do it (not using HA or Containers).
Ideally zwave-js would restart the stick (soft reset), which does the same, but this is disabled out of the box for most setups since it causes the connection to the stick to get lost when the host isn't configured correctly.

As for the NoAck thing, I appreciate you trying to help gain some insight, but you'll have to trust me that nothing that changed in v10 (or that the host software can do) can cause the acknowledgements not to be received.