Persist pending re-interview requests in the application database#1852
Draft
TheJulianJES wants to merge 10 commits into
Draft
Persist pending re-interview requests in the application database#1852TheJulianJES wants to merge 10 commits into
TheJulianJES wants to merge 10 commits into
Conversation
When a Zigbee device boots after an OTA, it sends a `QueryNextImageCommand` with its running `current_file_version`. If that version differs from the previously cached value for this cluster, schedule `device.reinterview()` to rebuild endpoints, clusters, and quirks against the new firmware. This makes post-OTA recovery stateless: it works for slow-rebooting devices that miss the post-flash polling window in `update_firmware()`, devices flashed outside HA, and HA restarts during the reboot window.
Replace the fixed post-OTA read_attributes / image_notify / reinterview sequence with a confirmation step that resolves on whichever signal arrives first: - A successful active read of `current_file_version` (kept from the old flow as a probe for quietly-rebooting mains devices, but now best-effort: exhausted retries no longer abort the post-OTA steps). - A new `QueryNextImageCommand` whose `current_file_version` differs from the value the OTA cluster cached pre-flash. The OTA cluster's listener also schedules a re-interview when this happens. - A `Device_annce` from this device after the post-flash reboot. If none arrive within `POST_OTA_CONFIRMATION_TIMEOUT` (5 min), update_firmware() still returns SUCCESS — recovery is deferred to the version-change listener if the device wakes up later. After confirmation, a best-effort `image_notify` is sent so the device refreshes its OTA query cache, and a re-interview is driven directly unless the version-change listener already replaced this device with a re-interviewed one. This fixes the slow-reboot race where the read_attributes retry budget was exhausted before the device finished rebooting (e.g. Hue LLC020 ~102s), causing update_firmware() to throw and skip image_notify and reinterview. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
Switch the post-OTA wait's announce signal from a `device.zdo` listener for `device_announce` to an `application.add_listener` for `device_joined`. `device_joined` is the existing application-level event for new joins / NWK changes (including the Device_annce path), so we don't add a new ZDO-level listener. The version-change signal still covers same-NWK rejoins. Also adds a test verifying that a `device_joined` event for a different device does NOT resolve this device's wait.
This was referenced Jul 3, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #1852 +/- ##
==========================================
- Coverage 99.47% 99.47% -0.01%
==========================================
Files 57 57
Lines 12026 12167 +141
==========================================
+ Hits 11963 12103 +140
- Misses 63 64 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes “pending re-interview” requests durable across zigpy restarts by persisting the pending flag in the application database and restoring it on startup, so sleepy devices can be reliably re-interviewed on their next check-in after post-OTA reboots.
Changes:
- Add DB schema v16
devices.reinterview_pendingand migration, plus load/save plumbing inappdb. - Introduce/extend the per-device “check-in action” mechanism used to retry re-interviews when devices wake.
- Hook OTA
QueryNextImageversion changes and the post-OTA confirmation flow to queue/schedule re-interviews reliably.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| zigpy/zcl/clusters/general.py | Schedules re-interview (and queues it for next check-in) on OTA firmware version change via QueryNextImage. |
| zigpy/device.py | Adds pending re-interview state, check-in action registry/execution, and updates post-OTA re-interview confirmation logic. |
| zigpy/application.py | Clears reinterview_pending after successful re-interview swap; triggers queued actions on stack-reported join/announce. |
| zigpy/appdb.py | Bumps DB to v16; persists/loads reinterview_pending; adds migration to v16. |
| zigpy/appdb_schemas/schema_v16.sql | New v16 schema with devices.reinterview_pending column. |
| tests/test_zcl_clusters.py | Adds tests for re-interview scheduling/queuing on OTA version-change QNI traffic. |
| tests/test_device.py | Adds tests for check-in actions and post-OTA confirmation/re-interview behaviors. |
| tests/test_appdb.py / tests/test_appdb_migration.py | Adds persistence and migration tests for reinterview_pending. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+533
to
+540
| existing = self._checkin_actions.get(name) | ||
| self._checkin_actions[name] = _CheckinAction( | ||
| name=name, | ||
| coro_factory=coro_factory, | ||
| cooldown=cooldown, | ||
| task=existing.task if existing is not None else None, | ||
| last_attempt=existing.last_attempt if existing is not None else None, | ||
| ) |
- Rename `POST_OTA_PROBE_ATTEMPTS` to `POST_OTA_PROBE_RETRIES`: `read_attributes(retries=N)` performs N+1 total attempts, so the old name misrepresented the behavior (which matches dev's existing `retries=10`). - Document that the QueryNextImage confirmation deliberately accepts any query when no pre-flash baseline was cached: the wait only establishes that the device is alive after the flash. - Use `asyncio.create_task()` instead of `asyncio.ensure_future()` for the probe task. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
Sleepy end devices only receive requests for a short window after they send something themselves. Add a small named-action registry on `Device`: `register_checkin_action()` queues a coroutine factory that is run whenever the device shows signs of being awake (any inbound packet, or a stack-reported join/announce). The coroutine returns `True` to unregister the action or `False`/raises to retry on a later check-in, gated by a per-action cooldown (default 5 minutes) so a frequently reporting device does not hammer a failing action. Actions run as application tasks, not device tasks: device-owned tasks are cancelled when a re-interview swaps the device object, which would cancel an action that is itself driving the re-interview. Poll Control check-ins now also request fast polling while check-in actions are pending, keeping the device awake for the queued work. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
- Re-registering an existing action now mutates the stored `_CheckinAction` in place instead of replacing it: a running attempt unregisters itself by object identity on success, so a replacement object would have left completed actions permanently registered. - Document why neither `remove_checkin_action()` nor `on_remove()` cancels a running attempt: `_device_reinterviewed()` calls the old device's `on_remove()` mid-swap, and an action driving that very re-interview must not cancel its own call stack. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
Add a `reinterview_pending` flag to `Device` (a request timestamp, mirroring `last_seen`), settable via the public `schedule_reinterview_on_checkin()`. While set, a check-in action is armed that drives `reinterview()` whenever the device is next heard from, retrying on later check-ins (with a cooldown) until the re-interview succeeds. The flag setter fires `device_reinterview_pending_updated` so it can be persisted. Post-OTA wiring: `update_firmware()` queues the re-interview right after a successful flash, before waiting for confirmation — a sleepy end device that misses the whole confirmation window is re-interviewed when it next reports in. The confirmation wait also resolves when a `device_reinterviewed` event for the device fires (e.g. the check-in path already completed the re-interview mid-wait). The OTA cluster's version-change listener queues the flag too, so its immediate re-interview attempt is retried if the device goes back to sleep. A successful re-interview clears the flag on the new device object; the flag is deliberately not copied to the re-interview shadow. `clone()` copies the raw field without firing events or arming actions. The coordinator itself is never queued. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
f159b13 to
fa8993f
Compare
Schema v16 adds a `reinterview_pending` column to the `devices` table (REAL request timestamp, 0 = none). The flag is written whenever `device_reinterview_pending_updated` fires and on the full device save, and restored in `_load_devices()`, where the property setter re-arms the re-interview check-in action. The restored flag is copied across the quirk/resolver swap during `load()`, like `original_signature`. This makes post-OTA recovery for sleepy devices survive restarts: if zigpy restarts while a device is still rebooting from a flash (or before it ever woke up again), the queued re-interview is re-armed on startup and runs when the device next checks in. Claude-Session: https://claude.ai/code/session_01VnDASUDhbLfANUM9UbnAeo
fa8993f to
75af8c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT / EXPERIMENTAL.
Note
Part of a four-PR series making post-OTA re-interviews reliable for sleepy end devices. Merge order (first to last):
Makes queued re-interviews survive restarts. Branch:
tjj/reinterview-pending-persistence, stacked ontjj/reinterview-on-checkin.Problem
The pending re-interview flag from
tjj/reinterview-on-checkinis in-memory only. Restarting zigpy during the post-flash reboot window (HA restarts to apply an update are a common time for OTAs to be running) loses the queued re-interview; recovery then depends on the device eventually reporting a changed version in an OTA query.Changes
devicestable gainsreinterview_pending REAL NOT NULL DEFAULT 0(request timestamp, 0 = none), added as the last column since_load_devices()unpacksSELECT *positionally._migrate_to_v16is a plain_migrate_tablescopy — the SQL default fills the new column, no manual row loop.device_reinterview_pending_updatedlistener updates the row (noMIN_UPDATE_DELTAthrottling — the event is rare and a clear must never be dropped), and the full device save (raw_device_initialized) includes the column from the snapshot clone. The re-interview swap therefore persists the cleared state automatically: the old row is cascade-deleted, the new row is inserted with 0, and the explicit clear event updates it._load_devices()restores the flag through the property setter, which re-arms the re-interview check-in action. The flag is copied across the quirk/resolver swap inload()(likeoriginal_signature), so it lands on the resolved device.Notes
tjj/ota-index-persistencealso claims v16 (ota_provider_index_cache). Whichever lands second renames to v17 (new schema file,_migrate_to_v17,DB_VERSIONbump, chaining the other's tables).last_attempt, monotonic) intentionally resets on restart — the first check-in after startup may attempt immediately.Testing
New tests: persist/restore/clear round-trips with the action re-armed on load, restore across a device-resolver (quirk-style clone) swap, and defaulting to no pending re-interview after the chain migration from a v5 fixture. The parametrized migration-chain tests and
test_db_version_is_latest_schema_versioncover v15→v16 automatically. Full suite passes.