A watchdog that is kicked by a timer thread proves only that one thread was scheduled. The UI may be frozen, the database locked, the network stack dead, and the product unable to do its job while that timer happily reports success.
For an unattended Android SBC, the watchdog heartbeat should represent service health, not process existence. Pair it with a hardware watchdog that can reset the processor even when the kernel stops scheduling, and preserve enough evidence to explain why the reset happened.
Use layers because failures occur at different levels
No single recovery mechanism covers every fault. Android’s activity manager can restart an application process, init can restart a native service, a userspace watchdog can request a reboot, and a hardware watchdog can reset the board when software no longer responds.
| Layer | Detects well | Recovery action | Limitation |
|---|---|---|---|
| Application health | Stuck workflow, failed peripheral transaction, bad local state | Restart activity, process, or workflow | Cannot recover a dead kernel |
| Android service supervision | Crashed system or vendor service | Restart service | Repeated crash loops may persist |
| Userspace supervisor | Cross-service health and deadlines | Controlled reboot or subsystem reset | Depends on scheduling and I/O |
| Kernel watchdog client | Kernel scheduling progress | Stop feeding or trigger reset | A weak heartbeat can miss functional failure |
| Hardware watchdog | CPU or kernel lockup | Board reset | Cannot diagnose the cause alone |
| External supervisor | PMIC, SoC, or rail failure | Power-cycle rails | Adds hardware and sequence complexity |
Define ownership between the layers. If the application and a native daemon can both reboot the device, simultaneous recovery attempts can corrupt evidence or create loops. We prefer one supervisor to evaluate health and choose the escalation, while components report their state.
Verify the watchdog hardware path first
Most Linux watchdog drivers expose a device such as /dev/watchdog. Opening it often starts the watchdog; periodic keepalive operations prevent the timeout. Driver behavior varies, so verify timeout range, boot status, pretimeout support, stop behavior, and whether nowayout is enabled.
Test the physical result early. Stop feeding the watchdog and measure the time until reset. Confirm which rails reset, whether peripherals also reset, and whether the bootloader reports the cause. Some board resets leave a USB hub, modem, or display bridge powered, preserving the exact fault that required recovery.
If the product needs a full power cycle, connect an external supervisor or PMIC sequence that removes the affected rails for a defined interval. A warm SoC reset and a cold power cycle are different tools.
The timeout should cover the worst healthy delay with margin, but still meet the product’s maximum outage requirement. Storage maintenance, thermal throttling, OTA installation, and first boot after an update can all extend normal execution. A ten-second timeout chosen on an idle bench may reset healthy units in the field.
Feed only after health is proven
A useful heartbeat combines several signals. The exact set depends on the product, but common checks include UI progress, main event-loop responsiveness, database transaction completion, peripheral response, storage writability, and completion of a real business transaction.
Network connectivity requires nuance. A kiosk should not reboot forever because the upstream server is offline. Instead, distinguish local network stack health, link state, DNS or server reachability, and business-service availability. Reboot only when a local fault is likely to improve; show an offline state for remote outages.
Use monotonic deadlines. Each component reports progress with a timestamp and a reason code. The supervisor feeds the hardware watchdog only when mandatory components are inside their deadlines or a declared maintenance state extends them.
Do not let every component write the hardware watchdog directly. Multiple feeders hide failures because one healthy service can keep the board alive while the critical application is dead.
Design escalation instead of rebooting immediately
Recovery should be proportional and bounded. For example:
- Retry a peripheral transaction with a strict limit.
- Reset the peripheral or reopen its driver.
- Restart the affected application or service.
- Request an orderly Android reboot if the system remains responsive.
- Stop feeding the hardware watchdog if controlled reboot fails.
- Enter a safe or rollback mode after repeated boots with the same fault.
Store a boot-attempt counter outside volatile memory and clear it only after the product has remained healthy for a defined period. Otherwise a broken application update can produce an endless reboot cycle that looks like flaky power.
Coordinate repeated-failure handling with A/B OTA rollback. The watchdog should supply evidence that a new slot failed health checks, not arbitrarily switch partitions on every transient network outage.
Preserve evidence before and after reset
A watchdog that restores service but erases the cause creates expensive support work. Capture the last health vector, failed component, escalation stage, uptime, temperature, storage state, application version, BSP build, and reset request.
Use a small append-safe or atomic record and cap its write frequency. The record itself must not create an eMMC endurance problem. Kernel pstore or equivalent persistent crash storage can preserve panic and console information across reboot when the BSP supports it.
On the next boot, read the hardware reset cause before another component clears it. Classify watchdog reset, brownout, thermal shutdown, software reboot, and external reset separately. The brownout-proof power guide explains why these events can otherwise be confused.
Upload evidence after connectivity returns, but keep a local bounded history for technicians. A single “unexpected reboot” counter is not enough.
Avoid the common false-positive traps
Suspend is an obvious one. Decide whether the watchdog pauses, extends, or uses a low-power-aware source while Android sleeps. Test resume with the production wake sources.
Long flash operations are another. OTA writes, filesystem checks, database migration, and log compression can create latency spikes. Do not disable the watchdog for an unbounded maintenance window. Enter a named state with a longer deadline and continue checking progress.
Thermal throttling slows work without necessarily making the system unhealthy. Set deadlines using worst-case validated performance at maximum ambient. If heat can prevent the product from meeting its service requirement, fix the fanless thermal design instead of hiding it behind a very long timeout.
Finally, avoid boot-time reset loops. Start the hardware watchdog only when the boot chain can keep feeding it or set a bootloader timeout that safely covers verified worst-case startup. Ensure recovery mode and factory provisioning also understand the watchdog.
Fault-injection test matrix
Watchdog validation should deliberately break the product. Run enough cycles to expose races and record both recovery time and retained evidence.
| Injected fault | Expected response | Evidence to retain |
|---|---|---|
| Kill kiosk application | App or service restarts; no board reboot unless restart fails | Process exit, restart count, recovery time |
| Block application main thread | Health deadline expires and escalation begins | Missed heartbeat and thread state |
| Hang a USB peripheral | Port or driver reset before system reboot | Device identity and transaction timeout |
| Fill writable storage | Enter degraded mode; avoid reboot loop | Free space, I/O errors, cleanup result |
| Stop userspace supervisor | Hardware watchdog resets board | Watchdog reset cause and last feed time |
| Trigger kernel panic | Hardware or panic policy restarts system | Persistent kernel record |
| Disconnect network upstream | Offline behavior, normally no reboot | Link and reachability classification |
| Interrupt OTA boot | Rollback or recovery after bounded attempts | Slot, boot count, health failure reason |
| Heat to rated maximum | Deadlines remain valid under throttling | Temperature, severity, and task latency |
Also test repeated faults. One clean recovery says little about the tenth cycle, when counters, storage, or peripherals may be left in a different state.
Frequently asked questions
What watchdog timeout should an Android SBC use?
There is no universal value. Measure the longest healthy scheduling and service delay under boot, update, storage, thermal, and workload stress, then add controlled margin while meeting the allowed outage time.
Should an app feed /dev/watchdog directly?
Usually no. A privileged system supervisor should combine health from required services and own the hardware feed. Direct feeding by one app cannot prove the rest of the product is healthy.
Is automatic reboot enough for field reliability?
No. Reboot must be one step in an escalation plan with failure classification, evidence retention, boot-loop protection, safe mode, and update rollback.
Bottom line
A production watchdog is a chain from meaningful health checks to a reset path that really removes the fault. Give one supervisor ownership, feed only after essential work progresses, retain reset evidence, and inject failures on the assembled product. The goal is not frequent rebooting. It is bounded, explainable recovery when normal supervision can no longer restore service.