SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsfw-hil-testing

fw-hil-testing

Testing & quality

View repository

Updated 10-16 05:47

About this skill

Firmware HIL in-the-loop acceptance (evidence-chain hub)

This Skill is the firmware-skills pack'sevidence-chain hub: the Pending HIL items that other skills suspend, as well as the determination of "how is firmware considered shippable once it's written", all land as evidence here. Core mantra:Build success ≠ hardware usable— without real-device evidence, there is no such word as "usable".

Determine Task Type

  1. OpenWrt gateway HIL: the target device is an OpenWrt gateway/box (N1/Amlogic, x86 soft router, mt7981 class) → Workflow A (gateway-type matrix).
  2. ESP32 MCU HIL: the target is ESP32 on-chip firmware → Workflow B (MCU-type matrix).
  3. HIL failure troubleshooting: flashing fails, no output after flashing, behavior on the real device does not match → Workflow C (failure three-category routing).
  4. Shipping determination / evidence review: some evidence already exists, asking "can it ship" → go by badge rating (completion-badges); for missing rows, fill the rows first, then rate.

Prerequisites / Preflight

Runtime verification comes first:

# ESP32：工具链与串口设备
idf.py --version 2>/dev/null || echo "IDF not in PATH"
ls /dev/tty.usb* /dev/ttyUSB* /dev/ttyACM* 2>/dev/null || echo "no serial device listed"
# OpenWrt 网关：构建产物是否在手
ls -l bin/targets/*/ 2>/dev/null || echo "no image-builder output"

Pre-Flash Checks (go through them one by one before flashing; if any one is not satisfied, stop first):

  1. Serial port probe ready: can list the target serial device and open it at 115200 8N1(common value; subject to actual board measurement); for devices whose serial port cannot be observed, at least identify one alternative observation channel (HDMI, LED, network reachability).
  2. Partition red line: make clear which partitions this flash will write;bootloader / factory calibration / NVS are by default not touched. Partition facts are subject to runtime verification (ESP32:idf.py partition-table; boxes: subject to device documentation and the ophub tool matrix); if uncertain, stop.
  3. A rollback firmware must be preserved first: preserve the currently usable firmware before flashing — for OpenWrt boxes use ddbr backup (→ openwrt-serial-recovery); for ESP32 preserve a "known good" complete image + sha256. The rollback plan must be able to answer: where is the file, what is the checksum, and how to flash it back.
  4. Destructive confirmation: restate the target device and the partitions to be erased to the user, and execute only after confirmation.

Offline Baseline

  • Walkthrough date 2026-09-02, not auto-updated.
  • OpenWrt baseline 25.12.5; ophub upstream HEAD c593d56; ESP-IDF baseline is subject to the Phase 2 baseline document (at that time v6.1）。
  • The HIL method itself is not tied to a version; the referenced commands (idf.py flash/monitor, IB make image, ddbr) are subject to the above baselines, and subsequent behavior changes are subject to the official sources.

Contracts (hardware contracts)

Record them before starting; if items are missing, ask first, do not guess (refer to rust-embedded's Mandatory Hardware Contracts pattern):

  • Exact device model/board version, power supply method, serial parameters (subject to actual board measurement);
  • Flashing tool and port, list of partitions to be written this time (excluding red-line areas);
  • Observation channels: serial log / SSH / Web interface / LED / multimeter·power meter (needed for abnormal paths and power sampling);
  • Location and checksum of the rollback firmware (the record of Pre-Flash item 3);
  • Anti-hallucination red line: pin, partition offsets, DTB filename, memory layout — when uncertain, fabricating is forbidden; mark Pending HIL and defer to runtime verification.

Evidence Contract (acceptance matrix)

All "verified / shippable" assertions must be attached to theacceptance matrix: each row = one acceptance item, six columns complete (template and instance row sets see evidence-matrix）：

Column Requirement
Acceptance item A single decidable statement ("service automatically recovers after 10 power pulls")
Preconditions Things that must hold before executing this row (X version already flashed, rollback image already retained)
Operation Deterministic actions (how to cut power, for how long, what abnormal input to send)
Observation means What is used to view the result (serial log / SSH command / Web / measuring instrument)
Expected Decidable pass line
Evidence form Screenshot | log | measurement value — at least one of them, saved to file

Five categories of rows must be covered; missing any one category means the matrix is not complete:

  1. Power-on boot: time from cold start to usable and state;
  2. Core function: the reason the device exists, one row per item;
  3. Exception paths: power loss / network loss / abnormal input — most easily missed, most expensive to rework;
  4. Long-duration run: soak sampling >=24h to start (memory, temperature, service liveness, restart count);
  5. Upgrade path: upgrade success + upgrade interruption/rollback (mechanism side → esp32-ota）。

Evidence discipline: each piece of evidence carriesdevice identifier + firmware version + date, and can be matched to a matrix row; re-shooting, shooting on behalf of others, and cross-device misappropriation of evidence are forbidden.

Tiered completion standard (badges)

Three-level badge semantics,skip-level claims are forbidden(for definitions, skip-level wording comparison, and self-check see completion-badges）：

Badge Obtaining conditions Allowed claim ceiling
Build Verification Only Build passes only "Compiled output image, not verified by running"
Boot Verified (QEMU) Bootable in boardless environment + smoke test (→ fw-emulation） "Bootable in QEMU, smoke test passed"
HIL Verified Full-row real-device evidence for the acceptance matrix complete "All real-device matrix rows pass, evidence traceable"

Whichever level is not obtained, claims can only be made up to that level; the legacy marker Pending HIL continues to be used as asingle-row-levelmarker (that row has not yet obtained real-device evidence), while badges aredeliverable-levelstatus; the two are used together.

Capability Boundaries + Hand-off

This Skill only does acceptance and evidence; it does not do building, does not do mechanism implementation, and does not make hardware design conclusions when there is no instrumentation:

User Intent Skill
Build / repackage image openwrt-image-build / openwrt-amlogic-remake
No hardware, first get it running in a boardless environment fw-emulation
OTA / rollback mechanism implementation esp32-ota
Flashing failure has caused unbootable state (OpenWrt side) openwrt-serial-recovery
ESP32 crash / reset localization esp32-debug
Evidence complete → release and naming validation fw-release-gate
Task classification entry fw-core

Failure three-category routing (details go to Workflow C):

Failure category Characteristics Routing
Build failure Image not produced or won't boot, suspected artifact issue fw-emulation No-board reproduction, separate "bad image" vs "board issue"; pure build errors go back to fw-toolchain / esp32-idf / openwrt-image-build
Flashing failure Flashing errors, unbootable after flashing (OpenWrt side) openwrt-serial-recovery(software-first then hardware gradient)
Functional failure Flashed successfully but crashes/resets/behavior mismatch esp32-debug(ESP32); OpenWrt runtime first logread / dmesg triage

Commercial gate mode(how the matrix maps to batch release; the mode is distilled from TinyNAS closed-loop practice, mapping table see evidence-matrix §4): a single unit with an all-green matrix is only a G1 engineering prototype-level ticket; batch shipping follows G1 engineering prototype continuous operation → G2 single-channel small-batch controlled observation (activation rate/boot rate/refund rate-type metrics) → G3 multi-channel scale-up comparison → G4 after-sales metric gaterelease level by level, never jump from "single-unit HIL Verified" straight to batch.

Workflow A — OpenWrt gateway HIL

  1. Record the hardware contract, pass Pre-Flash Checks (ddbr / image backup first).
  2. From evidence-matrix §2 instantiate the gateway-type matrix: power-on to usable, storage mount, core services, web admin, power-loss recovery loop, network-down degradation, soak, upgrade/rollback.
  3. Flash the target version, execute line by line and collect evidence (serial logs + SSH command output + web screenshots saved to files).
  4. Exception paths must actually be exercised: power loss using a timer socket or unplugging for N cycles; network disconnection by unplugging WAN to see LAN-side service survival; abnormal input to fill the disk, malformed configuration.
  5. All green → apply the HIL Verified badge and archive the evidence package; if any line is missing → downgrade the badge to the actual level, and after routing the missing line to the corresponding skill, re-verify.

Workflow B — ESP32 MCU HIL

  1. Record the hardware contract, pass Pre-Flash (retain a known-good image + sha256).
  2. From evidence-matrix §3 instantiate the MCU-type matrix: power-on log and reset reason, peripheral readiness, network provisioning success/failure and retry, OTA upgrade + power-loss rollback, low-power measured (if low power is claimed, µA-level values are based on actual measurement, do not copy datasheet verbal values), soak.
  3. idf.py -p <PORT> flash monitor flash and observe; exception lines such as power loss during upgrade and provisioning failure retry must be executed.
  4. Crash/reset-type failures are not characterized in this Skill → route to esp32-debug; after characterization and fix, return to the matrix to re-verify that line.

Workflow C — HIL failure troubleshooting (including "no serial output after flashing")

  1. Classify first, then act(three categories: build/flashing/functional), do not blindly re-flash on hardware repeatedly.
  2. "No serial output" troubleshooting order:
    • Observe the link: baud rate (115200 is a common value, based on board measurement), TX/RX crossover, common ground, power, whether the correct port is selected;
    • Flashing evidence: does the flashing log show successful writing? Was it written to the correct partition/medium?
    • Minimal boot: swap in a known-good image (rollback firmware) to verify "the board can respond" — if it responds, it is an image issue (→ build failure routing); if not, escalate to a board/flashing issue;
    • OpenWrt box completely unresponsive → openwrt-serial-recovery; has output but crashes → esp32-debug。
  3. During troubleshootingdo not erase red-line partitions; before each re-flash, confirm the rollback image is still present and its checksum is unchanged.

Validation Gates

  • Matrix integrity: all five mandatory rows are present; each row has an evidence file or an explicit N/A reason; there is no "checked but not executed".
  • Badge consistency: declared badges correspond one-to-one with evidence; where there is no real-device evidence, it must not appear HIL Verified。
  • Rollback drill: successfully performed at least once restoring from a retained image; those not drilled are explicitly marked and listed as known risks.
  • Failure closed loop: every failed row has been routed according to the three categories and has a conclusion (fix / degrade / mark Pending HIL).

Pitfalls

  1. Do notdescribe Boot Verified (QEMU) as "verified and shippable"—QEMU has no real electrical, storage media, power-loss, or wireless environment, and badges must not be upgraded beyond their level.
  2. Do noterase or write the bootloader/calibration/NVS before retaining rollback firmware—think about how to roll back before the first flash.
  3. Do notreplace matrix rows with "should be fine"—each row must either have evidence or state an N/A reason.
  4. Do notskip exception paths (power loss/network loss/abnormal input) and directly claim production readiness—this is the number one source of rework and returns.
  5. Do notforge or retrospectively stage evidence—evidence with mismatched dates, device identifiers, or versions is treated as invalid.
  6. Do notextrapolate a single prototype's conclusion into "no problem in batch production"—batch decisions go through the commercial gate mode and are released level by level.
  7. Do notconclude in this Skill that "this is a power/hardware design defect"—hardware suspicions must also be conclusively measured; do not assert without instruments.

Official Sources

Privacy

HIL evidence (serial logs, screenshots, measurement records) may contain device identifiers, MAC addresses, network credentials, and user data: redact before archiving and obtain user confirmation before external transmission; this Skill does not collect, store, or transmit any device data; treat rollback images as user data and store them encrypted.