RadioFoundry

Testing

How to run every test of RadioFoundry, from the host tests to the radio on the SDR bench. Each tier is repeatable and skips cleanly what it can't reach. What each level is for and where a new test goes: docs/dev/testing.md.

Tier Needs Command Where it is described
Release gate (host tests, build matrix, license checks) a Mac or Linux host with the ARM toolchain tools/ci.sh this page, section 1
Coverage, sanitizers, fuzzing the host (gcovr for coverage) tools/coverage.sh, make -C tests/host SANITIZE=1 docs/dev/ci.md
Hardware in the loop, radio only the radio on USB python3 -m pytest tests/hil -v -rs tests/hil/README.md
Hardware in the loop, on air the radio and the SDR bench the same with opt-in flags tests/hil/README.md, section 3 here
Manual bench cases the SDR bench, a person at the radio tools/bench/dmr_bench.py run CASE docs/testing/dmr-bench.md
DM-32UV first hardware test a DM-32UV on its programming cable, a dummy load step by step docs/testing/dm32-bringup.md
MD-380 first hardware test an MD-380 / MD-390 / RT3 on USB, a dummy load, an SDR or signal generator step by step docs/testing/md380-bringup.md

1. The release gate: tools/ci.sh

tools/ci.sh                    # everything below
tools/ci.sh --quick            # each board's default build only (no feature matrix)
tools/ci.sh --board <b>        # one board, for example dm1701
tools/ci.sh --matrix           # only the feature matrix (no default builds)
tools/ci.sh --steps host,python   # only some steps: host, python, firmware, licenses, similarity
JOBS=8 tools/ci.sh             # parallel make jobs (default: the CPU count)
RFOUNDRY_CI_PRUNE=1 tools/ci.sh     # drop each build tree once its size is recorded (logs and images stay)
RFOUNDRY_CI_SANITIZE=1 tools/ci.sh  # the host step also runs the host tests under ASan and UBSan

The CI workflows run these pieces per push and nightly (docs/dev/ci.md).

It runs, and reports each as ok / FAIL / skip (exit status 0 only when nothing failed):

  1. Host tests: make -C tests/host (the same as make test), built with no warnings. They need only the host's C compiler. Their binaries, .dSYM bundles and outputs stay in tests/host/ and are ignored by git (tests/host/.gitignore tracks only sources); make -C tests/host clean removes them.

  2. Python: every tool and HIL test parses, and pytest collects tests/hil (skipped without pytest: pip install pytest pyserial). Every HIL skip names its kind (tools/check_hil_skips.py), and the feature checklist is consistent (tools/bringup.py check and the bring-up mode tests, ci.md). The flashing tools' tests run: tools/rfoundry/tests (sector planning, manifests, the codec library and key derivation, the codec decision, whole flashes against a fake bootloader with a flash model; release.md) and the --self-test of tools/codec_extract.py, tools/stm32_dfu.py, tools/md380_migrate.py, tools/serial_loader.py (against a simulated Baofeng bootloader) and tools/dm32_migrate.py (DfuSe files, the conversion report, CPS writes against a simulated DM-32 shim), and the stack check's (tools/stack_check.py). tools/check_doc_links.py checks that every relative link, image and anchor in the Markdown resolves.

  3. Firmware, every board with the feature matrix. For each boards/<b>/board.mk whose compiler is installed:

    • default: the board's defaults;
    • no-<FEATURE>: every switch that is on by default, turned off alone (its sub-features follow it, docs/build-config.md);
    • fm-minimal: FM only, without DMR, USB, VFO, diagnostics and sub-audio tones;
    • maximal: every switch on, except the board's BOARD_UNSUPPORTED, BOARD_NOT_WRITTEN and MAXIMAL_SKIP, with its MAXIMAL_VARS (for example LTO=1 on the DM-1701);
    • vhf: the MD-380's VHF image (MD-380 only).

    Every configuration must build with zero compiler and linker warnings. The default and maximal ones must also pass the stack check: every task's deepest call chain fits its stack (docs/dev/stack-check.md). A board without its compiler (the DM-32's csky GCC comes from tools/build-csky-toolchain.sh or the Docker image, docs/dev/ci-runners.md) is skipped with a notice. Build trees are build/ci/<board>-<config>/, each with its log next to it. The sizes go to build/ci/size-report.md: image bytes, the change against the board's default, and the space left in the linker's application region (the codec windows lie outside it, so "free" is what code can still grow into).

  4. Licenses: tools/license_grep.sh (markers of code that must not be here, docs/CLEANROOM.md) and reuse lint. Without reuse installed the lint is skipped with a notice; install it with pipx install reuse (or pip install --user reuse).

  5. Similarity scan, only when RFOUNDRY_SIMILARITY_REFS names the reference trees (space separated). The reference trees are never part of this repository or of public CI; the project lead runs it against the aftermarket firmware trees before a release:

    RFOUNDRY_SIMILARITY_REFS="/path/to/aftermarket-tree /path/to/other-tree" tools/ci.sh
tools/similarity_scan.py /path/to/aftermarket-tree                    # the scan alone, every region with lines
tools/similarity_scan.py --ids --report scan.txt /path/to/tree   # also finds copies with renamed identifiers

    Each reported region gives our file and lines and the reference file and line it shares a run of 25 tokens with. Every region is reviewed by hand; #include blocks and generic idioms are expected noise. Options: tools/similarity_scan.py --help.

The firmware version

The image carries the commit it was built from (RFOUNDRY_GIT, shown on the splash screen, in the debug link's VERSION and in the MMDVM GET_VERSION description) with -dirty appended when tracked files differ from that commit or untracked sources sit in the built directories. The build prints it on the image line (image ...: N bytes, version c1544f29-dirty). An image for anyone else is built from a clean, committed tree.

The UI without a radio

The UI's host tests (make -C tests/host run-ui, part of the host tests above) replay every task flow of docs/ux/task-flows.md against the press budgets of docs/ux/usability-tests.md on four display and key map pairs, and crawl every screen for the rules that hold everywhere (docs/spec/ui.md 7). make -C tests/host ui-gallery renders every screen at every display class into docs/ux/screens/; run it after a UI change and commit the PNGs with it, so the review shows what changed. tests/host/uiSim renders one screen after a key script, to PNG.

2. Hardware in the loop, radio only

python3 -m pytest tests/hil -v -rs

The radio on USB with firmware built with CONFIG_DEBUGLINK (the default). Nothing transmits: version and features, screenshots, injected keys, FM receive on NOAA weather, MMDVM requests. Without a radio every test is skipped with the reason, which starts with the skip's kind ([bench]). Every test file, rig assumptions and pass criteria: tests/hil/README.md. A radio's feature checklist run is tools/bringup.py run --radio <id> (its "Bring-up mode").

The USB network tests (test_usb_network*.py, test_tcp_services.py, test_net_services.py, test_snmp.py) switch the radio to the network personality and back, so they need --allow-usb-switch. On macOS a missed NCM carrier can leave the radio a network adapter until it is power cycled (docs/dev/ci-runners.md, "Gotchas").

CPS and settings (test_cps.py): the radio-only tests read the codeplug twice over the open-firmware CPS protocol and compare, and check a channel change is saved. Two opt-in flags change the radio but don't transmit:

python3 -m pytest tests/hil -v -rs -s -k cps --allow-codeplug-write   # backs up, renames a channel, restores
python3 -m pytest tests/hil -v -rs -s -k cps --allow-reboot           # channel position survives a restart

The write test saves the backup in its tmp directory and prints the path. If the restore fails, the path and the tools/cps_client.py write command are in the failure message.

The UI controls (test_ui.py): the Quick tiles drive the radio core with injected keys and read it back (timeslot, color code and mode for now, hear all, monitor latch, VOX), the picker hands its focus row to the radio core, Settings reach the input layer (long press time) and the UI (auto lock, about 35 s). The DMR tests need a DMR channel, the monitor and VOX tests an FM channel; the others skip. Three change the radio and are opt-in:

python3 -m pytest tests/hil/test_ui.py -v -rs -k save --allow-codeplug-write          # timeslot saved to the channel and back
RFOUNDRY_HIL_DELETE_MESSAGE=1 python3 -m pytest tests/hil/test_ui.py -v -rs -k delete      # deletes the newest inbox message
RFOUNDRY_HIL_FACTORY_RESET=1 python3 -m pytest tests/hil/test_ui.py -v -rs -k factory --allow-reboot   # erases all settings

Factory reset puts every setting back to its default (brightness, privacy keys, radio settings, the channel position) and erases the native area; the codeplug and calibration stay. Note the settings before running it.

3. Hardware in the loop, on air (opt-in)

Each flag allows one more kind of action, and a test runs only when all it needs is allowed:

Flag Allows
--allow-rf anything that transmits: the SDR bench's modem, the radio itself
--allow-service-stop stopping the SDR bench's gateway service (always restarted by a bench-side guard) to use its modem and RTL-SDR
--allow-usb-switch, --allow-codeplug-write, --allow-reboot as in section 2 (no RF) 
python3 -m pytest tests/hil -v -rs -s --allow-rf --ambe SPEECH.ambe --dmr-freq MHZ   # the default bench pass, DMR audio smoke included
python3 -m pytest tests/hil -v -rs -s --allow-rf --allow-service-stop                  # DMR RX / voice, DMR TX, FM TX
python3 -m pytest tests/hil -v -rs -s --allow-rf --allow-service-stop -k "dmr_tx or fm_tx"
python3 -m pytest tests/hil -v -rs -s --allow-rf --allow-service-stop -k dmr_data   # DMR data services
python3 -m pytest tests/hil -v -rs -s --allow-rf --allow-service-stop -k aprs --aprs-call CALL   # APRS, 446.050 MHz
RFOUNDRY_SPLIT_VARIANT=0 python3 -m pytest tests/hil/test_split_duplex.py -v -rs -s --allow-rf --allow-service-stop
Test What is checked
test_dmr_rx, test_dmr_voice the bench modem sends calls; the radio's counters show every burst, every voice frame reaches the speaker path
test_dmr_audio_smoke DMR audio both ways, which no counter shows: the bench's speech decodes to audio on the radio, frames not repeated; the radio's test signal reaches the air as distinct, non-silent frames that decode to audio when played back. Part of every bench pass on every radio with DMR (tests/hil/README.md, Default bench pass); a run without it prints DMR AUDIO NOT CHECKED
test_dmr_tx the radio keys 3 s; an SDR capture (sdrmon.py) and the bench modem's listener check source ID, talkgroup, color code, headers / voice / terminator, slot timing, ramps, carrier offset
test_fm_tx the radio's FM VFO keys with CTCSS 100.0 Hz, DCS D023N, at 12.5 and 25 kHz, the 1750 Hz burst, the roger beep, three power levels; tools/sdr_fm_check.py decodes the tones (--tones: audio tones) and compares the deviations
test_aprs APRS at the lowest power on a 70 cm simplex frequency (446.050 MHz by default, never an APRS frequency, no path): position, status and message decoded off air by tools/aprs_afsk_check.py (Bell 202 / AX.25 from the RTL-SDR's I/Q), with the deviation, twist, phase continuity and TX delay; positions over DMR (LRRP, UDT) decoded from the bench modem
test_dmr_data the bench sends data, a text message, a call alert and radio checks to the radio, which answers (its dmr_data counters, the answers heard by the bench modem); the radio sends a call alert, a radio check and a text message the bench modem hears (docs/spec/dmr-data.md)
test_split_duplex a CONFIG_HOTSPOT_SPLIT=y build: the 0xD6 protocol (no RF); the TX half's continuous BS outbound at the lowest power checked off air with the CACH decode of the BS framing experiment (variant from RFOUNDRY_SPLIT_VARIANT); the RX half labeling bench calls, the label flipping with the epoch (docs/spec/split-duplex.md 10)
test_hotspot_voice the hotspot on air both ways: bench calls and data up the USB link, host calls and data heard on air
test_radio_features (rf) the zone scan, VFO scan and dual watch holding on a bench call; battery save waking for it

Which feature is covered on air, and what the last run found: on-air-coverage.md. On this bench the radio overloads both receivers even at 50 mW; tests/hil/README.md ("On this bench") says what the tests do about it.

The SDR bench's names come from the environment: RFOUNDRY_SDR_BENCH_HOST (ssh host or alias, default sdr-bench), RFOUNDRY_SDR_BENCH_SERVICE, RFOUNDRY_SDR_BENCH_DIR, RFOUNDRY_SDR_BENCH_PY, RFOUNDRY_SDR_BENCH_ENV_FILE (docs/testing/dmr-bench.md section 1).

4. Manual bench cases

docs/testing/dmr-bench.md: the DMR test cases that need a person at the radio (talking, reading the display), run end to end by tools/bench/dmr_bench.py with automatic checks where possible; --dry-run prints every command first. docs/testing/mmdvm-audit.md: the MMDVM protocol audit of the hotspot. docs/testing/dm32-bringup.md: the DM-32UV port's first test on a radio, in order, with what to change when a step fails; docs/testing/md380-bringup.md is the same for the MD-380.

The split duplex repeater has no hardware-free gap: make -C tests/host runs splitDuplexTest (the CACH and short LC against the standard's tables, the outbound buffer, the epoch, both roles through the MMDVM core), hrc6000BsTest (the HR_C6000 BS outbound engine on a fake chip) and sim/splitPairTest.py (two radios' cores in lockstep virtual time with drift, a clock wrap and USB jitter, a repeated call end to end); the bench steps are docs/spec/split-duplex.md 10.

View the source of this page