Settings store and safe flash writes

How the radio keeps its own state across power cycles (core/settings) and how the codeplug's flash sectors are rewritten without a power cut corrupting them (core/storage). Both live in the board's SPI NOR. Both are generic: the board gives the addresses in its hal_flash_map_t (board.flash->map) and core code never names a radio.

1. Areas

The board's flash map (hal/hal_flash.h) names these areas. A board without a map runs everything from RAM: the settings start from the defaults at every boot and nothing can be written.

The settings ring and journal go where no open-firmware tool reads or writes, and the CPS shim refuses to write there. Each board fixes the values in its hal_flash_map_t, and its hardware page lists the map. The usual order, low to high, is the open-firmware EEPROM image, the settings ring (32 KB, 8 sectors), the journal (2 sectors), the open-firmware flash image (up to openfw_end), the native codeplug (1 MB) and, on boards with room, a 256 KB staging area. A board that keeps its calibration in the vendor layout lists no calibration window. The boards are in feature-matrix.md.

The core works in 4 KB erase units (STORAGE_SECTOR). A board whose flash erases in other units gets no writes. A NAND board puts drivers/flash/nand_sectors in between, which gives NOR behavior in 4 KB sectors (nand-sectors.md).

2. Settings records

The settings area is a ring of sectors. Each sector holds 16 slots of 256 bytes, and each slot holds at most one record (little endian):

Load (settings_init, once at boot, after storage_init): every slot is read. A slot is free when all its bytes are 0xFF. It is valid when the magic, the length and the CRC match and the commit word is 0. Anything else (a torn write) is counted in settings.skipped and ignored. The valid page 0 record with the highest sequence number is in force (2.1 has the other pages). When its layout is not SETTINGS_LAYOUT, or no page 0 record is valid, the radio starts from the defaults (settings.load=defaults). A shorter or longer payload of the same layout fills the fields both firmwares know, and the rest keep their defaults (upgraded).

Save: the records go to the next free slots after the newest record. When a sector is full, the next sector of the ring is erased first. Each record is programmed, then its commit word, then the slot is read back. A cut before page 0's commit word leaves the previous set in force. A slot that doesn't read back is skipped and the next one is tried, up to 4 slots per record. No save erases a sector that holds a page of the set in force, or a page this save already wrote: the walk passes over such a sector to the next one. When every sector is protected that way the save fails, and the set in force stays. Erases go round the ring: 8 sectors (every board's ring today) hold 128 records, so with two pages a sector is erased once every 64 saves.

2.1 Pages

settings_t is longer than one slot's payload, so it is stored as pages: page k holds bytes 236 * k on. SETTINGS_PAGES is the number its size needs (2 now, 268 bytes); the page field allows 16 (3776 bytes). Each page is a record with its own length, so settings_t grows by appending fields, and a short page fills the fields it has.

Firmware from before pages reads a page 1 record as an ordinary record of layout 0x1001, which it doesn't know, and never sees it as newest: page 0 is always written last.

When: settings_begin() returns the live copy under a short RAM-only lock (storage_quick_lock) and settings_end(now) releases it. If anything changed, it notes the time. No flash operation happens inside, so the radio task may change settings. The housekeeping task calls settings_poll() every 100 ms. It saves once the changes have rested SETTINGS_SAVE_DELAY_MS (2 s), so turning the knob through 20 channels writes one record. settings_flush() saves at once: at power-off and on the CPS's "save" (C 6 0 and C 6 2).

Who writes what:

• The radio core (radio_sm.c persist()): VFO or channel mode, the zone, the channel number and its position, the VFO's frequency and settings, and the privacy key in use.
• The radio core, for radio_set_settings(): the rs_* fields (TOT, busy channel lockout, scan, beeps, VOX, battery saving, backlight times; radio-features.md 1). rs_stored 0 means the radio core's defaults.
• The radio core's data tick, for the DMR data service (dmr-data.md): the ds_* fields (ARS server, radio check reply, call alert ACK, LRRP reply, the formats). ds_stored 0 means the service's defaults.
• The UI and the debug link's radio op 27: usb_mode, net_flags, net_link_ip (usb-network.md 5.4); the host link reads them at start and on hostlink_settings_changed().
• The UI: brightness.
• The CPS shim: the VFO when a CPS writes a changed VFO A record (cps-protocol.md 2.3).

At boot, apply_prefs and restore_vfo read the squelch, the master power, the VFO step, the own DMR ID override, the radio name override and the privacy key. The radio name override is the talker alias text; empty means the codeplug's callsign.

3. Fields and the append-only rule

settings_t (core/settings/settings.h) is the payload. The rules that keep old records readable:

1. Fields are only appended. A new field goes at the end, before nothing. Its default comes from settings_defaults(), which a record written by an older firmware doesn't overwrite.
2. No padding. Members are fixed width and ordered so the compiler adds none. sizeof is a multiple of 4, which a static assert checks. Page 1 starts at byte 236 (a field there must not straddle the page boundary; a static assert checks the first page 1 field's offset).
3. Reserved bytes stay 0 in records we write. A later firmware may give them a meaning only if 0 means "as before".
4. SETTINGS_LAYOUT is bumped only when a field moves or changes meaning. Old records are then ignored and the radio starts from the defaults. Avoid this: append a new field instead.
5. 8-bit "not set" is SETTINGS_UNSET8 (0xFF), meaning the radio's or the codeplug's default applies.

All of these are still layout 1: they were appended before any firmware with the settings store shipped. Page 0 is full: its later fields went into reserved bytes where 0 already meant "as before" (net_flags, snmp_rf_manager[3], aprsis_range_km, snmp_flags, aprsis_port, snmp_rf_gap_s, aprs_power, aprs_cp_path, volume, vfo_talk_lo), or the new defaults (accent: the filled status strip, a new look on purpose). No reserved byte is left. One-byte fields that hold small numbers (hotspot_allowed, usb_mode) still have free upper bits (theme and updown_home hold the crash report's target; updown_home bit 7 is free; accent bits 6-7 are free); vfo_power does not (its "not set" is 0xFF). New settings are appended to page 1 as plain fields; a page 2 follows when page 1 is full (2.1).

3.0 Who writes, and storage jobs

The settings are written through settings_save (debounced, written by settings_poll on the housekeeping task in app.c). Other flash writes that a screen asks for (save to channel, the permanent talk target, factory reset) are storage jobs: storage_job_post(fn, ctx) hands a function to the housekeeping task, which runs it from storage_job_poll right after settings_poll. There is one slot: a second post while a job is pending is refused (the UI says so), and storage.job_busy on the debug link shows a pending job. A job runs with no UI lock held, so the UI keeps drawing during the flash write; the job reports back through ui_notify.

Factory reset (a job): settings_defaults into the live copy and save it at once, then storage_native_erase. That erases the native area only when its first word is the native magic STORAGE_NATIVE_MAGIC (0x4E52444F, bytes 4F 44 52 4E) or when it is blank already; anything else is reported as foreign and left alone. The open-firmware codeplug, the calibration and the foreign-guarded regions (section 5) are never touched; a reset of those would need its own second confirmation and is not offered.

3.1 Privacy key slots (the hook for DMR basic privacy)

The key store for core/dmr/dmrPrivacy.c (dmr-voice.md 5). Four slots, each a key value (privacy_key[i], u32: 16 bits are used today by Motorola and Anytone basic privacy, and the slot is wide enough for a 32-bit key later) and a type (privacy_type[i], dmrPrivacyType_t). privacy_active picks the slot in use radio-wide: 1 to 4, or 0 for privacy off. Turning privacy off keeps the key in its slot.

Today the radio core writes slot 1 when the key is set (the debug link's RADIO privacy op) and loads the active slot at boot, checking it with dmrPrivacyKeyIsValid. A per-channel choice of key will be a slot number in the native channel record (codeplug-native.md 3: 0 = the radio-wide choice, 1 to 4 = that slot). The keys themselves stay in the settings and never go into a codeplug file, so a codeplug can be shared without its keys.

Open: the keys are stored in plain text in the SPI NOR. Anyone with the radio and a flash reader, or with the CPS shim's area 1 reads, can see them. The settings area is outside what the open-firmware CPS reads by region, but area 1 reads any address. Options: refuse area 1 reads of the settings area, or obfuscate with a key from the MCU's UID. Neither stops a determined reader, and basic privacy is weak anyway.

4. The sector journal

A codeplug edit, or a CPS commit, rewrites a whole 4 KB sector: erase, then program. A cut between the two would leave the sector erased: channels, zones or contacts gone. The journal (core/storage/storage.c) makes the rewrite repeatable. It uses two sectors at journal_addr:

storage_sector_write(addr, data):

1. If the target already holds exactly data, nothing is done (storage.sector_skips).
2. Erase J1, program the copy, read it back and check its CRC.
3. Erase J0, program the header.
4. Erase the target, program data, read it back.
5. Program the done byte to 0.

storage_init() at boot, before the settings and the codeplug are read: a J0 with a valid header and a pending done byte means a cut after step 3. If J1's CRC matches, steps 4 and 5 run again (storage.replays). A cut before the header is complete leaves the target untouched, because the header is the commit point. A cut during the replay is replayed again at the next boot.

storage_write(addr, buf, len) writes any range (the codeplug editor's records, names and bitmaps). Open-firmware channel, zone and contact records don't divide 4 KB, so some cross a sector end; two separate sector rewrites would leave such a record half old, half new after a cut between them (a channel with the new frequency and the old timeslot). So a write of up to 4072 bytes, which touches one or two sectors, is one transaction:

1. If the flash already holds exactly the data, nothing is done.
2. Erase J0, program the header and the data, read them back (the commit point).
3. For each sector: J1 erased and programmed with the sector merged with the data, "staged"; the sector erased and programmed from J1, read back, "done".
4. Program "all done".

Replay: a header with a good CRC and "all done" pending finishes the sectors not done. A sector not staged has not been erased yet, so it is merged with the data from J0 again; a staged one is copied from J1. A longer write (none of the core's callers makes one) goes sector by sector through storage_sector_write. The bytes are merged in the shared 4 KB buffer, which is shared with the CPS shim's pending sector (storage_buffer_acquire), so an edit while a CPS write is going on fails cleanly instead of waiting.

A flash that is copy on write underneath offers hal_flash_t.replace (a NAND flash, nand-sectors.md 5): the sector rewrite then calls it and checks the result. There is no journal on such a flash (J0 and J1 stay blank), because the replace is one atomic record, and the replay at boot finds nothing to finish. The patch transaction of storage_write (up to 4 KB across one or two sectors) still uses the journal.

Wear: J0 and J1 are erased once per sector rewrite. A full qdmr write is about 46 sectors, so the journal sectors take 46 erases per write. At 100,000 cycles, that is about 2,000 full codeplug writes. Spreading the journal over a ring is possible later. The map has room after 0x01B000.

Locks: storage_lock() is a recursive FreeRTOS mutex around every flash sequence of the core: the codeplug calls, a settings save, a CPS commit. storage_quick_lock() guards only RAM: the live settings copy. The radio task takes only the quick lock on its own, so it never waits for an erase. A codeplug read from the radio task (a channel change) does take storage_lock and may wait out a sector rewrite (about 100 ms) during a CPS write.

5. Foreign region guard

The flash of a radio that ran other firmware may hold that firmware's data where our settings ring and journal go. Before anything is erased or programmed there, storage_init() classifies the region:

Foreign = some part holds other bytes and no valid record of ours (a settings record or a J0 header with a good CRC) exists anywhere in the region. Then:

• storage_init() returns false, storage_ready() is false, nothing is replayed.
• Every erase and program in the region is refused. storage_sector_write and storage_write fail (storage.foreign_refusals), so codeplug edits and CPS commits fail too (the journal is the region). The CPS shim refuses X 1 (cps-protocol.md 2.3).
• settings_init() returns SETTINGS_FOREIGN and the radio runs on the defaults in RAM; changes are kept in RAM and not saved.
• The debug link's view shows settings.load=foreign, storage.region=foreign, storage.journal=foreign and storage.foreign_slots (slots and journal sectors with other bytes).

Junk beside a valid record of ours (a torn erase of our own ring) is not foreign: the slots are skipped and reused once their sector comes round.

Ownership mark. When the region is ours but holds no valid record (a blank flash), storage_init() programs one valid settings slot of layout 0 and sequence 0 (no payload) in the first free slot. The settings load their defaults past it (no settings layout is 0) and the first real save gets sequence 1. From then on a valid record always exists, because saves never erase the sector with the newest record, so a power cut inside a journal rewrite (J0 erased, J1 full) can't make the region look foreign at the next boot.

Claim. Only through the debug link, behind the chip UID token like REBOOT (debuglink.md 3.13). Two clients send it:

• The web radio programming utility, right after a full backup (the backup holds the whole SPI flash, the region included), when STORAGE INFO reports the region foreign. The Codeplug Editor checks before every write and, on a foreign region, explains and claims only when the user confirms. Both poll INFO until the region is ours (20 s limit).
• rfoundry_debug.py storage claim --out DIR:

1. rfoundry_debug.py storage claim reads the settings ring, the journal and the calibration copy through the debug link's read-only SPI flash window (twice, compared) into --out DIR (refused inside any git work tree), then sends the claim with the token.
2. The firmware notes the request (storage.claim_pending). The housekeeping task (settings_poll) erases every settings sector and both journal sectors, checks they read blank, marks the region ours and writes the ownership mark, then saves the settings the radio is running with.
3. The tool polls until storage.region=ours.

rfoundry_debug.py storage dump saves the same files without claiming; storage info shows the state and addresses.

6. Testing

View the source of this page