Both Clyde North observers stopped answering over WiFi after days of uptime while LoRa and MQTT kept working. `memory` showed heap_min ~1KB and a largest internal block of 19KB: the HTTPS listener was alive but no mbedTLS handshake (~40KB contiguous) could be allocated. The trigger was the Aug 9 lru_purge change: with a 2-socket pool, every browser page-load burst evicted a live session and forced a fresh TLS handshake, and the repeated 40KB alloc/free cycles fragmented internal RAM. Before that change the pool simply jammed, so nothing churned. - Send `Connection: close` and trigger a session close on one-shot responses (pages, favicon, redirect, login, 401s) so page-load bursts release their sockets immediately. The authenticated /api/* polling connection keeps keep-alive so it does not pay a handshake per poll. - Gate WebPanelServer::start() on internal heap headroom (56KB free / 32KB largest), retrying every 15s instead of every loop tick. - Self-heal: when the panel is idle and the largest internal block drops below 24KB, stop and re-create the server to return its pools. - Count the MQTT teardown path that deliberately abandons a client on a heap-integrity failure. - Expose the above as `heals:`/`deferred:` in `get web.status` and `leaked:` in `get mqtt.status`. - Fix the stats page Channel/Gateway cells showing `--`: the /api/stats summary JSON never carried channel, gateway health, or the watchdog count (only the `get wifi.status` string did). - Add eastmesh-tools/web-heap-check.sh to read heap/service health over the API and optionally stress the panel with browser-style bursts. - Release notes 2026.8.3 for both observer tracks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
327 lines
17 KiB
Markdown
327 lines
17 KiB
Markdown
# Custom CLI Commands
|
|
|
|
This page covers the EastMesh-specific CLI commands added in this repository.
|
|
|
|
It does not try to repeat the full upstream MeshCore CLI surface.
|
|
|
|
## Start Here
|
|
|
|
If you are doing first-time setup, these are the commands most users need before anything else:
|
|
|
|
```text
|
|
set wifi.ssid <your-ssid>
|
|
set wifi.pwd <your-password>
|
|
get wifi.status
|
|
set mqtt.iata <code>
|
|
get mqtt.status
|
|
```
|
|
|
|
For observers with the local web panel, these are also useful:
|
|
|
|
```text
|
|
set web on
|
|
get web.status
|
|
set web off
|
|
```
|
|
|
|
Use `set web on` while setting up or troubleshooting, then use `set web off` when a fixed observer needs maximum memory headroom.
|
|
|
|
## Observer Commands
|
|
|
|
These commands are available on `*_repeater_observer` firmware targets.
|
|
|
|
No-argument `get` commands must be entered exactly as shown.
|
|
|
|
### MQTT Status And Routing
|
|
|
|
- `get mqtt.status`: shows Wi-Fi, NTP, IATA, endpoint status, custom endpoint transport, status publishing state, TX state, and `leaked:<n>` (MQTT clients intentionally left unfreed after a heap-integrity failure at teardown; should stay `0`).
|
|
- `get mqtt.statuscfg`: shows whether periodic status messages are enabled as a simple `on` or `off` value. Most users can just use `get mqtt.status`.
|
|
- `get mqtt.client_version`: shows the MQTT `client_version` string published by the repeater.
|
|
- `get mqtt.client_env`: shows the PlatformIO env used to build the repeater firmware.
|
|
- `get mqtt.iata`: shows the IATA/location code used in MQTT topics.
|
|
- `set mqtt.iata <code>`: sets the IATA/location code, for example `MEL`.
|
|
- `set mqtt.iata UNSET`: marks MQTT IATA as not configured yet. While it is `UNSET`, enabled MQTT brokers stay disconnected until a real code is saved.
|
|
|
|
### MQTT Identity
|
|
|
|
- `get mqtt.owner`: shows the configured owner public key.
|
|
- `set mqtt.owner <64-hex-char-public-key>`: sets the owner public key used in JWT metadata.
|
|
- `mqtt.owner <64-hex-char-public-key>`: shorthand for setting the owner public key.
|
|
- `get mqtt.email`: shows the configured owner email.
|
|
- `set mqtt.email <email>`: sets the owner email used in JWT metadata.
|
|
- `mqtt.email <email>`: shorthand for setting the owner email.
|
|
|
|
### MQTT Message Controls
|
|
|
|
- `get mqtt.packets`: shows whether packet messages are published.
|
|
- `set mqtt.packets on|off`: enables or disables packet publishing.
|
|
- `get mqtt.raw`: shows whether raw packet payloads are published.
|
|
- `set mqtt.raw on|off`: enables or disables the separate `raw` MQTT topic.
|
|
- `set mqtt.status on|off`: enables or disables periodic MQTT status publishing.
|
|
- `get mqtt.tx`: shows whether TX packets are included.
|
|
- `set mqtt.tx on|off`: enables or disables TX packet publishing.
|
|
|
|
### MQTT Endpoints
|
|
|
|
- `get mqtt.eastmesh-au`
|
|
- `set mqtt.eastmesh-au on|off`
|
|
- `get mqtt.meshmapper`
|
|
- `set mqtt.meshmapper on|off`
|
|
- `get mqtt.waev`
|
|
- `set mqtt.waev on|off`
|
|
- `get mqtt.letsmesh-eu` (retired)
|
|
- `set mqtt.letsmesh-eu on|off` (retired)
|
|
- `get mqtt.letsmesh-us` (retired)
|
|
- `set mqtt.letsmesh-us on|off` (retired)
|
|
- `get mqtt.custom`
|
|
- `set mqtt.custom on|off`
|
|
- `get mqtt.custom.host`
|
|
- `set mqtt.custom.host <host>`
|
|
- `get mqtt.custom.port`
|
|
- `set mqtt.custom.port <port>`
|
|
- `get mqtt.custom.transport`: shows `tcp` or `wss`.
|
|
- `set mqtt.custom.transport tcp|wss`
|
|
- `get mqtt.custom.username`
|
|
- `set mqtt.custom.username <username>`
|
|
- `get mqtt.custom.password`: shows `set` when a custom password is configured.
|
|
- `set mqtt.custom.password <password>`
|
|
|
|
Notes:
|
|
|
|
- new observer installs default `mqtt.iata` to `UNSET`
|
|
- a maximum of two MQTT brokers can be enabled at once
|
|
- `meshmapper` is the curated global broker (`wss://mqtt.meshmapper.net:443/mqtt`); like `eastmesh-au` it uses WSS, verified TLS, and MeshCore JWT auth with the broker host as the token audience
|
|
- `waev` is a curated broker (`wss://mqtt.waev.app:443/mqtt`); like `eastmesh-au` it uses WSS, verified TLS, and MeshCore JWT auth with the broker host as the token audience, but enforces a 1-hour auth token lifetime (the firmware mints a shorter-lived token for it automatically)
|
|
- `letsmesh-eu` and `letsmesh-us` are retired (LetsMesh is no longer maintained) and off by default; any saved selections are cleared once on upgrade, though the endpoints stay selectable for legacy use
|
|
- if `mqtt.iata` is `UNSET`, enabled MQTT brokers will not connect
|
|
- custom MQTT uses the configured username and password, not JWT authentication
|
|
- custom MQTT defaults to TCP; set `mqtt.custom.transport wss` for MQTT over secure WebSockets using the fixed `/mqtt` websocket path and the ESP-IDF x509 root CA bundle
|
|
- `get mqtt.status` reports the two enabled broker slots as `p:<broker>:<state>` (primary) and `s:<broker>:<state>` (secondary), or `-` when a slot is empty; custom appears as `custom:<transport>:<state>` (for example `p:custom:wss:up`); `conn` means connecting and `up` means connected
|
|
- custom MQTT uses the same `meshcore/<IATA>/<device>/<leaf>` topics as the curated brokers
|
|
- turning off a connected broker publishes a retained MQTT status update with `"status":"offline"` before the client disconnects
|
|
- changing `mqtt.iata` away from a configured value also publishes retained offline status to the old status topic, restarts connected broker clients, and reconnects under the new topic path
|
|
|
|
Legacy dotted aliases are also accepted:
|
|
|
|
- `mqtt.eastmesh.au`
|
|
- `mqtt.letsmesh.eu`
|
|
- `mqtt.letsmesh.us`
|
|
|
|
### Wi-Fi Settings For Observers
|
|
|
|
- `get wifi.status`: shows SSID, connection state, raw Wi-Fi status code, IP, channel, and signal when connected, plus gateway health (`gw:ok|lost`) and the watchdog reconnect count (`wd:<n>`).
|
|
- `get wifi.ssid`: shows the configured Wi-Fi SSID.
|
|
- `set wifi.ssid <ssid>`: sets the Wi-Fi SSID.
|
|
- `set wifi.pwd <password>`: sets the Wi-Fi password.
|
|
- `get wifi.powersaving`: shows the current Wi-Fi power save mode.
|
|
- `set wifi.powersaving none|min|max`: sets Wi-Fi power saving mode.
|
|
- `wifi reconnect`: drops the current association and rejoins from a full channel scan. Use when a node reports connected but is unreachable over the network.
|
|
|
|
Observers also run a connectivity watchdog: while Wi-Fi reports connected, the node ARP-probes its gateway every 30 seconds. If the gateway stays silent for 3 minutes (for example, an AP that keeps beaconing after losing its wired uplink), the node forces a full reconnect on its own, backing off up to 48 minutes between attempts while the outage persists. `wd:<n>` in `get wifi.status` counts these forced reconnects since boot.
|
|
|
|
### NTP Settings For Observers
|
|
|
|
- `get ntp.server1`: shows the primary NTP server.
|
|
- `get ntp.server2`: shows the secondary NTP server.
|
|
- `get ntp.server3`: shows the tertiary NTP server.
|
|
- `set ntp.server1 <host>`: sets the primary NTP server and restarts time sync.
|
|
- `set ntp.server2 <host>`: sets the secondary NTP server and restarts time sync.
|
|
- `set ntp.server3 <host>`: sets the tertiary NTP server and restarts time sync.
|
|
|
|
Default servers are `au.pool.ntp.org`, `time.google.com`, and `time.cloudflare.com`.
|
|
|
|
### ESP-NOW Bridge Settings For Observer ESP-NOW Builds
|
|
|
|
These commands are available on `*_repeater_observer_espnow` firmware targets that use the ESP-NOW bridge transport.
|
|
|
|
Bridge commands are for local ESP-NOW bridge use between nearby repeaters, such as linking repeaters on `Australia (Narrow)` and `Australia (Mid)`. They are not MQTT-over-WAN, VPN, or internet bridge controls.
|
|
|
|
- `get bridge.channel`: shows the configured ESP-NOW bridge channel.
|
|
- `set bridge.channel <channel>`: sets the ESP-NOW bridge channel and restarts the bridge. Use a value from `1` to `14`.
|
|
- `get bridge.secret`: shows the configured ESP-NOW bridge secret.
|
|
- `set bridge.secret <secret>`: sets the shared ESP-NOW bridge secret and restarts the bridge.
|
|
|
|
After running `set bridge.channel`, expect the bridge and web panel connection to drop briefly while the radio restarts. On current observer ESP-NOW builds, this can look like the board rebooted.
|
|
|
|
For `*_repeater_observer_espnow` builds that are connected to Wi-Fi, the ESP-NOW bridge channel must match the active 2.4 GHz Wi-Fi channel:
|
|
|
|
1. Run `get wifi.status`.
|
|
2. Read the `channel:<n>` value from the connected Wi-Fi status.
|
|
3. Run `get bridge.channel`.
|
|
4. If the values differ, run `set bridge.channel <n>` using the Wi-Fi channel value.
|
|
5. Use the same `bridge.channel` and `bridge.secret` on every ESP-NOW bridge node that should talk together.
|
|
|
|
Example:
|
|
|
|
```text
|
|
> get wifi.status
|
|
> ssid:EastMesh-IoT status:connected code:3 state:connected ip:192.168.1.50 channel:6 rssi:-61 quality:78% signal:good gw:ok wd:0
|
|
> get bridge.channel
|
|
> 1
|
|
> set bridge.channel 6
|
|
OK
|
|
```
|
|
|
|
### Web Panel Controls
|
|
|
|
- `get web`
|
|
- `get web.status`: shows whether the local HTTPS panel is available, plus `heals:<n>` (times the panel was restarted because internal heap was too fragmented for a TLS handshake) and `deferred:<n>` (start attempts postponed until heap headroom recovered).
|
|
- `get web.stats.status`: shows whether the dedicated stats page and history subsystem are enabled, whether recent history is active, whether PSRAM-backed history is available, and whether the SD-backed archive is mounted. When enabled, the history capture now covers supported environment telemetry too, not just the original battery/radio series. GPS-active boards also record per-minute satellites samples for the `/stats` history view. If archive access drops while stats remain enabled, the repeater retries the SD mount periodically.
|
|
- At boot, archive-backed stats restore uses bounded reads of the latest summary, events, and neighbour snapshots so malformed or unexpectedly large archive files do not delay MQTT or web startup.
|
|
- `set web on|off`
|
|
- `set.web on|off`: enables or disables the local HTTPS panel.
|
|
- `set web.stats on|off`
|
|
- `set.web.stats on|off`: enables or disables the dedicated `/stats` page and historical stats collection.
|
|
- `purge sd`: deletes files and directories from the mounted SD card archive, then recreates the empty `/stats` archive directory so runtime stats capture can continue. It does not erase the internal repeater filesystem or stored settings.
|
|
|
|
### Runtime Diagnostics
|
|
|
|
- `memory`: shows current heap and PSRAM usage.
|
|
- `stats-core`: shows battery, uptime, sticky error count, and outbound queue depth.
|
|
- `stats-radio`: shows radio noise floor, last RSSI, last SNR, and TX/RX airtime.
|
|
- `stats-packets`: shows packet receive/send totals, flood/direct breakdown, and receive errors.
|
|
|
|
> If `noise_floor` reports `0`, check `get agc.reset.interval`; if it is not `0`, try `set agc.reset.interval 0` and test again.
|
|
|
|
### Flood Forwarding Limit
|
|
|
|
- `get flood.max.unscoped`: shows the hop limit for unscoped flood packets.
|
|
- `set flood.max.unscoped <0-64>`: sets the hop limit for unscoped flood packets.
|
|
- `get flood.max.advert`: shows the hop limit for flooded advert packets.
|
|
- `set flood.max.advert <0-64>`: sets the hop limit for flooded advert packets.
|
|
|
|
Observer builds default `flood.max.unscoped` to `64`. Lower values can limit how far unscoped flood traffic is repeated while leaving scoped/region flood forwarding controlled by `flood.max`.
|
|
|
|
### Board Battery Reporting
|
|
|
|
- On observer builds, background battery sampling used for MQTT/status history is rate-limited to about once per minute. Explicit status and telemetry requests still refresh the reading immediately.
|
|
|
|
### T-Beam 1W Fan Control
|
|
|
|
These commands are only available on `LilyGo_TBeam_1W_*` repeater builds.
|
|
|
|
- `get fan`: shows the current fan mode, current fan state, and the last NTC-based board temperature when available.
|
|
- `set fan auto`: returns the fan to automatic control and persists that mode across reboot.
|
|
- `set fan on`: forces the fan on and persists that mode across reboot.
|
|
- `set fan off`: forces the fan off and persists that mode across reboot.
|
|
- `set fan timeout <Ns>`: changes the automatic post-TX hold window in seconds and persists it across reboot, for example `set fan timeout 45s`.
|
|
|
|
Auto mode behaviour:
|
|
|
|
- forces the fan on during TX and keeps it on for the configured timeout afterward
|
|
- otherwise turns the fan on at `48C`
|
|
- turns it back off at `42C`
|
|
- keeps the fan on if the NTC reading is unavailable
|
|
|
|
Notes:
|
|
|
|
- default repeater fan mode is `auto`
|
|
- default post-TX timeout is `30s`
|
|
- fan mode and timeout are stored in repeater prefs and survive reboot
|
|
- only `LilyGo_TBeam_1W_*` repeater builds use these persisted fan settings
|
|
- accepted range is `0s` to `600s`
|
|
|
|
## Web Panel CLI Access
|
|
|
|
When the repeater web panel is enabled and you are authenticated, the browser CLI panel can run the same CLI commands accepted by the repeater.
|
|
|
|
Notes:
|
|
|
|
- the panel still uses the repeater admin password for access
|
|
- commands run with the same care as if you typed them into the repeater CLI directly
|
|
- this is intended for local admin use on a trusted network
|
|
- `start ota` releases the local HTTP redirect listener on port `80` so the OTA HTTP listener can take over without stopping the rest of the repeater services, regardless of whether the command is run from the web panel, serial CLI, or a remote companion/app CLI session
|
|
- if the old redirect listener has not fully released port `80`, the OTA listener retries for up to about 30 seconds before giving up
|
|
- the web `Purge SD` button runs `purge sd` after browser confirmation
|
|
- `start ota` uses the repeater's existing Wi-Fi address when already connected, or starts the `MeshCore-OTA` access point when Wi-Fi is not connected
|
|
- the `/app` Regions shortcut runs the existing MeshCore region commands in sequence: `region put au`, `region put au-STATE`, `region allowf au`, `region allowf au-STATE`, then `region save`
|
|
|
|
## Companion Wi-Fi Rescue Commands
|
|
|
|
These commands are available in the serial rescue CLI for `*_companion_radio_wifi` builds.
|
|
|
|
To enter `CLI Rescue`:
|
|
|
|
- open a serial monitor at `115200` baud
|
|
- reboot the device
|
|
- long-press the user button within the first 8 seconds after boot
|
|
- wait for `========= CLI Rescue =========`
|
|
|
|
- `get wifi.status`: shows configured SSID, connection status, raw Wi-Fi status code, IP, channel, and signal when connected.
|
|
- `get wifi.ssid`: shows the configured Wi-Fi SSID.
|
|
- `get wifi.powersaving`: shows the current Wi-Fi power saving mode.
|
|
- `set wifi.ssid <ssid>`: saves a Wi-Fi SSID and immediately retries connection.
|
|
- `set wifi.pwd <password>`: saves a Wi-Fi password and immediately retries connection.
|
|
- `set wifi.powersaving none|min|max`: changes the Wi-Fi power save mode.
|
|
|
|
Companion Wi-Fi builds also still support the existing rescue commands such as:
|
|
|
|
- `set pin <6-digit-pin>`
|
|
- `rebuild`
|
|
- `erase`
|
|
- `ls ...`
|
|
- `cat ...`
|
|
- `rm ...`
|
|
- `reboot`
|
|
|
|
### Recovery Access Point
|
|
|
|
When a companion Wi-Fi device has no Wi-Fi credentials configured, or cannot connect
|
|
to its configured network for 60 seconds, it broadcasts its own `EastMesh-WiFi`
|
|
access point so it can be rescued without a serial cable.
|
|
|
|
**The AP password is the device pin, zero-padded to 8 digits** (WPA2 requires at
|
|
least 8 characters). The pin follows the same rules as the Bluetooth pairing pin on
|
|
BLE builds:
|
|
|
|
| Device | Active pin | `EastMesh-WiFi` password |
|
|
| --- | --- | --- |
|
|
| Has a screen, no pin set | random 6-digit pin each boot, shown as `Pin:NNNNNN` on the home screen | `00NNNNNN` — read it off the screen |
|
|
| No screen, no pin set | `123456` (default) | `00123456` |
|
|
| Pin set via `set pin <pin>` | your configured pin (any device) | your pin zero-padded to 8 digits, e.g. pin `4242` → `00004242` |
|
|
|
|
Notes on the pin:
|
|
|
|
- on devices with a screen, the pin is only random while no pin has been saved; run
|
|
`set pin <pin>` for a fixed password (takes effect next boot)
|
|
- headless devices in public areas should always get a custom pin — `00123456` is a
|
|
documented default, so treat it like a default router password
|
|
|
|
Recovery steps:
|
|
|
|
1. join the `EastMesh-WiFi` network with the password from the table above
|
|
2. open the rescue CLI with `telnet 192.168.4.1` (or `nc 192.168.4.1 23`) — all
|
|
rescue commands above are available
|
|
3. `set wifi.ssid <ssid>`, then `set wifi.pwd <password>` — the device immediately
|
|
retries the network with the new credentials
|
|
4. `reboot` (or just wait — see below)
|
|
|
|
While the AP is up the device keeps retrying its configured network about once a
|
|
minute (expect a brief AP hiccup at each attempt); as soon as the station
|
|
connection succeeds, the recovery AP shuts down automatically (this also drops
|
|
your rescue session — that's the sign it worked).
|
|
|
|
### Using the Companion App over the AP (Roaming)
|
|
|
|
The recovery AP is not just for fixing credentials — the full companion protocol
|
|
is available through it, which makes it a roaming access mode when the device is
|
|
away from its configured network:
|
|
|
|
1. join `EastMesh-WiFi` with the pin-derived password (see the table above)
|
|
2. in the MeshCore companion app, add/connect a Wi-Fi device with host
|
|
`192.168.4.1` and port `5000`
|
|
3. use the app normally — messages, contacts, channels all work over the AP
|
|
|
|
Notes for roaming use:
|
|
|
|
- run `set pin <pin>` once so the AP password stays fixed across reboots;
|
|
otherwise devices with a screen pick a new random pin every boot
|
|
- the AP appears about 60 seconds after boot (the device tries its configured
|
|
network first), and stays up for as long as that network is unreachable
|
|
- back in range of its configured network, the device joins it and shuts the AP
|
|
down automatically — reconnect the app via the LAN address instead
|
|
|
|
## Heap Health Check Script
|
|
|
|
`eastmesh-tools/web-heap-check.sh <host> [--stress] [--rounds N]` reads `memory`, `get web.status`, `get mqtt.status` and `get wifi.status` over the HTTPS API and, with `--stress`, simulates browser page-load bursts before re-reading `memory`. Authenticate with `REPEATER_PASSWORD` or `REPEATER_TOKEN` in the environment (or enter the password at the prompt). Watch `heap_max` (largest internal block; TLS needs ~40KB), `heap_min`, and the `heals`/`deferred`/`leaked` counters.
|