When startOTAUpdate() was called while the device was already connected
as a STA (web panel or MQTT active), it would bring up an open
"MeshCore-OTA" AP alongside the existing connection and report its IP
(softAPIP, typically 192.168.4.1) in the reply. The OTA AsyncWebServer
also listens on the STA interface, so the update page was reachable via
the existing network IP, but that address was not reported.
Two problems with the old behaviour:
- Raising an open AP while a STA connection is active is a security
concern.
- The reported AP address is not useful to a user who is already on the
same network as the device; they would naturally try the STA IP.
New behaviour:
- If the device is already connected as a STA (WiFi.status() ==
WL_CONNECTED), skip softAP() entirely and report WiFi.localIP() in
the reply. The OTA server is already reachable on that address.
- If there is no STA connection, keep the original behaviour: bring up
the "MeshCore-OTA" AP and report softAPIP().
In both cases the reply now includes the network name so the user knows
which interface to connect to:
STA: "Started: http://<ip>/update (WiFi: <ssid>)"
AP: "Started: http://<ip>/update (AP: MeshCore-OTA)"
If a user specifically wants to perform OTA via the dedicated AP, they
can disable the current WiFi interface first, after which startOTAUpdate
will fall into the AP path as before.
Calling `set web off` through the web panel caused the device to either
hang permanently or crash with a Guru Meditation Error (LoadProhibited).
Root cause: the HTTP command handler runs inside the httpd task and
synchronously called `setWebEnabled(false)`, which called
`_panel.stop()` -> `httpd_ssl_stop()` on the very connection that was
serving the request. This is unsafe in two ways:
1. `httpd_ssl_stop()` blocks waiting for the httpd task to finish, but
the httpd task is the one executing the handler — a self-deadlock.
2. Even if the stop proceeds, closing the active TLS connection from
within its own handler triggers lwIP teardown (esp_netif_down_api ->
dhcp_stop -> TCP RST) while the WiFi driver is in an inconsistent
state, causing a null-pointer dereference in ieee80211_output_do.
The deadlock was further compounded by a second concurrent call:
while the httpd task was stuck in httpd_ssl_stop(), the main loop kept
running, called ensureWebServer(), saw _server != nullptr (never cleared
because httpd_ssl_stop() never returned), and issued a second
httpd_ssl_stop() on the same handle — blocking the main loop as well
and stopping all radio packet processing until reboot.
The explicit start/stop calls in setWebEnabled() were redundant:
WebService::loop() already calls ensureWebServer() on every iteration,
which starts the server when enabled and WiFi is up, and stops it
otherwise. Remove the block entirely and let loop() handle both
transitions safely from the main loop context, outside the httpd task.
When the "start ota" command was issued from the web panel, a race
condition could occur between the main loop thread and the HTTP server
thread:
- HTTP thread: prepareForOTAStart() -> stopRedirectServer() ->
httpd_stop(_redirect_server) [blocking]
- Main thread: loop() -> ensureWebServer() -> stopRedirectServer() ->
httpd_stop(_redirect_server) [double free!]
Because stopRedirectServer() sets _redirect_server = nullptr only after
httpd_stop() returns, both threads could pass the nullptr check
simultaneously, resulting in a double free and heap corruption:
CORRUPT HEAP: Bad head at 0x3fcb1f24. Expected 0xabba1234 got 0x3fca5f34
assert failed: multi_heap_free multi_heap_poisoning.c:259 (head != NULL)
The call in ensureWebServer() was added as a safety net in commit
4b83142b, but is redundant: prepareForOTAStart() already calls
stopRedirectServer() synchronously before startOTAUpdate() occupies
port 80. Removing it eliminates the race condition.
The bug only manifested when OTA was triggered from the web panel
(HTTP thread), not from radio (main thread), because in the latter
case both calls happen on the same thread and cannot race.
The previous while loop computed a ternary `chunk_len` on every
iteration to handle the final partial chunk as a special case, and
called vTaskDelay(1) after every chunk including the last one,
adding an unnecessary yield immediately before the terminating null
chunk.
Restructure the loop using the identity:
last_size = ((len - 1) % kWebPageChunkSize) + 1
This gives `last_size` in [1, kWebPageChunkSize] for any non-zero `len`,
so the final partial chunk is always non-empty and can be sent after
the loop without a special case. The for loop then iterates only over
full kWebPageChunkSize chunks, making every iteration identical and
branch-free, and vTaskDelay(1) is called only after full chunks.
Also add a `len == 0` guard to short-circuit immediately when there is
nothing to send, and remove the best-effort terminator send on the
error path since a failed send makes a follow-up send equally likely
to fail.
The HTML pages (login, app, stats) are string literals compiled into
PROGMEM. Their sizes are known at compile time, but the previous
sendProgmemChunked() called strlen() on every request to determine
the length, requiring a full scan of potentially tens of kilobytes of
PROGMEM data before the first byte was sent.
Replace sendProgmemChunked() with a sendWhole() function that accepts
an explicit length parameter, and introduce a sendProgmem() macro that
passes sizeof(mem) - 1 at the call site. For array literals this
resolves to a compile-time constant, eliminating the runtime strlen
scan entirely.
The rename from sendProgmemChunked() to sendWhole() also better reflects
what the function does — it sends the entire buffer in chunks — rather
than where the data comes from.
The login, app, and stats HTML pages are static assets compiled into
PROGMEM. All dynamic content is fetched separately via JavaScript API
calls after the page loads — the HTML itself never changes between
requests within the same firmware version.
The "Cache-Control: no-store" header forced the browser to re-download
the full HTML on every visit, including a complete chunked transfer
from the ESP32. Given that the ESP32 is slow and serves responses in
small chunks, this was an unnecessary repeated cost that added latency
before the page became interactive.
Removing no-store allows the browser to cache the HTML pages locally.
Subsequent requests are served from the browser cache instantly, with no
transfer from the ESP32 at all, leaving the server free to handle the
API requests that actually carry dynamic data.
The HTTP redirect server has max_open_sockets = 2. A client that
connects and then disappears silently (network drop, browser crash,
mobile radio loss) leaves its socket occupying a slot indefinitely —
the server has no way to detect the loss until it tries to write to
the socket again.
Enable lru_purge_enable on the redirect server so that when both
slots are occupied by such stale connections, the least-recently-used
one is evicted automatically to make room for a new incoming
connection, rather than refusing it outright.
Add a "Connection: close" header to the 302 response as a complementary
measure. For well-behaved clients this triggers an immediate TCP
teardown after the redirect is received, shrinking the window during
which a connection can turn into a zombie. Together the two changes
provide defence in depth: "Connection: close" prevents stale connections
from forming in the first place; lru_purge_enable cleans them up when
they do.
The ESP32 processes HTTP requests sequentially on a single task.
Under load — generating chunked HTML pages, running commands, or
serving sequential stats API calls — it can be slow to send data
or loop back to accept the next request. The previous 2-second
recv/send timeouts were too tight for this: the httpd layer would
drop connections mid-transfer, causing the browser to hang or
show incomplete pages.
Raise recv_wait_timeout and send_wait_timeout from 2 s to 10 s
on both the HTTPS and HTTP-redirect servers, giving the ESP32
enough headroom to finish generating and sending responses without
the transport layer tearing down the connection prematurely.
Set backlog_conn to 0 on both servers. With the previous backlog
of 2, incoming connections completed the TCP handshake and queued
in the kernel while the ESP32 was busy. The browser saw the
connection as open but received no HTTP response, causing it to
freeze indefinitely. With backlog 0 (lwIP minimum), connections
that cannot be immediately accepted are refused outright, giving
the browser a fast, recoverable error instead of a silent hang.
Enable lru_purge_enable on the HTTPS server. Browsers hold
keep-alive connections open for reuse. With max_open_sockets = 2,
both slots can be occupied by idle keep-alive connections from the
same session, blocking a new connection attempt entirely. LRU
purge automatically closes the least-recently-used idle keep-alive
connection to make room, ensuring the single client can always
reconnect without a server restart.
When sending large PROGMEM content in chunks over HTTPS, the lwIP TCP/IP
task (tiT) could monopolize CPU 0 for an extended period without ever
yielding, starving the IDLE0 task and triggering the task watchdog timer.
Add vTaskDelay(1) at the end of each iteration in sendProgmemChunked()
to yield to the scheduler between chunks, allowing the IDLE task to reset
the watchdog and preventing spurious reboots during web panel page loads.
The hardcoded kWebPageChunkSize of 768 bytes caused the task watchdog
to trigger when serving the web panel over HTTPS. Each call to
httpd_resp_send_chunk() results in a separate TLS record encryption
via mbedTLS, which on ESP32-S3 uses DMA-backed AES-GCM (esp_aes_process_dma).
The gdma_disconnect() call inside that path enters a critical section,
blocking the IDLE0 task. With 768-byte chunks, a large page response
requires many such DMA operations in tight succession, starving the IDLE
task long enough to trip the watchdog.
Replacing the hardcoded value with MBEDTLS_SSL_OUT_CONTENT_LEN aligns the
chunk size to the TLS output record buffer, minimising the number of TLS
records (and thus DMA encryption operations) needed to send a full page,
and keeping the httpd task within the watchdog timeout.
Fixes: task_wdt abort in sendProgmemChunked() -> httpd_ssl_send() ->
esp_aes_process_dma() -> gdma_disconnect() on ESP32-S3.
Browsers negotiate ECDHE cipher suites by default. On ESP32-S3 the
hardware RSA accelerator handles RSA key exchange efficiently, but
there is no ECP hardware accelerator. ECDHE requires the server to
compute an ephemeral key pair: ecp_precompute_comb() builds a comb
table through many sequential ECP point doublings, each dispatched
to the hardware bignum unit (esp_bignum.c), but the ECP layer has
no RTOS yield points between iterations. The entire computation runs
to completion on CPU 0 without ever resetting the task watchdog.
A single handshake does not exceed the watchdog timeout on its own,
but two consecutive handshakes (e.g. a browser retry after a failed
attempt) accumulate enough uninterrupted runtime to starve IDLE0:
E (54924) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x0050
E (57208) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7280
E (57638) task_wdt: Task watchdog got triggered.
E (57638) task_wdt: - IDLE0 (CPU 0)
E (57638) task_wdt: Tasks currently running:
E (57638) task_wdt: CPU 0: httpd
The crash occurs in ecp_precompute_comb() → ecp_double_jac() →
mbedtls_mpi_mul_mpi() during the ServerKeyExchange step.
Fix by wrapping mbedtls_ssl_config_defaults() via the linker --wrap
mechanism. The wrapper intercepts server-side SSL config init
(MBEDTLS_SSL_IS_SERVER) and replaces the cipher suite list with
RSA key exchange only, routing handshakes through the hardware RSA
accelerator and eliminating the ECDH path entirely. MQTT connections
(MBEDTLS_SSL_IS_CLIENT) are unaffected.
Also switch the self-signed cert generator from EC (prime256v1) to
RSA 2048 so the generated certificate matches the restricted cipher
suites.
Changes:
- arch/esp32/tls_cipher_restrict.c: new file implementing the
mbedtls_ssl_config_defaults wrap; restricts server cipher suites
to RSA_WITH_AES_{128,256}_{GCM,CBC}_SHA{256,384}
- platformio.ini: add -Wl,--wrap=mbedtls_ssl_config_defaults and
include tls_cipher_restrict.c in the esp32_base build
- arch/esp32/extra_scripts/generate_web_panel_cert.py: switch key
generation from `openssl ecparam -name prime256v1` to `openssl
genrsa 2048`
On dual-core ESP32-S3 (ARDUINO_RUNNING_CORE=1), the Arduino loop and
LoRa processing run exclusively on core 1. ESP-IDF v4 creates mqtt_task
with tskNO_AFFINITY, meaning FreeRTOS may schedule them on core 1 under
load, preempting the LoRa loop.
ESP-IDF v4 provides no public API to change a task's core affinity after
creation (vTaskCoreAffinitySet is IDF v5+ only), and the esp_mqtt_client
config struct has no task_core_id field. The precompiled Arduino-ESP32
framework cannot be patched via sdkconfig.
Instead, use the GCC/LD --wrap linker mechanism to intercept every call
to xTaskCreatePinnedToCore. Any task created with tskNO_AFFINITY is
redirected to core 0. Tasks that are already explicitly pinned (Wi-Fi
driver, LwIP, httpd, esp_timer, ipc0/ipc1) are passed through unchanged.
We intentionally do not filter by task name. Pinning all unpinned tasks
makes the approach robust against internal ESP-IDF task name changes and
catches any future tasks that may be added with tskNO_AFFINITY.
Verified task layout after the change:
loopTask pri=1 core=1 (Arduino loop / LoRa — unchanged)
mqtt_task pri=5 core=0 (was tskNO_AFFINITY, now pinned)
httpd pri=2 core=0 (core_id set explicitly in WebPanelServer)
tiT pri=18 core=0 (LwIP, already pinned by ESP-IDF)
wifi pri=23 core=0 (Wi-Fi driver, already pinned)
esp_timer pri=22 core=0 (already pinned)
ipc0/ipc1 pri=24 core=0/1 (IPC, already pinned per-core)
The -Wl,--wrap flag and arch/esp32/task_pinning.c are added only to
[esp32_base] (IDF v4). The ESP32-C6 pioarduino target (IDF v5) is
unaffected and can use vTaskCoreAffinitySet() if needed in the future.
Set task_priority = tskIDLE_PRIORITY + 2 and core_id = 0 for both the
HTTPS server and the HTTP-to-HTTPS redirect server. This keeps web
serving off core 1, which handles radio and application logic, reducing
interference with time-sensitive operations.
_have_time_sync is reset to false whenever WiFi disconnects, even
though the ESP32 RTC continues to hold accurate time after a
successful SNTP sync. This caused hasTimeSync() to return false
during transient WiFi outages, unnecessarily tearing down MQTT
broker connections and suppressing packet publishing.
Introduce _last_time_sync to record the wall-clock time of the
most recent confirmed sync. Move hasTimeSync() out of the header
into NetworkService.cpp and extend its logic: in addition to the
existing _have_time_sync flag, return true if the system clock is
still sane (>= kMinSaneEpoch) and no more than kMaxOutOfSync (24h)
has elapsed since the last confirmed sync.
This makes the MQTT uplink resilient to brief WiFi dropouts without
requiring any changes to callers of hasTimeSync().
Also bump kMinSaneEpoch from 2025-01-01 to 2026-01-01.
There's no known reason why additional firmware build of this board
with serial logging enabled was needed.
So, removing it for now to optimize building process and cleanup
firmware list.
