Inspecting backtrace / stack trace after ESP32 panic using espressif xtensa toolchain
Background
There are time when the ESP32 got panic and reboot itself, and the output is as follow:
It’s pretty frustrating for someone with application development background who rely on stack trace output or using a debugger to setup breakpoint to debug.
ESP32 does not come with a debugger for most of their variety of boards and so if we need a debugger, we might need to purchase additional gears.
So here comes the question, how can I figure out where does it break?
Development Environment Setup
Before I step into the troubleshooting steps, I would like to mention that instead of using the Arduino IDE, I instead use Platform.io extension on Visual Studio Code on MS Windows, which is a more familiar IDE for me as well as better support like intellisense (code suggestion) or navigate to source code of imported libraries.
Reference: https://platformio.org/
Troubleshooting
The immediate objective is to look at the output and “guess” where it break, and further narrow the areas need to be inspected.
Looking at the first part of the output messages:
The first thing is the Debug exception reason, in the bracket, it mentioned (nimble_host), which seems to point to the NimBLE library I included.
The second thing to spot, is the register “PC”, PC is the Program Counter, which store the memory address that the program run to (when it panic).
So the immediate question would be where is this address point to at my source code? Is there a way to find out?
The solution is the the espressif tool chain have a tool called xtensa, which have a command (or executable) “xtensa-esp-elf-addr2line”, for those would use Platform.io plugin like myself, the installed location of this tool is at “C:\Users\<your user name>\.platformio\packages\toolchain-xtensa-esp32\bin”
Running the “xtensa-esp-elf-addr2line” command
The command’s help as follow:
The actual command I use as follow:
xtensa-esp32-elf-addr2line.exe -pfiaC -e C:\Users\<usename>\OneDrive\Documents\PlatformIO\Projects\ESP32Cam\.pio\build\esp32cam\firmware.elf -a 0x40099733And the output is as follow:
This does not help much (we can go to source code to check, but it’s at the leaf level of the stack, so we need more information up the stack)
Let’s inspect the second part of the panic output, the backtrace:
This looks like having some pattern: 0x400XXXXX:0x3ffdYYYY, they looks like memory addresses. The first part (0x400XXXXX) are close to our PC register and the 2nd part (0x3ffdYYYY) look close to our A1 register (before it panic)
By running the last address pair and going up, we can see something like a stack trace:
Based on the above, we can see the last 3 addresses looks like stack trace chain call of the bluetooth library (from code of BLE Characteristics to code about GATT).
The story can suspend here, but the debug goes on…
Supplementary information
Digging deeper, the address space is well defined in ESP32 technical specification document as follow:
And for embedded memory:
That align with our understanding, the PC fall into the instruction memory block, and interestingly the backtrace address for instructions fall between embedded memory and external memory addresses.
