Debugging NuttX

Finding and fixing bugs is an important part of the hardware and software development process. Sometimes you also need to use debugging techniques to understand how the system works. Two tools that are helpful are debug logging and debugging using the Gnu Debugger (gdb).

Debug Logging

NuttX has a powerful logging facility with info, warn, and error levels. You can enable debugging for your build for the net feature (TCP/IP stack) by using the menuconfig system to enable various logging levels. Messages will appear on the console by default, or to other places (memory, etc.) if you configure them.

Debug logging in the Apache NuttX documentation

You may need to do a little bit of experimenting to find the combination of logging settings that work for the problem you’re trying to solve. See the file debug.h for available debug settings that are available. This can also be configured via the menuconfig system.

There are also subsystems that enable USB trace debugging, and you can log to memory too, if you need the logging to be faster than what the console can output

Custom Debug Logging

Sometimes you need to see debug logs specific to your feature, and you don’t want the rest of the built-in logs because they’re either not relevant or have too much information. Debugging using logs is surprising powerful.

You can add your own custom debug logging by adding the following lines to debug.h:

/* after the CONFIG_DEBUG_WATCHDOG_INFO block near line 721 */
#ifdef CONFIG_DEBUG_CUSTOM_INFO
#  define custinfo    _info
#else
#  define custinfo    _none
#endif

You need to add the following line to your .config file:

CONFIG_DEBUG_CUSTOM_INFO=y

You would use it like this:

/* Custom debug logging */
custinfo("This is a custom log message.");
custinfo("Custom log data: %d", my-integer-variable);

JTAG Debugging

JTAG is a set of standards that specify a way to attach a hardware device to your embedded board, and then remotely control the CPU. You can load code, start, stop, step through the program, and examine variables and memory.

This guide assumes your board has a JTAG connector, and you have a JTAG hardware debugger that supports the SAMA5D27C like a Segger J-Link or OpenOCD.

OpenOCD now has support for the SAMA5D27C, and has been tested with the Olimex ARM-TINY-USB-H adapter. Other adapters should also work if they support OpenOCD.

See the section Running NuttX for a brief tutorial on how to use GDB.

Thread-aware Debugging

Viewing NuttX threads, and being able to switch to them and debug them, is an essential debugging tool.

See NuttX Debugging for info on how to set up thread-aware debugging. Under OpenOCD, which the adapters above support, the first approach is to set up a GDB init script that will support NuttX thread inspection. Using J-Link’s software, you can use the second approach, GDB scripting. Both approaches are described in the link above.

Starting OpenOCD

If you are using the Olimex ARM-USB-TINY-H JTAG adapter (info in resources), here’s how to start OpenOCD:

$ sudo openocd -f interface/ftdi/olimex-arm-usb-tiny-h.cfg -f target/at91sama5d2.cfg -c '$_TARGETNAME.0 configure -rtos nuttx'

Once you have thread aware debugging set up GDB, you can start it and look at NuttX’s running threads.

Here’s a sample GDB session that shows how the thread info looks. The ^C in the output below indicates where you would type the <Control-C> character to stop the program. This GDB session would be started in another window.

[adam@tolk]$ arm-none-eabi-gdb
GNU gdb (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 8.3.0.20190709-git
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-linux-gnu --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target extended-remote :3333
Remote debugging using :3333
warning: while parsing threads: not well-formed (invalid token)
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
warning: while parsing threads: not well-formed (invalid token)
0x2000926c in ?? ()
(gdb) file nuttx
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from nuttx...
(gdb) mon halt
(gdb) load nuttx
Loading section .text, size 0x46a29 lma 0x20008000
Loading section .ARM.exidx, size 0x8 lma 0x2004ea2c
Loading section .data, size 0x2e0 lma 0x2004ea34
Start address 0x20008040, load size 290065
Transfer rate: 49 KB/sec, 14503 bytes/write.
(gdb) c
Continuing.
^Cat91sama5d2.cpu_a5.0 rev 1, partnum c05, arch f, variant 0, implementor 41
[New Thread 537253616]
[New Thread 537263120]
[New Thread 537268224]
[New Thread 537271376]
[New Thread 537805696]
[New Thread 537256784]
[New Thread 537259952]

Thread 1 received signal SIGINT, Interrupt.
nx_start () at init/nx_start.c:805
805        for (; ; )
(gdb) info threads
  Id   Target Id                                               Frame
* 1    Thread 537198592 (Name: Idle Task, pid:0, RUNNING)      nx_start () at init/nx_start.c:805
  2    Thread 537253616 (Name: hpwork, pid:1, WAIT_SEM)        0x600000d2 in ?? ()
  3    Thread 537263120 (Name: init, pid:4, WAIT_SEM)          0x600000d2 in ?? ()
  4    Thread 537268224 (Name: OHCI Monitor, pid:5, WAIT_SEM)  0x600000d2 in ?? ()
  5    Thread 537271376 (Name: EHCI Monitor, pid:6, WAIT_SEM)  0x600000d2 in ?? ()
  6    Thread 537805696 (Name: Telnet daemon, pid:7, WAIT_SEM) 0x600000d2 in ?? ()
  7    Thread 537256784 (Name: lpwork, pid:2, WAIT_SEM)        0x600000d2 in ?? ()
  8    Thread 537259952 (Name: lpwork, pid:3, WAIT_SEM)        0x600000d2 in ?? ()
(gdb)

Next: Creating a Bootable NuttX SD Card From Scratch