Repository : ssh://git@diktynna/doc

On branches: backup-redmine/2021-08-07,backup-redmine/2021-09-11,backup-redmine/2021-10-09,backup-redmine/2021-11-13,backup-redmine/2021-12-11,backup-redmine/2022-01-08,backup-redmine/2022-02-12,backup-redmine/2022-03-12,backup-redmine/2022-04-09,backup-redmine/2022-05-07,backup-redmine/2022-06-11,backup-redmine/2022-08-06,backup-redmine/2022-10-07,backup-redmine/2022-11-14,backup-redmine/2023-01-14,main

commit 5c3d455a6bd98ce83f5d7c366afc6d08c4ceb9fe Author: Sven Eckelmann sven@narfation.org Date: Thu Jul 22 14:08:30 2021 +0000

doc: open-mesh/Kernel_debugging_over_JTAG

5c3d455a6bd98ce83f5d7c366afc6d08c4ceb9fe open-mesh/Kernel_debugging_over_JTAG.textile | 228 +++++++++++++++++++++++++++ 1 file changed, 228 insertions(+)

diff --git a/open-mesh/Kernel_debugging_over_JTAG.textile b/open-mesh/Kernel_debugging_over_JTAG.textile new file mode 100644 index 00000000..d445a2da --- /dev/null +++ b/open-mesh/Kernel_debugging_over_JTAG.textile @@ -0,0 +1,228 @@ +h1. Kernel debugging over JTAG + +"debugged using JTAG":https://openwrt.org/docs/techref/hardware/port.jtag + +As shown in the [[Kernel debugging with_qemu's GDB server]] documentation, it is easy to debug the Linux kernel in an [[OpenWrt_in_QEMU|emulated system]]. But some problems might only be reproducible on actual hardware ([[Mixing_VM_with_gluon_hardware|connected to the emulation setup]]). It is therefore sometimes necessary to debug a whole system. + +"JTAG":https://en.wikipedia.org/wiki/JTAG is a good way to get access to the hardware state independent of the software currently running on it. It is therefore a good choice when the used hardware "exposes JTAG (or SWD)":https://openwrt.org/docs/techref/hardware/port.jtag + +Requirements: + +* target board which exposes JTAG +** "the 8devices lima board":https://www.8devices.com/products/lima is used in this document +* debug adapter hardware +** "Bus Blaster v3":http://dangerousprototypes.com/docs/Bus_Blaster_v3_design_overview +** Raspberry Pi's GPIO pins of the expansion header +** "Bus Pirate":http://dangerousprototypes.com/docs/Bus_Blaster_v3_design_overview (really slow and not recommended) +** ... +* On Chip Debugger for JTAG/SWD +** "OpenOCD":http://openocd.org/ + +h2. Preparing the hardware + +The connection between the debug adapter and the target board must established by connecting a couple of pins with each other. + +The debug and target board via at least following of the debug adapter hardware: + +* TDI (Test Data In) +* TDO (Test Data Out) +* TCK (Test Clock) +* TMS (Test Mode Select) +* GND (common ground) +* VTG (voltage reference) + +The adapter boards must then also be select in the OpenOCD configuration. + +h3. Bus Blaster + +The Bus Blaster v3 with the 20 pin header and the default buffer logic the usage of the pins already written next to header. + +The configuration of the adapter board and the target board is also straight forward since the release of OpenOCD 0.11 + +<pre><code class="diff"> +cat > jtag_debug.cfg << "EOF" +source [find interface/ftdi/dp_busblaster.cfg] +adapter speed 2000 +transport select jtag + +# in case to start debugging session without "reset halt": +# "reset halt" would be necessary here because otherwise the flash chip cannot +# be detected and the gdb attach fails with +# "Unknown flash device (ID 0x00000000)" +gdb_memory_map disable + +source [find board/8devices-lima.cfg] +EOF +</code></pre> + +h3. 8devices target board + +The board which should run the firmware must be connected to at least following pins of the debug adapter hardware: + +* TDI (Test Data In) +* TDO (Test Data Out) +* TCK (Test Clock) +* TMS (Test Mode Select) +* GND (common ground) +* VTG (voltage reference) + +The 8devices lima reference board exposes all over its GPIO pins: + +* TDI: J2 - GPIO1 +* TDO: J2 - GPIO2 +* TCK: J2 - GPIO0 +* TMS: J2 13 - GPIO3 +* GND: J1 16 - GND +* VTG: J1 15 - 3.3V + + +h2. Preparing OpenWrt + +There is nearly no requirements from OpenWrt but there are several things which can make the debugging a lot easier. + +h3. Enable debug info + +The actual configuration has to be set in the target kernel configuration: + +<pre> +CONFIG_DEBUG_INFO=y +CONFIG_DEBUG_INFO_DWARF4=y +# CONFIG_DEBUG_INFO_REDUCED is not set +CONFIG_GDB_SCRIPTS=y +</pre> + +The kernel address space layout randomization complicates the resolving of addresses of symbols. It is highly recommended to start the kernel with the parameter "nokaslr". For example by adding it to <code>CONFIG_CMDLINE</code> or by adjusting the <code>bootargs</code> in the bootloader. It should be checked in <code>/proc/cmdline</code> whether it was really booted with this parameter. + +For ar71xx (8devices lima in my case), it would look like: + +<pre><code class="diff"> +diff --git a/target/linux/ar71xx/config-4.14 b/target/linux/ar71xx/config-4.14 +index 9a524fae4316caa10431bd6b3b4dadbe8660f14c..397e15bcecd4e9c696a2321174969541b673cbd3 100644 +--- a/target/linux/ar71xx/config-4.14 ++++ b/target/linux/ar71xx/config-4.14 +@@ -308,10 +310,14 @@ CONFIG_CPU_SUPPORTS_MSA=y + CONFIG_CRYPTO_RNG2=y + CONFIG_CRYPTO_WORKQUEUE=y + CONFIG_CSRC_R4K=y ++CONFIG_DEBUG_INFO=y ++CONFIG_DEBUG_INFO_DWARF4=y ++# CONFIG_DEBUG_INFO_REDUCED is not set + CONFIG_DMA_NONCOHERENT=y + CONFIG_EARLY_PRINTK=y + CONFIG_ETHERNET_PACKET_MANGLE=y + CONFIG_FIXED_PHY=y ++CONFIG_GDB_SCRIPTS=y + CONFIG_GENERIC_ATOMIC64=y + CONFIG_GENERIC_CLOCKEVENTS=y + CONFIG_GENERIC_CMOS_UPDATE=y +diff --git a/target/linux/ar71xx/image/Makefile b/target/linux/ar71xx/image/Makefile +index 804532b55cb145134acf47accd095bbb24dee059..c485389f56c34ca8216c1016d515be2836ab2349 100644 +--- a/target/linux/ar71xx/image/Makefile ++++ b/target/linux/ar71xx/image/Makefile +@@ -58,7 +58,7 @@ define Device/Default + PROFILES = Default Minimal $$(DEVICE_PROFILE) + MTDPARTS := + BLOCKSIZE := 64k +- CONSOLE := ttyS0,115200 ++ CONSOLE := ttyS0,115200 nokaslr + CMDLINE = $$(if $$(BOARDNAME),board=$$(BOARDNAME)) $$(if $$(MTDPARTS),mtdparts=$$(MTDPARTS)) $$(if $$(CONSOLE),console=$$(CONSOLE)) + KERNEL := kernel-bin | patch-cmdline | lzma | uImage lzma + COMPILE := +</code></pre> + +h3. Enabling python support for gdb + +OpenWrt will build a gdb when <code>CONFIG_GDB=y</code> is set in .config. But this version is missing python support. But it can be enabled with following patch: + +<pre><code class="diff"> +diff --git a/toolchain/gdb/Makefile b/toolchain/gdb/Makefile +index 41ba9853fd26d5ea2ba3759946a9591c668d92e9..afe4f01201fca21adc465a3fbd3c3751ec23df25 100644 +--- a/toolchain/gdb/Makefile ++++ b/toolchain/gdb/Makefile +@@ -45,7 +45,7 @@ HOST_CONFIGURE_ARGS = \ + --without-included-gettext \ + --enable-threads \ + --with-expat \ +- --without-python \ ++ --with-python \ + --disable-binutils \ + --disable-ld \ + --disable-gas \ +@@ -56,9 +56,11 @@ define Host/Install + $(INSTALL_BIN) $(HOST_BUILD_DIR)/gdb/gdb $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb + ln -fs $(TARGET_CROSS)gdb $(TOOLCHAIN_DIR)/bin/$(GNU_TARGET_NAME)-gdb + strip $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb ++ -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory install + endef + + define Host/Clean ++ -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory uninstall + rm -rf \ + $(HOST_BUILD_DIR) \ + $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb \ +</code></pre> + +It is often possible (and in case of memory access/symbol relocation problems even recommended) to just use the normal Distro's multiarch gdb. This would be in Debian "gdb-multiarch". + +h2. Start debugging session + +h3. Starting OpenOCD + +The start of OpenOCD couldn't be more trivial: + +<pre><code class="shell"> +openocd -f jtag_debug.cfg +</code></pre> + +It should start a telnet server (for manual intervention) on TCP port 3333, scan the JTAG chains and afterwards start the internal gdbserver on port 3333. + +h3. Connecting gdb + +I would use following folder in my ar711x build environment but they will be different for other architectures or OpenWrt versions: + +* <code>LINUX_DIR=${OPENWRT_DIR}/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/linux-4.14.236/</code> +* <code>GDB=${OPENWRT_DIR}/staging_dir/toolchain-mips_24kc_gcc-7.5.0_musl/bin/mips-openwrt-linux-gdb</code> +* <code>BATADV_DIR=${OPENWRT_DIR}/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/batman-adv-2019.2/</code> + +When openocd was started, we can configure gdb. It has to connect via the local openocd gdbstub to the target device. We must change to the LINUX_DIR first and can then start our target specific GDB with our uncompressed kernel image. + +<pre><code class="shell"> +cd "${LINUX_DIR}" +"${GDB}" -iex "set auto-load safe-path scripts/gdb/" -ex "target extended-remote localhost:3333" ./vmlinux +</code></pre> + +The debug information for each module must be loaded using <code>lx-symbols</code> or otherwise only debug information for builtin code will be accessible + +<pre> +lx-symbols .. + +continue +</pre> + +You should make sure that it doesn't load any <code>*.ko</code> files from <code>ipkg-*</code> directories. These files are stripped and doesn't contain the necessary symbol information. When necessary, just delete these folders or specify the folders with the unstripped kernel modules: + +<pre> +lx-symbols ../batman-adv-2019.2/.pkgdir/ ../backports-4.19.66-1/.pkgdir/ ../button-hotplug/.pkgdir/ +</pre> + +The rest of the process works similar to debugging using gdbserver. Just set some additional breakpoints and let the kernel run again. + +Some other ideas are documented in [[GDB Linux_snippets]]. + +The kernel hacking debian image page should also be checked to [[Kernel_hacking_Debian_image#Building-the-batman-adv-module|increase the chance of getting debugable modules]] which didn't had all information optimized away. The relevant flags could be set directly in the routing feed like this: + +<pre><code class="diff"> +diff --git a/batman-adv/Makefile b/batman-adv/Makefile +index a7c6a79..c18f978 100644 +--- a/batman-adv/Makefile ++++ b/batman-adv/Makefile +@@ -89,7 +89,7 @@ define Build/Compile + CROSS_COMPILE="$(TARGET_CROSS)" \ + SUBDIRS="$(PKG_BUILD_DIR)/net/batman-adv" \ + $(PKG_EXTRA_KCONFIG) \ +- EXTRA_CFLAGS="$(PKG_EXTRA_CFLAGS)" \ ++ EXTRA_CFLAGS="$(PKG_EXTRA_CFLAGS) -fno-inline -Og -fno-optimize-sibling-calls" \ + NOSTDINC_FLAGS="$(NOSTDINC_FLAGS)" \ + modules + endef +</code></pre> \ No newline at end of file