Repository : ssh://git@diktynna/doc

On branches: backup-redmine/2022-11-14,backup-redmine/2023-01-14,main

commit 626903329ef8abda27aee71b93adaa6f88d5341a Author: Sven Eckelmann sven@narfation.org Date: Sun Oct 23 09:10:02 2022 +0000

doc: devtools/Kernel_debugging_with_kgdb: update instructions for OpenWrt 22.03

626903329ef8abda27aee71b93adaa6f88d5341a devtools/Kernel_debugging_with_kgdb.textile | 336 ++++++++++------------------ 1 file changed, 115 insertions(+), 221 deletions(-)

diff --git a/devtools/Kernel_debugging_with_kgdb.textile b/devtools/Kernel_debugging_with_kgdb.textile index 5d0bb4e5..71399d14 100644 --- a/devtools/Kernel_debugging_with_kgdb.textile +++ b/devtools/Kernel_debugging_with_kgdb.textile @@ -29,234 +29,127 @@ ubus call system watchdog '{"stop":true}'

h3. Enabling KGDB in kernel

+OpenWrt must be modified slightly to expose the kernel gdbstub (@CONFIG_KERNEL_KGDB@): + The actual kernel gdbstub cannot be enabled via OpenWrt's .config. Instead the actual configuration has to be set in the target configuration:

-<pre> -# CONFIG_STRICT_KERNEL_RWX is not set -CONFIG_FRAME_POINTER=y -CONFIG_KGDB=y -CONFIG_KGDB_SERIAL_CONSOLE=y -CONFIG_DEBUG_INFO=y -CONFIG_DEBUG_INFO_DWARF4=y -# CONFIG_DEBUG_INFO_REDUCED is not set -CONFIG_GDB_SCRIPTS=y - -# optional: to allow activation of kgdb over serial by agent-proxy -# instead of manually triggering it over /proc/sysrq-trigger -CONFIG_MAGIC_SYSRQ_SERIAL=y -</pre> +<pre><code class="diff"> +From: Sven Eckelmann sven@narfation.org +Date: Thu, 13 Oct 2022 16:40:21 +0200 +Subject: openwrt: Add support for easily selectable kernel debugger support

-For x86-64, the change (mostly created using <code>make kernel_menuconfig</code>) would be: +See https://www.open-mesh.org/projects/devtools/wiki/Kernel_debugging_with_kgdb#... +for more information.

-<pre><code class="diff"> -diff --git a/target/linux/x86/config-5.4 b/target/linux/x86/config-5.4 -index 6676f9501a..2518c8f57b 100644 ---- a/target/linux/x86/config-5.4 -+++ b/target/linux/x86/config-5.4 -@@ -78,6 +78,7 @@ CONFIG_COMMON_CLK=y - CONFIG_COMPAT_32=y - CONFIG_COMPAT_32BIT_TIME=y - # CONFIG_COMPAT_VDSO is not set -+CONFIG_CONSOLE_POLL=y - CONFIG_CONSOLE_TRANSLATIONS=y - # CONFIG_CPU5_WDT is not set - CONFIG_CPU_FREQ=y -@@ -116,6 +117,9 @@ CONFIG_DCACHE_WORD_ACCESS=y - # CONFIG_DCDBAS is not set - # CONFIG_DEBUG_BOOT_PARAMS is not set - # CONFIG_DEBUG_ENTRY is not set -+CONFIG_DEBUG_INFO=y -+CONFIG_DEBUG_INFO_DWARF4=y -+# CONFIG_DEBUG_INFO_REDUCED is not set - CONFIG_DEBUG_MEMORY_INIT=y - CONFIG_DEBUG_MISC=y - # CONFIG_DEBUG_NMI_SELFTEST is not set -@@ -161,6 +165,7 @@ CONFIG_FUSION=y - CONFIG_FUSION_MAX_SGE=128 - CONFIG_FUSION_SPI=y - CONFIG_FW_LOADER_PAGED_BUF=y -+CONFIG_GDB_SCRIPTS=y - CONFIG_GENERIC_ALLOCATOR=y - CONFIG_GENERIC_BUG=y - CONFIG_GENERIC_CLOCKEVENTS=y -@@ -306,6 +311,11 @@ CONFIG_KALLSYMS=y - CONFIG_KEXEC=y - CONFIG_KEXEC_CORE=y - CONFIG_KEYBOARD_ATKBD=y -+CONFIG_KGDB=y -+# CONFIG_KGDB_KDB is not set -+# CONFIG_KGDB_LOW_LEVEL_TRAP is not set -+CONFIG_KGDB_SERIAL_CONSOLE=y -+# CONFIG_KGDB_TESTS is not set - # CONFIG_LEDS_CLEVO_MAIL is not set - CONFIG_LOCK_DEBUGGING_SUPPORT=y - # CONFIG_M486 is not set -@@ -314,6 +324,8 @@ CONFIG_M586MMX=y - # CONFIG_M586TSC is not set - # CONFIG_M686 is not set - # CONFIG_MACHZ_WDT is not set -+CONFIG_MAGIC_SYSRQ=y -+CONFIG_MAGIC_SYSRQ_SERIAL=y - # CONFIG_MATOM is not set - # CONFIG_MCORE2 is not set - # CONFIG_MCRUSOE is not set -@@ -421,6 +433,7 @@ CONFIG_SCx200HR_TIMER=y - # CONFIG_SCx200_GPIO is not set - # CONFIG_SCx200_WDT is not set - CONFIG_SERIAL_8250_PCI=y -+# CONFIG_SERIAL_KGDB_NMI is not set - CONFIG_SERIO=y - CONFIG_SERIO_I8042=y - CONFIG_SERIO_LIBPS2=y -diff --git a/target/linux/x86/image/Makefile b/target/linux/x86/image/Makefile -index f61e4ff802..e8c05c58e5 100644 ---- a/target/linux/x86/image/Makefile -+++ b/target/linux/x86/image/Makefile -@@ -9,7 +9,7 @@ GRUB2_VARIANT = - GRUB_TERMINALS = - GRUB_SERIAL_CONFIG = - GRUB_TERMINAL_CONFIG = --GRUB_CONSOLE_CMDLINE = -+GRUB_CONSOLE_CMDLINE = nokaslr - - ifneq ($(CONFIG_GRUB_CONSOLE),) - GRUB_CONSOLE_CMDLINE += console=tty0 -</code></pre> +When enabling this KERNEL_KGDB, make sure to clean some packages to make +sure that they are compiled with the correct settings:

-For ath79 (GL.inet AR750 in my case), it would look like: + make toolchain/gdb/clean + make toolchain/gdb/compile -j$(nproc || echo 1) + make target/linux/clean + make -j$(nproc || echo 1)

-<pre><code class="diff"> -diff --git a/target/linux/ath79/config-5.4 b/target/linux/ath79/config-5.4 -index 60f57692e2..01b66897fe 100644 ---- a/target/linux/ath79/config-5.4 -+++ b/target/linux/ath79/config-5.4 -@@ -25,7 +25,7 @@ CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT=y - CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y - CONFIG_AT803X_PHY=y - CONFIG_ATH79=y --CONFIG_ATH79_WDT=y -+# CONFIG_ATH79_WDT is not set - CONFIG_BLK_MQ_PCI=y - CONFIG_CEVT_R4K=y - CONFIG_CLKDEV_LOOKUP=y -@@ -34,6 +34,8 @@ CONFIG_CMDLINE="rootfstype=squashfs,jffs2" - CONFIG_CMDLINE_BOOL=y - # CONFIG_CMDLINE_OVERRIDE is not set - CONFIG_COMMON_CLK=y -+CONFIG_CONSOLE_POLL=y -+CONFIG_CONSOLE_TRANSLATIONS=y - # CONFIG_COMMON_CLK_BOSTON is not set - CONFIG_COMPAT_32BIT_TIME=y - CONFIG_CPU_BIG_ENDIAN=y -@@ -52,9 +54,13 @@ CONFIG_CPU_SUPPORTS_HIGHMEM=y - CONFIG_CPU_SUPPORTS_MSA=y - CONFIG_CRYPTO_RNG2=y - CONFIG_CSRC_R4K=y -+CONFIG_DEBUG_INFO=y -+CONFIG_DEBUG_INFO_DWARF4=y -+# CONFIG_DEBUG_INFO_REDUCED is not set - CONFIG_DMA_NONCOHERENT=y - CONFIG_DMA_NONCOHERENT_CACHE_SYNC=y - CONFIG_DTC=y -+CONFIG_DUMMY_CONSOLE=y - CONFIG_EARLY_PRINTK=y - CONFIG_EFI_EARLYCON=y - CONFIG_ETHERNET_PACKET_MANGLE=y -@@ -63,6 +69,7 @@ CONFIG_FONT_8x16=y - CONFIG_FONT_AUTOSELECT=y - CONFIG_FONT_SUPPORT=y - CONFIG_FW_LOADER_PAGED_BUF=y -+CONFIG_GDB_SCRIPTS=y - CONFIG_GENERIC_ATOMIC64=y - CONFIG_GENERIC_CLOCKEVENTS=y - CONFIG_GENERIC_CMOS_UPDATE=y -@@ -132,18 +139,27 @@ CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y - CONFIG_HAVE_RSEQ=y - CONFIG_HAVE_SYSCALL_TRACEPOINTS=y - CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y -+CONFIG_HW_CONSOLE=y - CONFIG_HZ_PERIODIC=y - CONFIG_IMAGE_CMDLINE_HACK=y - CONFIG_INITRAMFS_SOURCE="" -+CONFIG_INPUT=y - CONFIG_IRQCHIP=y - CONFIG_IRQ_DOMAIN=y - CONFIG_IRQ_FORCED_THREADING=y - CONFIG_IRQ_MIPS_CPU=y - CONFIG_IRQ_WORK=y -+CONFIG_KGDB=y -+# CONFIG_KGDB_KDB is not set -+# CONFIG_KGDB_LOW_LEVEL_TRAP is not set -+CONFIG_KGDB_SERIAL_CONSOLE=y -+# CONFIG_KGDB_TESTS is not set - CONFIG_LEDS_GPIO=y - # CONFIG_LEDS_RESET is not set - CONFIG_LIBFDT=y - CONFIG_LOCK_DEBUGGING_SUPPORT=y -+CONFIG_MAGIC_SYSRQ=y -+CONFIG_MAGIC_SYSRQ_SERIAL=y - CONFIG_MDIO_BITBANG=y - CONFIG_MDIO_BUS=y - CONFIG_MDIO_DEVICE=y -@@ -161,6 +177,7 @@ CONFIG_MIPS_CLOCK_VSYSCALL=y - # CONFIG_MIPS_CMDLINE_FROM_BOOTLOADER is not set - CONFIG_MIPS_CMDLINE_FROM_DTB=y - # CONFIG_MIPS_ELF_APPENDED_DTB is not set -+CONFIG_MIPS_FP_SUPPORT=y - CONFIG_MIPS_L1_CACHE_SHIFT=5 - # CONFIG_MIPS_NO_APPENDED_DTB is not set - CONFIG_MIPS_RAW_APPENDED_DTB=y -@@ -217,6 +234,7 @@ CONFIG_RESET_ATH79=y - CONFIG_RESET_CONTROLLER=y - CONFIG_SERIAL_8250_NR_UARTS=1 - CONFIG_SERIAL_8250_RUNTIME_UARTS=1 -+# CONFIG_SERIAL_KGDB_NMI is not set - CONFIG_SERIAL_AR933X=y - CONFIG_SERIAL_AR933X_CONSOLE=y - CONFIG_SERIAL_AR933X_NR_UARTS=2 -@@ -248,3 +266,8 @@ CONFIG_TICK_CPU_ACCOUNTING=y - CONFIG_TINY_SRCU=y - CONFIG_USB_SUPPORT=y - CONFIG_USE_OF=y -+# CONFIG_VGACON_SOFT_SCROLLBACK is not set -+# CONFIG_VGA_CONSOLE is not set -+CONFIG_VT=y -+# CONFIG_VT_CONSOLE is not set -+# CONFIG_VT_HW_CONSOLE_BINDING is not set -</code></pre> +The session can (after initializing agent-proxy) on serial via:

-h3. Enabling python support for gdb + ubus call system watchdog '{"magicclose":true}' + ubus call system watchdog '{"stop":true}'

-OpenWrt will build a gdb when <code>CONFIG_GDB=y</code> is set in .config. But it is important to also enable the python support via <code>CONFIG_GDB_PYTHON=y</code> or otherwise the Linux helper will not be able to correctly scan for modules. This feature was only added *after* the OpenWrt 21.02 release. + echo ttyMSM0,115200 > /sys/module/kgdboc/parameters/kgdboc + echo g > /proc/sysrq-trigger

-For older versions of OpenWrt (including 21.02.x), following script can also be used: +The rest has then to be done with the gdb(-remote) instance on the host. +But the system must not be stopped too long because the external (GPIO) +will otherwise kill the system.

-<pre><code class="diff"> -diff --git a/toolchain/gdb/Makefile b/toolchain/gdb/Makefile -index 05e3c7de3c..0ab20cb2d5 100644 ---- a/toolchain/gdb/Makefile -+++ b/toolchain/gdb/Makefile -@@ -36,7 +36,7 @@ HOST_CONFIGURE_ARGS = \ - --without-included-gettext \ - --enable-threads \ - --with-expat \ -- --without-python \ -+ --with-python \ - --disable-unit-tests \ - --disable-ubsan \ - --disable-binutils \ -@@ -49,9 +49,11 @@ define Host/Install - $(INSTALL_BIN) $(HOST_BUILD_DIR)/gdb/gdb $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb - ln -fs $(TARGET_CROSS)gdb $(TOOLCHAIN_DIR)/bin/$(GNU_TARGET_NAME)-gdb - strip $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb -+ -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory install - endef +Important here, is that OpenWrt 22.03 (on IPQ807x/IPQ60xx) doesn't provide +the correctly mapped vmlinux in the source directory, So it is necessary to +run it like this: + + $ cd "{LINUX_DIR}" + $ cp -r vmlinux-gdb.py vmlinux.debug-gdb.py + $ cp ../vmlinux.debug vmlinux.debug + $ "${GDB}" -iex "set auto-load safe-path `pwd`/scripts/gdb/" -iex "target remote localhost:5551" vmlinux.debug + (gdb) lx-symbols .. + +Signed-off-by: Sven Eckelmann sven@narfation.org

- define Host/Clean -+ -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory uninstall - rm -rf \ - $(HOST_BUILD_DIR) \ - $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb \ +diff --git a/config/Config-kernel.in b/config/Config-kernel.in +index 21a56e864098b8f652f06e319ce795a9456d5dcb..d0bc5e5d8b45cf6a0c63d86f5a2140980605373b 100644 +--- a/config/Config-kernel.in ++++ b/config/Config-kernel.in +@@ -11,6 +11,43 @@ config KERNEL_IPQ_MEM_PROFILE + This option select memory profile to be used,which defines + the reserved memory configuration used in device tree. + ++config KERNEL_VT ++ bool ++ ++config KERNEL_GDB_SCRIPTS ++ bool ++ ++config KERNEL_HW_CONSOLE ++ bool ++ ++config KERNEL_CONSOLE_POLL ++ bool ++ ++config KERNEL_MAGIC_SYSRQ ++ bool ++ ++config KERNEL_MAGIC_SYSRQ_SERIAL ++ bool ++ ++config KERNEL_KGDB_SERIAL_CONSOLE ++ bool ++ ++config KERNEL_KGDB_HONOUR_BLOCKLIST ++ bool ++ ++config KERNEL_KGDB ++ select KERNEL_VT ++ select KERNEL_GDB_SCRIPTS ++ select KERNEL_HW_CONSOLE ++ select KERNEL_CONSOLE_POLL ++ select KERNEL_MAGIC_SYSRQ ++ select KERNEL_MAGIC_SYSRQ_SERIAL ++ select KERNEL_KGDB_SERIAL_CONSOLE ++ select KERNEL_KGDB_HONOUR_BLOCKLIST ++ select GDB_PYTHON ++ bool "Enable kernel debugger over serial" ++ ++ + config KERNEL_BUILD_USER + string "Custom Kernel Build User Name" + default "builder" if BUILDBOT +diff --git a/include/kernel-build.mk b/include/kernel-build.mk +index 80da4455bc04fccd1c7834fe8b94c29399289bd2..4cbb8a861ed01a48d65a28f2d0b6e34837284cf2 100644 +--- a/include/kernel-build.mk ++++ b/include/kernel-build.mk +@@ -143,6 +143,7 @@ define BuildKernel + $(LINUX_DIR)/.image: $(STAMP_CONFIGURED) $(if $(CONFIG_STRIP_KERNEL_EXPORTS),$(KERNEL_BUILD_DIR)/symtab.h) FORCE + $(Kernel/CompileImage) + $(Kernel/CollectDebug) ++ +[ -z "$(CONFIG_KERNEL_GDB_SCRIPTS)" ] || $(KERNEL_MAKE) scripts_gdb + touch $$@ + + mostlyclean: FORCE +diff --git a/target/linux/generic/config-5.10 b/target/linux/generic/config-5.10 +index 4a6efc88012580691b52493685992a2af7fa1c65..00238982863b98c2f340c7e6a76a19652c26d2c1 100644 +--- a/target/linux/generic/config-5.10 ++++ b/target/linux/generic/config-5.10 +@@ -7184,3 +7184,12 @@ CONFIG_ZONE_DMA=y + # CONFIG_ZRAM_MEMORY_TRACKING is not set + # CONFIG_ZSMALLOC is not set + # CONFIG_ZX_TDM is not set ++ ++# KGDB specific "disabled" options ++# CONFIG_CONSOLE_TRANSLATIONS is not set ++# CONFIG_VT_CONSOLE is not set ++# CONFIG_VT_HW_CONSOLE_BINDING is not set ++# CONFIG_SERIAL_KGDB_NMI is not set ++# CONFIG_KGDB_TESTS is not set ++# CONFIG_KGDB_KDB is not set ++# CONFIG_KGDB_LOW_LEVEL_TRAP is not set </code></pre>

h2. Start debugging session @@ -285,14 +178,15 @@ h3. Connecting gdb

I would use following folder in my x86-64 build environment but they will be different for other architectures or OpenWrt versions:

-* <code>LINUX_DIR=${OPENWRT_DIR}/build_dir/target-x86_64_musl/linux-x86_64/linux-5.4.143/</code> -* <code>GDB=${OPENWRT_DIR}/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/bin/x86_64-openwrt-linux-gdb</code> -* <code>BATADV_DIR=${OPENWRT_DIR}/build_dir/target-x86_64_musl/linux-x86_64/batman-adv-2021.1/</code> +* <code>LINUX_DIR=${OPENWRT_DIR}/build_dir/target-x86_64_musl/linux-x86_64/linux-5.10.146/</code> +* <code>GDB=${OPENWRT_DIR}/staging_dir/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/bin/x86_64-openwrt-linux-gdb</code> +* <code>BATADV_DIR=${OPENWRT_DIR}/build_dir/target-x86_64_musl/linux-x86_64/batman-adv-2022.0/</code>

When kgdb is activated using sysrq, we can configure gdb. It has to connect via a serial adapter to the target device. We must change to the LINUX_DIR first and can then start our target specific GDB with our uncompressed kernel image before we will connect to the remote device.

<pre><code class="shell"> cd "${LINUX_DIR}" +cp ../vmlinux.debug vmlinux "${GDB}" -iex "set auto-load safe-path scripts/gdb/" -iex "set serial baud 115200" -iex "target remote /dev/ttyUSB0" ./vmlinux </code></pre>

@@ -307,7 +201,7 @@ continue You should make sure that it doesn't load any <code>*.ko</code> files from <code>ipkg-*</code> directories. These files are stripped and doesn't contain the necessary symbol information. When necessary, just delete these folders or specify the folders with the unstripped kernel modules:

<pre> -lx-symbols ../batman-adv-2021.1/.pkgdir/ ../backports-5.10.42-1/.pkgdir/ ../button-hotplug/.pkgdir/ +lx-symbols ../batman-adv-2022.0/.pkgdir/ ../backports-5.15.58-1/.pkgdir/ ../button-hotplug/.pkgdir/ </pre>

The rest of the process works similar to debugging using gdbserver. Just set some additional breakpoints and let the kernel run again. kgdb will then inform gdb whenever a breakpoints was hit. Just keep in mind that it is not possible to interrupt the kernel from gdb (without a Oops or an already existing breakpoint) - use the sysrq mechanism again from Linux to switch back to kgdb.