Talk

Tools and Techniques to Debug an Embedded Linux System

Watch Now

58:53

EOC 2022

Login to Watch

Sergio Prado

Sergio Prado has been working with embedded systems for more than 25 years, providing consulting and training services for companies worldwide. He also writes on his blog at sergioprado.blog and contributes to several free and open-source projects, including Buildroot, Yocto Project, and the Linux kernel.

View full profile

Topics Covered

Debugging Tools Embedded Linux

Talk

Tools and Techniques to Debug an Embedded Linux System

Sergio Prado

58:53 EOC 2022

Sergio Prado

58:53

Summary: In this talk, we will learn how to use different tools and techniques to debug an embedded Linux system.

Description: There are several tools and techniques to debug an embedded Linux system that can be applied in both user space and kernel space. Depending on the problem, you may need different tools like addr2line for crash dump analysis, GDB for interactive debugging, ftrace for kernel tracing, valgring to catch memory-related issues, gprof for application profiling, etc! In this hands-on oriented talk, we will learn how these and many other tools and techniques can be used when debugging an embedded Linux system.

1 / 5

Please log in or create an account to test your knowledge and see the answers.

Sergio described a four-step debugging process he uses for any software or system. Which sequence correctly captures those four steps?

A Understand the problem → Reproduce the problem → Identify the root cause → Fix the problem

B Reproduce the problem → Fix the problem → Identify the root cause → Verify the fix

C Fix the problem → Reproduce the problem → Understand the problem → Identify the root cause

D Identify the root cause → Reproduce the problem → Fix the problem → Understand the problem

E Understand the problem → Fix the problem → Reproduce the problem → Identify the root cause

When doing post-mortem analysis of a user-space crash on an embedded device, what did Sergio say is required to analyze the core dump effectively on your development machine?

A Enable core dumps with ulimit -c unlimited, then use GDB with the crashed binary (built with debug symbols) and the core file: gdb <binary> -c <corefile>

B Core dumps are always generated automatically, so you only need the binary without debug symbols to inspect the crash

C You must run the program under gdbserver on the device before it crashes to get a usable core dump

D Upload the raw core file to any online analyzer; no binary or debug symbols are needed

E Use Valgrind on the core file directly to reproduce the crash interactively

For analyzing a kernel oops message (NULL pointer dereference) on the board, which approach did Sergio recommend to map the reported address to the exact source line?

A Use the vmlinux file built with CONFIG_DEBUG_INFO and tools like addr2line or GDB to convert the address to the source line

B Only read the oops text on the console; mapping addresses to source is not needed to find the bug

C Rebuild the kernel without debug symbols to reduce size and then retry to reproduce the oops for a clearer message

D Run Valgrind on the kernel image to get source-line mapping for kernel addresses

E Use trace-cmd to map the oops address to a source line directly on the device without any kernel debug symbols

Sergio warned about relying on print statements for debugging. What key point did he make about using prints versus other techniques?

A Adding print statements (tracing) is a valid technique but it's wrong to use it for every bug; other tools may be more appropriate

B Print statements are always the best and fastest way to debug any embedded Linux problem

C Print statements should never be used under any circumstances because they always break the system

D To use tracing you must first remove all print statements from the codebase

E Print statements automatically give you kernel-level backtraces without extra tooling

When Sergio demonstrated interactive debugging of the Linux kernel, what did he say you must enable and what did he emphasize about hardware requirements?

A Enable CONFIG_KGDB to use GDB over serial; this does not require additional hardware like JTAG

B Enable Valgrind in the kernel config and attach a JTAG debugger to the board

C You must boot a special minimal kernel image without debug symbols and have a JTAG to connect GDB

D Enable CONFIG_PRINTK to allow GDB to attach; hardware JTAG is mandatory

E Use trace-cmd instead of GDB for kernel interactive debugging because GDB can't connect to the kernel

Formatting help

italics	surround text with asterisks
bold	surround text with two asterisks
hyperlink	[hyperlink](https://example.com) or just a bare URL
code	surround text with `backticks`
~~strikethrough~~	surround text with ~~two tilde characters~~
quote	prefix with >

Upvotes Newest Oldest

Carlos.Amaya

Score: 2 | 4 years ago | no reply

Thank you very much Sergio, great presentation!

phfbertoleti

Score: 2 | 4 years ago | 1 reply

Sergio, absolutely fantastic presentation! Thanks for sharing such an amazing content!

I've a question for you: to get kernel logs when we're developing a product, it's usual to configure a serial console for kernel and check the messages sent there (as I believe you've done when checking kernel crash in USB sub-system) during developing and testing phases. However, sometimes we're in a situation where there isn't a "free" uart for this (like debugging a product which a bug has been discovered in field operation, for example). In these cases, what's the best way to get kernel crash logs? Could you please share some good practices in this situation?

Thanks in advance.

sergio_pradoSpeaker

Score: 0 | 4 years ago | no reply

Hi phfbertoleti! I am glad you enjoyed it!

About your question, you have a few options:

You could redirect kernel logs (including kernel oops messages) to some persistent storage that you could retrieve later. Usually, log daemons like journald or rsyslogd are able to do that.
You could use the kernel pstore framework. This framework enables you to store kernel logs (including crashes) in a separated region of memory that you could retrieve later (after a soft reset for example).

Hope that helps.

Thanks!

glennk

Score: 1 | 4 years ago | 1 reply

This was a fantastic talk, you are spot on debugging is not given anywhere near enough attention which it certainly deserves! I also loved "THE SIX STAGES OF DEBUGGING", classic!

sergio_pradoSpeaker

Score: 0 | 4 years ago | no reply

I am glad you liked it!

LeeT1

Score: 1 | 4 years ago | 1 reply

Will the video be available for download? You covered a lot more information during the debugging exercises than are covered in the slides.

Stephane.Boucher

Score: 1 | 4 years ago | no reply

If you mean the video of the presentation, they cannot be downloaded. But they can be watched at will on the website.

jgaitan

Score: 1 | 4 years ago | 1 reply

Awesome session, extremely informative!

sergio_pradoSpeaker

Score: 0 | 4 years ago | no reply

Thanks @jgaitan! I am glad you enjoyed it!

Partners & Sponsors

Percepio

Edge Impulse

STMicroelectronics

Renesas

Amazon AWS

The Qt Company

NXP

TUXERA

XMOS

Timesys

Saleae

Become a Sponsor