Using Visual Trace Diagnostics to Evaluate Linux User space and Kernel space Implementations
Developing performant implementations for Linux-based embedded systems can be difficult, due to the sheer complexity of the Linux kernel and the volume of processes and applications that generally exist on a Linux-based embedded system. For kernel space implementations, such as kernel modules, it is important to leverage the appropriate interfaces to the kernel and pay careful attention to the hardware interface. For user space applications, performance is key to ensure that the application can scale without substantial rework.
Software tracing can be an invaluable tool to ensure that an implementation is performant. It enables a developer to identify any anomalies that may exist while the system is running. While there exist numerous open-source utilities that can provide textual cues towards the performance of the implementation, they are generally not amenable to quickly identify any clear issues, due to the way the human brain operates.
Visual trace diagnostics can significantly reduce the time to identify anomalies in a Linux-based implementation. It leverages software tracing, but presents data in a manner that is easily consumed by the human brain. This results in quickly identifying and rectifying anomalies in an implementation.
Join Mohammed Billoo as he explores the different facets of visual trace diagnostics. His talk will consist of hands-on demonstrations that show the different elements of capturing and analyzing software traces on a Linux-based embedded system, in both user space and kernel space.
What is the primary advantage of Visual Trace Diagnostics over traditional step-by-step debugging when investigating difficult-to-reproduce bugs?
Hi, Johan from Percepio here. If you mean live trace streaming over a local network, that is indeed supported by Tracealyzer and LTTng. You need to use the LTTng relay daemon (relayd) and that can sometimes be a bottleneck, but it works.
Hi, thank you for your question! Tracealyzer by itself cannot be used for online real-time tracing. However, Percepio does offer a DevAlert service where traces are automatically uploaded to the cloud whenever a fault occurs in your system (where you can define a set of faults).
Thanks for this nice introduction! As I'm using Tracealyzer for nearly 3 years now I can confirm it's usefulness investigating how your software operates under different circumstances. But as each medal has two sides, tracing always adds additional runtime to your code. In my project we figured out where an how much tracing is beneficial and when does tracing overhead distort your system. In my opinion you can't get enough details out of you software (especially in complex systems) but you have always to pay a price for getting insights.
Do you have made any measurements regarding the overhead for user trace points on linux?
Thanks for your insight! I absolutely agree on the statement of paying a price for getting insights. I haven't made any measurements yet but will soon and expect to write a blog post about it. Keep an eye for it!
This seems to be very helpful for developers. How would you say that Tracealyzer timelines can be helfpful for higher level software testers?
Glad you found the talk useful. For a tester, I would say that you can use the Actor Instance Graph or the Custom Interval Graph to get a sense for the baseline performance when there are no regressions. As new releases are made, you could review these two graphs to identify any regressions. If new regressions are identified, you could send the trace to the developer for further debugging.
Can LTTng/Tracealyzer be used for online real-time tracing?