Home > On-Demand Archives > Talks >
How to Get the Bugs Out of your Embedded Product
Dave Nadler - Watch Now - EOC 2021 - Duration: 45:39
What approaches really work for development and especially shaking out the bugs?
Test-driven development?
Simulation?
Code and design reviews?
Memory access checkers?
Scripted tests with coverage analysis?
Dumb Luck?
This is case study of a project to 'fix a small setup bug' in an existing product, which proved a wee bit larger than expected.
We'll review a bunch of actual bugs, techniques used, and statistics.
The approaches discussed could help you with your current project!
Thanks Anders for the kind words. I can't cite any references off-hand. Its closer to waterfall then some goofy stuff sometimes proposed. Using requirements to immediately define tests (before diving in and starting development) clarifies requirements and minimizes late surprises - do this and you will always find holes in the initial requirements. Decomposing top-level tests (which may only pass near project completion) into constituent/prerequisite tests helps define the lower-level interfaces needed for testing.
Hope that helps!,
Best Regards, Dave
Hi Dave,
Congratulations for an excellent presentation full of precious, hard-won lessons. I really hope that the younger fellow embedded developers are paying ATTENTION! Learning from experienced developers is much cheaper than repeating all the mistakes by yourself.
The most important, recurring message of your presentation is to develop as much of the embedded code as only possible ON THE DESKTOP. It's amazing to me how much your experience in this area matches mine. I also always start every embedded project with a PC-based "simulation" or a "prototype" of the device.
But unfortunately, this development strategy (also known as "Dual Targeting") is generally misunderstood and sometimes causes adverse reactions. For example, here is a link to my post on Statckoverflow "Prototyping and simulating embedded software on Windows". As you can read there, my post was generally misunderstood, criticized and finally closed as a "speculation".
Anyway, in your talk you mention wxWidgets as your prototyping platform. I'd like to second the requests from the Q&A session for you to publish an example device simulation with that GUI.
But if anybody is interested, an even simpler GUI prototyping platform called "QWin" is available for download NOW on GitHub as part of the QTools collection
The complete QWin code with a working example and a detailed Application Note explaining the process of simulating embedded devices on Windows (with the free Visual Studio) is also available:
The QWin prototyping platform is based directly on the Win32 API in C, so it binds directly to pure C code (as well as to C++ code). This is as simple as it gets and no third-party libraries or DLLs are needed. Still, QWin supports graphic LCD (monochrome and color), segmented displays, LEDs, bush-buttons as well as anything else you can put in a Windows dialog box.
Thanks Miro for the kind words. In https://www.embeddedonlineconference.com/meeting/Live_Discussion_How_to_Get_the_Bugs_Out_of_your_Embedded_Product I did mention your product. I suspect your Windows simulator perhaps frightens people off with the raw Windows API (though of course some of us have built products that way). We do prefer an OS-agnostic easier framework, latest one we're using being wxWidgets. Perhaps I should recode your QP Windows example using wxWidgets...
Best would be a PC framework (Windows and Linux etc) that provides:
- running from test scripts with result capture,
- capture of manually entered events into script for later playback
- compare run results (including things like LCD screen image)
This would give a great basis for CI/CD...
Thanks again for the kind words,
Best Regards, Dave
Hi Dave... I have a question... In software field exist tools that see the quality of the code developed showing the areas to improve... like a Sonar (https://www.sonarqube.org/)... Is there a tool focus on the quality of the code developed on embedded system?
I'm not aware of tools specific to embedded. I'm told some high-end tools can detect concurrency issues but don't know any specifics, sorry. Much static analysis is generic and will work fine for embedded. In the example project of this talk, I tried CPPcheck and it only found 1 of the top 12 bugs (in this example 6 of the 12 could theoretically have been found by static analysis). Hope this helps!
I'm OK with ASSERT() 'as long as' you can guaranty that the assert WILL be executed during testing. I've seen cases where programmers use asserts and they never create test cases where the assert is executed during testing and of course, some of these asserts can crash the system in the field which is not desirable.
As with many applications, this one is built with NDEBUG for target builds so no explosions on target possible. In Mars Perseverance presentation, IIRC Steve alluded to assert failure triggering a system reset. We don't want asserts to fire in regression testing; assert failure means application logic is incorrect and must be fixed. They are intended to help find and fix application logic errors as quickly as possible during development. So the assert should be executed but never fire during testing. Hope I explained my intent OK...
I agree with Jean in that assertions need to be tested, just as any other code. Or, actually, even more so, because they can bring down the whole system. I'm assuming here that assertions ARE enabled in the production software, as they should.
Great talk Dave, and one I'll be going back to watch again.
Just a +1 to the Q&A session plea for you to share a stripped back example of the simulator and application code if you can. Having a template/example makes it much easier to get started!
Excellent presentation!
Really great talk Dave. I learned a lot by you thinking out loud your dececions.
One question I have is regarding Test Driven Design, I'm familiar with Test Driven Development. But hadn't heard about this concept.
Could you give some example test cases for a Test Driven Design?
And at what point should they pass? I'm guessing only when the product is finished or near finished..