Flexible and Layered Embedded Firmware through Test Driven Development (TDD)
Recent years the software industry has developed different methodologies with camps to support them many of them claiming better quality of work and speed. Embedded real-time firmware due to it's challenges makes adoption of these tools more difficult as we need to test systems interacting with the hardware that have timing constraints. Not all methods work well or there is often the question if the effort is worth the benefit.
In this session we will discuss the application of TDD,
- what is TDD and the difference with unit testing,
- example application of the method,
- how we can model the hardware registers transparently,
- how to tackle challenges porting to different architectures,
- using object oriented techniques for configurability
- the benefits and pitfalls of the method,
The session will be based on actual application of the method on real medium scale bare-bones systems projects.
According to the talk, what is the primary difference between Test-Driven Development (TDD) and traditional unit testing?
Happy that you found the speech useful.
Thank you, Nice presentation!
Nice presentation!!! Interesting to have some ideas on how complex systems/modules can be validated using TDD.
Other than that, hope I still have time to ask a question regarding integration/system-level tests: is it possible to create one that can validate timing constraints? I was thinking if there's some framework or recommended ways to validate in-between modules communication (which can be logical/software level only, or maybe even in-between chips), which can be critical to some systems.
Thanks in advance.
NOTE: In case someone is passing by, this is complementary to James Grenning presentation, so watching both is definitely valuable :)
Thank you Cesar_Santos. Timing constraints in a host environment do not make any sense. The only thing you can do is sequencing tests through Mock objects. In this case the Mock objects may test that the sequence of actions were executed as per spec (ie. having a state machine in the mock and check that it passes all transitions). I can think of ways to do timing constrained tests using Unity to execute target tests and use a hardware timer register to check timings. But still I am not sure if this would be realistic ie. passing the test implies that the code is correct. In complex systems interrupts and other stuff runs and may throw you out if you make a mistake. I prefer to handle these problems on paper. Design it right into the paper and make it sure that it will work by design, if this is possible.
Great presentation and introduction to TDD with clear and detailed examples. Many thanks!
Thank you Iramage, yes LLVM is another option. Creating test cases for legacy code is a pain. Myself I am doing partial testing on critical modules which through my experience I believe are more probable to fail. At the end I want to reduce development time, creating tests only for the most difficult parts of the code helps me to have peace of mind as the rest of the code is manageable.
Hi Ilias,
Thanks for the talk. Which tool would you recommend to generate code coverage reports without adding instrumentation in the production code?
Dear MJ,
I am not using any code coverage tools, but I think GCC has such a free tool (gcov). The source will be unaffected but the compiler will inject its own instructions in the binary file to gather the data. I guess you are not supposed to ship this code to the field and later on gather such data (although you could). I would not use coverage tools in production code (ie that runs into the field), only to support lab tests as part of test phase (depending the project). I do not know how other tools work.
TDD has the advantage of providing functional coverage (which is a super-set), is automated after the completion and do not disrupt production code in any level.
Thanks for the tip, I will look into gcov into detail. I think it has some integration with eclipse which would be useful if I can get it working correctly. I guess a way to set this would involve having a Debug build with gcov enabled and then a Release build without that.
As Ilias already mentioned, one can use gcov/lcov if using the GNU Compiler Collection. However, another great alternative is scan-build, part of the clang/llvm suite, https://clang-analyzer.llvm.org/scan-build.html
Really a very good webinar. I got a few ideas how to overcome some problems I have doing TDD within our projects.
Glad to see that the provided information was useful. If you have specific problems we can discuss them.
What other test frameworks other than CppUTest have you used? And what are your thoughts and experience using them?
I'm doing my test stuff with ceedling which is using CMock and Unity for the test stuff. It's very easy to setup. Maybe you want to give it a try (http://www.throwtheswitch.org/ceedling)
Thanks for the pointers. I believe many people will find these useful.
I have heard of those and looked at them. I did not see the merit myself as it is really straight forward in CppUTest to do all the stuff with not much copying of headers etc. In fact I only need to create the mock/test doubles functions only and headers only for very special cases. So my test setup uses all the headers of my actual code and I just put stubs of functions (or actual mock functions) for the doubles. No really time consuming. I haven't researched much myself to these tools, though.
I personally haven't tried other ones. I guess there should not be much different. I know that Unity is very similar, and that's why CppUTest has conversion scripts. The important thing I would like to have is to be able to test double through the linker instead of having to modify sources etc.
I think a big challenge in adopting TDD in embedded systems, is that abstracting hardware, is only going to be achievable for rather simple cases like simulating the contents of a register or a GPIO as demonstrated in your video. Embedded systems keep on getting more sophisticated, so simulating all their functions properly is really hard, or really long. And the requirements keep changing, so a team that does a good job at completing that task, may find themselves with an inoperative set of tests in a surprisingly short time. But still, I do think TDD is applicable in many cases, and should always be considered, for making products that are highly reliable.
In my video I also show part of the more complex queued SPI module that Coldfire has. I also have created USB tests as well and actually I can say USB implementation of the mocks was much simpler than Queued SPI. In a presentation like this it is not possible to show all the parts and the complexity. The reason I found USB easier, is that the hardware engine provides a packetized approach (per USB also principles) that was much easier to replicate. So although theoretically we have more complex hardware, it is not necessarily more difficult to mock. For standard microcontroller applications you use SPI, I2C/TWI, Uarts, Timers, maybe DMA. You can cover lots of these as standard devices. Of course if you change platforms around all the time, yes then you have to re-develop or modify. On the other hand having programmable logic with standard components (or your own components) can keep the longevity of the code easier as the hardware is controlled by you (another approach though). You may also stick with TDD up to the driver level, depends on how confident you are about them. Sometimes of course there is no option.
I would like to see a systematic way of creating mocks that can test the system timing and synchronization of data.
Do you mean actual timing or sequencing? Mocks will allow you sequencing. Timing does not make sense in absolute ways on your PC, your system will differ. If you are concerned about relative timings the method proposed on the top level could work also. Can you provide an example case to help me understand your problem?
My baseline assumption was regarding an embedded system performing real time operations. Would like a more systematic way to prove that we will meet deadlines, and routines will not deadlock etc., and would like to make these proofs before all hardware and software are integrated. We architect our software to meet real time events, but we often don't discover our actual performance until we build all of our code and exercise it with real hardware and real data IO. Meeting real time requirements assumes threads meat deadlines, and of course no deadlocks, etc. occur. We look at our interrupt routines and we make assumptions and make analysis on the time required to service interrupts. We should be able to simulate the system long before it is built.
Welcome to this session. I hope you enjoy it!
Excellent show on the full test spectrum coverage on how to do it all the way through.