Home > On-Demand Archives > Talks >

Testing of Event-Driven Embedded Software with Python

Miro Samek - Watch Now - EOC 2022 - Duration: 41:04

Testing of Event-Driven Embedded Software with Python
Miro Samek

Testing event-driven software typically involves injecting events to the system and verifying that the software reacts by executing the expected sequence of actions. These actions might be quite complex. For example, a single transition in a hierarchical state machine might trigger several exit actions from the source state configuration, followed by entry actions and nested initial transitions into the target state configuration. The actions themselves might involve interactions with multiple hardware components, such as toggling GPIOs, sending data through various interfaces, etc.

In traditional unit testing frameworks, the verification of such potentially complex interactions typically requires applying mock objects as the test doubles for the various hardware components. But even though generating mocks can be automated, it's difficult to verify sequences of expected interactions among multiple mocks (multiple hardware components). Also, working with mocks reverses the natural sequencing, because the expectations must be specified before the actual call to the CUT (code under test).

This session will show a more intuitive approach, conceptually similar to "debugging with printf", where you instrument the code with printf statements to output the software trace information. You then run the CUT with a controlled set of inputs, and examine the produced trace to determine whether the CUT operates correctly. The main differences from using actual printfs will be: (1) that the much more efficient software tracing mechanism will be used instead and (2) that both generating the inputs and the checking of the trace output will be automated.

This approach has many interesting implications:

  • It separates the execution of the CUT from checking of the "test assertions". The embedded target is concerned only with running a test fixture that calls the CUT and produces the trace, but it does not check the test expectations.
  • Checking the generated trace output against the expectations is performed separately on the host computer and can be done in a different programming language (e.g., Python) than the CUT (C or C++).
  • Virtually all test doubles can be implemented as simple "spy" test doubles without the need to applying mocks or other complex test doubles. Verifying trace output from multiple such "spy" test doubles is trivial.
  • The testing process follows the natural sequencing where the generating of inputs to the CUT is naturally followed by the expectations of the performed actions. 

This session will utilize hands-on demonstrations of the QUTest (pronounced "cutest") unit testing harness running on both the host computer and STM32-NUCLEO board. Specifically, you will see a comparison between the traditional approach with a mock test double and the QUTest approach with software tracing and a "spy" test double.

italicssurround text with
boldsurround text with
**two asterisks**
or just a bare URL
surround text with
strikethroughsurround text with
~~two tilde characters~~
prefix with

Score: 0 | 2 years ago | 1 reply

Are QSpy and QUTest part of the QP/C licensing model ? or are they selfcontained tools provided with their own licenses ?

Score: 0 | 2 years ago | no reply

Yes, QSpy and QUTest use the same licensing model as QP/C and QP/C++, which means that they are available under GPLv3 and potentially under a commercial license from Quantum Leaps. Please note, however, that most of the time you don't ship the test/trace code inside embedded devices. Instead, the tracing and testing occurs in-house, where you can easily comply with the GPL.

Score: 0 | 2 years ago | 1 reply

Hi Miro, regarding your question the only comparable solution I know is from a relativly small consulting company located in Munich. They are doing many things that Quantum Leaps does with their framework called eTrice (https://www.protos.de/produkte/etrice-modellierung-in-eclipse/). They model state machines, generate code and also use traces to verify behaviour on real hardware. They worked as project partner and developed theit mini HIL project based on thos thechnique. I also tried to do something simillar on another project, but it never got beyond the demo stage as customer acceptance was not really present (meaning the time to setup the testframework was considered to high, compared to the benefits they saw) sadly.
But I think I will get your demo, play a little bit an give another try on my next project!

Score: 0 | 2 years ago | 1 reply

Thanks a lot for the pointer. It seems that I haven't yet heard of eTrice and the PROTOS company. They introduce eTrice as "Modeling of embedded systems with ROOM". QP also started as a "minimal ROOM virtual machine". Very interesting!

Score: 0 | 2 years ago | 1 reply

Correctly they started by using ROOM for 2 reasons

  • technical background of founders
  • ROOM is much easier to start from than current UML2 which brings much more features and confusion
    As far as I remember they are also integrated somehow in the Eclipse framework.
Score: 0 | 2 years ago | no reply

Yes, I'm just looking at eTrice on the Eclipse Foundation website ( https://www.eclipse.org/etrice).

Score: 0 | 2 years ago | no reply

If anybody is interested in actually running the examples, all the code presented is available on GitHub:
Specifically, the projects from the presentation are in the sub-directories: unity_strlen, unity_ledbar, and unity_ledbar2.

Score: 1 | 2 years ago | 1 reply

Excellent presentation!

Score: 1 | 2 years ago | no reply

I'm glad that you liked it!