Accelerating System-Level Embedded Testing with OpenHTF and AI
In many embedded development workflows, system-level and black-box testing are naturally positioned later in the lifecycle, once firmware functionality and system behavior have stabilized. While effective for validation, this timing can limit early feedback on real device behavior.
This talk presents a practical approach to accelerating system-level testing by combining OpenHTF with AI-assisted development. By leveraging AI to bootstrap test frameworks and focusing firmware engineers on test design rather than infrastructure, teams can bring automated device-under-test validation online much earlier in the development cycle.
Through real embedded firmware experience, we demonstrate how early system-level automation complements unit and integration tests, enabling faster iteration, improved reliability, and smoother collaboration between firmware and QA teams.
Github Repo
https://github.com/bbmegareve/stm32_openhtf_demo
Video Timeline:
- 0:00 Introduction & About the Speaker
- 2:40 The Reality of Embedded System Testing
- 4:53 Late Testing = Expensive Feedback
- 6:08 The Testing Pyramid Explained
- 7:32 The Missing Piece: Automated System Tests
- 11:17 Introducing OpenHTF
- 14:32 Framework Architecture & Folder Organization
- 16:02 Live Demo — Quick Manual Test
- 17:57 Demo OpenHTF key files overview
- 26:10 Demo setup and live run output explanation
- 32:05 What Just Happened? Boundary moved on pyramid
- 34:16 AI Acceleration Workflow
- 35:09 Writing and running new test case with AI help
- 40:29 Writing a new OpenHTF plug
- 43:49 Key Takeaways & What You Can Do Tomorrow
What this presentation is about and why it matters
How do you pull system-level embedded tests earlier, when today they are still manual, late, and hard to reproduce? Everardo Garcia tackles that tension with a practical walkthrough built around OpenHTF, a Python-based hardware testing framework, and a simple STM32 Nucleo setup over USB. He shows the shape of the workflow, from flashing and reading UART output to structuring tests, logs, and configuration so they fit into day-to-day development. The talk also adds a second layer, using GitHub Copilot to speed up test and plug creation. It is a good fit for teams trying to make hardware validation less ad hoc and more repeatable.
Who will benefit the most from this presentation
- Embedded software engineers who currently validate firmware through manual terminal sessions and spreadsheets.
- Test or validation engineers who need repeatable system-level checks on real hardware.
- Firmware developers working with STM32, UART, SWD, or similar board-level interfaces.
- Team leads who want earlier feedback without building a large test lab first.
- Engineers curious about using AI to scaffold test code while keeping human review in the loop.
What you need to know
There are no deep prerequisites, but the session will land better if you are comfortable with:
- Basic embedded development and firmware flashing.
- Serial console workflows such as UART terminals.
- Reading simple Python code and YAML configuration.
- The idea of system-level tests versus unit tests.
Glossary (terms used in this talk)
- TDD (Test-Driven Development): A development technique where tests are written before implementation and code is iterated until the tests pass.
- Python: A high-level programming language commonly used for modeling, scripting, and test automation in engineering workflows. In DSP and FPGA development it is often used for algorithm prototyping and verification.
- JTAG (Joint Test Action Group): A standardized hardware debugging and test interface used for low-level access to electronic devices.
- Test phase: A named step in a hardware or system test sequence, often used to separate setup, actions, and validation. Breaking a test into phases makes hardware interaction, logging, and failure handling easier to reason about.
- Plug: A hardware interface component that encapsulates a specific way of talking to a device or accessory, such as flashing, reset, or serial communication. Plugs isolate transport details from the test logic that uses them.
- Structured JSON reporting: A reporting approach that stores test results in machine-readable JSON with fields such as measurements, timestamps, and execution history. This makes test outcomes easier to archive, compare, and debug later.
- SWD (Serial Wire Debug): ARM's two-pin debug transport protocol used by debug probes to access target memory and debug interfaces without a full JTAG port.
- UDS: Unified Diagnostic Services, a diagnostic communication protocol often used over CAN for vehicle diagnostics and other embedded diagnostic exchanges.
Toolbox (mentioned in this talk)
- ST-LINK: A debug probe and programmer used with STM microcontrollers. Tools that support it can connect to and debug STM-based targets.
- PuTTY: A terminal emulator and serial client commonly used to connect to embedded devices over UART or similar interfaces.
- OpenHTF: An open-source hardware testing framework for defining phases, measurements, and structured test execution against physical devices. It is designed to capture repeatable results and detailed logs.
- GitHub Copilot: An AI coding assistant that can generate or complete code from prompts and surrounding context. It is often used to scaffold repetitive code and boilerplate faster than writing it manually.
- STM32CubeProgrammer: STMicroelectronics software for programming and configuring STM32 microcontrollers. It is commonly used from the command line for flashing firmware and verifying device state.
- Minicom: A terminal emulator commonly used for serial console access on embedded targets. It provides interactive text I/O over UART or similar connections.
Final thoughts
Practical and demo-driven, this session gives you a usable mental model for moving from manual board poking to repeatable system-level tests, then shows how AI can help with the scaffolding without taking over the judgment. It should be especially useful if you own firmware validation, write test harnesses, or want earlier feedback on hardware behavior. The talk’s appeal is in how it makes a messy workflow feel tractable with a few disciplined building blocks.
This overview is AI-generated from the session transcript. Spot an issue? Let us know.
Hi Roberto, I am glad you enjoyed it!! .
Interesting question of using Pytest, Pytest is very popular and we have used it in some projects as well on the past , however pytest is aimed for "unit testing" at the software level not at the hardware level, for managing equipment to test devices for testing at the functional / system level testing pytest will start to have limitations that you need to figureout and debugging it becomes harder. So pretty much pytest will end up as an adapted implementation for Hardware In the loop test and that is where in the long rung you will notice it becomes hard to maintain if you switch to different equipment or even if you switch to a similar product to test and the reuse becomes hard, I have also seen on some long running test its does not manage very some of the waiting.
OpenHTF was build to have a clear architecture on how to handle the lifecycle of your hardware using "plugs", the plug concept can handle automatically the setup/tearDown, also testing hardware often you need a set of "ranges" to identify a PASS or FAIL (voltage output, signal strength, time to settle, etc ) vs using strict "asserts" as true or false.
So this is my criteria and priority order, I may have more but these are my top 5
- OpenHTF is build thinking for Device Under Test (hardware test), not unit test . It has the concept also for test phases where you need to usually do a lot of sequence before getting to a particular state on the system
- Hardware lifecycle is handled automatic (or manual if you need it)
- The plugs concept helps to clearly separate the hardware or interface from the test logic (think of it as your hardware driver layer).
- The decorator pattern is much cleaner to inject plugs, every plug shares the BasePlug contract, you can substitute implementations. And this make it more maintainable when building any custom tool or similar too that will do a same job (example: power supplies to turn on or off)
- If you need to run a set of test in factory with an operator (or your self with a box of untested hardware), its a direct port (has a UI for that also), for hardware bring-ups it helps to have something like this when running same test on more than 5 , 10 , 20 units.
- Also the decorator pattern is used on the measurements criteria, I like the way this type of criteria vs assets fits on our embedded systems world:
@htf.measures( htf.Measurement("temperature") # Declares the measurement name .in_range(0, 100) # Defines the validator .with_units(units.DEGREE_CELSIUS) # Specifies the unit .with_precision(1) # Rounds to 1 decimal place )
Thanks a lot for the quick and detailed response! I'll definitely give it a try. :)
Hello Everardo! I really enjoyed your presentation, and I completely agree about the importance of integrating automated system tests early in the dev cycle. I am curious, though: what led you to choose OpenHTF over pytest, given that pytest seems much more popular today?