Keynote

Learning from Disaster

Watch Now

01:17:57

EOC 2022

Login to Watch

Jack Ganssle

Jack Ganssle is an internationally-recognized embedded systems engineer, author and speaker. He designed some of the very first embedded systems. He started three high tech companies, built or managed the development of over 100 embedded systems, and now advises companies, lectures, and writes about embedded systems. He was the only embedded person on NASA's Super Problem Resolution Team, created by the Columbia Accident Investigation Board to harness the expertise of highly-respected people from outside the agency. He has worked on many classified government projects. Full bio here.

View full profile

Topics Covered

Design Cycle Software Design

Keynote

Learning from Disaster

Jack Ganssle

01:17:57 EOC 2022

Jack Ganssle

01:17:57

Civil engineers have learned how to avoid failure from their rich history of bridge collapses, tunnel floodings, and building disintegrations. The firmware world is quite different; it seems we all make the same mistakes, repeatedly. Yet most problems have similar root causes. In this class we’ll examine a number of embedded disasters, large and small, and extract lessons we must learn to improve our code. It’s a fun, wild ride though avoidable catastrophes!

1 / 6

Please log in or create an account to test your knowledge and see the answers.

According to Jack's talk, what practical step could have prevented missions like Clementine and NEAR from losing fuel after software crashes?

A Enable and use a hardware watchdog timer so the system resets before runaway thruster code consumes fuel.

B Rely solely on frequent manual ground resets and operator intervention during anomalies.

C Remove all thruster control from flight software and control thrusters manually from Earth.

D Trust that intermittent software crashes will never affect critical actuators if the code looks stable.

E Only test thruster code in simulation; hardware watchdogs are unnecessary if simulation passes.

What did Jack emphasize as one of the most effective ways to find bugs before running the code on hardware?

A Relying on automated unit tests alone without human review.

B Performing systematic code inspections early and routinely.

C Waiting until after a field failure to perform a full code audit.

D Using only static analysis tools and avoiding human review to save time.

E Outsourcing bug finding to beta users and customer field testing.

What stance did Jack recommend regarding assertions (assert) in C code for embedded systems?

A Remove all asserts in production builds because they only help during development.

B Use asserts only for logging and never change their behavior at runtime.

C Keep asserts in the code (avoid turning them off with NDEBUG) and, if needed, change how they report or mitigate errors rather than removing them.

D Replace asserts with complex exception-handling frameworks in all embedded code.

E Use asserts as normal control flow statements (e.g., assignments) to save lines of code.

How did Jack describe the role of simulation in testing embedded systems?

A Simulation is a complete substitute for hardware testing and is sufficient for launch decisions.

B Simulation is useless and should never be used; only live hardware tests matter.

C Simulations can replace unit tests but you should avoid integration testing.

D Simulation is a valuable model but is always fiction and must be augmented with real, lifelike tests.

E If a simulation runs millions of cycles, there's no need to test on the actual hardware.

What approach to handling sensor readings did Jack recommend in order to avoid failures like the 737 MAX crashes?

A Always trust a single sensor reading and act quickly to minimize pilot workload.

B Compare multiple sensors and validate readings against physical limits and laws; take mitigating action if data is impossible.

C Ignore sudden sensor spikes and never trigger any automatic mitigation to avoid false alarms.

D Rely solely on sensor filtering (e.g., moving average) to smooth out bad data and never cross-check.

E Disable sensors that produce any intermittent anomalies and rely on pilot judgement only.

Which process failure did Jack highlight as a root cause in several space and launch disasters, and what is the recommended remedy?

A Overusing formal methods; remedy: write code faster with less paperwork.

B Avoiding redundancy; remedy: always use only single-bank systems to reduce complexity.

C Lack of version control and release discipline; remedy: use version control and rigorous configuration management to ensure the correct flight code is launched.

D Excessive simulation; remedy: remove all simulation from the development process.

E Relying on third-party RTOSes; remedy: always build a custom RTOS to match mission needs.

Formatting help

italics	surround text with asterisks
bold	surround text with two asterisks
hyperlink	[hyperlink](https://example.com) or just a bare URL
inline code	surround text with single `backticks`
code block (multi-line)	wrap on its own lines with three backticks: ``` your code here ```
~~strikethrough~~	surround text with ~~two tilde characters~~
quote	prefix with >

Upvotes Newest Oldest

Kevin H

Score: 0 | 3 years ago | no reply

And the most recent spacecraft failure, the Japanese Lunar Lander, which crashed into a crater. The spacecraft had originally been targeted to land on a flat plain, but at the last minute someone decided it would be better to land inside a crater (how cool!). Unfortunately, the crater walls confused the sensors during landing, because the software had not been programmed to recognize them as normal (there are no walls on a flat plain).

kwadrat

Score: 0 | 4 years ago | no reply

Boeing 737 pilot explains what can be learnt/learned from flight disasters:
https://www.youtube.com/c/MentourPilotaviation/videos
It shows me that details matter.

SimonSmith

Score: -1 | 4 years ago | no reply

Great talk. +1 for asserts for detecting design bugs and contracts. I often wonder how requirements complexity affects reliability - we treat firmware as able to implement almost any functionality, but should it? Can we also write less buggy manual code, by using qualified code generators such as SCADE?

Stephane.Boucher

Score: 1 | 4 years ago | no reply

The recording has been posted.

Unknown

Score: 0 | 4 years ago | 2 replies

Cannot access the Zoom call because maximum capacity has been reached. Well done!

Stephane.Boucher

Score: 0 | 4 years ago | 2 replies

Should be able to join now.
There was a setting on Zoom that wasn't properly set.
Sorry about that.

Bill

Score: 0 | 4 years ago | 1 reply

I had to punt after being shut out. How soon before the no longer live playback is available?

Stephane.Boucher

Score: 0 | 4 years ago | 1 reply

We'll try to process and upload the recording this afternoon.

Bill

Score: 0 | 4 years ago | 1 reply

Thank you!

Stephane.Boucher

Score: 0 | 4 years ago | no reply

Available now!

pintert3

Score: 0 | 4 years ago | no reply

Thanks!

pintert3

Score: 0 | 4 years ago | no reply

Same here. :(

Jay.Cosper

Score: 0 | 4 years ago | no reply

Great talk! Will the slides be available for download?

Partners & Sponsors

Percepio

Edge Impulse

STMicroelectronics

Renesas

Amazon AWS

The Qt Company

NXP

TUXERA

XMOS

Timesys

Saleae

Become a Sponsor