Embodied AI Safety: Reimagining safety engineering for artificial intelligence in physical systems

Embodied AI (eAI), also called Physical AI, uses artificial intelligence based on machine learning to interact with the physical world. We are already seeing eAI deployed in the real world in robotaxis, smart medical devices, household robots, and other applications. However, everyone is struggling with the safety of these devices: how to design for safety, how to evaluate safety, and how to think about whether any particular eAI system is acceptably safe.

This talk provides an overview of my new book on this topic, with robotaxi safety as a concrete example. Anyone working in this area needs a basic understanding of four core areas: safety engineering, cybersecurity engineering, machine learning technology, and human/computer interaction. The talk also discusses eAI safety issues in the wild, the complexities of establishing what risks might be acceptable, and open challenges in eAI safety. A proposal for reimagining safety engineering responds to the huge disruption that eAI technology creates when applying traditional computer-based system safety approaches. The talk finishes with a call to build justifiable trust in eAI safety. 

Highlights:

• Identifies key principles in the areas of system safety, cybersecurity, machine learning, human/computer interaction, and liability
• Illustrates how things change when a human operator is replaced by a computer, using examples from the robotaxi industry
• Explains the need to re-frame system safety from risk optimization to a multi-constraint satisfaction approach

What this presentation is about and why it matters

What changes when a system with sensors and actuators stops being just an embedded controller and starts making machine learning driven decisions in the physical world? Phil Koopman approaches that question as a case for rethinking safety engineering, not a checklist update. He moves across four literacy areas, safety engineering, cybersecurity, machine learning, and human factors, using robotaxi incidents, drones, and other embodied AI examples to show where familiar assumptions break down. This is a broad, opinionated talk grounded in real safety concerns. It will be especially useful if you work anywhere that AI can affect physical systems, people nearby, or public trust.

Who will benefit the most from this presentation

• Safety engineers working on autonomy or robotics, especially if your current process comes from conventional embedded systems
• Security engineers who need to think about malicious behavior as a safety issue in physical systems
• ML engineers on perception or planning systems who need a better safety framing than accuracy alone
• Product, systems, or compliance leads responsible for risk, accountability, or trust in deployed autonomous systems
• Human factors or UX practitioners who support supervision, handoff, or fallback behavior in safety-critical automation

What you need to know

No deep background is required, but the talk will land better if you are already familiar with a few basic ideas:

• What an embedded system is, and how sensors and actuators connect software to the physical world
• Basic safety engineering vocabulary such as hazards, risk, mitigation, and redundancy
• General awareness of machine learning systems, especially classification and training on data
• Interest in autonomous systems, robotaxis, drones, or other safety-critical cyber-physical products

Glossary (terms used in this talk)

• Duty of care: A legal and ethical obligation to act with reasonable care when an activity can affect others. In safety-critical systems, it helps define whether behavior is acceptable, not just whether a crash occurred.
• Safety integrity level (SIL): A graded measure of how much rigor is expected for a given level of safety risk. Higher levels usually require stronger processes, more evidence, and additional safeguards.
• V model: A development and validation structure that traces requirements down into design and implementation, then back up through verification and validation. It is often used in safety standards to tie engineering process to confidence in system behavior.
• Automation bias: The tendency to trust computer output over human judgment, even when that trust is not justified. It can reduce vigilance and delay correction when an automated system behaves unexpectedly.
• Automation complacency: A reduction in attention that can happen when an automated system has performed well for a long time. It becomes harder for a person to stay actively engaged and ready to intervene.
• UL 4600: A safety standard for autonomous products and systems, focused on assuring safety without relying on a human driver or operator as the primary safety mechanism.
• Embodied AI: AI systems that operate in and interact with the physical world using sensors and actuators, combining traditional embedded systems with machine learning components.
• Perception response time: The time it takes a human to perceive an event, interpret it, decide on an action, and begin to execute that action; relevant for human-in-the-loop safety and supervision.

Final thoughts

Practical and deliberately expansive, this talk gives you a way to think about embodied AI that is bigger than model performance or isolated compliance checks. The value is a sharper lens for judging safety claims, understanding why physical systems are different, and spotting the gaps where process, supervision, and accountability have to do real work. It will help engineers, managers, and reviewers who are responsible for systems that act in the world. The through-line is clear: once software can move things, safety stops being a side topic.

This overview is AI-generated from the session transcript. Spot an issue? Let us know.