TAS Governance Node Seminar Lorenzo Strigini City University of London

Claiming that a system is "safe enough" implies a prediction that it will cause accidents infrequently enough. These quantitative predictions are especially hard for some autonomous systems, like self-driving road vehicles for unrestricted use. I will briefly discuss the possible quantitative requirements and the difficulties of demonstrating that they are satisfied. These difficulties may arise from insufficient evidence (we may simply not have enough relevant experience with these novel systems) and/or from weak arguments (statistical reasoning that is undermined by relying on doubtful assumptions). I will outline research using forms of probabilistic argument that try to avoid the latter problem. These formal probabilistic arguments mimic common, informal arguments for system safety but allow one to study to what extent they should bolster confidence and what are their limits. These forms of mathematical argument might support more modest safety claims than desired; but they support risk-aware decisions about these novel systems, and indicate directions for progress towards greater confidence in greater safety.