NeurIPS-2019 Workshop on

Safety and Robustness in Decision Making

December 13th, East Ballroom A

Workshop Summary

Interacting with increasingly sophisticated decision-making systems is becoming more and more a part of our daily life. This creates an immense responsibility for designers of these systems to build them in a way to guarantee safe interaction with their users and good performance, in the presence of noise and changes in the environment, and/or of model misspecification and uncertainty. Any progress in this area will be a huge step forward in using decision-making algorithms in emerging high stakes applications, such as autonomous driving, robotics, power systems, health care, recommendation systems, and finance.

This workshop aims to bring together researchers from academia and industry in order to discuss main challenges, describe recent advances, and highlight future research directions pertaining to develop safe and robust decision-making systems. We aim to highlight new and emerging theoretical and applied research opportunities for the community that arise from the evolving needs for decision-making systems and algorithms that guarantee safe interaction and good performance under a wide range of uncertainties in the environment.

The research challenges we are interested in addressing in this workshop include (but are not limited to):

  • Counterfactual reasoning and off-policy evaluation.
  • Specifying appropriate definitions of safety and robustness, and quantifying/certifying them under model uncertainty.
  • Safe exploration.
  • Trading off robustness/safety with performance.
  • Understanding the connections between risk and robustness, and other related definitions.

Call for Papers

IMPORTANT DATES

    • Paper Submission Deadline: September 22, 2019
    • Notification of Acceptance: September 30, 2019
    • Workshop: Friday December 13, 2019

SUBMISSION INSTRUCTIONS

Papers submitted to the workshop should be between 4 to 8 pages long, excluding references and appendix, and in NeurIPS 2019 format (NOT ANONYMIZED). Accepted papers will be presented as posters or contributed oral presentations.

Submissions should be sent as a pdf file by email to [email protected]

Organizers

Yinlam Chow

(Google Research)

Mohammad Ghavamzadeh

(Facebook AI Research)

Shie Mannor

(Technion)

Marek Petrik

(University of New Hampshire)

Yisong Yue

(Caltech)

Invited Speakers

Finale Doshi-Velez

(Harvard University)

Title: Combining Statistical methods with Human Input for Evaluation and Optimization in Batch Settings

Abstract

Statistical methods for off-policy evaluation and counterfactual reasoning will have fundamental limitations based on what assumptions can be made and what kind of exploration is present in the data (some of which is being presented here by other speakers!). In this talk, I'll discuss some recent directions in our lab regarding ways to integrate human experts into the process of policy evaluation and selection in batch settings. The first deals with statistical limitations by seeking a diverse collection of statistically-indistinguishable (with respect to outcome) policies for humans to eventually decide from. The second involves directly integrating human feedback to eliminate or validate specific sources of sensitivity in an off-policy evaluation to get more robust estimates (or at least better understand the source of their non-robustness). More broadly, I will discuss open directions for moving from purely-statistical (e.g. off-policy evaluation) or purely-human (e.g. interpretability-based) approaches for robust/safe decision-making toward combining the advantages of both.

Dimitar Filev

(Henry Ford Technical Fellow at Ford Motor Company)

Title: Practical Approaches to Driving Policy Design for Autonomous Vehicles

Abstract

The presentation deals with some practical facets of application of AI methods to designing driving policy for autonomous vehicles. Relationship between the reinforcement learning (RL) based solutions and the use of rule-based and model-based techniques for improving their robustness and safety are discussed. One approach to obtaining explainable RL models by learning alternative rule-based representations is proposed. The presentation also elaborates on the opportunities for extending the AI driving policy approaches by applying game theory inspired methodology to addressing diverse and unforeseen scenarios, and representing the negotiation aspects of decision making in autonomous driving.

Thorsten Joachims

(Cornell University)

Title: Fair Ranking with Biased Data

Abstract

Search engines and recommender systems have become the dominant matchmaker for a wide range of human endeavors -- from online retail to finding romantic partners. Consequently, they carry immense power in shaping markets and allocating opportunity to the participants. In this talk, I will discuss how the machine learning algorithms underlying these system can produce unfair ranking policies for both exogenous and endogenous reasons. Exogenous reasons often manifest themselves as biases in the training data, which then get reflected in the learned ranking policy and lead to rich-get-richer dynamics. But even when trained with unbiased data, reasons endogenous to the algorithms can lead to unfair allocation of opportunity. To overcome these challenges, I will present new machine learning algorithms that directly address both endogenous and exogenous unfairness.

Nathan Kallus

(Cornell University)

Title: Efficiently Breaking the Curse of Horizon with Double Reinforcement Learning

Abstract

Off-policy evaluation (OPE) is crucial for reinforcement learning in domains like medicine with limited exploration, but OPE is also notoriously difficult because the similarity between trajectories generated by any proposed policy and the observed data diminishes exponentially as horizons grow, known as the curse of horizon. To understand precisely when this curse bites, we consider for the first time the semi-parametric efficiency limits of OPE in Markov decision processes (MDP), establishing the best-possible estimation errors and characterizing the curse as a problem-dependent phenomenon rather than method-dependent. Efficiency in OPE is crucial because, without exploration, we must use the available data to its fullest. In finite horizons, this shows standard doubly-robust (DR) estimators are in fact inefficient for MDPs. In infinite horizons, while the curse renders certain problems fundamentally intractable, OPE may be feasible in ergodic time-invariant MDPs. We develop the first OPE estimator that achieves the efficiency limits in both setting, termed Double Reinforcement Learning (DRL). In both finite and infinite horizons, DRL improves upon existing estimators, which we show are inefficient, and leverages problem structure to its fullest in the face of the curse of horizon. We establish many favorable characteristics for DRL including efficiency even when nuisances are estimated slowly by blackbox models, finite-sample guarantees, and model double robustness.

Daniel Kuhn

(EPFL)

Title: From Data to Decisions: Distributionally Robust Optimization is Optimal

Abstract

We study stochastic optimization problems where the decision-maker cannot observe the distribution of the exogenous uncertainties but has access to a finite set of independent training samples. In this setting, the goal is to find a procedure that transforms the data to an estimate of the expected cost function under the unknown data-generating distribution, i.e., a predictor, and an optimizer of the estimated cost function that serves as a near-optimal candidate decision, i.e., a prescriptor. As functions of the data, predictors and prescriptors constitute statistical estimators. We propose a meta-optimization problem to find the least conservative predictors and prescriptors subject to constraints on their out-of-sample disappointment. The out-of-sample disappointment quantifies the probability that the actual expected cost of the candidate decision under the unknown true distribution exceeds its predicted cost. Leveraging tools from large deviations theory, we prove that this meta-optimization problem admits a unique solution: The best predictor-prescriptor pair is obtained by solving a distributionally robust optimization problem over all distributions within a given relative entropy distance from the empirical distribution of the data.

Scott Niekum

(University of Texas at Austin)

Title: Scaling Probabilistically Safe Learning to Robotics

Abstract

In recent years, high-confidence reinforcement learning algorithms have enjoyed success in application areas with high-quality models and plentiful data, but robotics remains a challenging domain for scaling up such approaches. Furthermore, very little work has been done on the even more difficult problem of safe imitation learning, in which the demonstrator's reward function is not known. This talk focuses on three recent developments in this emerging area of research: (1) a theory of safe imitation learning; (2) scalable reward inference in the absence of models; (3) efficient off-policy policy evaluation. The proposed algorithms offer a blend of safety and practicality, making a significant step towards safe robot learning with modest amounts of real-world data.

Marco Pavone

(Stanford University)

Title: On Safe and Efficient Human-robot Interactions via Multi-modal Intent Modeling and Reachability-based Safety Assurance

Abstract

In this talk I will present a decision-making and control stack for human-robot interactions by using autonomous driving as a motivating example. Specifically, I will first discuss a data-driven approach for learning multimodal interaction dynamics between robot-driven and human-driven vehicles based on recent advances in deep generative modeling. Then, I will discuss how to incorporate such a learned interaction model into a real-time, interaction-aware decision-making framework. The framework is designed to be minimally interventional; in particular, by leveraging backward reachability analysis, it ensures safety even when other cars defy the robot's expectations without unduly sacrificing performance. I will present recent results from experiments on a full-scale steer-by-wire platform, validating the framework and providing practical insights. I will conclude the talk by providing an overview of related efforts from my group on infusing safety assurances in robot autonomy stacks equipped with learning-based components, with an emphasis on adding structure within robot learning via control-theoretical and formal methods

Andy Sun

(Georgia Tech)

Title: Recent Advances in Multistage Decision-making under Uncertainty: New Algorithms and Complexity Analysis

Abstract

In this talk, we will review some recent advances in the area of multistage decision making under uncertainty, especially in the domain of stochastic and robust optimization. We will present some new algorithmic development that allows for exactly solving huge-scale stochastic programs with integer recourse decisions, and algorithms in a dual perspective that can deal with infeasibility in problems. This significantly extends the scope of stochastic dual dynamic programming (SDDP) algorithms from convex or binary state variable cases to general nonconvex problems. We will also present a new analysis of the iteration complexity of the proposed algorithms. This settles some open questions in regards of the complexity of SDDP.

Aviv Tamar

(Technion)

Title: Visual Plan Imagination - An Interpretable Robot Learning Framework

Abstract

How can we build autonomous robots that operate in unstructured and dynamic environments such as homes or hospitals? This problem has been investigated under several disciplines, including planning (motion planning, task planning, etc.), and reinforcement learning. While both of these fields have witnessed tremendous progress, each have fundamental drawbacks: planning approaches require substantial manual engineering in mapping perception to a formal planning problem, while RL, which can operate directly on raw percepts, is data hungry, cannot generalize to new tasks, and is ‘black box’ in nature.

Motivated by humans’ remarkable capability to imagine and plan complex manipulations of objects, and recent advances in imagining images such as GANs, we present Visual Plan Imagination (VPI) — a new computational problem that combines image imagination and planning. In VPI, given off-policy image data from a dynamical system, the task is to ‘imagine’ image sequences that transition the system from start to goal. Thus, VPI focuses on the essence of planning with high-dim perception, and abstracts away low level control and reward engineering. More importantly, VPI provides a safe and interpretable basis for robotic control — before the robot acts, a human inspects the imagined plan the robot will act upon, and can intervene if necessary.

I will describe our approach to VPI based on Causal InfoGAN, a deep generative model that learns features that are compatible with strong planning algorithms. We show that Causal InfoGAN can generate convincing visual plans, and we demonstrate learning to imagine and execute real robot rope manipulation from image data. I will also discuss our VPI simulation benchmarks, and recent efforts in novelty detection, an important component in VPI, and in safe decision making in general.

Workshop Schedule

List of Accepted Papers

  1. Andrew Bennett, Nathan Kallus. Policy Evaluation with Latent Confounders via Optimal Balance
  2. Carroll L. Wainwright, Peter Eckersley. SafeLife 1.0: Exploring Side Effects in Complex Environments
  3. Chao Tang, Yifei Fan, Anthony Yezzi. An Adaptive View of Adversarial Robustness from Test-time Smoothing Defense
  4. Daniel S. Brown, Scott Niekum. Deep Bayesian Reward Learning from Preferences
  5. David I. Inouye, Liu Leqi, Joon Sik Kim, Bryon Aragam, Pradeep Ravikumar. Diagnostic Curves for Black Box Models
  6. David Venuto, Léonard Boussioux,Junhao Wang, Rola Dali, Jhelum Chakravorty, Yoshua Bengio, Doina Precup. Avoidance Learning Using Observational Reinforcement Learning
  7. Divyam Madaan, Sung Ju Hwang. Adversarial Neural Pruning
  8. Dmitrii Krasheninnikov, Rohin Shah, Herke van Hoof. Combining Reward Information from Multiple Sources
  9. Eleanor Quint, Dong Xu, Haluk Dogan, Zeynep Hakguder, Stephen Scott, Matthew Dwyer. Formal Language Constraints for Markov Decision Processes
  10. Gavin S. Hartnett, Andrew J. Lohn, Alexander P. Sedlack. Adversarial Examples for Cost-Sensitive Classifiers
  11. Geoffroy Dubourg-Felonneau, Omar Darwish, Christopher Parsons, Dàmi Rebergen, John W Cassidy, Nirmesh Patel, Harry Clifford. Deep Bayesian Recurrent Neural Networks for Somatic Variant Calling in Cancer
  12. Gokul Swamy, Siddharth Reddy, Sergey Levine, Anca D. Dragan. Scaled Autonomy: Enabling Human Operators to Control Robot Fleets
  13. Jason Carter, Marek Petrik. Robust Risk-Averse Sequential Decision Making
  14. Javier Garcia-Barcos, Ruben Martinez-Cantin. Parallel Robust Bayesian Optimization with Off-policy Evaluations
  15. Jesse Zhang, Brian Cheung, Chelsea Finn, Dinesh Jayaraman, Sergey Levine. Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents
  16. Julian Bolick, Gino Brunner, Oliver Richter, Roger Wattenhofer. Tunnel Vision Attack on IMPALA - Questioning the Robustness of Reinforcement Learning Agents
  17. Justin Cosentino, Federico Zaiter, Dan Pei, Jun Zhu. The Search for Sparse, Robust Neural Networks
  18. Kai Y. Xiao, Sven Gowal, Todd Hester, Rae Jeong, Daniel J. Mankowitz, Yuanyuan Shi, Tsui-Wei Weng. Learning Neural Dynamics Simulators with Adversarial Specification Training
  19. Mahdieh Abbasi, Changjian Shui, Arezoo Rajabi, Christian Gagné, Rakesh B. Bobba. Metrics for Differentiating Out-of-Distribution Sets
  20. Marco Gallieri, Seyed Sina Mirrazavi Salehian, Nihat Engin Toklu, Alessio Quaglino, Jonathan Masci, Jan Koutník, Faustino Gomez. Safe Interactive Model-Based Learning
  21. Mathieu Seurin, Philippe Preux, Olivier Pietquin. "I’m sorry Dave, I’m afraid I can’t do that" Deep-Q Learning From Forbidden Actions
  22. Matthew Sotoudeh, Aditya V. Thakur. Correcting Deep Neural Networks with Small, Generalizing Patches
  23. Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi. Striving for Simplicity in Off-Policy Deep Reinforcement Learning
  24. Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, Krzysztof Czarnecki. Out-of-distribution Detection in Classifiers via Generation
  25. Sai Kiran Narayanaswami, Nandan Sudarsanam, Balaraman Ravindran. An Active Learning Framework for Efficient Robust Policy Search
  26. Samuel Daulton, Shaun Singh, Vashist Avadhanula, Drew Dimmery, Eytan Bakshy. Thompson Sampling for Contextual Bandit Problems with Auxiliary Safety Constraints
  27. Stephen Mell, Olivia Brown, Justin Goodwin and Sung-Hyun Son. Safe Predictors for Enforcing Input-Output Specifications
  28. Vedant Nanda, Junaid Ali, Krishna P. Gummadi, Muhammad Bilal Zafar. Unifying Model Explainability and Accuracy through Reasoning Labels
  29. .Victoria Krakovna, Laurent Orseau, Miljan Martic, Shane Legg. Avoiding Side Effects By Considering Future Tasks
  30. Yuanyuan Shi, Kai Y. Xiao, Daniel J. Mankowitz, Rae Jeong, Nir Levine, Sven Gowal, Timothy Mann, Todd Hester. Data-Driven Robust Reinforcement Learning for Continuous Control
  31. Ahana Ghosh, Sebastian Tschiatschek, Hamed Mahdavi, Adish Singla. Towards Deployment of Robust AI Agents for Human-Machine Partnerships
  32. Ahmadreza Jeddi, Mohammad Javad Shafiee, Michelle Karg, Christian Scharfenberger, Alexander Wong. Learn2Perturb: Improving Adversarial Robustness on Deep Neural Networks through End-to-end Feature Perturbation Learning
  33. Akhilan Boopathy, Tsui-Wei Weng, Sijia Liu, Pin-Yu Chen, Luca Daniel. DoubleMargin: Efficient Training of Robust and Verifiable Neural Networks
  34. Alex Tamkin, Ramtin Keramati, Christoph Dann, Emma Brunskill. Distributionally-Aware Exploration for CVaR Bandits
  35. Ali Baheri, Subramanya Nageshrao,Ilya Kolmanovsky, Anouck Girard, H. Eric Tseng, Dimitar Filev. Deep Q-Learning with Dynamically-Learned Safety Module: A Case Study in Autonomous Driving
  36. Amir Nazemi, Paul Fieguth. Identifying threatening samples for adversarial robustness
  37. Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, Yisong Yue. Robust Regression for Safe Exploration in Control
  38. Anqi Liu, Hao Liu, Anima Anandkumar, Yisong Yue. Triply Robust Off-Policy Evaluation
  39. Arushi Jain, Doina Precup. Learning Options using Constrained Return Variance
  40. Benjamin Eysenbach, Jacob Tyo, Shane Gu, Ruslan Salakhutdinov, Zachary Lipton, Sergey Levine. Reinforcement Learning with Unknown Reward Functions
  41. Benjie Wang, Stefan Webb, Tom Rainforth. Statistically Robust Neural Network Classification
  42. Boxiao Chen, Selvaprabu Nadarajah, Stefanus Jasin. Robust Demand Learning
  43. Chandramouli S Sastry, Sageev Oore. Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices
  44. Dhruv Ramani, Benjamin Eysenbach. Avoiding Negative Side-Effects And Promoting Safe Exploration With Imaginative Planning
  45. Dimitrios I. Diochnos, Saeed Mahloujifar, Mohammad Mahmoody. Lower Bounds for Adversarially Robust PAC Learning
  46. Dotan Di Castro, Joel Oren, Shie Mannor. Practical Risk Measures in Reinforcement Learning
  47. Doyup Lee, Yeongjae Cheon, Woo-sung Jung. Anomaly Detection in Visual Question Answering
  48. Elmira Amirloo Abolfathi, Jun Luo, Peyman Yadmellat, Kasra Rezaee. CoachNet: An Adversarial Sampling Approach for Reinforcement Learning
  49. Evrard Garcelon, Mohammad Ghavamzadeh, Alessandro Lazaric and Matteo Pirotta. Improved Algorithms for Conservative Exploration in Bandits
  50. Evrard Garcelon, Mohammad Ghavamzadeh, Alessandro Lazaric and Matteo Pirotta. Conservative Exploration in Finite Horizon Markov Decision Processes
  51. I-Te Danny Hung, Chao-Han Huck Yang, Yi Ouyang, Pin-Yu Chen, Chin-Hui Lee, Xiaoli Ma. Toward Resilient Reinforcement Learning: Causal Inference Q-Networks
  52. Ilija Bogunovic, Andreas Krause, Jonathan Scarlett. Bayesian Optimization with Adversarial Corruption
  53. Jeet Mohapatra, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, Luca Daniel. Towards Verifying Robustness of Neural Networks Against Semantic Perturbations
  54. Jian Qian, Merwan Barlier, Igor Colin, Ludovic Dos Santos. Safe Reinforcement Learning with Probabilistic Constraints
  55. John D. Martin, Michal Lyskawinski, Xiaohu Li, Brendan Englot. Stochastically Dominant Distributional Reinforcement Learning
  56. Karthik Abinav Sankararaman, Anand Louis, Navin Goyal. Robust Identifiability in Linear Structural Equation Models of Causal Inference
  57. Kyriakos Polymenakos, Luca Laurenti, Andrea Patane, Jan-Peter Calliess, Luca Cardelli, Marta Kwiatkowska, Alessandro Abate, Stephen Roberts. Safety Guarantees for Planning Based on Iterative Gaussian Process
  58. Lan Hoang, Edward Pyzer-Knapp. Distributional Actor-Critic for Risk-Sensitive Multi-Agent Reinforcement Learning
  59. Matteo Papini, Andrea Battistello, Marcello Restelli. Safe Exploration in Gaussian Policy Gradient
  60. Matteo Turchetta, Andreas Krause, Sebastian Trimpe. Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimization
  61. Nader Asadi, Amir M. Sarfi, Mehrdad Hosseinzadeh, Sahba Tahsini, Mahdi Eftekhari. Diminishing the Effect of Adversarial Perturbations via Refining Feature Representation
  62. Nathan Fulton, Nathan Hunt, Nghia Hoang, Subhro Das. Formal Verification of End-to-End Learning in Cyber-Physical Systems: Progress and Challenges
  63. Nathan Kallus, Masatoshi Uehara. Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes
  64. Niranjani Prasad, Barbara E Engelhardt, Finale Doshi-Velez. Defining Admissible Rewards for High-Confidence Policy Evaluation
  65. Patrik Kolaric, Devesh K. Jha, Diego Romeres, Arvind U Raghunathan, Mouhacine Benosman, Daniel Nikovski. Robust Optimization for Trajectory-Centric Model-based Reinforcement Learning
  66. Philipp Renz, Sepp Hochreiter, Gunter Klaumauer. Uncertainty Estimation Methods to Support Decision-Making in Early Phases of Drug Discovery
  67. Prateek Jaiswal, Harsha Honnappa, Vinayak A. Rao. Variational Inference for Risk-Sensitive Decision-Making
  68. Reazul Hasan Russel, Bahram Behzadian, Marek Petrik. Optimizing Norm-bounded Weighted Ambiguity Sets for Robust MDPs
  69. Riashat Islam, Raihan Seraj, Samin Yeasar Arnob, Doina Precup. Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning
  70. Ryan-Rhys Griffiths, Miguel Garcia-Ortegon, Alexander A. Aldrick, Alpha A. Lee. Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation
  71. Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, Anima Anandkumar. Regret Analysis of Partially Observable Linear Quadratic Systems
  72. Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody. Adversarially Robust Learning Could Leverage Computational Hardness
  73. Sebastian East, Marco Gallieri, Jonathan Masci, Jan Koutník, Mark Cannon. Infinite-Horizon Differentiable Model Predictive Control
  74. Shun Zhang, Edmund H. Durfee, and Satinder Singh. Querying to Find a Safe Policy Under Uncertain Safety Constraints in Markov Decision Processes
  75. Soheil Sibdari. On the Robustness of Data-driven Pricing Decisions with Revenue Maximization
  76. Theodore Vasiloudis, Gianmarco De Francisci Morales, Henrik Bostrom. Quantifying Uncertainty in Online Regression Forests
  77. Wenhao Luo, Ashish Kapoor. Airborne Collision Avoidance Systems with Probabilistic Safety Barrier Certificates
  78. Wesley Chung, Shin-ichi Maeda, Yasuhiro Fujita. Safe Reinforcement Learning with Adversarial Threat Functions
  79. Yuh-Shyang Wang, Tsui-Wei Weng, Luca Daniel. Verification of Neural Network Control Policy Under Persistent Adversarial Perturbation
  80. Z. Cranko, Z. Shi, X. Zhang, S. Kornblith, R. Nock, M. Li, Y. Ma, H. Jin. Certifying Distributional Robustness using Lipschitz Regularisation
  81. Ziping Xu, Ambuj Tewari . Worst-Case Regret Bound for Perturbation Based Robust Exploration in Reinforcement Learning
  82. Ibrahim Ben Daya, Mohammad Javad Shafiee, Michelle Karg, Christian Scharfenberger, Alexander Wong. SANER: Efficient Stochastically Activated Network Ensembles for Adversarial Robustness Through Randomized Assembly
  83. Maxime Wabartha, Andrey Durand, Vincent Francois-Lavet, Joelle Pineau. Handling Black Swan Events in Deep Learning with Diversely Extrapolated Neural Networks
  84. Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster, Thomas Lukasiewicz, Phil Blunsom. Sequence-to-Sequence Adversarial Attack for Detecting Inconsistent Neural Explanations
  85. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, Phil Blunsom. Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations