Learning Effective Abstractions for Planning (LEAP)

Overview

Complex long-horizon and sparse reward robotics tasks are challenging for scaling end-to-end learning methods. By contrast, planning approaches have shown great potential to handle such complex tasks effectively. One of the major criticisms of planning-based approaches has been the lack of availability of accurate world models (aka abstractions) to utilize.

There has been a renewed interest in using learning-based approaches to learn symbolic representations that support planning. However, this research is often fragmented in disjoint sub-communities such as task and motion planning, reinforcement learning (hierarchical, model-based), planning with formal logic, planning with natural language (language models), and neuro-symbolic AI. This workshop aims to create a common forum to share insights, discuss key questions, and chart a path forward via abstraction.

We aim to facilitate this bridge-building in two ways: (1) a diverse selection of papers and invited speakers; and (2) a highly interactive workshop. Concretely, the workshop will highlight approaches that use different learning methods, mainly to learn symbolic and composable representations of world models. Key questions for discussion include:

1. What is the right objective for abstraction learning for robotic planning? To what extent should we consider factors such as soundness, completeness, target planner and planning efficiency, and task distribution?
2. What level of abstraction is needed for it to be effective? How general-purpose or specific do these abstractions have to be for long-term autonomy? Do learned abstractions need to be hierarchical or at a single level?
3. To what extent should the abstractions used for robotic planning be interpretable or explainable to a human? How can this be achieved?
4. How can existing pre-trained foundational models (large language models (LLMs) and vision-language models (VLMs)) be utilized for learning symbolic abstractions while ensuring guarantees about correctness?
5. When, where, and from what data should abstractions be learned? Should they be learned as priors in the robot factory, using expert demonstrations, or in the “wild” from interaction with humans or the world?

Following the success of the previous offering of the workshop at CoRL 2023, we propose a second iteration of the workshop at CoRL 2024. Specifically, the previous iteration of the workshop received a total of 26 submissions highlighting important characteristics and challenges of learning abstractions. They highlighted how pre-trained foundational models, specifically LLMs or VLMs, enable learning various forms of abstractions for a diverse set of robotics tasks. This iteration of the workshop would have a stronger emphasis on learning provably correct symbolic abstractions using different techniques.

Areas of Interest

We solicit papers of the following topics:

• Learning generalizable and composable representations for robot planning
• Learning for task and motion planning (TAMP)
• Learning state abstractions and action abstractions
• Natural language as an abstraction for learning-based planning
• Learning other knowledge representations for planning
• Learning for hierarchical planning
• Learning for LTL-based planning
• Neuro-symbolic approaches for task and motion planning
• Hierarchical reinforcement learning for robotics

Best Paper: ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

Important Dates

Schedule

Invited Speakers

Georgia Chalvatzaki

TU Darmstadt Germany

Caelan Garrett

Nvidia

Leslie Pack Kaelbling

Massachusetts Institute of Technology
USA

Eric Rosen

The AI Institute
Boston Dynamics

Siddharth Srivastava

Arizona State University USA

Gregory Stein

George Mason University USA

Marc Toussaint

TU Berlin
Germany

Organizing Committees

Naman Shah

Brown University
USA

David Paulius

Brown University
USA

Nishanth Kumar

Massachusetts Institute of Technology
USA

Jiayuan Mao

Massachusetts Institute of Technology
USA

Yiqing Xu

Natitonal University of Singapore
Singapore

Nakul Gopalan

Arizona State University
USA

Rudolph Lioutikov

Karlsruhe Institute of Technology
Germany

George Konidaris

Brown University
USA

Accepted Papers (Read them here)

• Illustrated Landmark Graphs for Long-Horizon Policy Learning
  Christopher Watson, Arjun Krishna, Rajeev Alur, Dinesh Jayaraman
• BMP: Bridging the Gap between B-Spline and Movement Primitives
  Weiran Liao, Ge Li, Hongyi Zhou, Rudolf Lioutikov, Gerhard Neumann
• A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards
  Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, Yunzhu Li
• SkillGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment
  Caelan Reed Garrett, Ajay Mandlekar, Bowen Wen, Dieter Fox
• FlowNav: Learning Efficient Navigation Policies via Conditional Flow Matching
  Samiran Gode, Abhijeet Nayak, Wolfram Burgard
• Planning with Adaptive World Models for Autonomous Driving
  Arun Balajee Vasudevan, Neehar Peri, Jeff Schneider, Deva Ramanan
• Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learning
  Meng Feng, Viraj Parimi, Brian C. Williams
• Language Guided Operator Learning for Goal Inference
  Zachary S Siegel, Jiayuan Mao, Nishanth Kumar, Tianmin Shu, Jacob Andreas
• Least Commitment Planning for the Object Scouting Problem - Preliminary Results
  Max Merlin, David Paulius, George Konidaris
• Learning Compositional Behaviors from Demonstration and Language
  Weiyu Liu, Neil Nie, Jiayuan Mao, Ruohan Zhang, Jiajun Wu
• Enhancing Object Search by Augmenting Planning with Predictions from Large Language Models
  Shahriar Hossain, Gregory J. Stein, Abhishek Paudel
• Environment as Policy: Learning to Race in Unseen Tracks
  Hongze Wang, Jiaxu Xing, Nico Messikommer, Davide Scaramuzza
• Local Policies Enable Zero-shot Long-horizon Manipulation
  Murtaza Dalal, Min Liu, Walter Talbott, Chen Chen, Deepak Pathak, Jian Zhang, Russ Salakhutdinov
• Latent Space Exploration and Trajectory Space Update in Temporally-Correlated Episodic Reinforcement Learning
  Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, Gerhard Neumann
• Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
  Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Chris Paxton, Lerrel Pinto, Soumith Chintala, Nur Muhammad Mahi Shafiullah
• TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning
  Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, Gerhard Neumann
• Structured Exploration in Reinforcement Learning by Hypothesizing Linear Temporal Logic Formulas
  Yichen Wei, Xiaochen Li, Jason Xinyu Liu, Naman Shah, Benedict Quartey, George Konidaris, Stefanie Tellex, Akhil Bagaria
• AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
  Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Ajay Mandlekar, Dieter Fox, Yijie Guo
• RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
  Hanxiao Jiang, Binghao Huang, Ruihai Wu, Zhuoran Li, Shubham Garg, Hooshang Nayyeri, Shenlong Wang, Yunzhu Li
• MotIF: Motion Instruction Fine-tuning
  Minyoung Hwang, Joey Hejna, Dorsa Sadigh, Yonatan Bisk
• Efficient long-horizon planning and learning for locomotion and object manipulation
  Victor Dhédin, Huaijiang Zhu, Ludovic Righetti, Majid Khadiv
• ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
  Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, Li Fei-Fei
• Learning Contact-rich Abstractions using Tensor Factorization
  Teng Xue, Amirreza Razmjoo, Suhan Shetty, Sylvain Calinon
• CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph
  Yixuan Wang, Leonor Fermoselle, Tarik Kelestemur, Jiuguang Wang, Yunzhu Li
• Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
  Marvin Alles, Philip Becker-Ehmck, Patrick van der Smagt, Maximilian Karl
• On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer
  Elie Aljalbout, Felix Frank, Maximilian Karl, Patrick van der Smagt
• From Language to Action with Object-level Planning
  David Paulius, Alejandro Agostini, George Konidaris
• Learning to Act for Perceiving in Partially Unknown Environments
  Leonardo Lamanna, Mohamadreza Faridghasemnia, Alfonso Gerevini, Alessandro Saetti, Alessandro Saffiotti, Luciano Serafini, Paolo Traverso
• Zero-Shot Offline Imitation Learning via Optimal Transport
  Thomas Rupf, Marco Bagatella, Nico Gürtler, Jonas Frey, Georg Martius
• SPIRE: Synergistic Planning, Imitation, and Reinforcement for Long-Horizon Manipulation
  Zihan Zhou, Animesh Garg, Dieter Fox, Caelan Reed Garrett, Ajay Mandlekar
• VQ-CNMP: Neuro-Symbolic Skill Learning for Bi-Level Planning
  Hakan Aktas, Emre Ugur
• DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Models
  Yuchen Liu, Luigi Palmieri, Ilche Georgievski, Marco Aiello
• BaB-ND: Long-Horizon Motion Planning with Branch-and-Bound and Neural Dynamics
  Keyi Shen, Jiangwei Yu, Huan Zhang, Yunzhu Li
• Successor Representations Enable Emergent Compositional Instruction Following
  Vivek Myers, Chunyuan Zheng, Anca Dragan, Kuan Fang, Sergey Levine
• Guiding Long-Horizon Task and Motion Planning with Vision Language Models
  Zhutian Yang, Caelan Reed Garrett, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling
• Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics
  Yixuan Huang, Christopher Agia, Jimmy Wu, Tucker Hermans, Jeannette Bohg
• Lang2LTL-2: Grounding Spatiotemporal Navigation Commands Using Large Language and Vision-Language Models
  Jason Xinyu Liu, Ankit Shah, George Konidaris, Stefanie Tellex, David Paulius
• Representing Positional Information in Generative World Models for Object Manipulation
  Bart Dhoedt, Pietro Mazzaglia, Stefano Ferraro, Tim Verbelen, Sai Rajeswar
• TAMPering with RLBench: Enabling joint developments in Task and Motion Planning and Reinforcement Learning research
  Alexander Dürr, Elin A. Topp, Jacek Malec
• Let's Make a Splan: Risk-Aware Trajectory Optimization in a Normalized Gaussian Splat
  Jonathan Michaux, Seth Isaacson, Challen Enninful Adu, Adam Li, Parker Ewen, Katherine A. Skinner, Ram Vasudevan
• Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight
  Jiaxu Xing, Leonard Bauersfeld, Yunlong Song, Chunwei Xing, Davide Scaramuzza
• LTL-Transfer: Skill Transfer for Temporal Task Specification
  AJason Xinyu Liu, Ankit Shah, Eric Rosen, Mingxi Jia, George Konidaris, Stefanie Tellex
• Online Neuro-Symbolic Predicate Invention for High-Level Planning
  Yichao Liang, Nishanth Kumar, Hao Tang, Adrian Weller, Joshua B. Tenenbaum, Tom Silver, Joao F. Henriques, Kevin Ellis
• Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph
  Utkarsh Aashu Mishra, Yongxin Chen, Danfei Xu
• Say-REAPEx: An LLM-Modulo UAV Online Planning Framework for Search and Rescue
  Björn Döschl, Jane Jean Kiam
• Neural MP: A Generalist Neural Motion Planner
  Deepak Pathak, Jiahui Yang, Murtaza Dalal, Russ Salakhutdinov, Russell Mendonca, Youssef Khaky
• ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter
  Yaoyao Qian, Xupeng Zhu, Ondrej Biza, Shuo Jiang, Linfeng Zhao, Haojie Huang, Yu Qi, Robert Platt
• Language Models can Infer Action Semantics for Classical Planners from Environment Feedback
  Wang Zhu, Ishika Singh, Robin Jia, Jesse Thomason
• SkillWrapper: Skill Abstraction in the Era of Foundation Models
  Shreyas Sundara Raman, Ziyi Yang, Benned Hedegaard, Stefanie Tellex, David Paulius, Naman Shah
• Sequential Object-Centric Relative Placement Prediction for Long-horizon Imitation Learning
  Ben Eisner, Eric Cai, Octavian Donca, Teeratham Vitchutripop, David Held
• One-Shot Manipulation Strategy Learning by Making Contact Analogies
  Yuyao Liu, Jiayuan Mao, Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling
• DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation
  Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

Program Committee