Open-Ended Learning through Imagination(RSS 2024)

July 15, 2024

Open-Ended Robot Learning through Imagination:
How can robots use imagination to facilitate continual learning?

Service robots are expected to be more autonomous and efficiently work in human-centric environments. For these robots, object perception and grasping are two challenging tasks due to the high demand for the accurate and real-time response under changing and unpredictable environmental conditions. Although many problems have already been understood and solved successfully, many challenges still remain. Open-ended learning is one of these challenges waiting for many improvements.

From a human cognition standpoint, imagination is often considered a foundational ability of planning and is defined as the ability of the brain to form images or sensations without any immediate sensory input. In robotics, imagination refers to the ability of robots to comprehend information across multiple modalities inputs (e.g., visual, auditory, and textual), to generate a representation of the desired situation based on the input description and the current state of the environment. This capability enables robots to understand and respond to a wide range of information, enhancing their overall abilities and making them more adaptable in various environments.

This workshop explores the pivotal role of imagination in robotics and how imagination could facilitate continual learning, spanning various levels of perception, action, and planning. Our discussion will be mainly around how imagination enhances robotic capabilities including generative and diffusion models in perception and action, as well as visual task planning. Beyond enhancing the capabilities of robots, we explore how these imagination models serve as versatile tools, offering insights into explaining complex concepts to non-expert users and generating invaluable training data.

Topics of Discussion

In this workshop, we would like to discuss the critical role of open-ended robot learning and how imagination could facilitate continual learning, spanning various levels of perception, action, and planning. We invite a number of renowned experts in the field who will highlight the current successes and future challenges these robots face. In particular, we will discuss the current and future challenges and opportunities for open-ended approaches. Together, we will dive deep into fundamental questions, such as:

Human-Robot Interaction: How can imaginative machines bridge the communication gap between robots and non-expert users, serving as effective proxies for an explanation?
Task Planning: What are the benefits of imagination models in improving the adaptability of robot task planning in complex environments?
Interdisciplinary: How can imagination models integrate insights from large pre-trained models, robot vision, natural language processing, and artificial intelligence to enhance robotic cognition?
Ethical Considerations: As we empower robots with imaginative capabilities, what ethical considerations should be taken into account, particularly in terms of user interaction, data privacy, and societal impact?
Future Directions: What are the key challenges and exciting avenues for future research in advancing imagination in robotics, and how can the field collectively address these challenges?

Topics of Interest

Topics of interest include but not limited to the following:

Various forms of imagination for continual robot learning
Architectures for open-ended learning
Transfer learning from one to another type of robot
Lifelong learning and adaptation for autonomous robots
Robot task planning through natural language
Generative models to support continual learning
Open-ended grasping of deformable objects
Continual task-informed grasping
Deep transfer learning for object perception
Knowledge transfer and avoidance of catastrophic forgetting
Continual affordance learning
Challenges of Human-Robot collaborative manipulation
Grasping and object manipulation
Lifelong 3D object category learning and recognition
Active perception and scene interpretation
Coupling between object perception and manipulation
Learning from demonstrations

Call for Papers

Great line-up of speakers

Tamim Asfour / Karlsruhe Institute of Technology (KIT), Germany
Pieter Abbeel & Sherry Yang / UC Berkeley, US
Jens Kober / TU Delft, Netherlands
Shuran Song / Standford and Columbia University, US
Gregory S. Chirikjain / National University of Singapore, and the University of Delaware/ Singapore, US
Ingmar Posner / Oxford University, UK
Leslie Kaelbling and Yilun Du / MIT, US

Program

Time	Speaker	Topic	Presentation
9:00-9:10	Organizers	Welcome & Introduction	download
9:10-9:50	Invited Talk #1	--	download
9:50-10:30	Invited Talk #2	--	download
10:30-11:00	Coffee break	Interactive poster presentations
11:00-12:00	Invited Talk #3	--	download
12:0-13:30	Lunch break	--	--
13:30-14:00	Poster Presentations	highlight poster presentations (3 mins each)	--
14:00 - 14:40	Invited Talk #4	--	download
14:40-15:20	Invited Talk #5	-	download
15:20 - 15:50	Hao Su	Learning to Model the Environment for Interaction.	download
15:20-15:50	Coffee break	Interactive poster presentations	--
15:50-16:30	Invited Talk #6	--	download
16:30-17:30	Panel Discussion	Interactive discussion
17:30	Closing remarks

Call for Papers

Submissions

We encourage contributions with either a contributed paper ( IEEE conference format, 6 pages without references), an extended abstract of a published work (IEEE conference format, 2 pages maximum).
All papers are reviewed using a blind review process. All papers must be written in English and submitted electronically in PDF format by emailing it to oel.workshop@gmail.com

Important Dates

Submission Deadline: TBD
Notification: TBD
Workshop Date: TBD

Accepted Papers

Authors	Title

Photos of the workshop

Organizers

Hamidreza Kasaei, University of Groningen, Netherlands

Hamidreza Kasaei an Assistant Professor in the Department of Artificial Intelligence at the University of Groningen in the Netherlands. With a strong background in robot learning, computer vision, machine learning, my research interests primarily focus on continual robot learning, 3D Object Perception and Object Manipulation. My current work is centered around enabling robots to learn from past experiences, interact safely with non-expert human users, and use data-efficient open-ended machine-learning techniques. To achieve this, my research group concentrates on Lifelong Interactive Robot Learning (IRL-Lab) and we strive to push the boundaries of robotics research. Prior to joining the University of Groningen, I was a visiting scholar with the Imperial College London.

Georgia Chalvatzaki, TU Darmstadt, Germany

Georgia Chalvatzaki is an Full Professor at TU Darmstadt since April 2023. Before that, she was an Independednt Research Group Leader from March 2021, after getting the renowned Emmy Noether Programme (ENP) fund of the German Research Foundation (DFG). --> In her research group iROSA, they will research the topic of "Robot Learning of Mobile Manipulation for Assistive Robotics". Dr. Chalvatzaki proposes new methods at the intersection of machine learning and classical robotics, taking one step further the research for embodied AI robotic assistants. The research in iROSA proposes novel methods for combined planning and learning for enabling mobile manipulator robots to solve complex tasks in house-like environments, with the human-in-the-loop of the interaction process.

Hao Su, UC San Diego, US

Hao Su is an Associate Professor of Computer Science at the University of California, San Diego. He is the Director of the Embodied AI Lab at UCSD, a founding member of the Data Science Institute, and a member of the Center for Visual Computing and the Contextural Robotics Institute. He works on algorithms to model, undertand, and interact with the physical world. His interests span computer vision, machine learning, computer graphics, and robotics -- all areas in which he has published and lectured extensively. Hao Su obtained his Ph.D. in Computer Science from Stanford. Prior to that, he obtained a Ph.D. in Mathematics from Beihang University, China. At Stanford and UCSD he developed widely used datasets and softwares such as ImageNet, ShapeNet, PointNet, PartNet, SAPIEN, and more recently, ManiSkill. He also developed new courses to promote machine learning methods for 3D geometry and embodied AI. He served as the Area Chair or Associate Editor for top conferences and journals in computer vision (ICCV/ECCV/CVPR), computer graphics (SIGGRAPH/ToG), robotics (IROS/ICRA), and machine learning (NeurIPS/ICLR). He received the SIGGRAPH Best Ph.D. Thesis Award Honorable Mention and the NSF CAREER Award.