Robots that Learn, Interact and Perceive using Multiple Modalities with Jivko Sinapov 4/24

04/19/2023
By Paul Robinette

What: Robots that learn, interact, and perceive using multiple modalities: lessons learned and on-going challenges
When: Monday April 24, 2023 at 11 a.m.
Where: DAN-321
Who: Prof Jivko Sinapov from Tufts University

Abstract:
Robots have the potential to transform the way we live and are increasingly deployed in applications ranging from assistive care settings to collaborative manufacturing. Enabling such robots to adapt in real time when facing novel situations, and problems, however, remains a challenge. In this talk, I will argue for a multimodal approach to learning, interaction, and perception for achieving robot autonomy in ever-changing environments. First, I will describe how robots can transfer embodied knowledge across modalities (e.g., touch, sound, and vision) so that new robots, with different embodiments, sensors, and behaviors can still make use of the knowledge learned by other, more experienced, robots. Next, I will present results on how learned skills can be transferred from simple to complex environments as to afford the use of reinforcement learning methods that typically scale poorly in robotics domains. Finally, I will highlight multimodal approaches to interaction with people, including augmented reality and language, that help robots learn skills and concepts in order to be better partners and collaborators. We will conclude with a discussion on open questions and problems, along with my group’s strategy to address them.

Bio:
Jivko Sinapov is an assistant professor in Computer Science at Tufts University where he leads the Multimodal Learning, Interaction, and Perception (MuLIP) lab. He received his Ph.D. in computer science and human-computer interaction at Iowa State University in 2013 and subsequently worked as a postdoctoral associate at UT Austin prior to joining Tufts in 2017. His research interests include cognitive and developmental robotics, computational perception, human-robot interaction, and reinforcement learning. Jivko received the NSF CAREER award in 2023 and is also the recipient of the Tufts ROUTE award for undergraduate research advising in 2022.