Skip to Main Content

Computer Scientists Communicate with Robot Using Natural Language

Technology Can Be Used in Assessing Threats, Search and Rescue

UMass Lowell Image

Using natural language, the iRobot ATRV-JR can be instructed to identify and defuse bombs as well as to report if it finds any hostages or criminals.

By Edwin L. Aguirre

Imagine being able to control a robot by simply speaking to it or sending it a text message.

That’s the goal of a joint project being conducted by a team of researchers from UMass Lowell, UMass Amherst, the University of Pennsylvania, Cornell University, Stanford University and George Mason University.

Called Situation Understanding Bot Through Language and Environment (SUBTLE), the project uses a fully autonomous robot — in this case, an iRobot ATRV-JR — to carry out tasks, particularly in situations that are far too dangerous for humans.

“We are working with computational linguists to enable a person to command a robot using voice commands similar to those between people,” says Daniel Brooks, a graduate student in the University’s Robotics Lab. “While such systems have been created in the past, the language was usually limited to pre-scripted commands. Our goal is to allow people to communicate with the robot more naturally.”

Enhancing Human-Robot Interaction

Brooks says a user types in commands on a tablet computer using an interface similar to text- or instant-messaging. The robot receives the text input and translates it into a set of instructions which it then carries out autonomously.

“The robot can be given commands to be executed immediately, such as ‘Search the floor for hostages,’” he says. “It can also be given standing orders for use over the entire run, like ‘Let me know if you see any bombs.’”

The robot relays information to the user through the interface on the tablet computer. 

“The system can add icons to the map displayed and highlight areas of the map to convey concepts such as ‘I am here,’” adds Brooks.

In addition to military and police operations and search-and-rescue missions, such technology could be applied to disaster relief as well as environmental monitoring, mapping and cleanups.

Other members of the University research team include students Mikhail Medvedev, Abraham Shultz, Sean McSheehy, Adam Norton, Eric McCann, Munjal Desai and Prof. Holly Yanco. SUBTLE is funded through a grant from the Army Research Office. For more information, go to