Skip to Main Content

All 2020 Spring classes online. Building access restricted. Check uml.edu/alert for more info.

Benchmarking Tools for Evaluating Robotic Assembly of Small Parts

Workshop at Robotics: Science and Systems (RSS) 2020

Robot manipulating a gear on the NIST-ATB #1

Robot inserting a gear on the NIST-ATB #1

Half-day workshop on Monday, July 13, 2020

Abstract

Benchmarking tools have advanced the research and development of robotic manipulation capabilities, particularly for grasping (e.g., YCB Object Set). As robot capabilities progress and domains like manufacturing become populated with robot systems, performance artifacts for more advanced capabilities are needed. This workshop aims to promote the usage of a performance artifact for benchmarking robotic manipulation and assembly capabilities. The NIST Assembly Task Boards (NIST-ATB) are designed to quantify a robot system’s grasping, manipulation, and perception capabilities when used in small parts assembly operations. NIST-ATB #1 requires competencies including peg insertions, gear meshing, electrical connector insertions, and nut threading. It has been used in robotic manipulation competitions (e.g., IROS 2017 and 2019) and is currently being proliferated throughout the research community. Workshop content includes presentations of robotics research that utilizes the NIST-ATB for benchmarking purposes, such as the development of force sensing capabilities for insertion or the design of dexterous end-effectors to manipulate small objects. Related benchmarking and evaluation efforts involving the NIST-ATB will also be covered, as well as a discussion of the current and future plans for the benchmarking tool. 

Submissions are elicited for extended abstracts of ongoing or proposed research using the NIST-ATB #1 benchmarking tool, which will be provided to authors of accepted contributions for free (a $450 value). Task boards can be shipped to your institution (if outside of the US, recipient may have to pay import duties/taxes upon delivery which can range between $50-150) or the task board can be picked up at the workshop. The goal of this workshop is to spur the development of advanced robotic capabilities and begin building a user community around robotic assembly research.

See the NIST website for more information on the assembly task boards. A paper was also recently published in the IEEE RA-L Special Issue on Benchmarking Protocols for Robotic Manipulation, Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems, that provides an overview of the NIST assembly task boards. The paper is freely available as open access.

Organizers

  • Adam Norton, University of Massachusetts Lowell
  • Holly Yanco, University of Massachusetts Lowell
  • Joseph Falco, National Institute of Standards Technology (NIST)
  • Kenneth Kimble, National Institute of Standards and Technology (NIST)

Submissions

Extended abstracts (1-2 pages, following RSS style guidelines) are being solicited to cover research towards robotic assembly and manufacturing, which may include end-to-end solutions and/or development of relevant components including vision algorithms, tactile sensing, end-effector design, and teaching/programming techniques. All contributions should cover ongoing or proposed research that will utilize the NIST-ATB #1 as part of their experimentation plan.

We are requesting 1-2 page extended abstracts using the RSS 2020 format to be submitted via e-mail to: nistatb.rss2020@gmail.com. Anonymization is not required. When emailing the paper, please include the title, author list, affiliations, and e-mail addresses in the body of the email. All submissions will be reviewed and authors of accepted papers will be asked to give a 3-5 minute lightning talk at the workshop and provide a poster for a session following the lighting talks. At least one author of each accepted submission must register for the workshop. Authors of accepted submissions will be given a NIST-ATB #1 kit for free that will either be shipped to them (if outside of the US, recipient may have to pay import duties/taxes upon delivery which can range between $50-150) or can be picked up at the workshop.

Authors of accepted submissions will be offered the option of having their papers uploaded to a workshop-specific archive on arXiv.org. Inclusion in this archive will not be mandatory since it may create problems for authors who wish to submit follow-on work to venues with strict prior publication rules. Accepted authors will also be asked about how they would like to receive their task board.

  • March 6, 2020: Call for submissions open
  • April 9, 2020, 23:59 Anywhere on Earth (AoE): Submission deadline for extended abstracts
  • April 16: Notification of acceptance for workshop submissions
All questions regarding the workshop can be e-mailed to: nistatb.rss2020@gmail.com

Speakers

Invited speakers use the NIST-ATB in their current research, either to benchmark the performance of robotic solutions they have developed (e.g., perception algorithms, end effector design) or other uses of the tool in evaluation methods. Accepted extended abstracts and posters will present either ongoing or proposed research that will use the NIST-ATB.

  • Kenneth Kimble, National Institute of Standards and Technology (NIST): Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems (organizer)
  • James Watson, University of Colorado, Boulder: Test of Assembly Algorithms with an Industry-Inspired Benchmark (invited; confirmed)
  • Adam Norton, University of Massachusetts Lowell: Test and Evaluation of Robot Capabilities in Industrial Environments (organizer)
  • Juan Aparicio, Siemens: Developing Advanced Robotic Capabilities in Manufacturing Using Test and Evaluation Artifacts (invited; confirmed)
  • Cindy Grimm, Oregon State University: Automating Grasp Testing and Improving Ground Truth Estimations Using Sensorized Objects (invited; confirmed)

Schedule

  • 9 a.m. - Welcome and overview
  • 9:10 a.m. - Talk 1: Kenneth Kimble, NIST
  • 9:30 a.m. - Talk 2: James Watson, University of Colorado, Boulder
  • 9:50 a.m. - Talk 3: Adam Norton, UMass Lowell
  • 10:10 a.m. - Lightning talks (3-5 minutes each)
  • 10:45 a.m. - Break and poster session
  • 11:20 a.m. - Talk 4: Juan Arpacio, Siemens
  • 11:40 a.m. - Talk 5: Cindy Grimm, Oregon State University
  • Noon - Discussion
  • 12:30 p.m. - End